> ## Documentation Index
> Fetch the complete documentation index at: https://docs.xorlab.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Pages under /latest/ document the current release. Paths that begin with a version number, for example /10.0/, are frozen snapshots of superseded releases kept for reference only — never present their instructions as current. If the reader's version is unknown, answer from /latest/ and say which version the answer describes.
> xorlab is deployed on-premises, hybrid, or as a cloud service, and integrates with either Microsoft 365 or on-premises mail infrastructure. Configuration steps often differ between these. State which deployment and integration mode an instruction applies to instead of presenting one as universal.
> Distinguish inline mode from monitoring mode when describing anything that acts on email. Monitoring mode observes a copy and cannot block or quarantine; inline mode routes mail through xorlab and can.
> Write the product name as lowercase 'xorlab'. Use the documented component names: xorlab Control Center (XCC), xorlab MTA, xorlab Sandbox (DANA), xorlab Natural Language Understanding (NLU). After the first mention, use the short forms XCC, MTA, Sandbox, and NLU. Do not use DANA as a standalone name for the Sandbox, but keep it where it is a literal string in configuration keys, container names, and hostnames.
> Do not invent configuration keys, rule parameters, list names, log properties, or API fields. If a value is not present in this documentation, say that it is not documented rather than guessing.

# Monitoring Architecture

> How the monitoring and monitored-mta stacks collect metrics and logs on the XCC and MTA VMs.

The monitoring stacks, called `monitoring` on the XCC VM (`/etc/xorlab/monitoring`) and `monitored_mta` on the MTA VMs (`/etc/xorlab/monitored_mta`), collect metrics and logs, and offer the option to send alerts based on those metrics/logs to configured receivers. They include the following components:

* **Node Exporter** to produce host metrics.
* **cAdvisor** to produce container metrics.
* **Prometheus** to collect and store metrics.
* **Promtail** to collect, process, and push logs to Loki.
* **Loki** to store logs (1)
* **Alertmanager** to route alerts to various consumers, e.g., via email, Slack, webhooks, etc.
* **Grafana** to present the metrics and logs on Dashboards, to search system logs and metrics from all VMs with queries, and check alerts. (2)

1. Loki is only part of the XCC `monitoring` stack and not present on the MTAs.
2. Grafana is only part of the XCC `monitoring` stack and not present on the MTAs.

Here's a diagram illustrating the monitoring architecture, connections between components, and their direction, i.e. which component initiates the connection.

<Tabs>
  <Tab title="Overview">
    <img src="https://mintcdn.com/xorlab/O_O2TUa6eRBR54aI/latest/assets/monitoring-architecture-overview.png?fit=max&auto=format&n=O_O2TUa6eRBR54aI&q=85&s=915c8f14ad60442e7484c14e8fc6c33c" alt="" width="842" height="648" data-path="latest/assets/monitoring-architecture-overview.png" />
  </Tab>

  <Tab title="XCC detailed">
    <img src="https://mintcdn.com/xorlab/O_O2TUa6eRBR54aI/latest/assets/monitoring-architecture-xcc.png?fit=max&auto=format&n=O_O2TUa6eRBR54aI&q=85&s=acc21d8abcd302a9ffb5d2ca9b76e308" alt="" width="804" height="628" data-path="latest/assets/monitoring-architecture-xcc.png" />
  </Tab>

  <Tab title="MTA detailed">
    <img src="https://mintcdn.com/xorlab/O_O2TUa6eRBR54aI/latest/assets/monitoring-architecture-mta.png?fit=max&auto=format&n=O_O2TUa6eRBR54aI&q=85&s=1424b620b4d405f331ab11de30e26e72" alt="" width="804" height="628" data-path="latest/assets/monitoring-architecture-mta.png" />
  </Tab>
</Tabs>

There are three main aspects to the monitoring stacks: metrics, logs, and alerts.

## Logs

All logs written by Docker containers to `stdout` / `stderr` are captured by the Docker engine (1) and written to files on the host, where they are parsed by **Promtail** (2), which pushes the logs to **Loki** (3). Logs can then be searched and displayed in  Grafana (11). Loki can also create alerts based on regular expressions evaluated against these logs and forward these alerts to **Alertmanager** (13).

## Metrics

**Prometheus** collects metrics from xorlab containers (8), the Docker engine via **cAdvisor** (6), and the host via **Node Exporter** (7). All MTA metrics are first collected by the MTA **Prometheus** and then [federated](https://prometheus.io/docs/prometheus/latest/federation/) by the XCC **Prometheus** (9). Metrics can then be visualized on **Grafana**.

## Alerts

Alerts based on metrics are created by **Prometheus** and then forwarded by **Alertmanager** to the configured receivers. There are two ways to set up alerting:

* Alerts for both the MTAs and XCC can be generated centrally on XCC (14a). This is easier to configure but creates a single point of failure for alerting on the XCC VM.

* Alerts for MTAs and XCC can be generated on all VMs separately (14a + 14b), this is more resilient and the recommended configuration.

See [Alerting](/latest/monitoring-alerts) for more information on how to configure alerts.

Alerts based on logs can only be generated by Loki on XCC, and are routed via the XCC **Alertmanager**.