> ## Documentation Index > Fetch the complete documentation index at: https://docs.xorlab.com/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > Pages under /latest/ document the current release. Paths that begin with a version number, for example /10.0/, are frozen snapshots of superseded releases kept for reference only — never present their instructions as current. If the reader's version is unknown, answer from /latest/ and say which version the answer describes. > xorlab is deployed on-premises, hybrid, or as a cloud service, and integrates with either Microsoft 365 or on-premises mail infrastructure. Configuration steps often differ between these. State which deployment and integration mode an instruction applies to instead of presenting one as universal. > Distinguish inline mode from monitoring mode when describing anything that acts on email. Monitoring mode observes a copy and cannot block or quarantine; inline mode routes mail through xorlab and can. > Write the product name as lowercase 'xorlab'. Use the documented component names: xorlab Control Center (XCC), xorlab MTA, xorlab Sandbox (DANA), xorlab Natural Language Understanding (NLU). After the first mention, use the short forms XCC, MTA, Sandbox, and NLU. Do not use DANA as a standalone name for the Sandbox, but keep it where it is a literal string in configuration keys, container names, and hostnames. > Do not invent configuration keys, rule parameters, list names, log properties, or API fields. If a value is not present in this documentation, say that it is not documented rather than guessing. # Alerting ## Architecture Alerts based on metrics are generated by **Prometheus** and routed via **Alertmanager**. They can be either send centrally from the **Alertmanager** on XCC or from the **Alertmanager** on all VMs (MTAs & XCC). While the former is simpler to set up, it creates a single point of failure on the XCC **Prometheus** / **Alertmanager** and we recommend to generate alerts on all VMs. Alerts based on logs can only be generated by Loki on XCC, and are routed via the XCC **Alertmanager**. xorlab does currently not ship with log based alerts by default. ## Alert receivers ### Configuration examples **Alertmanager** configuration comes with the following example configurations: * `alertmanager.yml` - No alerts (default) * `alertmanager.email.yml` - Email notification examples, one basic example and one with severity based routing. * `alertmanager.webhook.yml` - Generic webhook example * `alertmanager.slack.yml` - Slack integaration example. The diffent examples can be mixed by configuring different `receivers` and adjusting the `route` configuration accordingly, similar to the *Email receiver (with routing)* example. **Alertmanager** supports more receivers and allows for more advanced configuration (e.g. alerts being sent to more than one receiver, complex routing, etc.), see also [prometheus.io - Route-related settings](https://prometheus.io/docs/alerting/latest/configuration/#route-related-settings). By default, **Alertmanager** does not send alerts to any real receivers, all alerts are routed to a no-op / blackhole receiver. ```yaml theme={null} route: receiver: blackhole routes: - receiver: blackhole receivers: - name: blackhole ``` This example configuration sends all alerts to a specified email address, see also [prometheus.io - email\_config](https://prometheus.io/docs/alerting/latest/configuration/#email_config). ```yaml hl_lines="4-7 19 20 24 25 29 30 34 35" theme={null} --- global: smtp_smarthost: smtp_from: smtp_auth_username: # Only required if smtp auth is used smtp_auth_password: # Only required if smtp auth is used smtp_require_tls: true smtp_hello: localhost resolve_timeout: 2m templates: - /config/monitoring/alertmanager/template/email.tmpl # - /config/monitoring/alertmanager/template/custom.tmpl (..) receivers: - name: email email_configs: - to: from: html: '{{ template "email.default.html" . }}' ``` This example configuration sends alerts to different email addresses, based on the severity of the alerts, see also [prometheus.io - email\_config](https://prometheus.io/docs/alerting/latest/configuration/#email_config). There are four levels of alert severities used by the default alerts: normal, medium, high, and critical. ```yaml hl_lines="4-6 38 39 43 44 48 49 53 54" theme={null} --- global: smtp_smarthost: smtp_from: smtp_auth_username: # Only required if smtp auth is used smtp_auth_password: # Only required if smtp auth is used smtp_require_tls: true smtp_hello: localhost resolve_timeout: 2m templates: - /config/monitoring/alertmanager/template/email.tmpl # - /config/monitoring/alertmanager/template/custom.tmpl route: receiver: email-normal # default receiver group_by: - alertname group_wait: 1m # initial wait to ensure alerts can be grouped group_interval: 5m # subsequent alerts wait 30m for new alerts if in an active group repeat_interval: 30m # wait for 'repeat_interval' before resending active alerts continue: false # whether an alert should continue matching against subsequent routes routes: - receiver: email-medium matchers: - severity="medium" - receiver: email-high matchers: - severity="high" - receiver: email-critical matchers: - severity="critical" receivers: - name: email-normal email_configs: - to: ', ' from: html: '{{ template "email.default.html" . }}' - name: email-medium email_configs: - to: ', ' from: html: '{{ template "email.default.html" . }}' - name: email-high email_configs: - to: from: html: '{{ template "email.default.html" . }}' - name: email-critical email_configs: - to: from: html: '{{ template "email.default.html" . }}' ``` This example configuration sends all alerts to a specified webhook, see also [prometheus.io - Receiver integration settings](https://prometheus.io/docs/alerting/latest/configuration/#receiver-integration-settings) ```yaml hl_lines="5" theme={null} --- global: route: receiver: webhook group_by: - alertname group_wait: 1m group_interval: 5m repeat_interval: 30m routes: - receiver: webhook receivers: - name: webhook webhook_configs: - url: https://example.com/webhook ``` This example configuration sends all alerts to a specified **Slack** channel, see also [prometheus.io - Receiver integration settings](https://prometheus.io/docs/alerting/latest/configuration/#receiver-integration-settings) and the [Slack documentation](https://api.slack.com/messaging/webhooks). ```yaml hl_lines="6 12 17 22 27" theme={null} --- global: # REPLACE the following slack_api_url with a valid one. # NB: We have to use this placeholder here, so that alertmanager does not crash during start-up slack_api_url: https://hooks.slack.com/services/XXXXXXXXXXX/YYYYYYYYYYY/ZZZZZZZZZZZZZZZZZZZZZZZZ resolve_timeout: 2m (...) receivers: - name: slack-alert slack_configs: channel: text: '{{ template "slack.text.alert" . }}' ``` ### Applying the configuration The configuration can be found in the [Expert Editor](/9.0/expert-editor) under `/monitoring/alertmanager` for the XCC monitoring and in the `/monitored_mta/alertmanager` for the MTA monitoring. The file `alertmanager.yml` always holds the currently active configuration. To use the config provided in the example files, e.g. for email notifications, you can either: * rename `alertmanager.yml` to `alertmanager.noop.yml`, and then rename `alertmanager.email.yml` to `alertmanager.yml`, or * copy the content of `alertmanager.email.yml` and paste it into to `alertmanager.yml` To apply the configuration, execute the following steps: 1. Open the [Expert Editor](/9.0/expert-editor). 2. The various config examples described above are saved in `/monitoring/alertmanager` 3. Optionally you can back up `alertmanager.yml` by duplicating the file, e.g. into `alertmanager.noop.yml` 4. To use a specific configuration copy and paste the config into: * `/monitoring/alertmanager/alertmanager.yml` for XCC alerting, and * `/monitored_mta/alertmanager/alertmanager.yml` for MTA alerting 5. Click **PUBLISH** to commit the changes and wait one minute for the **Prometheus** to reload the config. See also [*How to Activate the Configuration*](/9.0/activate-configuration). ## Testing alerts To trigger a test alert, and verify that the routing and configuration is correct, SSH to the host and execute the following command: ```shell theme={null} docker exec -it monitoring_config trigger_an_alert "Summary" "Description" "Severity" ``` ```shell theme={null} docker exec -it monitored_mta_config trigger_an_alert "Summary" "Description" "Severity" ``` Be patient, as it takes some time for the **Alertmanager** to send the alert. To find out how long you should wait, check the value of `route.group_wait` (or the same value under the specific severity you have used) in the `/monitoring/alertmanager/alertmanager.yml` file opened in the [Expert Editor](/9.0/./expert-editor) (the default value is one minute). ## Troubleshooting alerts When something is not working as expected, you can check the logs of **Prometheus** (which creates the Alerts) and **Alertmanager** (which sends the alerts to the external systems \[receivers]) to see if there are any errors: Prometheus logs: ```shell theme={null} docker logs monitoring_prometheus ``` Alertmanager logs: ```shell theme={null} docker logs monitoring_alertmanager ``` Prometheus logs: ```shell theme={null} docker logs monitored_mta_prometheus ``` Alertmanager logs: ```shell theme={null} docker logs monitored_mta_alertmanager ``` If those logs are not enough, you can use the `log.level` setting to enable further debug logs of **Prometheus**. Go to the `/monitoring/prometheus/prometheus.properties` (for XCC monitoring) or `/monitored_mta/prometheus/prometheus.properties` (for MTA monitoring) and uncomment the following line: ```yaml theme={null} #log.level=debug ``` You can also use this setting to only log messages with these severities (or above): * `info` * `warn` * `error` You can check if monitoring is running properly by executing the following command and confirming that the status of the containers is `healthy`: ```shell theme={null} docker ps --filter "name=monitor*" ``` ## Optional Settings ### Adding a custom alert Custom alerting rules should be added to `/monitoring/prometheus/alerts/custom.yml` for alerting rules evaluated on XCC and to `/monitored_mta/prometheus/alerts/custom.yml` for alerting rules evaluated on the MTAs. In addition, custom rules must be activated in `/monitoring/prometheus/prometheus.yml`, respectively `/monitored_mta/prometheus/prometheus.yml`. Alerting rules are described in the official [documentation](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) 1. Open the [Expert Editor](/9.0/./expert-editor). 2. Navigate to the `/monitoring/prometheus/alerts`. 3. Add rules to `custom.yml` 4. Add `custom.yml` to the list of `rule_files` in `/monitoring/prometheus/prometheus.yml`: ```yaml hl_lines="7" theme={null} rule_files: # Default alerting rules - /config/monitoring/prometheus/alert/monitoring_alerts.yml - /config/monitoring/prometheus/alert/system_alerts.yml - /config/monitoring/prometheus/alert/xcc_alerts.yml # Custom alerting rules - /config/monitoring/prometheus/alert/custom.yml ``` **Do not edit default rules files! You can deactivate default rule files by commenting them out in, or removing them from the `rule_files` list. Changes made to the default files will be overwritten by software updates.** 5. Click **PUBLISH** to commit the changes and wait one minute for the **Prometheus** to reload the config. See also [*How to Activate the Configuration*](/9.0/activate-configuration). ### Customizing a notification template 1. Open the [Expert Editor](/9.0/./expert-editor). 2. Navigate to the `/monitoring/alertmanager/template/custom.tmpl`, and edit the template. These templates use [Go templating](https://golang.org/pkg/text/template/), some examples can be found in the [Alerting documentation](https://prometheus.io/docs/alerting/latest/notification_examples/). 3. Enable the template by uncommenting the corresponding line in the current Alertmanager config file `monitoring/alertmanager/alertmanager.yml` (provided you have already choose a config example, e.g. `alertmanager.email.yml` or `alertmanager.slack.yml`, see [Alert receivers](#alert-receivers)), and comment out the default template line: ```yaml theme={null} templates: # - /config/monitoring/alertmanager/template/slack.tmpl - /config/monitoring/alertmanager/template/custom.tmpl ``` 4. Click **PUBLISH** to commit the changes and wait one minute for the **Alertmanager** to reload the config. See also [*How to Activate the Configuration*](/9.0/activate-configuration). ### Enabling notifications for resolved alerts If you want to enable notifications about resolved alerts, you have to add the `send_resolved` property and set it to `true` for your selected receiver. 1. Open your active `alertmanager.yml` config file. 2. Find the `receivers` group, and under the chosen receiver config add `send_resolved: true`. For example, the receivers config for **email** alerts should look like this: ```yaml hl_lines="8" theme={null} (...) receivers: - name: email-normal email_configs: - to: alerts@example.com from: alertmanager@xsp.example.com html: '{{ template "email.default.html" . }}' send_resolved: true (...) ``` 3. Click **PUBLISH** to commit the changes and wait one minute for the **Alertmanager** to reload the config. See also [*How to Activate the Configuration*](/9.0/activate-configuration).