Alerting - xorlab

Architecture

Alerts based on metrics are generated by Prometheus and routed via Alertmanager. They can be either send centrally from the Alertmanager on XCC or from the Alertmanager on all VMs (MTAs & XCC). While the former is simpler to set up, it creates a single point of failure on the XCC Prometheus / Alertmanager and we recommend to generate alerts on all VMs. Alerts based on logs can only be generated by Loki on XCC, and are routed via the XCC Alertmanager. xorlab does currently not ship with log based alerts by default.

Alert receivers

Configuration examples

Alertmanager configuration comes with the following example configurations:

alertmanager.yml - No alerts (default)
alertmanager.email.yml - Email notification examples, one basic example and one with severity based routing.
alertmanager.webhook.yml - Generic webhook example
alertmanager.slack.yml - Slack integaration example.

The diffent examples can be mixed by configuring different receivers and adjusting the route configuration accordingly, similar to the Email receiver (with routing) example. Alertmanager supports more receivers and allows for more advanced configuration (e.g. alerts being sent to more than one receiver, complex routing, etc.), see also prometheus.io - Route-related settings.

No alerts (default)
Email receiver (basic)
Email receiver (with routing)
Webhook receiver
Slack receiver

By default, Alertmanager does not send alerts to any real receivers, all alerts are routed to a no-op / blackhole receiver.

route:
  receiver: blackhole
  routes:
    - receiver: blackhole

receivers:
  - name: blackhole

This example configuration sends all alerts to a specified email address, see also prometheus.io - email_config.

---
global:
  smtp_smarthost: <localhost:25>
  smtp_from: <alertmanager@xorlab.example.com>
  smtp_auth_username: <alertmanager>     # Only required if smtp auth is used
  smtp_auth_password: <password>         # Only required if smtp auth is used
  smtp_require_tls: true
  smtp_hello: localhost
  resolve_timeout: 2m

templates:
  - /config/monitoring/alertmanager/template/email.tmpl
#  - /config/monitoring/alertmanager/template/custom.tmpl
(..)
receivers:
  - name: email
    email_configs:
      - to: <alert@on-call.example.com>
        from: <alertmanager@xorlab.example.com>
        html: '{{ template "email.default.html" . }}'

This example configuration sends alerts to different email addresses, based on the severity of the alerts, see also prometheus.io - email_config. There are four levels of alert severities used by the default alerts: normal, medium, high, and critical.

---
global:
  smtp_smarthost: <localhost:25>
  smtp_from: <alertmanager@xorlab.example.com>
  smtp_auth_username: <alertmanager>     # Only required if smtp auth is used
  smtp_auth_password: <password>         # Only required if smtp auth is used
  smtp_require_tls: true
  smtp_hello: localhost
  resolve_timeout: 2m

templates:
  - /config/monitoring/alertmanager/template/email.tmpl
#  - /config/monitoring/alertmanager/template/custom.tmpl

route:
  receiver: email-normal                 # default receiver
  group_by:
    - alertname
  group_wait:       1m                   # initial wait to ensure alerts can be grouped
  group_interval:   5m                   # subsequent alerts wait 30m for new alerts if in an active group
  repeat_interval: 30m                   # wait for 'repeat_interval' before resending active alerts
  continue: false                        # whether an alert should continue matching against subsequent routes

  routes:
    - receiver: email-medium
      matchers:
        - severity="medium"
    - receiver: email-high
      matchers:
        - severity="high"
    - receiver: email-critical
      matchers:
        - severity="critical"

receivers:
  - name: email-normal
    email_configs:
      - to: '<user1@example.com>, <user2@example.com>'
        from: <alertmanager@xorlab.example.com>
        html: '{{ template "email.default.html" . }}'
  - name: email-medium
    email_configs:
      - to: '<user1@example.com>, <user2@example.com>'
        from: <alertmanager@xsp.example.com>
        html: '{{ template "email.default.html" . }}'
  - name: email-high
    email_configs:
      - to: <on-call@example.com>
        from: <alertmanager@xsp.example.com>
        html: '{{ template "email.default.html" . }}'
  - name: email-critical
    email_configs:
      - to: <on-call@example.com>
        from: <alertmanager@xsp.example.com>
        html: '{{ template "email.default.html" . }}'

This example configuration sends all alerts to a specified webhook, see also prometheus.io - Receiver integration settings

---
global:

route:
  receiver: webhook
  group_by:
    - alertname
  group_wait: 1m
  group_interval: 5m
  repeat_interval: 30m

  routes:
    - receiver: webhook

  receivers:
    - name: webhook
      webhook_configs:
        - url: https://example.com/webhook

This example configuration sends all alerts to a specified Slack channel, see also prometheus.io - Receiver integration settings and the Slack documentation.

---
global:
  # REPLACE the following slack_api_url with a valid one.
  # NB: We have to use this placeholder here, so that alertmanager does not crash during start-up
  slack_api_url: https://hooks.slack.com/services/XXXXXXXXXXX/YYYYYYYYYYY/ZZZZZZZZZZZZZZZZZZZZZZZZ
  resolve_timeout: 2m
(...)
receivers:
  - name: slack-alert
    slack_configs:
        channel: <alert>
        text: '{{ template "slack.text.alert" . }}'

Applying the configuration

The configuration can be found in the Expert Editor under /monitoring/alertmanager for the XCC monitoring and in the /monitored_mta/alertmanager for the MTA monitoring. The file alertmanager.yml always holds the currently active configuration. To use the config provided in the example files, e.g. for email notifications, you can either:

rename alertmanager.yml to alertmanager.noop.yml, and then rename alertmanager.email.yml to alertmanager.yml, or
copy the content of alertmanager.email.yml and paste it into to alertmanager.yml

To apply the configuration, execute the following steps:

Open the Expert Editor.
The various config examples described above are saved in /monitoring/alertmanager
Optionally you can back up alertmanager.yml by duplicating the file, e.g. into alertmanager.noop.yml
To use a specific configuration copy and paste the config into:
- /monitoring/alertmanager/alertmanager.yml for XCC alerting, and
- /monitored_mta/alertmanager/alertmanager.yml for MTA alerting
Click PUBLISH to commit the changes and wait one minute for the Prometheus to reload the config. See also How to Activate the Configuration.

Testing alerts

To trigger a test alert, and verify that the routing and configuration is correct, SSH to the host and execute the following command:

docker exec -it monitoring_config trigger_an_alert "Summary" "Description" "Severity"

docker exec -it monitored_mta_config trigger_an_alert "Summary" "Description" "Severity"

Be patient, as it takes some time for the Alertmanager to send the alert. To find out how long you should wait, check the value of route.group_wait (or the same value under the specific severity you have used) in the /monitoring/alertmanager/alertmanager.yml file opened in the Expert Editor (the default value is one minute).

Troubleshooting alerts

When something is not working as expected, you can check the logs of Prometheus (which creates the Alerts) and Alertmanager (which sends the alerts to the external systems [receivers]) to see if there are any errors:

Prometheus logs:

docker logs monitoring_prometheus

Alertmanager logs:

docker logs monitoring_alertmanager

Prometheus logs:

docker logs monitored_mta_prometheus

Alertmanager logs:

docker logs monitored_mta_alertmanager

If those logs are not enough, you can use the log.level setting to enable further debug logs of Prometheus. Go to the /monitoring/prometheus/prometheus.properties (for XCC monitoring) or /monitored_mta/prometheus/prometheus.properties (for MTA monitoring) and uncomment the following line:

#log.level=debug

You can also use this setting to only log messages with these severities (or above):

info
warn
error

You can check if monitoring is running properly by executing the following command and confirming that the status of the containers is healthy:

docker ps --filter "name=monitor*"

Optional Settings

Adding a custom alert

Custom alerting rules should be added to /monitoring/prometheus/alerts/custom.yml for alerting rules evaluated on XCC and to /monitored_mta/prometheus/alerts/custom.yml for alerting rules evaluated on the MTAs. In addition, custom rules must be activated in /monitoring/prometheus/prometheus.yml, respectively /monitored_mta/prometheus/prometheus.yml. Alerting rules are described in the official documentation

Open the Expert Editor.
Navigate to the /monitoring/prometheus/alerts.
Add rules to custom.yml

Add custom.yml to the list of rule_files in /monitoring/prometheus/prometheus.yml:

rule_files:
  # Default alerting rules
  - /config/monitoring/prometheus/alert/monitoring_alerts.yml
  - /config/monitoring/prometheus/alert/system_alerts.yml
  - /config/monitoring/prometheus/alert/xcc_alerts.yml
  # Custom alerting rules
  - /config/monitoring/prometheus/alert/custom.yml

Do not edit default rules files! You can deactivate default rule files by commenting them out in, or removing them from the rule_files list. Changes made to the default files will be overwritten by software updates.

Click PUBLISH to commit the changes and wait one minute for the Prometheus to reload the config. See also How to Activate the Configuration.

Customizing a notification template

Open the Expert Editor.
Navigate to the /monitoring/alertmanager/template/custom.tmpl, and edit the template. These templates use Go templating, some examples can be found in the Alerting documentation.
Enable the template by uncommenting the corresponding line in the current Alertmanager config file monitoring/alertmanager/alertmanager.yml (provided you have already choose a config example, e.g. alertmanager.email.yml or alertmanager.slack.yml, see Alert receivers), and comment out the default template line:
templates: # - /config/monitoring/alertmanager/template/slack.tmpl - /config/monitoring/alertmanager/template/custom.tmpl
Click PUBLISH to commit the changes and wait one minute for the Alertmanager to reload the config. See also How to Activate the Configuration.

Enabling notifications for resolved alerts

If you want to enable notifications about resolved alerts, you have to add the send_resolved property and set it to true for your selected receiver.

Open your active alertmanager.yml config file.

Find the receivers group, and under the chosen receiver config add send_resolved: true. For example, the receivers config for email alerts should look like this:

(...)
receivers:
  - name: email-normal
    email_configs:
      - to: alerts@example.com
        from: alertmanager@xsp.example.com
        html: '{{ template "email.default.html" . }}'
        send_resolved: true
(...)

Click PUBLISH to commit the changes and wait one minute for the Alertmanager to reload the config. See also How to Activate the Configuration.

​Architecture

​Alert receivers

​Configuration examples

​Applying the configuration

​Testing alerts

​Troubleshooting alerts

​Optional Settings

​Adding a custom alert

​Customizing a notification template

​Enabling notifications for resolved alerts

Architecture

Alert receivers

Configuration examples

Applying the configuration

Testing alerts

Troubleshooting alerts

Optional Settings

Adding a custom alert

Customizing a notification template

Enabling notifications for resolved alerts