Test and Troubleshoot Alerts

Trigger a test alert

To trigger a test alert, and verify that the routing and configuration is correct, SSH to the host and execute the following command:

docker exec -it monitoring_config trigger_an_alert "Summary" "Description" "Severity"

docker exec -it monitored_mta_config trigger_an_alert "Summary" "Description" "Severity"

Be patient, as it takes some time for the Alertmanager to send the alert. To find out how long you should wait, check the value of route.group_wait (or the same value under the specific severity you have used) in the /monitoring/alertmanager/alertmanager.yml file opened in the Expert Editor (the default value is one minute).

Troubleshoot alerts

When something is not working as expected, you can check the logs of Prometheus (which creates the Alerts) and Alertmanager (which sends the alerts to the external systems [receivers]) to see if there are any errors:

Prometheus logs:

docker logs monitoring_prometheus

Alertmanager logs:

docker logs monitoring_alertmanager

Prometheus logs:

docker logs monitored_mta_prometheus

Alertmanager logs:

docker logs monitored_mta_alertmanager

If those logs are not enough, you can use the log.level setting to enable further debug logs of Prometheus. Go to the /monitoring/prometheus/prometheus.properties (for XCC monitoring) or /monitored_mta/prometheus/prometheus.properties (for MTA monitoring) and uncomment the following line:

#log.level=debug

You can also use this setting to only log messages with these severities (or above):

info
warn
error

You can check if monitoring is running properly by executing the following command and confirming that the status of the containers is healthy:

docker ps --filter "name=monitor*"

​Trigger a test alert

​Troubleshoot alerts

Trigger a test alert

Troubleshoot alerts