Operation Reference

xorlab Security Platform architecture

Every xorlab Security Platform component (Sandbox, MTA, and XCC) comprises a set of Docker containers which have different responsibilities. MTA and XCC components are delivered to on-prem customers as separate VMs. xorlab Cloud (xCloud) and the xorlab Sandbox are components in the SaaS infrastructure that are needed by our platform. In rare cases, the Sandbox is run on-prem. The following picture shows the most important containers, their dependencies, and a typical mail flow:

Mail flow, containers and their dependencies in xorlab Security Platform (not exhaustive) Note that this is not an exhaustive list of dependencies. For all inter-component dependencies, full network traffic information, and other details regarding data flow from/to xorlab and within the platform, please see Network Guide. The mail flow can be described in the following steps:

An email arrives at the MTA (xorlab Mail Transfer Agent) component using SMTP and is received by the mta container (basically, a postfix mail server).
The mta container forwards the email—still using SMTP—to the core container. If this transmission fails—e.g., because core is not ready to process the email—the email will be deferred by core, and you should see the following alert: Emails Older Than 100 Seconds Stuck In the Deferred Queue. core then performs a static analysis of the message. Depending on the settings and the message itself, the message attachments can additionally be scanned by sending them to the xorlab Sandbox (DANA). When the Sandbox scan is done, the email will be shown in Messages → Pending in the web interface.
The analysis results are published to the XCC component, where the results are persisted in a database and accessible through the XCC web interface. If the result publishing fails, you will receive the following alert: Core System Test Failed, with the description containing the name of the failing test—here, ResultPublishSystemTest.
If the message is allowed to be delivered, the core container hands it over back to the mta container for sending to the recipient. As in step #2, errors will trigger the following alert: Emails Older Than 100 Seconds Stuck In the Deferred Queue.

Generally, all containers are grouped into stacks. You can find all containers with their respective stacks in the following picture. Based on the stack and container name, you can later derive the location of the container log files on the host, e.g., xcc_backend.

Container description

MTA

Container	Description
`core`	Email processing engine. Parses the email, applies rules and policies, classifies the email, and decides if it is legitimate or not
`mta`	Responsible for email routing. Accepts and forwards emails. Sends the emails to `core` for processing before forwarding them
`rspamd`	Spam detection module used by `core`
`rproxy_mta`	Reverse proxy for the MTA

XCC

Container	Description
`backend`	Heart of XCC. Provides the API for MTAs, maintains the configuration, computes the trust model, provides the web interface, etc.
`db`	Database for all email related data
`web`	Reverse proxy handling access to the `backend` API
`redis`	Fast, in-memory key-value pair data store
`elasticsearch`	Refines the data from `db` to support advanced data search and processing. Used internally to provide search in the web interface
`kibana`	Provides the dashboards accessible through the Insights menu
`auth`	Performs user authentication. Can integrate external systems through, for example, LDAP or SAML. Provides user and role information to the backend for access control

Sandbox (DANA)

Container	Description
`analysisnode`	Responsible for the Sandbox. Receives files and scans them in internal VMs
`tika`	Text processing engine. Extracts text from various documents and supports OCR
`nginx`	Reverse proxy handling access to the `analysisnode` API

Shared monitoring containers

Container	Description
`prometheus`	Monitoring system and time series database: stores all monitoring metrics and applies the alert rules to see if an alert must be triggered. Additionally, Prometheus on XCC can collect the monitoring metrics from other Prometheus instances on MTAs and DANAs for a centralized view of all metrics
`grafana`	Data analytics and monitoring
`node_exporter`	Gathers metrics from the host (node)
`cadvisor`	Gathers Docker metrics
`alertmanager`	Collects and sends alerts to users
`config`	Pulls config from the config providers (`xcloud` for global and `xcc_backend` for local configuration)
`loki`	Log aggregation system for storing and querying logs
`promtail`	Sends the contents of local logs to the `loki` service

Start and stop

Generic procedure to start and stop stacks on the xorlab Security Platform components is performed within the stack folder and looks like this.

## Execute inside the stack folder, e.g., for xcc:
cd /etc/xorlab/xcc

## Stop stack
docker compose down

## Start stack (starts all not running containers)
docker compose up -d

## Stop single container
docker compose stop <container>

MTA

Stop and start:

cd /etc/xorlab/activeguard

# Stop activeguard stack; make sure to stop the core container first.
docker compose down

# Start it again
docker compose up -d

If you only want to restart the core container due to configuration changes, use the restart button in the MTA GUI.

XCC

Stop and start:

## The ordering is important

# Stop all stacks
cd /etc/xorlab/auth
docker compose down
cd /etc/xorlab/xcc
docker compose down
cd /etc/xorlab/elastic
docker compose down

# Start all stacks again
cd /etc/xorlab/elastic
docker compose up -d
cd /etc/xorlab/xcc
docker compose up -d
cd /etc/xorlab/auth
docker compose up -d

Sandbox (DANA)

Stop and start:

cd /etc/xorlab/dana

# Stop the Sandbox stack
docker compose down

# Start it again
docker compose up -d

Note that current dynamic scans will be lost and there will be no re-scan. To prevent this, make sure the queue is empty before stopping.

Shell aliases

From release 11.0.5, each VM comes with shell aliases for the commands on this page. The upgrade installs them into /etc/profile.d/xorlab-aliases-<component>.sh, so they are available in every new SSH session after the upgrade. Only the aliases for the stacks present on the VM are installed. Run alias to list the ones on the VM you are logged in to.

Generic Docker Compose aliases

These are available on every component and act on the stack folder you are in, e.g., /etc/xorlab/xcc.

Alias	What it does
`dcup`	Starts the stack: `docker compose up -d`
`dcdown`	Stops the stack: `docker compose down`
`dcdownup`	Stops the stack and starts it again
`dcdownuplogf`	Stops the stack, starts it again, and follows its logs
`dclogf`	Follows the logs of the stack, starting from the latest line

MTA

Alias	What it does
`cdactiveguard`	Changes to the stack folder `/etc/xorlab/activeguard`
`mta-up`	Starts the activeguard stack
`mta-up-logf`	Starts the activeguard stack and follows its logs. Leaves you in the stack folder
`mta-down-now`	Stops the activeguard stack immediately, without draining the queues. To shut down gracefully, use `mta-down` instead
`mta-stop-core`	Stops the `core` container
`mta-start-core`	Starts the `core` container
`mta-restart-core`	Restarts the `core` container
`mta-restart-core-logf`	Restarts the `core` container and follows its log
`mta-core-health`	Prints the health status of the `core` container (release 11.0.7 and later)
`xorlab-mailq`	Lists all queued emails
`xorlab-postqueue-flush`	Flushes (resends) all queues
`xorlab-bash-core`	Opens a shell in the `core` container
`xorlab-bash-mta`	Opens a shell in the `mta` container
`tail-activeguard`	Follows `activeguard.log`
`tail-activeguarderr`	Follows `activeguard_err.log`
`tail-mailinfo`	Follows the `mta` container’s `mail.info` log
`tail-mailwarn`	Follows the `mta` container’s `mail.warn` log
`tail-mailerr`	Follows the `mta` container’s `mail.err` log

XCC

The -all aliases stop and start the auth, xcc, and elastic stacks in the order given in Start and stop.

Alias	What it does
`cdxcc`	Changes to `/etc/xorlab/xcc`
`cdauth`	Changes to `/etc/xorlab/auth`
`cdelastic`	Changes to `/etc/xorlab/elastic`
`cdmonitoring`	Changes to `/etc/xorlab/monitoring`
`cdlog`	Changes to the log folder `/var/log/xorlab/`
`cdaglocal`	Changes to the local MTA configuration folder `/var/lib/xorlab/xcc/config/work/ag_local/`
`xcc-up`	Starts the xcc stack
`xcc-up-logf`	Starts the xcc stack and follows its logs
`xcc-down`	Stops the xcc stack
`xcc-downup`	Stops the xcc stack and starts it again
`xcc-downup-logf`	Stops the xcc stack, starts it again, and follows its logs
`xcc-up-all`	Starts all stacks
`xcc-up-all-logf`	Starts all stacks and follows the logs of the xcc stack
`xcc-down-all`	Stops all stacks
`xcc-downup-all`	Stops all stacks and starts them again
`xcc-downup-all-logf`	Stops all stacks, starts them again, and follows the logs of the xcc stack
`tail-xcc`	Follows `xcc.log`
`tail-xccerr`	Follows `xcc_err.log`

Sandbox (DANA)

Alias	What it does
`cddana`	Changes to `/etc/xorlab/dana`
`cdlog`	Changes to the log folder `/var/log/xorlab/`
`dana-up`	Starts the dana stack
`dana-up-logf`	Starts the dana stack and follows its logs
`dana-down`	Stops the dana stack
`dana-downup`	Stops the dana stack and starts it again
`dana-downup-logf`	Stops the dana stack, starts it again, and follows its logs

Graceful MTA shutdown

The upgrade also installs the mta-down command on the MTA. It finishes the emails the MTA has already accepted before it shuts down the activeguard stack:

Blocks new incoming SMTP connections on port 25 of every physical network interface.
Flushes the Postfix queues.
Waits up to five minutes for the core container to finish processing, then up to one minute for it to publish the results to XCC.
Stops the activeguard stack and lifts the block again.

If the core container is not running, mta-down stops the stack right away. Use mta-down --block and mta-down --unblock to apply or lift only the SMTP block, as an alternative to the iptables commands in Stopping and resuming accepting emails by MTA.

Reboot

To reboot a component, execute the following command:

sudo reboot

XCC: can be rebooted at any time.
MTA: stop the ag stack before reboot, preferably with mta-down.

Emails that are being processed when a shutdown command is issued, will not be lost but are delayed until the MTA is fully operational again. To avoid this delay, you can block new incoming SMTP connections using the iptables commands described in the Email queue section below. The inserted firewall block, iptables -I, will be cleared by a reboot or the corresponding iptables -D command. It is strongly recommended to only reboot one MTA at a time to ensure continuous email processing.

Log files

Most xorlab Security Platform internal containers write log files to a standardized location:

/var/log/xorlab/<stack>/<container>/

For example:

/var/log/xorlab/xcc/backend/
/var/log/xorlab/auth/auth/
/var/log/xorlab/activeguard/core/
/var/log/xorlab/dana/analysisnode/

The following log files are available:

# Error log file
<container>_err.log
# Verbose log (INFO/DEBUG)
<container>.log

The most important container log files:

backend: xcc.log and xcc_err.log
auth: auth.log and auth_err.log
core: activeguard.log and activeguard_err.log
analysisnode: analysisnode.log and analysisnode_err.log

Standard outThe following containers do not have log files: elasticsearch, kibana, db, and all monitoring containers. For those, logs are written to the standard output which you can read by executing docker compose logs <container> inside the /etc/xorlab/<stack> folder.

Email queue

The email queue sizes can be conveniently observed in the grafana monitoring dashboards. For more advanced email queue handling, connect to each MTA separately. List all queued emails:

docker exec -it ag_mta mailq

Flush (resend) all queues:

docker exec -it ag_mta postqueue -f

Requeue all queued emails:

docker exec -it ag_mta postsuper -r ALL

Requeue vs. flushA flush is a lightweight operation where xorlab Security Platform will try to resend all queued emails. A requeue will treat all queued emails as if they were completely new emails. This will re-apply all header checks, address rewrites, routing decisions, etc. If you use this when there are emails in the deferred queue that already passed through core, they will be sent again to core (which will cause the email to be shown twice in the XCC).

Delete an email from the queue:

# Find the queue ID of the email that should be deleted
docker exec -it ag_mta mailq
# Replace <mail_queue_id> with the queue ID of the actual email
docker exec -it ag_mta postsuper -d <mail_queue_id>

Delete all emails in a specific queue:

# Deletes all emails in the deferred queue. "deferred" can be replaced with other queue names like "incoming" or "active".
docker exec -it ag_mta postsuper -d ALL deferred

Exporting an email from the queue to EML format

The email files of a particular queue—active, incoming, or deferred—are in the /var/lib/docker/volumes/activeguard_mta_spool/_data/<queue_name> directory on the MTAs. You must be root to access this folder (sudo su). The email file in the queue directory will have the queue ID as a name (e.g., 9A2821FE2). If you want to convert the email file to the EML format, you can use the following instructions:

Run
docker exec -it ag_mta bash
Execute
cd /var/spool/postfix/<queue_name>
Find the email file.

Run

postcat -bh <file_name> > /var/log/file_name.eml

Press Ctrl+D.
Now, you can find the EML file in the /var/log/xorlab/activeguard/mta/ directory.

Stopping and resuming accepting emails by MTA

Sometimes you might want to prevent an MTA from accepting emails because, for example, you would like to empty the MTA from messages before performing an update. In this case, use SSH and run the following command (port 25 is used here as an example):

# block incoming emails on MTA (e.g., for maintenance) on port 25
sudo iptables -I DOCKER-USER -i eth0 -p tcp --dport 25 -m state --state NEW -j REJECT

where eth0 is the interface name, which can be found, for example, with ip -4 a. When you execute the command, wait one minute—or check in XCC for emails being processed—then shut down xorlab Security Platform. After the maintenance, you can resume accepting emails:

# accept incoming emails again on port 25
sudo iptables -D DOCKER-USER -i eth0 -p tcp --dport 25 -m state --state NEW -j REJECT

Docker packet filteringTo learn more, see the Packet filtering and firewalls and Match the original IP and ports for requests from the official Docker Engine documentation.

​xorlab Security Platform architecture

​Container description

​MTA

​XCC

​Sandbox (DANA)

​Shared monitoring containers

​Start and stop

​MTA

​XCC

​Sandbox (DANA)

​Shell aliases

​Generic Docker Compose aliases

​MTA

​XCC

​Sandbox (DANA)

​Graceful MTA shutdown

​Reboot

​Log files

​Email queue

​Exporting an email from the queue to EML format

​Stopping and resuming accepting emails by MTA

xorlab Security Platform architecture

Container description

MTA

XCC

Sandbox (DANA)

Shared monitoring containers

Start and stop

MTA

XCC

Sandbox (DANA)

Shell aliases

Generic Docker Compose aliases

MTA

XCC

Sandbox (DANA)

Graceful MTA shutdown

Reboot

Log files

Email queue

Exporting an email from the queue to EML format

Stopping and resuming accepting emails by MTA