Backup & Recovery

Overview

xorlab supports manual as well as scheduled automatic backups that can be used to restore the system after a failure. Backups are either full or incremental:

Full backups are self-contained and do not depend on previous backups, which makes them more robust than incremental backups.
An incremental backup only saves changes made since the last backup (full or incremental) hence it is faster, but it depends on earlier full/incremental backups. XCC will always trigger a full backup if no full backup can be found.
A sequence that begins with a full backup and may include any number of incremental backups is called a backup chain.

Backups are disabled by default on on-premises VMs, as the host VM itself is typically backed up via the hypervisor. Should you choose to use xorlab’s backup functionality, it is recommended to schedule weekly full backups over the weekend and daily incremental backups. Backup (and recovery) are done only on the XCC VM, as MTAs do not store data. Host configuration files, e.g. files in /etc/xorlab/ are also not covered by the backups. Hence, it is recommended to regularly take snapshots of all VMs, at least after every update of xorlab.

Quick setup

Open the Expert Editor and navigate to the /xcc/backend/xcc.yml.
Enable the backup service, and if needed, adjust the schedule for full and incremental backups, and the retention policy.
backup: enabled: true incrementalBackupCron: 0 0 12,21 * * SUN-FRI fullBackupCron: 0 0 1 * * SAT deletion: enabled: true keepAdditional: 0 deleteDanglingChain: true
In this example, incremental backups occur daily from Sunday to Friday at 12:00 and 21:00. A full backup is taken on Saturday at 01:00. Incomplete chains, e.g. incremental backups without a full backup, will be removed. Long running backups get aborted after 72 hours, which should be sufficient for typical email load and data retention settings. See Backup Settings for more details.
SSH to the XCC, open /etc/xorlab/xcc/.env and adjust the location of BKP_BASE_DIR, the directory where backups will be saved:
BKP_BASE_DIR=/backups
Click Publish, then restart the xcc_backend container to apply the changes. The general backup settings in xcc/backend/xcc.yml require an XCC restart; see How to Activate the Configuration:
cd /etc/xorlab/xcc && docker compose stop backend && docker compose up -d
Optionally, we recommend integrating/sending the backup events sys.backup.* to your SIEM or centralized monitoring system, if available. This will allow you to be notified if a backup fails.

Backup

Software versions compatibilityThe compatibility of backups between major/minor versions (e.g. 9.1.0 and 9.0.0) is not guaranteed. Hence, when the XCC runs on a new version, the next scheduled backup will always be a full backup automatically.It is recommended to perform an incremental backup before a software upgrade and a full backup immediately afterward.

Backup Service does not back up any host related configuration, e.g. firewall, /etc/xorlab directory, etc.

Each backup (full or incremental) consists of the info.json metadata file and four module files:

Assets - contains the assets required by the XCC web interface and the quarantined messages.
ConfigRepos - contains the xorlab configuration for each platform component.
Redis - contains the metadata used for similarity matching.
Database - contains the database contents with the metadata of each message analyzed by xorlab.

An example of all files for a full and two incremental backups:

backup-2025-09-23T14-42-50.738330283Z-full-Assets
backup-2025-09-23T14-42-50.738330283Z-full-ConfigRepos
backup-2025-09-23T14-42-50.738330283Z-full-Database
backup-2025-09-23T14-42-50.738330283Z-full-Redis
backup-2025-09-23T14-42-50.738330283Z-full-info.json
backup-2025-09-23T14-47-02.611662359Z-incremental-Assets
backup-2025-09-23T14-47-02.611662359Z-incremental-ConfigRepos
backup-2025-09-23T14-47-02.611662359Z-incremental-Database
backup-2025-09-23T14-47-02.611662359Z-incremental-Redis
backup-2025-09-23T14-47-02.611662359Z-incremental-info.json
backup-2025-09-23T14-47-19.942717733Z-incremental-Assets
backup-2025-09-23T14-47-19.942717733Z-incremental-ConfigRepos
backup-2025-09-23T14-47-19.942717733Z-incremental-Database
backup-2025-09-23T14-47-19.942717733Z-incremental-Redis
backup-2025-09-23T14-47-19.942717733Z-incremental-info.json

Each file follows the naming schema:

backup-timestamp-type-module

It starts with the backup- prefix followed by the UTC timestamp, which corresponds to the time when the backup was scheduled/triggered. The name further contains the type of the backup and the name of the module that created it. The file ending in info.json contains meta-data about the backup. If the backup service is enabled, XCC saves backups to the chosen backup directory according to the backup schedules.

Backup directory

Backups will take up a significant amount of disk space. In order to not interfere with normal operations, e.g. by using up all available disk space, it is strongly recommended to keep the backup directory on a different mount point than the other production data. By default, the backup directory is set to /backups. You can change it in /etc/xorlab/xcc/.env:

BKP_BASE_DIR=/backups

This backup directory (BKP_BASE_DIR) is mapped to /var/lib/xorlab/xcc/backup/backups in the backend container, as configured in /etc/xorlab/xcc/docker-compose.yml:

- ${BKP_BASE_DIR}:/var/lib/xorlab/xcc/backup/backups               # Backup dir

Backup schedule

The backup schedule syntax uses the Quartz cron format, and can be edited in /xcc/backend/xcc.yml via the Expert Editor. The pattern represents: second, minute, hour, day, month, weekday. Month and weekday names can be given as the first three letters of the English names. Here are a few examples:

0 0 6,19 * * *: at 6:00 and 19:00, every day.
0 0 9-17 * * MON-FRI: every hour from 09:00 to 17:00, on weekdays.
0 0 0 * * SUN: every Sunday at midnight.
"-": Never/disabled.

The default configuration schedules incremental backups daily from Sunday to Friday at 12:00 and 21:00, and a full backup on Saturday at 01:00. Note that the cron schedules are mutually exclusive (no incremental backup scheduled for Saturday):

backup:
  enabled: false
  incrementalBackupCron: 0 0 12,21 * * SUN-FRI
  fullBackupCron: 0 0 1 * * SAT
  deletion:
    enabled: true
    keepAdditional: 0
    deleteDanglingChain: true

After editing the backup schedule in the Expert Editor, click Publish and restart the XCC. These general xcc/backend/xcc.yml settings require an XCC restart; see How to Activate the Configuration. You can use online tools, such as Cron Expression Generator & Explainer or crontab guru, to verify that your cron schedule. Be mindful, that crontab guru follows the standard cron syntax (no seconds field).

Incremental backup scheduleEven if an incremental backup is scheduled, XCC will do a full backup if one of the following conditions apply:

XCC runs on a more recent minor version (after an update full backup is scheduled automatically).
There is an incomplete full backup that can be resumed.
No previous backup is found in the backup directory.

Backup settings

By default, automatic backup deletion is enabled. Only the current backup chain is kept (and one additional one related to the last minor release), and the dangling incremental backups (without associated full backup as their base) will be deleted. Here’s the list of all backup configuration options that can be set in /xcc/backend/xcc.yml.

backup:
  enabled: false
  # Cron pattern for incremental backup. May still trigger a full backup if
  # a new minor version is detected after an update, which necessitates a full
  # backup. "-" means disabled.
  incrementalBackupCron: 0 0 12,21 * * SUN-FRI
  # Cron pattern for full backup, "-" means disabled.
  fullBackupCron: 0 0 1 * * SAT
  # Maximum backup time in hours after which the backup gets aborted.
  maxBackupTimeH: 72
  # Time in seconds to wait to acquire shared table locks at the beginning of the dump.
  dbTableLockTimeoutS: 3600
  # Time in hours to wait for a database dump.
  dbDumpTimeoutH: 24
  # Whether to check if the last backup is complete on startup and immediately resume
  # incomplete backup.
  resumeLastBackupOnStartup: true
  # Number of concurrent threads during backup.
  maxBackupModuleThreads: 20
  # Comma-separated list of module names to skip for full backups.
  # Can be one of Database, ConfigRepos, Assets, Redis
  skipModuleFull: ""
  # Comma-separated list of module names to skip for incremental backups:
  # Can be one of Database, ConfigRepos, Assets, Redis
  skipModuleIncremental: ""
  # Backup chains retention policy. Scanning for old backups gets triggered on startup and
  # after each scheduled backup.
  deletion:
     ## If backups should be deleted at all.
     enabled: true
     # Keep backup chain if it is no older than this number of days.
     keepForDays: 0
     # Keep X additional backup chains. The current chain will never be deleted.
     keepAdditional: 0
     # Keep last backup chain of previous (minor) release. Recommended for rolling
     # back an update in the worst case.
     keepLastPreviousRelease: true
     # If a backup chain without a full backup can be deleted.
     deleteDanglingChain: true

Manual backup

When you need to create backups manually—for example, when the automatic backup was performed a couple of days ago and you want to make some substantial changes such as upgrading to a new xorlab version—you can can trigger a backup manually from the XCC web interface.

Log in to the XCC web interface.
On the main screen, click the tiles icon next to your account name and click the Admin icon in the displayed menu:
Open the Backup page:
Press the New Incremental Backup button, or choose New Full Backup:

Recovery

xorlab will automatically start a recovery procedure during startup, if the recovery folder contains a valid backup. This recovery will overwrite the current data. Make sure to only populate the recovery folder if you want to recover from a backup.

If you require assistance, or are in doubt, do not hesitate to contact us via support@xorlab.com.

The recovery process starts automatically if the backend service finds backup files in the recovery directory during startup. It will commence with the most recent full backup and proceed to apply all subsequent incremental backups. It replaces the database with the one from the backup, so the existing database does not have to be deleted beforehand. Here are the steps needed to perform a recovery:

Restore host configuration
Populate recovery directory
Start recovery process
Clear recovery directory
Remove outdated search results
Re-start XCC stack

Restore host configuration

Host configuration, that was changed during an update, is backed up to /etc/xorlab/backup/.

Populate recovery directory

The recovery directory is ${WRK_BASE_DIR}/backup/recovery on the XCC, which is /var/lib/xorlab/xcc/backup/recovery with default settings. The default /etc/xorlab/xcc/docker-compose.yml already mounts it into the backend container, so it needs no setup. Move the backup chain you would like to recover from the backup directory to the recovery directory: its full backup, and the incremental backups after it up to the point in time you want to restore. Assuming /backups is used as the backup directory:

sudo mv /backups/backup-2025-10-10* /var/lib/xorlab/xcc/backup/recovery/

If the backup directory is on a different mount point than /var/lib/xorlab/xcc, as recommended, moving the files copies them, so make sure there is enough free disk space for the backup chain.

Start recovery process

Make sure all containers from the xcc stack are stopped:
cd /etc/xorlab/xcc && docker compose down
Start PostgreSQL—the database has to be running to recover its data:
docker compose up -d db

Start the backend service:

docker compose up backend

and observe the logs. After a few seconds status reports similar to the following will be shown:

xcc_backend  | INFO  [RecoveryServiceThread-6]: c.x.j.b.m.ModuleRunner - ### Status Report:
xcc_backend  | Assets [30s ago]: Extracting to /var/lib/xorlab/xcc/backend/assets (no progress will be reported until done.)
xcc_backend  | Database [18s ago]: pg_restore: finished item 3439 INDEX uq_asset_ggrid_name
xcc_backend  | RecoveryService [30s ago]: Starting recovery modules [com.xorlab.jxcc.backup.module.recovery.AssetsRecoveryModule@ac91282, com.xorlab.jxcc.backup.module.recovery.DatabaseRecoveryModule@7f79edee] for backup Backup(2025-10-29T04:00:00.016296214Z, full, /var/lib/xorlab/xcc/backup/recovery)

Once the recovery is complete, the JVM in the xcc_backend container will shut down with exit code 3, and you will see this line:
INFO exited: xcc (exit status 3; expected)
Ctrl+C to close the logs and stop the container.

Errors from pg_restorexorlab uses pg_restore to restore backups. All log lines containing Found expected (non-)error from pg_restore can be safely ignored as they do not influence the recovery.

Clear recovery directory

Move the backup chain back to the backup directory. The recovery directory must be empty before the backend service starts again, otherwise it starts a second recovery. xorlab also only cleans up old backup chains in the backup directory.

sudo mv /var/lib/xorlab/xcc/backup/recovery/backup-* /backups/

Remove outdated search results

The recovery marks all emails for re-synchronization with Elasticsearch, and the backend service re-synchronizes them from the database once it starts. Until then, the search index still holds the emails from before the recovery, and the XCC web interface shows them in the search overview. Delete the index while the backend service is still stopped; it creates the index again when it starts.

List the index and note its version number, xcc_result_publish_version_<N>:

docker exec es_0 curl -s "localhost:9200/_cat/indices/xcc_result_publish_*?v"

Delete the index, replacing <N> with the version number from the previous step:

docker exec es_0 curl -s -X DELETE "localhost:9200/xcc_result_publish_version_<N>"

Re-start XCC stack

cd /etc/xorlab/xcc && docker compose down && docker compose up -d

The analyst dashboard will appear empty or incomplete while data is being re-synchronized. Depending on the amount of emails, it may take minutes to hours until this process is completed.

Patch rollback

If the patch upgrade was not successful, it’s possible to revert the changes. All components can be restored to the previous patch version (unless stated otherwise in the release notes) by performing the following steps:

Set the version variables (*_VERSION) to the previous patch version in the /etc/xorlab/<component>/.env file.

Restart the component:

cd /etc/xorlab/<component> && docker compose up -d

For example, to roll back the xcc component from 11.0.1 to the 11.0.0:

Set the version variables to the 11.0.0:

/etc/xorlab/xcc/.env

(...)
XCCWEB_VERSION=11.0.0        # was 11.0.1
XCC_VERSION=11.0.0           # was 11.0.1
XCC_DB_VERSION=11.0.0        # was 11.0.1
XCC_REDIS_VERSION=11.0.0     # was 11.0.1
(...)

Restart the xcc component:

cd /etc/xorlab/xcc && docker compose up -d

In case the patch rollback fails, please refer to the Recovery section, to perform a restore from the backup chain created before the update.

​Overview

​Quick setup

​Backup

​Backup directory

​Backup schedule

​Backup settings

​Manual backup

​Recovery

​Restore host configuration

​Populate recovery directory

​Start recovery process

​Clear recovery directory

​Remove outdated search results

​Re-start XCC stack

​Patch rollback

Overview

Quick setup

Backup

Backup directory

Backup schedule

Backup settings

Manual backup

Recovery

Restore host configuration

Populate recovery directory

Start recovery process

Clear recovery directory

Remove outdated search results

Re-start XCC stack

Patch rollback