Backup & Recovery

Overview

xorlab supports manual as well as scheduled automatic backups that can be used to restore the system after a failure. Backups are either full or incremental:

Full backups are self-contained and do not depend on previous backups, which makes them more robust than incremental backups.
An incremental backup only saves changes made since the last backup (full or incremental) hence it is faster, but it depends on earlier full/incremental backups. XCC will always trigger a full backup if no full backup can be found.
A sequence that begins with a full backup and may include any number of incremental backups is called a backup chain.

Backups are disabled by default on on-premises VMs, as the host VM itself is typically backed up via the hypervisor. Should you choose to use xorlab’s backup functionality, it is recommended to schedule weekly full backups over the weekend and daily incremental backups. Backup (and recovery) are done only on the XCC VM, as MTAs do not store data. Host configuration files, e.g. files in /etc/xorlab/ are also not covered by the backups. Hence, it is recommended to regularly take snapshots of all VMs, at least after every update of xorlab.

Quick setup

Open the Expert Editor and navigate to the /xcc/backend/xcc.yml.
Enable the backup service, and if needed, adjust the schedule for full and incremental backups, and the retention policy.
backup: enabled: true incrementalBackupCron: 0 0 12,21 * * SUN-FRI fullBackupCron: 0 0 1 * * SAT deletion: enabled: true keepAdditional: 0 deleteDanglingChain: true
In this example, incremental backups occur daily from Sunday to Friday at 12:00 and 21:00. A full backup is taken on Saturday at 01:00. Incomplete chains, e.g. incremental backups without a full backup, will be removed. Long running backups get aborted after 72 hours, which should be sufficient for typical email load and data retention settings. See Backup Settings for more details.
SSH to the XCC, open /etc/xorlab/xcc/.env and adjust the location of BKP_BASE_DIR, the directory where backups will be saved:
BKP_BASE_DIR=/backups
Click Publish, then restart the xcc_backend container to apply the changes. The general backup settings in xcc/backend/xcc.yml require an XCC restart; see How to Activate the Configuration:
cd /etc/xorlab/xcc && docker compose stop backend && docker compose up -d
Optionally, we recommend integrating/sending the backup events sys.backup.* to your SIEM or centralized monitoring system, if available. This will allow you to be notified if a backup fails.

Backup

Software versions compatibilityThe compatibility of backups between major/minor versions (e.g. 9.1.0 and 9.0.0) is not guaranteed. Hence, when the XCC runs on a new version, the next scheduled backup will always be a full backup automatically.It is recommended to perform an incremental backup before a software upgrade and a full backup immediately afterward.

Backup Service does not back up any host related configuration, e.g. firewall, /etc/xorlab directory, etc.

Each backup (full or incremental) consists of the info.json metadata file and four module files:

Assets - contains the assets required by the XCC web interface and the quarantined messages.
ConfigRepos - contains the xorlab configuration for each platform component.
Redis - contains the metadata used for similarity matching.
Database - contains the database contents with the metadata of each message analyzed by xorlab.

An example of all files for a full and two incremental backups:

backup-2025-09-23T14-42-50.738330283Z-full-Assets
backup-2025-09-23T14-42-50.738330283Z-full-ConfigRepos
backup-2025-09-23T14-42-50.738330283Z-full-Database
backup-2025-09-23T14-42-50.738330283Z-full-Redis
backup-2025-09-23T14-42-50.738330283Z-full-info.json
backup-2025-09-23T14-47-02.611662359Z-incremental-Assets
backup-2025-09-23T14-47-02.611662359Z-incremental-ConfigRepos
backup-2025-09-23T14-47-02.611662359Z-incremental-Database
backup-2025-09-23T14-47-02.611662359Z-incremental-Redis
backup-2025-09-23T14-47-02.611662359Z-incremental-info.json
backup-2025-09-23T14-47-19.942717733Z-incremental-Assets
backup-2025-09-23T14-47-19.942717733Z-incremental-ConfigRepos
backup-2025-09-23T14-47-19.942717733Z-incremental-Database
backup-2025-09-23T14-47-19.942717733Z-incremental-Redis
backup-2025-09-23T14-47-19.942717733Z-incremental-info.json

Each file follows the naming schema:

backup-timestamp-type-module

It starts with the backup- prefix followed by the UTC timestamp, which corresponds to the time when the backup was scheduled/triggered. The name further contains the type of the backup and the name of the module that created it. The file ending in info.json contains meta-data about the backup. If the backup service is enabled, XCC saves backups to the chosen backup directory according to the backup schedules.

Backup directory

Backups will take up a significant amount of disk space. In order to not interfere with normal operations, e.g. by using up all available disk space, it is strongly recommended to keep the backup directory on a different mount point than the other production data. By default, the backup directory is set to /backups. You can change it in /etc/xorlab/xcc/.env:

BKP_BASE_DIR=/backups

This backup directory (BKP_BASE_DIR) is mapped to /var/lib/xorlab/xcc/backup/backups in the backend container, as configured in /etc/xorlab/xcc/docker-compose.yml:

- ${BKP_BASE_DIR}:/var/lib/xorlab/xcc/backup/backups               # Backup dir

Backup schedule

The backup schedule syntax uses the Quartz cron format, and can be edited in /xcc/backend/xcc.yml via the Expert Editor. The pattern represents: second, minute, hour, day, month, weekday. Month and weekday names can be given as the first three letters of the English names. Here are a few examples:

0 0 6,19 * * *: at 6:00 and 19:00, every day.
0 0 9-17 * * MON-FRI: every hour from 09:00 to 17:00, on weekdays.
0 0 0 * * SUN: every Sunday at midnight.
"-": Never/disabled.

The default configuration schedules incremental backups daily from Sunday to Friday at 12:00 and 21:00, and a full backup on Saturday at 01:00. Note that the cron schedules are mutually exclusive (no incremental backup scheduled for Saturday):

backup:
  enabled: false
  incrementalBackupCron: 0 0 12,21 * * SUN-FRI
  fullBackupCron: 0 0 1 * * SAT
  deletion:
    enabled: true
    keepAdditional: 0
    deleteDanglingChain: true

After editing the backup schedule in the Expert Editor, click Publish and restart the XCC. These general xcc/backend/xcc.yml settings require an XCC restart; see How to Activate the Configuration. You can use online tools, such as Cron Expression Generator & Explainer or crontab guru, to verify that your cron schedule. Be mindful, that crontab guru follows the standard cron syntax (no seconds field).

Incremental backup scheduleEven if an incremental backup is scheduled, XCC will do a full backup if one of the following conditions apply:

XCC runs on a more recent minor version (after an update full backup is scheduled automatically).
There is an incomplete full backup that can be resumed.
No previous backup is found in the backup directory.

Backup settings

By default, automatic backup deletion is enabled. Only the current backup chain is kept (and one additional one related to the last minor release), and the dangling incremental backups (without associated full backup as their base) will be deleted. Here’s the list of all backup configuration options that can be set in /xcc/backend/xcc.yml.

backup:
  enabled: false
  # Cron pattern for incremental backup. May still trigger a full backup if
  # a new minor version is detected after an update, which necessitates a full
  # backup. "-" means disabled.
  incrementalBackupCron: 0 0 12,21 * * SUN-FRI
  # Cron pattern for full backup, "-" means disabled.
  fullBackupCron: 0 0 1 * * SAT
  # Maximum backup time in hours after which the backup gets aborted.
  maxBackupTimeH: 72
  # Time in seconds to wait to acquire shared table locks at the beginning of the dump.
  dbTableLockTimeoutS: 3600
  # Time in hours to wait for a database dump.
  dbDumpTimeoutH: 24
  # Whether to check if the last backup is complete on startup and immediately resume
  # incomplete backup.
  resumeLastBackupOnStartup: true
  # Number of concurrent threads during backup.
  maxBackupModuleThreads: 20
  # Comma-separated list of module names to skip for full backups.
  # Can be one of Database, ConfigRepos, Assets, Redis
  skipModuleFull: ""
  # Comma-separated list of module names to skip for incremental backups:
  # Can be one of Database, ConfigRepos, Assets, Redis
  skipModuleIncremental: ""
  # Backup chains retention policy. Scanning for old backups gets triggered on startup and
  # after each scheduled backup.
  deletion:
     ## If backups should be deleted at all.
     enabled: true
     # Keep backup chain if it is no older than this number of days.
     keepForDays: 0
     # Keep X additional backup chains. The current chain will never be deleted.
     keepAdditional: 0
     # Keep last backup chain of previous (minor) release. Recommended for rolling
     # back an update in the worst case.
     keepLastPreviousRelease: true
     # If a backup chain without a full backup can be deleted.
     deleteDanglingChain: true

Manual backup

When you need to create backups manually—for example, when the automatic backup was performed a couple of days ago and you want to make some substantial changes such as upgrading to a new xorlab version—you can can trigger a backup manually from the XCC web interface.

Log in to the XCC web interface.
On the main screen, click the tiles icon next to your account name and click the Admin icon in the displayed menu:
Open the Backup page:
Press the New Incremental Backup button, or choose New Full Backup:

Recovery

xorlab will automatically start a recovery procedure during startup, if the recovery folder contains a valid backup. This recovery will overwrite the current data. Make sure to only populate the recovery folder if you want to recover from a backup.

If you require assistance, or are in doubt, do not hesitate to contact us via support@xorlab.com.

The recovery process starts automatically if the backend service finds backup files in the recovery directory during startup. It will commence with the most recent full backup and proceed to apply all subsequent incremental backups. Here are the steps needed to perform a recovery:

Restore Host Configuration
Populate Recovery Directory
Start Recovery Process
Clear Recovery Directory
Re-start xcc Stack
Re-synchronize Elastic indices

Restore host configuration

Host configuration, that was changed during an update, is backed up to /etc/xorlab/backup/.

Populate recovery directory

If there is only one backup chain in the backup folder, you can simply set the recovery folder to the backup folder and skip step 1. & 2. In case of several backup chains, move the backup chain you would like to recover to a separate recovery folder. It is recommended to use a path on the same mountpoint as the backup directory, so that moving the backups to the recovery directory does not involve actual disk IO. Assuming /backups is used as the backup directory:

Create a subfolder recovery owned by xcc_backend

cd /backups
sudo mkdir recovery
sudo chown xcc_backend: recovery/

Move the backup you want to restore to the recovery folder, e.g.:
sudo mv backup-2025-10-10* recovery/

Configure your recovery folder in docker-compose.yml. In the following example, we use ${BKP_BASE_DIR}/recovery:

/etc/xorlab/xcc/docker-compose.yml

backend:
    image: ${XOR_REGISTRY}/xcc/backend:${XCC_VERSION}
    container_name: xcc_backend
    volumes:
    [...]
    - ${WRK_BASE_DIR}:/var/lib/xorlab/xcc
    - ${BKP_BASE_DIR}/recovery:/var/lib/xorlab/xcc/backup/recovery        # Recovery dir
    - ${BKP_BASE_DIR}:/var/lib/xorlab/xcc/backup/backups                  # Backup dir

Start recovery process

Make sure all containers from the xcc stack are stopped:
cd /etc/xorlab/xcc && docker compose down

This step is only required when recovering from a backup that has been taken with an older major or minor version than the one currently running on XCC.

Delete the existing DB:

sudo find /var/lib/xorlab/xcc/postgres -mindepth 1 -delete

Delete existing Elastic data:

cd /etc/xorlab/elastic && docker compose down && \
sudo find /var/lib/xorlab/elastic -mindepth 1 -delete && \
docker compose up -d

Start PostgreSQL—the database has to be running to recover its data:
docker compose up -d db

Start the backend service:

docker compose up backend

and observe the logs. After a few seconds status reports similar to the following will be shown:

xcc_backend  | INFO  [RecoveryServiceThread-6]: c.x.j.b.m.ModuleRunner - ### Status Report:
xcc_backend  | Assets [30s ago]: Extracting to /var/lib/xorlab/xcc/backend/assets (no progress will be reported until done.)
xcc_backend  | Database [18s ago]: pg_restore: finished item 3439 INDEX uq_asset_ggrid_name
xcc_backend  | RecoveryService [30s ago]: Starting recovery modules [com.xorlab.jxcc.backup.module.recovery.AssetsRecoveryModule@ac91282, com.xorlab.jxcc.backup.module.recovery.DatabaseRecoveryModule@7f79edee] for backup Backup(2025-10-29T04:00:00.016296214Z, full, /var/lib/xorlab/xcc/backup/recovery)

Once the recovery is complete, the JVM in the xcc_backend container will shut down with exit code 3, and you will see this line:
INFO exited: xcc (exit status 3; expected)
Ctrl+C to close the logs and stop the container.

Errors from pg_restorexorlab uses pg_restore to restore backups. All log lines containing Found expected (non-)error from pg_restore can be safely ignored as they do not influence the recovery.

Clear recovery directory

Restore the original bind mount in docker-compose.yml. To avoid an accidental second recovery, the mapped folder ${WRK_BASE_DIR}/backup/recovery, which is /var/lib/xorlab/xcc/backup/recovery/ with default settings, must be empty.

/etc/xorlab/xcc/docker-compose.yml

backend:
    image: ${XOR_REGISTRY}/xcc/backend:${XCC_VERSION}
    container_name: xcc_backend
    volumes:
    [...]
    - ${WRK_BASE_DIR}:/var/lib/xorlab/xcc
    - ${WRK_BASE_DIR}/backup/recovery:/var/lib/xorlab/xcc/backup/recovery  # Recovery dir
    - ${BKP_BASE_DIR}:/var/lib/xorlab/xcc/backup/backups                   # Backup dir

Re-start XCC stack

Ensure that the Elastic stack is running, if it was previously stopped, run:

cd /etc/xorlab/elastic && docker compose up -d

and restart xcc stack:

cd /etc/xorlab/xcc && docker compose down && docker compose up -d

Re-synchronize elastic indices

During the recovery process, elastic indices are automatically marked for re-synchronization, but need to be deleted manually (they will be re-synchronized from the database automatically). Otherwise, the XCC web interface will still show the emails from before the recovery in the search overview. To trigger re-synchronization:

Ensure that the Auth stack is running, if it was previously stopped, run:
cd /etc/xorlab/auth && docker compose up -d
Open the XCC web interface and go to Insights from the sidebar menu.
Open any of the dashboards, a new tab will open.
Expand the Elastic side menu on the left and open Stack Management located under the Management section.
Open Index Management located under the Data section (from the updated left side menu).
Search for the xcc_result_publish_<version>.
Check the box to the left of the result.
Press the blue Manage index button, and choose Delete Index.

Stop the xcc_backend container:

cd /etc/xorlab/xcc && docker compose stop backend

Mark data for re-synchronization with:

docker compose exec db psql -U xcc -c "UPDATE xcc.result_publication SET es_sync_version=0, es_sync_attempts=0;"

Start the xcc_backend container:
docker compose up -d backend

The analyst dashboard will appear empty or incomplete while data is being re-synchronized. Depending on the amount of emails, it may take minutes to hours until this process is completed.

Recovery directory cleanup

If you created a separate recovery folder in step Populate Recovery Directory, move the backup chain back to the backup folder, as xorlab will not clean up old backup chains from folders other than the backup folder.

Patch rollback

If the patch upgrade was not successful, it’s possible to revert the changes. All components can be restored to the previous patch version (unless stated otherwise in the release notes) by performing the following steps:

Set the version variables (*_VERSION) to the previous patch version in the /etc/xorlab/<component>/.env file.

Restart the component:

cd /etc/xorlab/<component> && docker compose up -d

For example, to roll back the xcc component from 11.0.1 to the 11.0.0:

Set the version variables to the 11.0.0:

/etc/xorlab/xcc/.env

(...)
XCCWEB_VERSION=11.0.0        # was 11.0.1
XCC_VERSION=11.0.0           # was 11.0.1
XCC_DB_VERSION=11.0.0        # was 11.0.1
XCC_REDIS_VERSION=11.0.0     # was 11.0.1
(...)

Restart the xcc component:

cd /etc/xorlab/xcc && docker compose up -d

In case the patch rollback fails, please refer to the Recovery section, to perform a restore from the backup chain created before the update.

​Overview

​Quick setup

​Backup

​Backup directory

​Backup schedule

​Backup settings

​Manual backup

​Recovery

​Restore host configuration

​Populate recovery directory

​Start recovery process

​Clear recovery directory

​Re-start XCC stack

​Re-synchronize elastic indices

​Recovery directory cleanup

​Patch rollback

Overview

Quick setup

Backup

Backup directory

Backup schedule

Backup settings

Manual backup

Recovery

Restore host configuration

Populate recovery directory

Start recovery process

Clear recovery directory

Re-start XCC stack

Re-synchronize elastic indices

Recovery directory cleanup

Patch rollback