> ## Documentation Index > Fetch the complete documentation index at: https://docs.xorlab.com/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > Pages under /latest/ document the current release. Paths that begin with a version number, for example /10.0/, are frozen snapshots of superseded releases kept for reference only — never present their instructions as current. If the reader's version is unknown, answer from /latest/ and say which version the answer describes. > xorlab is deployed on-premises, hybrid, or as a cloud service, and integrates with either Microsoft 365 or on-premises mail infrastructure. Configuration steps often differ between these. State which deployment and integration mode an instruction applies to instead of presenting one as universal. > Distinguish inline mode from monitoring mode when describing anything that acts on email. Monitoring mode observes a copy and cannot block or quarantine; inline mode routes mail through xorlab and can. > Write the product name as lowercase 'xorlab'. Use the documented component names: xorlab Control Center (XCC), xorlab MTA, xorlab Sandbox (DANA), xorlab Natural Language Understanding (NLU). After the first mention, use the short forms XCC, MTA, Sandbox, and NLU. Do not use DANA as a standalone name for the Sandbox, but keep it where it is a literal string in configuration keys, container names, and hostnames. > Do not invent configuration keys, rule parameters, list names, log properties, or API fields. If a value is not present in this documentation, say that it is not documented rather than guessing. # Backup & Recovery > Configure manual and scheduled full or incremental backups, and restore xorlab after a failure. ## Overview xorlab supports [manual](#manual-backup) as well as [scheduled automatic](#backup) backups that can be used to restore the system after a failure. Backups are either **full** or **incremental**: * **Full** backups are self-contained and do not depend on previous backups, which makes them more robust than incremental backups. * An **incremental** backup only saves changes made since the last backup (full or incremental) hence it is faster, but it depends on earlier full/incremental backups. XCC will always trigger a full backup if no full backup can be found. * A sequence that begins with a **full** backup and may include any number of **incremental** backups is called a **backup chain**. Backups are **disabled** by default on on-premises VMs, as the host VM itself is typically backed up via the hypervisor. Should you choose to use xorlab's backup functionality, it is recommended to schedule *weekly* full backups over the weekend and *daily* incremental backups. Backup (and [recovery](#recovery)) are done only on the XCC VM, as MTAs do not store data. Host configuration files, e.g. files in `/etc/xorlab/` are also not covered by the backups. Hence, it is recommended to regularly take snapshots of all VMs, at least after every update of xorlab. ## Quick setup 1. Open the [Expert Editor](/latest/expert-editor) and navigate to the `/xcc/backend/xcc.yml`. 2. Enable the backup service, and if needed, adjust the [schedule](#backup-schedule) for **full** and **incremental** backups, and the retention policy. ```shell {2} theme={null} backup: enabled: true incrementalBackupCron: 0 0 12,21 * * SUN-FRI fullBackupCron: 0 0 1 * * SAT deletion: enabled: true keepAdditional: 0 deleteDanglingChain: true ``` > In this example, incremental backups occur daily from Sunday to Friday at 12:00 and 21:00. A full backup is taken on Saturday at 01:00. Incomplete chains, e.g. incremental backups without a full backup, will be removed. Long running backups get aborted after 72 hours, which should be sufficient for typical email load and data retention settings. See [Backup Settings](#backup-settings) for more details. 3. SSH to the XCC, open `/etc/xorlab/xcc/.env` and adjust the location of `BKP_BASE_DIR`, the directory where backups will be saved: ```shell theme={null} BKP_BASE_DIR=/backups ``` 4. Click **Publish**, then restart the `xcc_backend` container to apply the changes. The general backup settings in `xcc/backend/xcc.yml` require an XCC restart; see [How to Activate the Configuration](/latest/activate-configuration): ```shell theme={null} cd /etc/xorlab/xcc && docker compose stop backend && docker compose up -d ``` 5. Optionally, we recommend integrating/sending the backup events [`sys.backup.*`](/latest/logging-events#system-events) to your SIEM or centralized monitoring system, if available. This will allow you to be notified if a backup fails. ## Backup **Software versions compatibility** The compatibility of backups between major/minor versions (e.g. 9.1.0 and 9.0.0) is not guaranteed. Hence, when the XCC runs on a new version, the next scheduled backup will always be a full backup automatically. It is recommended to perform an **incremental** backup before a software upgrade and a **full** backup immediately afterward. **Backup Service does not back up any host related configuration, e.g. firewall, `/etc/xorlab` directory, etc.** Each backup (full or incremental) consists of the `info.json` metadata file and four **module** files: * **Assets** - contains the assets required by the XCC web interface and the quarantined messages. * **ConfigRepos** - contains the xorlab configuration for each platform component. * **Redis** - contains the metadata used for similarity matching. * **Database** - contains the database contents with the metadata of each message analyzed by xorlab. An example of all files for a full and two incremental backups: ``` backup-2025-09-23T14-42-50.738330283Z-full-Assets backup-2025-09-23T14-42-50.738330283Z-full-ConfigRepos backup-2025-09-23T14-42-50.738330283Z-full-Database backup-2025-09-23T14-42-50.738330283Z-full-Redis backup-2025-09-23T14-42-50.738330283Z-full-info.json backup-2025-09-23T14-47-02.611662359Z-incremental-Assets backup-2025-09-23T14-47-02.611662359Z-incremental-ConfigRepos backup-2025-09-23T14-47-02.611662359Z-incremental-Database backup-2025-09-23T14-47-02.611662359Z-incremental-Redis backup-2025-09-23T14-47-02.611662359Z-incremental-info.json backup-2025-09-23T14-47-19.942717733Z-incremental-Assets backup-2025-09-23T14-47-19.942717733Z-incremental-ConfigRepos backup-2025-09-23T14-47-19.942717733Z-incremental-Database backup-2025-09-23T14-47-19.942717733Z-incremental-Redis backup-2025-09-23T14-47-19.942717733Z-incremental-info.json ``` Each file follows the naming schema: ``` backup-timestamp-type-module ``` It starts with the `backup-` prefix followed by the UTC timestamp, which corresponds to the time when the backup was scheduled/triggered. The name further contains the type of the backup and the name of the module that created it. The file ending in `info.json` contains meta-data about the backup. If the backup service is enabled, XCC saves backups to the chosen [backup directory](#backup-directory) according to the [backup schedules](#backup-schedule). ### Backup directory Backups will take up a significant amount of disk space. In order to not interfere with normal operations, e.g. by using up all available disk space, it is strongly recommended to keep the backup directory on a different mount point than the other production data. By default, the backup directory is set to `/backups`. You can change it in `/etc/xorlab/xcc/.env`: ```shell theme={null} BKP_BASE_DIR=/backups ``` This backup directory (`BKP_BASE_DIR`) is mapped to `/var/lib/xorlab/xcc/backup/backups` in the `backend` container, as configured in `/etc/xorlab/xcc/docker-compose.yml`: ```yaml theme={null} - ${BKP_BASE_DIR}:/var/lib/xorlab/xcc/backup/backups # Backup dir ``` ### Backup schedule The backup schedule syntax uses the Quartz cron format, and can be edited in `/xcc/backend/xcc.yml` via the [Expert Editor](/latest/expert-editor). The pattern represents: second, minute, hour, day, month, weekday. Month and weekday names can be given as the first three letters of the English names. Here are a few examples: * `0 0 6,19 * * *`: at 6:00 and 19:00, every day. * `0 0 9-17 * * MON-FRI`: every hour from 09:00 to 17:00, on weekdays. * `0 0 0 * * SUN`: every Sunday at midnight. * `"-"`: Never/disabled. The default configuration schedules incremental backups daily from Sunday to Friday at 12:00 and 21:00, and a full backup on Saturday at 01:00. Note that the cron schedules are mutually exclusive (no incremental backup scheduled for Saturday): ```yaml theme={null} backup: enabled: false incrementalBackupCron: 0 0 12,21 * * SUN-FRI fullBackupCron: 0 0 1 * * SAT deletion: enabled: true keepAdditional: 0 deleteDanglingChain: true ``` After editing the backup schedule in the Expert Editor, click **Publish** and restart the XCC. These general `xcc/backend/xcc.yml` settings require an XCC restart; see [How to Activate the Configuration](/latest/activate-configuration). You can use online tools, such as [Cron Expression Generator & Explainer](https://www.freeformatter.com/cron-expression-generator-quartz.html) or [crontab guru](https://crontab.guru/), to verify that your cron schedule. Be mindful, that crontab guru follows the standard cron syntax (no seconds field). **Incremental backup schedule** Even if an incremental backup is scheduled, XCC will do a **full** backup if one of the following conditions apply: * XCC runs on a more recent minor version (after an update full backup is scheduled automatically). * There is an incomplete full backup that can be resumed. * No previous backup is found in the backup directory. ### Backup settings By default, automatic backup deletion is enabled. Only the current backup chain is kept (and one additional one related to the last minor release), and the dangling incremental backups (without associated full backup as their base) will be deleted. Here's the list of all backup configuration options that can be set in `/xcc/backend/xcc.yml`. ```yaml theme={null} backup: enabled: false # Cron pattern for incremental backup. May still trigger a full backup if # a new minor version is detected after an update, which necessitates a full # backup. "-" means disabled. incrementalBackupCron: 0 0 12,21 * * SUN-FRI # Cron pattern for full backup, "-" means disabled. fullBackupCron: 0 0 1 * * SAT # Maximum backup time in hours after which the backup gets aborted. maxBackupTimeH: 72 # Time in seconds to wait to acquire shared table locks at the beginning of the dump. dbTableLockTimeoutS: 3600 # Time in hours to wait for a database dump. dbDumpTimeoutH: 24 # Whether to check if the last backup is complete on startup and immediately resume # incomplete backup. resumeLastBackupOnStartup: true # Number of concurrent threads during backup. maxBackupModuleThreads: 20 # Comma-separated list of module names to skip for full backups. # Can be one of Database, ConfigRepos, Assets, Redis skipModuleFull: "" # Comma-separated list of module names to skip for incremental backups: # Can be one of Database, ConfigRepos, Assets, Redis skipModuleIncremental: "" # Backup chains retention policy. Scanning for old backups gets triggered on startup and # after each scheduled backup. deletion: ## If backups should be deleted at all. enabled: true # Keep backup chain if it is no older than this number of days. keepForDays: 0 # Keep X additional backup chains. The current chain will never be deleted. keepAdditional: 0 # Keep last backup chain of previous (minor) release. Recommended for rolling # back an update in the worst case. keepLastPreviousRelease: true # If a backup chain without a full backup can be deleted. deleteDanglingChain: true ``` ### Manual backup When you need to create backups manually—for example, when the automatic backup was performed a couple of days ago and you want to make some substantial changes such as upgrading to a new xorlab version—you can can trigger a backup manually from the XCC web interface. 1. Log in to the XCC web interface. 2. On the main screen, click the tiles icon next to your account name and click the **Admin** icon in the displayed menu: 3. Open the **Backup** page: Backup tile 4. Press the **New Incremental Backup** button, or choose **New Full Backup**: Backup page ## Recovery **xorlab will automatically start a recovery procedure during startup, if the recovery folder contains a valid backup. This recovery will overwrite the current data. Make sure to only populate the recovery folder if you want to recover from a backup.** **If you require assistance, or are in doubt, do not hesitate to contact us via [support@xorlab.com](mailto:support@xorlab.com).** The recovery process starts automatically if the `backend` service finds backup files in the recovery directory during startup. It will commence with the most recent full backup and proceed to apply all subsequent incremental backups. Here are the steps needed to perform a recovery: 1. [Restore Host Configuration](#restore-host-configuration) 2. [Populate Recovery Directory](#populate-recovery-directory) 3. [Start Recovery Process](#start-recovery-process) 4. [Clear Recovery Directory](#clear-recovery-directory) 5. [Re-start xcc Stack](#re-start-xcc-stack) 6. [Re-synchronize Elastic indices](#re-synchronize-elastic-indices) ### Restore host configuration Host configuration, that was changed during an update, is backed up to `/etc/xorlab/backup/`. ### Populate recovery directory If there is only one backup chain in the backup folder, you can simply set the recovery folder to the backup folder and skip step 1. & 2. In case of several backup chains, move the backup chain you would like to recover to a separate recovery folder. It is recommended to use a path on the same mountpoint as the backup directory, so that moving the backups to the recovery directory does not involve actual disk IO. Assuming `/backups` is used as the backup directory: 1. Create a subfolder `recovery` owned by `xcc_backend` ```shell theme={null} cd /backups sudo mkdir recovery sudo chown xcc_backend: recovery/ ``` 2. Move the backup you want to restore to the `recovery` folder, e.g.: ```shell theme={null} sudo mv backup-2025-10-10* recovery/ ``` 3. Configure your recovery folder in `docker-compose.yml`. In the following example, we use `${BKP_BASE_DIR}/recovery`: ```yaml /etc/xorlab/xcc/docker-compose.yml {7} theme={null} backend: image: ${XOR_REGISTRY}/xcc/backend:${XCC_VERSION} container_name: xcc_backend volumes: [...] - ${WRK_BASE_DIR}:/var/lib/xorlab/xcc - ${BKP_BASE_DIR}/recovery:/var/lib/xorlab/xcc/backup/recovery # Recovery dir - ${BKP_BASE_DIR}:/var/lib/xorlab/xcc/backup/backups # Backup dir ``` ### Start recovery process 1. Make sure all containers from the xcc stack are stopped: ```shell theme={null} cd /etc/xorlab/xcc && docker compose down ``` 2. This step is only required when recovering from a backup that has been taken with an older major or minor version than the one currently running on XCC. 1. Delete the existing DB: ```shell theme={null} sudo find /var/lib/xorlab/xcc/postgres -mindepth 1 -delete ``` 1. Delete existing Elastic data: ```shell theme={null} cd /etc/xorlab/elastic && docker compose down && \ sudo find /var/lib/xorlab/elastic -mindepth 1 -delete && \ docker compose up -d ``` 3. Start PostgreSQL—the database has to be running to recover its data: ```shell theme={null} docker compose up -d db ``` 4. Start the backend service: ```shell theme={null} docker compose up backend ``` and observe the logs. After a few seconds status reports similar to the following will be shown: ```java theme={null} xcc_backend | INFO [RecoveryServiceThread-6]: c.x.j.b.m.ModuleRunner - ### Status Report: xcc_backend | Assets [30s ago]: Extracting to /var/lib/xorlab/xcc/backend/assets (no progress will be reported until done.) xcc_backend | Database [18s ago]: pg_restore: finished item 3439 INDEX uq_asset_ggrid_name xcc_backend | RecoveryService [30s ago]: Starting recovery modules [com.xorlab.jxcc.backup.module.recovery.AssetsRecoveryModule@ac91282, com.xorlab.jxcc.backup.module.recovery.DatabaseRecoveryModule@7f79edee] for backup Backup(2025-10-29T04:00:00.016296214Z, full, /var/lib/xorlab/xcc/backup/recovery) ``` 5. Once the recovery is complete, the JVM in the `xcc_backend` container will shut down with exit code 3, and you will see this line: ```shell theme={null} INFO exited: xcc (exit status 3; expected) ``` 6. Ctrl+C to close the logs and stop the container. **Errors from `pg_restore`** xorlab uses `pg_restore` to restore backups. All log lines containing `Found expected (non-)error from pg_restore` can be safely ignored as they do not influence the recovery. ### Clear recovery directory Restore the original bind mount in `docker-compose.yml`. To avoid an accidental second recovery, the mapped folder `${WRK_BASE_DIR}/backup/recovery`, which is `/var/lib/xorlab/xcc/backup/recovery/` with default settings, must be empty. ```yaml /etc/xorlab/xcc/docker-compose.yml {7} theme={null} backend: image: ${XOR_REGISTRY}/xcc/backend:${XCC_VERSION} container_name: xcc_backend volumes: [...] - ${WRK_BASE_DIR}:/var/lib/xorlab/xcc - ${WRK_BASE_DIR}/backup/recovery:/var/lib/xorlab/xcc/backup/recovery # Recovery dir - ${BKP_BASE_DIR}:/var/lib/xorlab/xcc/backup/backups # Backup dir ``` ### Re-start XCC stack Ensure that the Elastic stack is running, if it was previously stopped, run: ```shell theme={null} cd /etc/xorlab/elastic && docker compose up -d ``` and restart xcc stack: ```shell theme={null} cd /etc/xorlab/xcc && docker compose down && docker compose up -d ``` ### Re-synchronize elastic indices During the recovery process, elastic indices are automatically marked for re-synchronization, but need to be deleted manually (they will be re-synchronized from the database automatically). Otherwise, the XCC web interface will still show the emails from before the recovery in the search overview. To trigger re-synchronization: 1. Ensure that the Auth stack is running, if it was previously stopped, run: ```shell theme={null} cd /etc/xorlab/auth && docker compose up -d ``` 2. Open the XCC web interface and go to **Insights** from the sidebar menu. 3. Open any of the dashboards, a new tab will open. 4. Expand the **Elastic** side menu on the left and open **Stack Management** located under the **Management** section. 5. Open **Index Management** located under the **Data** section (from the updated left side menu). 6. Search for the `xcc_result_publish_`. 7. Check the box to the left of the result. 8. Press the blue **Manage index** button, and choose **Delete Index**. 9. Stop the `xcc_backend` container: ```shell theme={null} cd /etc/xorlab/xcc && docker compose stop backend ``` 10. Mark data for re-synchronization with: ```shell theme={null} docker compose exec db psql -U xcc -c "UPDATE xcc.result_publication SET es_sync_version=0, es_sync_attempts=0;" ``` 11. Start the `xcc_backend` container: ```shell theme={null} docker compose up -d backend ``` **The analyst dashboard will appear empty or incomplete while data is being re-synchronized. Depending on the amount of emails, it may take minutes to hours until this process is completed.** ### Recovery directory cleanup If you created a separate recovery folder in step [Populate Recovery Directory](#populate-recovery-directory), move the backup chain back to the backup folder, as xorlab will not clean up old backup chains from folders other than the backup folder. ## Patch rollback If the patch upgrade was not successful, it’s possible to revert the changes. All components can be restored to the **previous** patch version (unless stated otherwise in the [release notes](/latest/upgrade-guide#reading-the-changelog)) by performing the following steps: 1. Set the version variables (`*_VERSION`) to the **previous** patch version in the `/etc/xorlab//.env` file. 2. Restart the component: ```shell theme={null} cd /etc/xorlab/ && docker compose up -d ``` For example, to roll back the xcc component from `11.0.1` to the `11.0.0`: 1. Set the version variables to the `11.0.0`: ```shell /etc/xorlab/xcc/.env theme={null} (...) XCCWEB_VERSION=11.0.0 # was 11.0.1 XCC_VERSION=11.0.0 # was 11.0.1 XCC_DB_VERSION=11.0.0 # was 11.0.1 XCC_REDIS_VERSION=11.0.0 # was 11.0.1 (...) ``` 2. Restart the xcc component: ```shell theme={null} cd /etc/xorlab/xcc && docker compose up -d ``` In case the patch rollback fails, please refer to the [Recovery](#recovery) section, to perform a restore from the backup chain created before the update.