> ## Documentation Index
> Fetch the complete documentation index at: https://docs.xorlab.com/llms.txt
> Use this file to discover all available pages before exploring further.
> ## Agent Instructions
> Pages under /latest/ document the current release. Paths that begin with a version number, for example /10.0/, are frozen snapshots of superseded releases kept for reference only — never present their instructions as current. If the reader's version is unknown, answer from /latest/ and say which version the answer describes.
> xorlab is deployed on-premises, hybrid, or as a cloud service, and integrates with either Microsoft 365 or on-premises mail infrastructure. Configuration steps often differ between these. State which deployment and integration mode an instruction applies to instead of presenting one as universal.
> Distinguish inline mode from monitoring mode when describing anything that acts on email. Monitoring mode observes a copy and cannot block or quarantine; inline mode routes mail through xorlab and can.
> Write the product name as lowercase 'xorlab'. Use the documented component names: xorlab Control Center (XCC), xorlab MTA, xorlab Sandbox (DANA), xorlab Natural Language Understanding (NLU). After the first mention, use the short forms XCC, MTA, Sandbox, and NLU. Do not use DANA as a standalone name for the Sandbox, but keep it where it is a literal string in configuration keys, container names, and hostnames.
> Do not invent configuration keys, rule parameters, list names, log properties, or API fields. If a value is not present in this documentation, say that it is not documented rather than guessing.
# Data Flow and Data Storage
This page provides a high-level overview of how email data flows through the xorlab Security Platform and how data is stored at each stage of processing. The xorlab platform is composed of multiple tightly integrated components:
* MTA (Mail Transfer Agent): Handles inbound and outbound email traffic. It performs the email analysis and makes a final decision for the email classification.
* XCC (xorlab Control Center): Serves as the central orchestration layer. It aggregates and post-processes analysis results from the MTA, applies configuration updates, enriches data with external threat intelligence (TI) if available, and presents users findings via the XCC interface.
* Centralized xorlab services: A set of centralized services operated by xorlab that are used by the MTA and XCC to enhance detection and classification accuracy. Some services are optional, some are mandatory. The xorlab Dynamic Analysis service is the only exception that can also be run on-premise.
This architecture covers data storage points, external communication, and protocol usage.
## Data Flow
The data flow diagram and the order in which an email passes through the processing stages are
documented in
[Data Flow Details → Email processing pipeline](/9.0/data-flow-details#email-processing-pipeline).
Password required.
Note: Optional threat intelligence sources (e.g., VirusTotal) can be integrated to assist analysts during manual investigations.
## Data used in external queries
The following table shows which data is used by the MTAs in external queries:
| Data attribute | Encrypted | Query endpoint | Mandatory | Comment |
| :----------------------------- | :-------- | :------------------------- | :-------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Domains, IPs, URLs | Yes | xorlab xCloud | Yes | Query global information for domains, IPs, and URLs that are found in an email |
| Detection indicators | Yes | xorlab ML | Yes | Analyze an email by sending data to the machine learning service. The data is not client-identifying, meaning no sensitive information is transmitted. What is sent is already visible under the **CONTEXT VARIABLES** tab in the message detail view and limited to the variables that start with `dce.`. The data includes SHA-256 hashes of attachments and files, and otherwise only inferred metadata of the email |
| Attachments | Yes | xorlab Dynamic Analysis | No | Certain suspicious attachments are temporarily uploaded to a secure cloud environment for dynamic analysis (sandbox). After the scan is completed, relevant data is automatically removed from the cloud, and the analysis results are stored locally. If a customer runs its own xorlab Dynamic Analysis locally, then the local service will be used instead. |
| Attachments | Yes | xorlab Tika | Yes | Used to extract text and URLs from attachments. Files are sent to Tika over HTTPS and processed in-memory; they are not retained after processing. Disabling this feature may reduce classification performance. If a customer runs its own xorlab Dynamic Analysis locally, then the local Tika service will be used instead (Tika runs on the same system as the xorlab Dynamic Analysis). |
| Email (full content) | Yes | xorlab NLU | No | Sensitive email content is sent to xorlab NLU for text analysis. This feature is optional and disabled by default for on-premises installations. An on-premises version is planned for a future release. |
| Domains, IPs (A, MX, PTR, TXT) | No | Local DNS server (on-prem) | Yes | Used to verify SPF/DKIM, resolve domains, and, if manually enabled, query third-party DNS-based TI services. DNS queries are made directly from the MTA and may appear in internal DNS logs. |
The following table shows which data is used by the XCC in external queries:
| Data attribute | Encrypted | Query endpoint | Mandatory | Comment |
| :------------------------- | :-------- | :----------------------- | :-------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Attachment hashes and URLs | Yes | External TI (VirusTotal) | No | On-demand VirusTotal query to get more threat intelligence for attachments and URLs ¡ |
| Telemetry | Yes | Telemetry | Yes | Information on the hardware the XCC VM is running on (CPU, memory, ...), information on the XCC VM (versions, packages, ...), information on the product usage. |
## Locally stored data
### Data sensitivity
For the purpose of this documentation, sensitive data is categorized as follows:
\####Sensitivity: Low
Data that does not contain Customer Identifying Data (CID) or Personally Identifiable Information (PII) and cannot typically be linked to a specific email. Examples are collections of [Tags](/9.0/tags) used by [xorlab ML](#xorlab-ml) to classify email or the DNS records of XSP infrastructure.
\####Sensitivity: Medium
Data that may include CID or PII, such as email metadata (Subject, Sender, Recipient), but does not include full email content or attachments.
\####Sensitivity: High
Data such as full email bodies and/or attachments.
### XCC
The Control Center stores both structured and unstructured data. Key data types, such as Email metadata and Log files, are detailed below. All stored data is subject to a retention time.
| Data type | Sensitivity | Stored | Format | Retention time |
| :--------------------------------------- | :---------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :----------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Email metadata | Medium | **Always** | Plaintext, structured | Configurable, default is 90 days |
| Email subject | Medium | Configurable, options:
- Do not store
- **Store only for malicious emails (default)**
- Store for all emails | Plaintext, structured | Configurable, default is 90 days |
| Email content | High | Always kept for quarantined or reported emails. For all other cases: configurable, options:
- **Do not store (default)**
- Store only for malicious emails
- Store for all emails | Plaintext (EML), unstructured | Configurable. Quarantined emails are kept according to the quarantine configuration. By default, quarantined emails are kept for 30 days and reported emails are kept for two years. |
| Email content screenshots | High | Configurable, options:
- Do not store
- **Store only for malicious emails (default)**
- Store for all emails | JPEG, rendered image | Configurable, default is 90 days |
| Email attachment screenshots and videos | High | Applies only to dynamically analyzed attachments. Configurable, options:
- Do not store
- **Store for all dynamically analyzed attachments (default)** | JPEG, MP4 | Configurable, default is 90 days |
| Email attachments | High | Configurable, options:
- **Do not store (default)**
- Store for all emails | Unstructured file, unencrypted | Configurable, default is 90 days |
| Trustmodel (communication relationships) | High | **Always** | Hashed, structured | Not configurable. Entities are removed from the trust model after three months of inactivity |
#### Email metadata
The email metadata contains the following attributes:
| Data attribute | Example |
| :----------------------------------------- | :------------------------------------------------------------------------------------------------------ |
| All sender and recipient email addresses | `account@xorlab.com` |
| Sender and recipient display names | `Mr. xorlab` |
| Sending MTA IP address | `1.2.3.4` |
| Received at (timestamp) | `2020-01-01T01:01:01+00:00` |
| Body URLs | `https://www.xorlab.com` |
| Matched email content and subject keywords | Popular phishing or spam keywords like `payment`, `drugs`, `banks`, etc. |
| All email headers (except subject) | `Received`, `Date`, `From`, `To`, etc. |
| All analysis results | Authentication information, extracted VBS from office attachments, events during dynamic analysis, etc. |
| Attachment names | `newsletter.pdf` |
### MTA
An MTA maintains a persistent cache for various information like local domain reputation, trust scores, shortened URLs, etc. The cache contains only aggregated information and does not reference or relate to single emails.
| Data type | Sensitivity | Stored | Format | Retention time |
| :--------------------------------------------------------------------------------------- | :---------- | :--------- | :----------------------------------------------------------- | :---------------------------------------------------------- |
| Cache for:
- Domain and IP information
- Email addresses
- Shortened URLs | Low | **Always** | Plaintext. Aggregated, data does not relate to single emails | Varies between the entities. Between one hour and two weeks |
### Storage during Dynamic Analysis
The xorlab Dynamic Analysis component only stores files during the dynamic analysis in the sandbox environment. Files are deleted immediately after processing, typically within minutes. In any case, files are automatically removed within a few hours, even if the requesting MTA is unavailable.
### Log files
Every xorlab Security Platform component writes local logfiles containing runtime events of different severity.
| Data type | Sensitivity | Stored | Format | Retention time |
| :-------- | :------------ | :---------------------------------------------- | :-------- | :--------------------------------- |
| Log files | Low to Medium | Configurable. Can be disabled (not recommended) | Plaintext | Configurable, default is one month |
* The MTA and XCC log files may contain the same data which is described in the *[Email metadata](#email-metadata)* section, and thus may contain email related data.
SaaS instances may send log files to a centralized logging server operated by xorlab in the same region as the SaaS instance, i.e. in Switzerland for CH instances and in Germany for instances outside of Switzerland.
* The log levels (DEBUG, INFO, WARN, ERROR) can be used to reduce the amount of email related data written to the log files, but overall there may always be email related data in the log files.
* The xorlab Dynamic Analysis component log files contain less sensitive data. The only email related data that those log files may contain are the names of the files (attachments) that are scanned.
* The xorlab NLU log files do not contain any sensitive data. Only properties such as the lengths of analyzed text or
the number of requests are stored.
* Important runtime events, such as operational and audit logs, can be sent to external receivers. Please see *[Logging](/9.0/logging)* for more information.
## Externally processed data
The information below is provided as an overview. Please refer to your Master Agreement or Data Processing Agreement for contractually binding details.
Some services are cloud-based and involve off-site processing. The following tables provide an overview of the different services, per customer regions.
The column *Data processed by* indicates whether the data is processed by xorlab or the sub-processor. If xorlab is listed, data is processed by xorlab software running on (virtualized) hardware of the sub-processor.
### Customers in Switzerland
| Service | Sensitivity | Sub-processor | Data processed by | Region | Typical retention time |
| :-------------------------------------------------- | :---------- | :-------------------------------------------------------------------------------- | :---------------- | :---------- | :--------------------- |
| [xorlab Dynamic Analysis](#xorlab-dynamic-analysis) | High | [Green Datacenter](https://www.green.ch/en/about-green/what-we-want/certificates) | xorlab | Switzerland | 2-3 minutes |
| [xorlab Tika](#xorlab-tika) | High | [Green Datacenter](https://www.green.ch/en/about-green/what-we-want/certificates) | xorlab | Switzerland | a few seconds |
| [xorlab xCloud](#xorlab-xcloud) | Low | [Green Datacenter](https://www.green.ch/en/about-green/what-we-want/certificates) | xorlab | Switzerland | less than a second |
| [xorlab ML](#xorlab-ml) | Low | [Azure CH](https://learn.microsoft.com/en-us/azure/compliance/) | xorlab | Switzerland | less than a second |
| [xorlab NLU](#xorlab-nlu) | High | [Azure CH](https://learn.microsoft.com/en-us/azure/compliance/) | xorlab | Switzerland | less than a second |
| [Logs](#log-files) (only for SaaS) | Medium | [Azure CH](https://learn.microsoft.com/en-us/azure/compliance/) | xorlab | Switzerland | 180 days |
| DNS records (only for SaaS) | Low | [Cloudflare](https://www.cloudflare.com/) | Cloudflare | not defined | less than a second |
### Customers outside of Switzerland
| Service | Sensitivity | Sub-processor | Data processed by | Region | Typical retention time |
| :-------------------------------------------------- | :---------- | :----------------------------------------------------------------------- | :---------------- | :---------- | :--------------------- |
| [xorlab Dynamic Analysis](#xorlab-dynamic-analysis) | High | [Hetzner](https://docs.hetzner.com/general/others/certificates/) | xorlab | Germany | 2-3 minutes |
| [xorlab Tika](#xorlab-tika) | High | [Hetzner](https://docs.hetzner.com/general/others/certificates/) | xorlab | Germany | a few seconds |
| [xorlab xCloud](#xorlab-xcloud) | Low | [Hetzner](https://docs.hetzner.com/general/others/certificates/) | xorlab | Germany | less than a second |
| [xorlab ML](#xorlab-ml) | Low | [Azure West Europe](https://learn.microsoft.com/en-us/azure/compliance/) | xorlab | Netherlands | less than a second |
| [xorlab NLU](#xorlab-nlu) | High | [Azure West Europe](https://learn.microsoft.com/en-us/azure/compliance/) | xorlab | Netherlands | less than a second |
| [Logs](#log-files) (only for SaaS) | Medium | [Hetzner](https://docs.hetzner.com/general/others/certificates/) | xorlab | Germany | 180 days |
| DNS records (only for SaaS) | Low | [Cloudflare](https://www.cloudflare.com/) | Cloudflare | not defined | less than a second |
### xorlab Dynamic Analysis
xorlab [Dynamic Analysis Service](/9.0/dynamic-analysis) scans selected email attachments for malicious behavior in a secure sandbox environment. Supported file types include:
* MS Office files: Word (including RTF), Excel (including CSV), PowerPoint, OneNote
* PDF files
These files are transmitted over HTTPS and are only stored temporarily—typically for a few minutes—during the analysis process. After analysis, they are automatically deleted.
You can [configure](/9.0/dynamic-analysis#configuration) which file types are sent for analysis. By default, the xorlab Security Platform does not dynamically scan attachments from trusted senders.
Disabling file submission effectively turns off dynamic analysis. You may also [disable xorlab Dynamic Analysis entirely](/9.0/dynamic-analysis#disabling-xorlab-sandbox), if required.
### xorlab Tika
The [Tika](/9.0/tika-onpremise) service is used to improve the URL extraction and text extraction from attachments (for example, through OCR). It processes a defined set of file formats.
Attachments are sent to Tika via HTTPS for processing, then the extracted text is sent back. Files are only kept during processing.
Tika sould not be disabled as it is a critical component for email classification.
### xorlab ML
The xorlab ML service uses machine learning based on indicators found in emails to further improve the message classification. The indicators are sent to the ML service over an HTTPS connection. The data does not contain personal or customer-identifying data, nor does it contain the message body data. The data is only kept during processing.
xorlab ML sould not be disabled as it is a critical component for email classification.
### xorlab NLU
The xorlab NLU service uses natural language understanding to enhance the classification accuracy of the xorlab Security Platform by analyzing the semantics and intent of email content.
When enabled, selected content such as the subject line, message body, and extracted attachment text is securely transmitted to the NLU service via HTTPS. This data may include customer-identifying information and is therefore treated as sensitive.
The transmitted data is processed in-memory only and is never persisted or stored after analysis.
For on-premise deployments this service is disabled by default and needs to be manually activated.
### xorlab xCloud
xorlab maintains a centralized service for information around IPs, domains and URLs that is queried by the xorlab Security Platform. These queries are sent via HTTPS and contain domain names or IP addresses found in messages analyzed by xorlab Security Platform. You can’t turn them off. In general, the xorlab xCloud does not store the IPs, domains URLs beyond the time of processing. It may do so only in case of an error.
For shortened URLs (e.g., `bit.ly/foo`), xorlab xCloud may forward only the shortened URL to the external shortener service to resolve the URL. However the final URL is not accessed at all. The traffic is via HTTPS.
The xorlab xCloud is active by default and cannot be disabled.
### Blocklist integrations via DNS
Customers may enable the integration with third-party DNS TI services such as Abusix or Spamhaus. In such a case, xorlab may send domains, IPs, URLs and hashes to the third party TI service via DNS requests.
This integration is disabled by default and needs to be manually activated.
### Misclassification reports
The data in these reports depends on the reporter's chosen options: it’s possible to share metadata, previews, and original email; share metadata only, or share anonymized metadata.
The metadata means the following information:
| Information | Description |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `ggrid` | Unique ID |
| `direction` | Incoming, outgoing, internal |
| `contextFields` | [Context variables](/9.0/context-vars) |
| `coreCreated` | Analysis start time |
| `coreCompleted` | Analysis end time |
| `xccReceived` | Time when XCC received the message |
| `coreVerdict` | Message verdict |
| `coreVersion` | – |
| `pipeline` | Email, AnalysisFile, AnalysisEmail, or Report |
| `tags` | Tags added by XSP |
| `userTags` | Tags added by the admin/analyst |
| `appliedRules` | Rules applied to this message |
| `passiveRules` | Passive rules |
| `ruleSetEssentialInfo` | Finger printing |
| `decisionHistory` | Series of decisions made by Core |
| `spamScore` | – |
| `rspamdScore` | Rspamd: external |
| `reporterClassification` | Reporter’s expected classification if the message is reported |
| `reporterVerdict` | Reporter’s expected verdict if the message is reported |
| `tenantId` | Unique customer ID |
| `principal` | Reporter’s username and permissions |
| `submissionDate` | – |
| `perceivedSeverity` | [Misclassification severity](/9.0/reporting-misclassification#severity-levels) as perceived by the reporter: low, moderate, major, critical |
| `expectedClassification` | Classification as expected by the reporter: benign, malicious, or other |
| `reporterMessage` | Message from the misclassification reporter |
| `event` | Where the misclassification is reported from: isolation, report, direct submission |
Most of this data is contained in the **Context** fields (you can see it by clicking the **Context** tab in the message detail view).
Reports are sent to xorlab Cloud via HTTPS, then they are processed by us and stored indefinitely on our infrastructure.