Data Flow and Data Storage

This page provides a high-level overview of how email data flows through the xorlab Security Platform and how data is stored at each stage of processing. The xorlab platform is composed of multiple tightly integrated components:

MTA (Mail Transfer Agent): Handles inbound and outbound email traffic. It performs the email analysis and makes a final decision for the email classification.
XCC (xorlab Control Center): Serves as the central orchestration layer. It aggregates and post-processes analysis results from the MTA, applies configuration updates, enriches data with external threat intelligence (TI) if available, and presents users findings via the XCC interface.
Centralized xorlab services: A set of centralized services operated by xorlab that are used by the MTA and XCC to enhance detection and classification accuracy. Some services are optional, some are mandatory. The xorlab Dynamic Analysis service is the only exception that can also be run on-premise.

This architecture covers data storage points, external communication, and protocol usage.

Data Flow

The data flow diagram and the order in which an email passes through the processing stages are documented in Data Flow Details → Email processing pipeline. Password required.

Note: Optional threat intelligence sources (e.g., VirusTotal) can be integrated to assist analysts during manual investigations.

Data used in external queries

The following table shows which data is used by the MTAs in external queries:

Data attribute	Encrypted	Query endpoint	Mandatory	Comment
Domains, IPs, URLs	Yes	xorlab xCloud	Yes	Query global information for domains, IPs, and URLs that are found in an email
Detection indicators	Yes	xorlab ML	Yes	Analyze an email by sending data to the machine learning service. The data is not client-identifying, meaning no sensitive information is transmitted. What is sent is already visible under the CONTEXT VARIABLES tab in the message detail view and limited to the variables that start with `dce.`. The data includes SHA-256 hashes of attachments and files, and otherwise only inferred metadata of the email
Attachments	Yes	xorlab Dynamic Analysis	No	Certain suspicious attachments are temporarily uploaded to a secure cloud environment for dynamic analysis (sandbox). After the scan is completed, relevant data is automatically removed from the cloud, and the analysis results are stored locally. If a customer runs its own xorlab Dynamic Analysis locally, then the local service will be used instead.
Attachments	Yes	xorlab Tika	Yes	Used to extract text and URLs from attachments. Files are sent to Tika over HTTPS and processed in-memory; they are not retained after processing. Disabling this feature may reduce classification performance. If a customer runs its own xorlab Dynamic Analysis locally, then the local Tika service will be used instead (Tika runs on the same system as the xorlab Dynamic Analysis).
Email (full content)	Yes	xorlab NLU	No	Sensitive email content is sent to xorlab NLU for text analysis. This feature is optional and disabled by default for on-premises installations. An on-premises version is planned for a future release.
Domains, IPs (A, MX, PTR, TXT)	No	Local DNS server (on-prem)	Yes	Used to verify SPF/DKIM, resolve domains, and, if manually enabled, query third-party DNS-based TI services. DNS queries are made directly from the MTA and may appear in internal DNS logs.

The following table shows which data is used by the XCC in external queries:

Data attribute	Encrypted	Query endpoint	Mandatory	Comment
Attachment hashes and URLs	Yes	External TI (VirusTotal)	No	On-demand VirusTotal query to get more threat intelligence for attachments and URLs ¡
Telemetry	Yes	Telemetry	Yes	Information on the hardware the XCC VM is running on (CPU, memory, …), information on the XCC VM (versions, packages, …), information on the product usage.

Locally stored data

Data sensitivity

For the purpose of this documentation, sensitive data is categorized as follows: ####Sensitivity: Low Data that does not contain Customer Identifying Data (CID) or Personally Identifiable Information (PII) and cannot typically be linked to a specific email. Examples are collections of Tags used by xorlab ML to classify email or the DNS records of XSP infrastructure. ####Sensitivity: Medium Data that may include CID or PII, such as email metadata (Subject, Sender, Recipient), but does not include full email content or attachments. ####Sensitivity: High Data such as full email bodies and/or attachments.

XCC

The Control Center stores both structured and unstructured data. Key data types, such as Email metadata and Log files, are detailed below. All stored data is subject to a retention time.

Data type	Sensitivity	Stored	Format	Retention time
Email metadata	Medium	Always	Plaintext, structured	Configurable, default is 90 days
Email subject	Medium	Configurable, options: - Do not store - Store only for malicious emails (default) - Store for all emails	Plaintext, structured	Configurable, default is 90 days
Email content	High	Always kept for quarantined or reported emails. For all other cases: configurable, options: - Do not store (default) - Store only for malicious emails - Store for all emails	Plaintext (EML), unstructured	Configurable. Quarantined emails are kept according to the quarantine configuration. By default, quarantined emails are kept for 30 days and reported emails are kept for two years.
Email content screenshots	High	Configurable, options: - Do not store - Store only for malicious emails (default) - Store for all emails	JPEG, rendered image	Configurable, default is 90 days
Email attachment screenshots and videos	High	Applies only to dynamically analyzed attachments. Configurable, options: - Do not store - Store for all dynamically analyzed attachments (default)	JPEG, MP4	Configurable, default is 90 days
Email attachments	High	Configurable, options: - Do not store (default) - Store for all emails	Unstructured file, unencrypted	Configurable, default is 90 days
Trustmodel (communication relationships)	High	Always	Hashed, structured	Not configurable. Entities are removed from the trust model after three months of inactivity

Email metadata

The email metadata contains the following attributes:

Data attribute	Example
All sender and recipient email addresses	`account@xorlab.com`
Sender and recipient display names	`Mr. xorlab`
Sending MTA IP address	`1.2.3.4`
Received at (timestamp)	`2020-01-01T01:01:01+00:00`
Body URLs	`https://www.xorlab.com`
Matched email content and subject keywords	Popular phishing or spam keywords like `payment`, `drugs`, `banks`, etc.
All email headers (except subject)	`Received`, `Date`, `From`, `To`, etc.
All analysis results	Authentication information, extracted VBS from office attachments, events during dynamic analysis, etc.
Attachment names	`newsletter.pdf`

MTA

An MTA maintains a persistent cache for various information like local domain reputation, trust scores, shortened URLs, etc. The cache contains only aggregated information and does not reference or relate to single emails.

Data type	Sensitivity	Stored	Format	Retention time
Cache for: - Domain and IP information - Email addresses - Shortened URLs	Low	Always	Plaintext. Aggregated, data does not relate to single emails	Varies between the entities. Between one hour and two weeks

Storage during Dynamic Analysis

The xorlab Dynamic Analysis component only stores files during the dynamic analysis in the sandbox environment. Files are deleted immediately after processing, typically within minutes. In any case, files are automatically removed within a few hours, even if the requesting MTA is unavailable.

Log files

Every xorlab Security Platform component writes local logfiles containing runtime events of different severity.

Data type	Sensitivity	Stored	Format	Retention time
Log files	Low to Medium	Configurable. Can be disabled (not recommended)	Plaintext	Configurable, default is one month

The MTA and XCC log files may contain the same data which is described in the Email metadata section, and thus may contain email related data. SaaS instances may send log files to a centralized logging server operated by xorlab in the same region as the SaaS instance, i.e. in Switzerland for CH instances and in Germany for instances outside of Switzerland.
The log levels (DEBUG, INFO, WARN, ERROR) can be used to reduce the amount of email related data written to the log files, but overall there may always be email related data in the log files.
The xorlab Dynamic Analysis component log files contain less sensitive data. The only email related data that those log files may contain are the names of the files (attachments) that are scanned.
The xorlab NLU log files do not contain any sensitive data. Only properties such as the lengths of analyzed text or the number of requests are stored.
Important runtime events, such as operational and audit logs, can be sent to external receivers. Please see Logging for more information.

Externally processed data

The information below is provided as an overview. Please refer to your Master Agreement or Data Processing Agreement for contractually binding details.

Some services are cloud-based and involve off-site processing. The following tables provide an overview of the different services, per customer regions. The column Data processed by indicates whether the data is processed by xorlab or the sub-processor. If xorlab is listed, data is processed by xorlab software running on (virtualized) hardware of the sub-processor.

Customers in Switzerland

Service	Sensitivity	Sub-processor	Data processed by	Region	Typical retention time
xorlab Dynamic Analysis	High	Green Datacenter	xorlab	Switzerland	2-3 minutes
xorlab Tika	High	Green Datacenter	xorlab	Switzerland	a few seconds
xorlab xCloud	Low	Green Datacenter	xorlab	Switzerland	less than a second
xorlab ML	Low	Azure CH	xorlab	Switzerland	less than a second
xorlab NLU	High	Azure CH	xorlab	Switzerland	less than a second
Logs (only for SaaS)	Medium	Azure CH	xorlab	Switzerland	180 days
DNS records (only for SaaS)	Low	Cloudflare	Cloudflare	not defined	less than a second

Customers outside of Switzerland

Service	Sensitivity	Sub-processor	Data processed by	Region	Typical retention time
xorlab Dynamic Analysis	High	Hetzner	xorlab	Germany	2-3 minutes
xorlab Tika	High	Hetzner	xorlab	Germany	a few seconds
xorlab xCloud	Low	Hetzner	xorlab	Germany	less than a second
xorlab ML	Low	Azure West Europe	xorlab	Netherlands	less than a second
xorlab NLU	High	Azure West Europe	xorlab	Netherlands	less than a second
Logs (only for SaaS)	Medium	Hetzner	xorlab	Germany	180 days
DNS records (only for SaaS)	Low	Cloudflare	Cloudflare	not defined	less than a second

xorlab Dynamic Analysis

xorlab Dynamic Analysis Service scans selected email attachments for malicious behavior in a secure sandbox environment. Supported file types include:

MS Office files: Word (including RTF), Excel (including CSV), PowerPoint, OneNote
PDF files

These files are transmitted over HTTPS and are only stored temporarily—typically for a few minutes—during the analysis process. After analysis, they are automatically deleted. You can configure which file types are sent for analysis. By default, the xorlab Security Platform does not dynamically scan attachments from trusted senders. Disabling file submission effectively turns off dynamic analysis. You may also disable xorlab Dynamic Analysis entirely, if required.

xorlab Tika

The Tika service is used to improve the URL extraction and text extraction from attachments (for example, through OCR). It processes a defined set of file formats. Attachments are sent to Tika via HTTPS for processing, then the extracted text is sent back. Files are only kept during processing. Tika sould not be disabled as it is a critical component for email classification.

xorlab ML

The xorlab ML service uses machine learning based on indicators found in emails to further improve the message classification. The indicators are sent to the ML service over an HTTPS connection. The data does not contain personal or customer-identifying data, nor does it contain the message body data. The data is only kept during processing. xorlab ML sould not be disabled as it is a critical component for email classification.

xorlab NLU

The xorlab NLU service uses natural language understanding to enhance the classification accuracy of the xorlab Security Platform by analyzing the semantics and intent of email content. When enabled, selected content such as the subject line, message body, and extracted attachment text is securely transmitted to the NLU service via HTTPS. This data may include customer-identifying information and is therefore treated as sensitive. The transmitted data is processed in-memory only and is never persisted or stored after analysis. For on-premise deployments this service is disabled by default and needs to be manually activated.

xorlab xCloud

xorlab maintains a centralized service for information around IPs, domains and URLs that is queried by the xorlab Security Platform. These queries are sent via HTTPS and contain domain names or IP addresses found in messages analyzed by xorlab Security Platform. You can’t turn them off. In general, the xorlab xCloud does not store the IPs, domains URLs beyond the time of processing. It may do so only in case of an error. For shortened URLs (e.g., bit.ly/foo), xorlab xCloud may forward only the shortened URL to the external shortener service to resolve the URL. However the final URL is not accessed at all. The traffic is via HTTPS. The xorlab xCloud is active by default and cannot be disabled.

Blocklist integrations via DNS

Customers may enable the integration with third-party DNS TI services such as Abusix or Spamhaus. In such a case, xorlab may send domains, IPs, URLs and hashes to the third party TI service via DNS requests. This integration is disabled by default and needs to be manually activated.

Misclassification reports

The data in these reports depends on the reporter’s chosen options: it’s possible to share metadata, previews, and original email; share metadata only, or share anonymized metadata. The metadata means the following information:

Information	Description
`ggrid`	Unique ID
`direction`	Incoming, outgoing, internal
`contextFields`	Context variables
`coreCreated`	Analysis start time
`coreCompleted`	Analysis end time
`xccReceived`	Time when XCC received the message
`coreVerdict`	Message verdict
`coreVersion`	–
`pipeline`	Email, AnalysisFile, AnalysisEmail, or Report
`tags`	Tags added by XSP
`userTags`	Tags added by the admin/analyst
`appliedRules`	Rules applied to this message
`passiveRules`	Passive rules
`ruleSetEssentialInfo`	Finger printing
`decisionHistory`	Series of decisions made by Core
`spamScore`	–
`rspamdScore`	Rspamd: external
`reporterClassification`	Reporter’s expected classification if the message is reported
`reporterVerdict`	Reporter’s expected verdict if the message is reported
`tenantId`	Unique customer ID
`principal`	Reporter’s username and permissions
`submissionDate`	–
`perceivedSeverity`	Misclassification severity as perceived by the reporter: low, moderate, major, critical
`expectedClassification`	Classification as expected by the reporter: benign, malicious, or other
`reporterMessage`	Message from the misclassification reporter
`event`	Where the misclassification is reported from: isolation, report, direct submission

Most of this data is contained in the Context fields (you can see it by clicking the Context tab in the message detail view). Reports are sent to xorlab Cloud via HTTPS, then they are processed by us and stored indefinitely on our infrastructure.

​Data Flow

​Data used in external queries

​Locally stored data

​Data sensitivity

​XCC

​Email metadata

​MTA

​Storage during Dynamic Analysis

​Log files

​Externally processed data

​Customers in Switzerland

​Customers outside of Switzerland

​xorlab Dynamic Analysis

​xorlab Tika

​xorlab ML

​xorlab NLU

​xorlab xCloud

​Blocklist integrations via DNS

​Misclassification reports

Data Flow

Data used in external queries

Locally stored data

Data sensitivity

XCC

Email metadata

MTA

Storage during Dynamic Analysis

Log files

Externally processed data

Customers in Switzerland

Customers outside of Switzerland

xorlab Dynamic Analysis

xorlab Tika

xorlab ML

xorlab NLU

xorlab xCloud

Blocklist integrations via DNS

Misclassification reports