DARPA Transparent Computing Program

• DARPA Transparent Computing Program is a research initiative advancing unified telemetry integration for detecting and investigating cyberattacks.
• It leverages fine-grained provenance tracking, host telemetry, and network data to build scalable provenance graphs for real-time analysis.
• The program validates its approach via structured red-team exercises, achieving near 100% true positive rates in complex multi-stage APT scenarios.

The DARPA Transparent Computing (TC) Program is a research initiative focused on advancing real-time, system-wide cyberattack detection, forensic analysis, and post-mortem investigation through the comprehensive collection and fusion of provenance, host telemetry, and network activity data. By leveraging fine-grained tracking of information flows and rigorous policy-driven analytics, the program seeks to systematically expose sophisticated adversarial behaviors—from initial ingress to exfiltration—across large-scale, operationally realistic enterprise networks (Anjum et al., 2021, Eshete et al., 2016).

1. Program Objectives and Conceptual Framework

The Transparent Computing Program’s primary objective is to develop a cohesive, scalable security architecture capable of identifying and reconstructing complex, multi-stage cyberattacks. Its foundational concepts include:

• Unification of Telemetry: Integration of host-level telemetry (e.g., system calls), network-flow data, and provenance to provide a global perspective on system and application behaviors.
• Fine-Grained Provenance Tracking: Continuous monitoring and linkage of information flows from low-trust sources (“untrusted”) to high-value assets (“secret”), operationalized through event stream semantics and labeling.
• Adversarial Engagements: Structured red-team/blue-team exercises where attackers (TA1) inject advanced persistent threat (APT) scenarios into instrumented environments, while defenders (TA3) apply detection and forensic analytics to streaming event data, blinded to the ground truth (Eshete et al., 2016).

2. Data Models, Instrumentation, and Provenance Representation

The TC program utilizes several formal data representations tailored to high-throughput, consistent event tracking:

• Common Data Model (CDM): Encapsulates all relevant system events—file accesses, process creation, network socket operations, memory events—in a sequenced stream. Entities (processes, files, sockets) form nodes; events (e.g., read, write, connect, execve) form directed, labeled edges.
• Provenance Graphs: Defined as $G = (V, E)$, with
  • $V$ representing system entities,
  • $E \subset V \times V \times T$ (edges labeled by event types $T$)
  • Event streams unified into high-performance in-memory graphs, facilitating scalable policy evaluation.
• Event Schema (OpTC): Each event $e = (t, h, O, A, F, idₒ, idₐ, actor)$ where $t$ is timestamp, $h$ host ID, $O$ object type (FLOW, FILE, etc.), $A$ action, $F$ object-specific fields, $idₒ$, $idₐ$ unique IDs, and $actor$ the principal responsible (Anjum et al., 2021).
• Implication: This data architecture enables both per-host and global graph analytics, supporting rich downstream tasks such as anomaly detection, provenance querying, and process-tree construction.

3. Operationally Transparent Cyber (OpTC) Dataset

The OpTC dataset, central to the TC program’s contemporary research utility, exhibits the following characteristics (Anjum et al., 2021):

• Scale and Complexity: 17,433,324,390 events (≈17.4B) over 6 days, covering 1000 Windows 10 hosts in a live enterprise testbed.
• Event Modalities: Includes both network-flow records (Zeek/Bro-derived, “FLOW” events) and host-level telemetry (system calls, process/thread/file events) in JSON/eCAR format.
• Provenance Construction: Events support the synthesis of implicit provenance graphs ($G = (V, E)$ with $V$ all events, $E$ connecting actor ID to object ID with chronological constraint).
• APT Coverage: Features three full multi-day APT kill-chain scenarios—PowerShell Empire-based lateral movement, data exfiltration via Netcat/RDP, and a software-update trojan campaign.
• Labeling: All events during the “evaluation” period are tagged as benign or malicious, based on detailed red-team timelines and actor identification.
• Class Imbalance: Only ≈0.0016% of events are labeled malicious, reflecting realistic but challenging analytic conditions.
• Object/Action Distribution: Approximately 71.7% FLOW, 12.4% FILE, 8.6% PROCESS, and 7% other types (THREAD, MODULE, etc.).
• Benchmark Comparisons: OpTC surpasses older public datasets (e.g., LANL 2015, 2018) in event richness, APT scenario diversity, and linkage between host and network contexts.

4. Attack Scenarios, Detection Methodologies, and Performance Metrics

Within adversarial engagements, distinct attack campaigns were orchestrated and detected via provenance-driven analytics (Eshete et al., 2016):

• Attack Vectors:
  • Bovia: Exploitation of nginx/browser, dropper binary deployment and execution, exfiltration via SMTP-like channels.
  • Pandex: SSHD exploitation, backdoor installation with dropbear, system reconnaissance, exfiltration over SCP/custom sockets.
  • Stretch: Local code-build compromise, nginx binary replacement, credential exfiltration.
• Detection Policies:
  • Entities are assigned integrity ($I$) and confidentiality ($C$) tags (e.g., $I = \{$Whitelist, Invulnerable, Benign, Untrusted, Malicious$\}$; $C = \{$Public, Private, Sensitive, Secret$\}$).
  • Automatic alarms are triggered if provenance analysis reveals a policy violation path—e.g., exfiltration of $C = \text{Secret}$ data by an $I = \text{Untrusted}$ subject via a write event.
  • Anomaly Scoring: $A(e)=1$ if event $e$ violates policy, $0$ otherwise.
• Systems and Metrics:
  • UIC system: Streaming tag propagation and policy enforcement with real-time latency; analysis throughput up to 24x real-time speed-up.
  • SBU: In-memory Common Semantic Representation (CSR) graphs, ~2–3% of CDM disk size, analysis speed-ups of 50K–110K× per core.
  • Performance: True positive rate ≈100% for all injected attacks; minor false positives due to data format inconsistencies; negligible detection latency under streaming analysis conditions.
  • Practical Selectivity: For Bovia (TRACE): 2.3B CDM events filtered to 28K provenance events with 12 distinct alarms.

5. Qualitative Assessment, Limitations, and Benchmarking

The TC program’s datasets and methodologies display multiple strengths and some limitations:

• Strengths:
  • High-fidelity capture of enterprise-scale activity, including multi-step APT chains with staging, lateral movement, privilege escalation, and exfiltration.
  • Integration of eCAR metadata (command lines, image paths, registry keys) supports fine-grained analytics and feature engineering.
  • Joint host-network context (via eCAR-Bro linkage) enables fusion-based detection unavailable in legacy datasets.
  • Streaming provenance systems achieve high throughput with modest resource utilization.
• Limitations:
  • No raw packet-capture (pcap) data—only flow-level abstraction.
  • Strong class imbalance complicates learning-based detection.
  • Labelling inconsistencies due to actor_id de-confliction and event metadata quality (e.g., FLOW.acuity_level field errors).
  • Underrepresented event types (SERVICE, SHELL, USER_SESSION).
• Comparison Table:

Dataset	Events	Malicious Events	Duration	Hosts
OpTC	17.4B	292,000	6 days	1000
LANL'15	1.6B	749	58 days	17,684
LANL'18	5.5B	none documented	90 days	~17,500

• Benchmark Metrics: Precision ($TP/(TP+FP)$), Recall ($TP/(TP+FN)$), and $F_1$-score ($$2 \cdot (\text{Precision} \cdot \text{Recall})/(\text{Precision} + \text{Recall})$$) are used for evaluation.

6. Illustrative Event Analytics and Research Directions

Empirical analyses within the TC program highlight:

• Malicious Host Profiling: Malicious activity is highly concentrated; for example, on Day 1, 3.4% of hosts generate nearly all FLOW-MESSAGE (malicious) events during lateral movement phases; similar concentration for exfiltration flows on Days 2–3.
• Event-Type Distributions: Malicious periods see pronounced spikes in SHELL-COMMAND events (>2% in malicious, nearly 0 in benign), as well as PROCESS-CREATE and FILE-MODIFY aligned to attack windows.
• Forensic Visualizations: Simple event-type or host-flow statistics already yield high-discriminative features for anomaly/attack classifiers.
• Proposed Research:
  • Anomaly Detection at Scale: The scale of OpTC (17B events) supports the training and evaluation of deep models, especially those robust to severe class imbalance.
  • Graph-based Representation Learning: Construction of provenance graphs for embedding and GNN-based detection, leveraging ground-truth APT chains.
  • Process/Tree Event Modeling: Hierarchical process/event trees exploit actor-object relations and temporal sparsity to improve classification of attack stages.
  • Forward-propagated Provenance Context: Attack narrative extraction via subgraph visualization enhances forensic response and triage.

7. Standardization, Lessons, and Future Work

• Data Standardization: Emphasis on rigorously specified semantics (e.g., unique UUIDs, event field consistency) to minimize analysis overhead—~70% of analyst effort was spent addressing format issues in adversarial engagement 1.
• Policy Tuning: Analytics frameworks require adjustable parameters for false positive reduction due to incomplete or inconsistent context.
• Systemic Gaps: CDM’s coverage of device, memory, and interprocess communication semantics remains incomplete; future directions call for extension to RDP, camera, and keyboard event capture.
• Future Engagements: The program plans more sophisticated adversarial challenges, including polymorphic code, encrypted exfiltration channels, and memory-only payloads to stress analytics robustness.
• Conclusion: Provenance-based, tag and policy-driven detection in high-throughput, unified event streams is empirically validated, but enduring progress depends on further advances in semantic standardization, attack scenario diversity, and real-world provenance fidelity (Anjum et al., 2021, Eshete et al., 2016).