OrgForge: Synthetic Corp Simulation

• OrgForge is a multi-agent simulation framework that deterministically generates verifiable synthetic corporate corpora with a strict physics-cognition boundary.
• It employs a deterministic Python engine to control simulation state, timestamps, actor identities, and event logging, ensuring consistent cross-artifact records.
• OrgForge-IT applies this architecture as an insider-threat benchmark, enabling reproducible evaluation of LLM-based detection against modern enterprise telemetry.

OrgForge is a multi-agent simulation framework for generating verifiable synthetic corporate corpora in which factual state is controlled by a deterministic Python engine and LLMs are restricted to rendering surface prose from validated proposals. Its central design principle is a strict “physics-cognition boundary”: simulation state, timestamps, actor identities, incident routing, and ground truth are owned by the engine, while LLMs cannot directly mutate state or write to the event log. This architecture was introduced for retrieval-augmented generation evaluation and later instantiated in OrgForge-IT, an insider-threat benchmark that uses the same verifiable synthetic paradigm to evaluate LLM-based detection under modern enterprise telemetry (Flynt, 16 Mar 2026, Flynt, 23 Mar 2026).

1. Problem setting and design objective

OrgForge addresses a corpus-quality problem in evaluation research. The framework is motivated by the requirement that synthetic corporate corpora be ground-truth traceable, temporally structured, cross-artifact coherent, and configurable in complexity. Existing resources such as the Enron corpus are described as carrying legal ambiguity, demographic skew, and no structured ground truth, while purely LLM-generated corpora solve the legal issue but introduce cross-document contradictions that cannot be externally adjudicated (Flynt, 16 Mar 2026).

The framework’s response is architectural rather than post hoc. Instead of asking a model to generate the corpus and then hoping it remains internally consistent, OrgForge ensures that LLMs never author facts directly. A deterministic engine controls organizational reality, and a shared SimEvent ground-truth bus records all significant actions. Artifacts such as Slack threads, JIRA tickets, Confluence pages, Git pull requests, emails, and server logs are therefore human-readable realizations of a single underlying simulation run rather than independent textual inventions.

This design later became the substrate for OrgForge-IT. In that setting, the same principle is applied to insider-threat detection: prior synthetic benchmarks are said to suffer from a consistency problem because an LLM-generated corpus cannot rule out cross-artifact contradictions, whereas OrgForge-IT guarantees that ground truth and observable telemetry come from the same deterministic run. A plausible implication is that OrgForge should be understood less as a document generator than as a state-constrained simulation environment whose outputs happen to include natural-language artifacts.

2. Physics-cognition boundary and ground-truth semantics

OrgForge formalizes the simulation as

$M = (S, P, V, E),$

where $S$ is simulation state, $P$ denotes planners, $V$ the validator, and $E$ the SimEvent log or ground-truth event bus. The “physics layer” comprises $S$, $E$, clocks, graph dynamics, routing, and incident propagation; the “cognition layer” comprises LLM-based department planners; and the boundary is the validator, which determines whether proposals are valid before they can affect the simulation (Flynt, 16 Mar 2026).

Every significant action emits a structured SimEvent stored in MongoDB. Each SimEvent includes event_type, actors, artifact_ids, and a facts payload containing state-relevant values at that moment. Examples of event types include incident_opened, pr_merged, external_contact_summarized, day_summary, and proposed_event_rejected. Because rejected proposals are also logged, the audit trail includes not only accepted events but failed candidate transitions.

The validator enforces several deterministic constraints. Actor integrity requires every actor to exist in the org chart or external contacts. Novel event triage accepts unknown events only when they specify a known artifact hint; otherwise they are rejected and logged. State plausibility blocks certain event types when system health is too low. Cooldown windows prevent some events from firing too frequently, and morale gating restricts morale interventions to states in which morale is sufficiently low. The consequence is that LLM output enters the system only as a proposal stream subject to deterministic admission control.

In OrgForge-IT the same logic is made explicit for insider-threat benchmarking: the engine owns the event bus, timestamps, actor identities, threat labels, and mutable simulation state; labels in ground truth.jsonl are guaranteed to match observable artifacts; and LLMs generate only surface prose at designated points. Cross-artifact consistency is thus not an empirical property of a successful generation run but an architectural guarantee.

3. Temporal semantics and organizational dynamics

A distinctive feature of OrgForge is its actor-local clock. The framework identifies a common synthetic-data failure mode in which timestamps are sampled independently per artifact, yielding causal inconsistencies. OrgForge replaces this with sim_clock.py, where each employee has an independent time cursor. Two primitives are central: advance_actor(i, Δ), which advances only actor $i$’s cursor to model parallel work, and sync_and_tick(A, Δ), which synchronizes all participants in a causal interaction to the maximum cursor time among them and then advances the shared cursor. This enforces causal timestamp correctness across artifact types: replies cannot predate triggers, parallel work remains possible, and same-day time travel is excluded (Flynt, 16 Mar 2026).

The simulation also maintains a weighted undirected social graph $G=(V_G,E_G)$ over employees and external contacts. Initial weights are defined by organizational relations: same-department ties receive $10.5$, department-lead ties receive $S$0, cross-department cold edges receive $S$1, and external contacts begin at a cold-start weight of $S$2; same-department leads therefore have an effective combined weight of $S$3. On top of this graph, OrgForge specifies three graph-dynamic subsystems.

First, stress propagation is governed by betweenness centrality. Key players are identified by

$S$4

with default key-player multiplier $S$5. Stress then evolves as

$S$6

Burned-out central actors therefore bleed stress along stronger ties.

Second, edge weights decay and reinforce over time:

$S$7

The interaction boost $S$8 takes values $S$9 for joint incident response, $P$0 for PR co-review, $P$1 for 1-on-1 or mentoring, $P$2 for a shared Slack thread, and $P$3 for no recorded interaction. With default $P$4 and floor $P$5, the model encodes organizational memory without allowing ties to disappear entirely.

Third, escalation routing is posed as shortest-path computation over inverse-weight edges,

$P$6

and solved with Dijkstra’s algorithm. Escalation targets are chosen by domain-expert lead match, otherwise highest direct-weight lead, otherwise highest-centrality non-source node. The resulting route is emitted as a SimEvent and used as prompt context so that generated prose matches actual routing.

Capacity and morale are also modeled deterministically. Engineer capacity is reduced by on-call duty and high stress, while morale decays multiplicatively during active incidents and recovers when no incidents remain at end of day. Ambient social interruptions such as watercooler chats, off-topic Slack, and informal interruptions consume actor-clock time and introduce benign noise.

4. Simulation loop, artifact system, and evaluation stack

OrgForge runs a discrete-time simulation over $P$7 days with a daily cycle of planning, execution, and end-of-day summarization. The execution flow updates graph dynamics, obtains structured JSON proposals from department-level LLM planners, validates them, generates typed artifacts, extends causal chains and email flows, and logs summaries and state updates. Artifact generation is activity-typed: for example, ticket_progress produces a JIRA comment and optionally a Slack thread, pr_review produces a GitHub bot message and optional reply, design_discussion produces a Slack thread and sometimes a Confluence stub, and deep_work produces no artifact (Flynt, 16 Mar 2026).

The CausalChainHandler accumulates ordered artifact IDs per incident. A causal chain may include an initial Slack bot alert, a JIRA ticket, PR merges, a Confluence postmortem, external emails, and vendor alerts. Snapshots of the chain are stored inside each SimEvent’s facts, so the chain is recoverable at any prior point in time. This supports evaluation tasks that require timeline reconstruction from arbitrary entry points.

For recurrence detection, OrgForge uses hybrid reciprocal-rank fusion over vector and text retrieval:

$P$8

with $P$9, $V$0, and $V$1, subject to thresholds $V$2 and $V$3. Accepted matches are annotated as recurrence_of with recurrence_gap_days.

The external email engine extends causal chains beyond internal artifacts. Tech-vendor inbound messages are routed by expertise similarity using find_expert_by_skill(). Customer complaints pass through a gated path in which Sales receives a Slack ping, Product makes a triage decision, and high-priority complaints may create a JIRA ticket; about 15% are probabilistically dropped, logged as email_dropped, and become verifiable gaps. HR outbound messages for hiring are sent 1–3 days before arrival and linked into the employee_hired causal chain.

The evaluation stack includes eval_harness.py, scorer.py, and export_to_hf.py. OrgForge generates eight question types: RETRIEVAL, CAUSAL, TEMPORAL, GAP_DETECTION, ROUTING, PLAN, ESCALATION, and KNOWLEDGE_GAP. Each question includes a difficulty label, a requires_reasoning flag, and an evidence chain of supporting artifact IDs. Scoring is deterministic:

$V$4

where evidence overlap is Jaccard overlap between retrieved artifact IDs and the ground-truth chain. In a 22-business-day run, the reported corpus contains 43 simulated personnel, 1,079 documents, and 83 evaluation questions; BM25 performs better overall than dense retrieval, dense retrieval is slightly better only on RETRIEVAL, BM25 is strongest on CAUSAL, and both methods fail on PLAN and ESCALATION.

5. OrgForge-IT as an insider-threat benchmark

OrgForge-IT is the insider-threat benchmark built on OrgForge. It is designed to replace the unconstrained synthetic generation paradigm with a verifiable synthetic benchmark in which a deterministic simulation engine maintains ground truth and LLMs generate only surface prose. The benchmark is framed explicitly against the CERT dataset, which is described as canonical but static, lacking cross-surface correlation scenarios, lacking modern enterprise artifacts such as Slack, JIRA, or Confluence, and predating the LLM era (Flynt, 23 Mar 2026).

The benchmark instance described in the paper spans 51 simulated days and 2,904 observable telemetry records at a 96.4% noise rate. It contains three threat classes—negligent, disgruntled, and malicious—and eight injectable behaviors spanning five artifact surfaces: secret in commit, unusual hours access, excessive repo cloning, sentiment drift, cross dept snooping, data exfil email, host data hoarding, social engineering, and idp anomaly. The IDP authentication subsystem generates realistic baseline logins for all employees, allowing anomalous authentication patterns to be evaluated against plausible SSO background traffic.

OrgForge-IT introduces four scenarios intended to defeat shallow triage strategies. “Ghost logins” require reasoning from the absence of corroborating downstream activity after an IDP authentication success. “Vishing” requires cross-actor temporal joining because a phone call is followed by an IDP authentication on the victim’s account rather than the attacker’s. “Three-phase host data hoarding” spreads bulk file copy, compression, and archive exfiltration across consecutive days, making single-day analysis insufficient. “Trust-building social engineering” separates benign contact from later attack by a delay that requires memory across an inter-event gap.

Its evaluation pipeline has three stages. Baseline calibration uses pre-onset clean records to estimate false positives. Triage and credential scan use a 7-day sliding-window agent with a two-signal escalation threshold plus a separate scanner for fatal single-surface events such as plaintext secrets. Correlation and verdict pass escalated employees to a Tier 2 investigator that sees the full timeline, Slack history, and behavior definitions, and returns a structured verdict with evidence citations. Triage scoring uses precision, recall, and F1 over escalation decisions; verdict scoring uses precision, recall, and F1 over final verdict objects; additional metrics include baseline false-positive rate, vishing detected, and host trail reconstructed.

6. Empirical findings, limitations, and significance

OrgForge-IT’s leaderboard is used to separate anomaly spotting from operationally usable attribution. Eight of ten models achieve identical triage $V$5, yet verdict $V$6 splits into Tier A at $V$7 and Tier B at $V$8. Tier A consists of Devstral 2 123B and Claude Opus 4.6; Tier B consists of DeepSeek v3.2, Mistral Large 675B, GLM-5, Claude Sonnet 4.6, Claude Haiku 4.5, Qwen3-Coder, Llama 4 Maverick, and Llama 3.3 70B. The benchmark therefore rejects the common assumption that triage accuracy is an adequate proxy for final investigative quality (Flynt, 23 Mar 2026).

A second result is that baseline false-positive rate is a necessary companion to verdict $V$9. DeepSeek, Mistral Large, GLM-5, Sonnet, Haiku, and Qwen3-Coder have clean baseline false-positive rates around 0.021–0.023, Llama 4 Maverick is at 0.063, and Llama 3.3 70B is at 0.813, escalating 39 innocent employees in the clean baseline period despite matching Tier B verdict $E$0. The benchmark’s position is that a system that obtains acceptable verdict scores by reviewing almost everyone behaves as a review system rather than a detection system.

The vishing scenario creates the sharpest capability separation. All ten models detect that an attack occurred, but only Tier A models exonerate the compromised account holder, Chris, by identifying the inconsistency between the anomalous IDP session and Chris’s established device and MFA profile. Tier B models detect the attack but still mark Chris as suspicious, which yields precision 0.667 in verdict scoring. The benchmark treats victim attribution as a substantive reasoning threshold rather than a cosmetic labeling issue.

The benchmark also shows that a rigid two-signal escalation threshold structurally excludes some negligent insiders. A developer who accidentally commits an AWS key may generate a single high-confidence event on one surface and never satisfy a multi-signal threshold. This is why the harness includes a separate credential-scanning path and argues for parallel, threat-class-specific triage pipelines.

Prompt sensitivity analysis reports that prompt structure changes expression more than underlying capability for the strongest models. Under the official prompt, verdict $E$1 remains stable across variants, but unstructured prompts induce vocabulary hallucination: semantically correct yet non-canonical behavior labels receive zero credit under exact-match behavior citation scoring. The proposed response is a two-track framework consisting of a primary exact-match track over the canonical ALL BEHAVIORS taxonomy and a secondary semantic track based on embeddings or judge models.

The broader OrgForge framework carries its own stated limitations. Persona diversity is hand-authored, prose naturalness remains LLM-dependent even when factual consistency is guaranteed, only retrieval baselines are reported for the original corpus benchmark, and the simulation is limited to a single-company scope with passive external contacts. Even so, the framework establishes a specific methodological contribution: verifiable synthetic corpora can be generated with immutable event semantics, causal timestamp enforcement, graph-dynamic organizational behavior, recoverable causal chains, and deterministic evaluation. That combination makes OrgForge both a corpus-generation system and a benchmark-construction methodology for settings where real corporate data are legally, ethically, or structurally unsuitable.