MANTRA Dataset Overview

• MANTRA datasets are a suite of rigorously defined benchmarks that use formal definitions and automated synthesis to enable trace-level evaluation in varied research domains.
• The LLM compliance dataset leverages SMT-validated, trace-level task synthesis from structured manuals, providing detailed error analysis and quantifiable success metrics.
• Complementary datasets for astronomical lightcurve classification and manifold triangulations offer robust applications in time-series analysis and topological deep learning experiments.

The term "MANTRA dataset" refers to multiple prominent datasets published under the acronym MANTRA, each addressing distinct research domains—compliance benchmarking for tool-using LLM agents, astronomical transient detection, and topological deep learning on manifold triangulations. Each dataset is defined rigorously in its source material, serves as a community benchmark, and is publicly accessible for research use. This article presents a comprehensive overview of the MANTRA datasets, focusing on their construction, formal definitions, unique features, primary results, and research implications.

1. MANTRA for SMT-Validated Compliance Benchmarks in Tool-Using LLMs

The MANTRA benchmark for tool-using LLM agents is a large-scale, automatically synthesized, and SMT-validated compliance dataset for evaluating the procedural correctness of LLM-driven agents operating under natural-language manuals and structured tool schemas (Anand et al., 7 May 2026). Its principal contribution lies in replacing ad-hoc or LLM-judged compliance benchmarks with formally validated, trace-level task suites derived directly from complex procedural documents.

Formal Definition

Each MANTRA task is built on three core abstractions:

• Symbolic World Model $W_S=(X,T_S,\delta)$ for each document region $S$, where $X$ denotes the state variables, $T_S$ the local tool subset, and $\delta$ gives tool-specific transition relations. Compliance is defined over traces $\tau=[(t_0,a_0),\ldots,(t_{h-1},a_{h-1})]$ that have valid stepwise executions according to $\delta$ given an initial state.
• Trace-level Compliance Checks $C_\lambda = \{c_1,\dots,c_k\}$ for each scenario $\lambda$, using a formal grammar of atomic tool call constraints (e.g., $\mathtt{call}(t,a)$, $S$0) and compound temporal/ordering/logical constructs (e.g., $S$1, $S$2).
• SMT Consistency: All combinations of world model and checks are translated to SMT (e.g., Z3) constraints $S$3, $S$4; soundness and completeness are assessed by forward conflict search and per-check backward audits.

Synthesis Pipeline and Construction

The automated pipeline operates as follows: a plain-text manual and structured tool schema are processed to construct a document dependence graph (via hierarchical chunking and LLM-extracted references). Subgraphs are sampled and relevant tool subsets are identified for each sample. Scenarios are generated, and for each scenario, LLMs instantiate symbolic compliance checks and world model DSL components, which are compiled into SMT formulas. An iterative cross-validation and repair loop eliminates inconsistencies using SMT traces and either automated or fallback human edits. Only scenarios passing all forward and backward audits are retained.

Scale and Composition

Summary statistics are as follows:

Domain	Manual Words	#Tools	Tasks	Fwd-Validated	Bwd-Validated	Final Set
Operations	566	20	166	82	65	63
Sneaky-Sasquatch	1,338	23	162	76	53	47
Cabin-Safety	16,644	148	164	71	43	43
Tau2-Retail	1,158	16	75	59	47	44
Tau2-Airline	1,313	14	80	49	57	57
Tau2-Telecom	3,715	13	56	36	32	31

The final dataset spans domains and manuals up to 50+ pages, comprising 285 tasks, each with validated compliance checks and symbolic formalism.

2. Benchmark Features and Comparative Context

Unlike prior benchmarks (ToolBench, τ-Bench, SOP-Bench, AgentIF), MANTRA exhibits:

Feature	MANTRA	Alternatives
Automated generation	✓	✗/Partial
Large unstructured manual support	✓	✗/Partial
Expandable to new domains	✓	✗/Partial
Trace-level compliance	✓	Partial/None
Deterministic grading	✓	Mixed

MANTRA is the only benchmark with automatic, formally validated task synthesis and minimal human intervention (Anand et al., 7 May 2026).

3. Empirical Results and Error Analysis

Evaluation of six LLM agent systems (Qwen3 8B, GPT-OSS 20B, Qwen3.6 36B, GPT-5.4-mini, Claude Haiku 4.5, Gemma4 8B) over MANTRA’s 285 tasks (five runs each) yields:

• Success rates: Pass@1 between 40–60% for most domains; “Retail” domain notably lower.
• Failure typology (across 10,895 failed checks):

Failure Mode	Count	Percent
Missing-Required-Call	6,349	58%
Missing-Anchor	2,006	18%
Forbidden-Call	2,184	20%
Pure ordering violation	250	2%
Compound or-clause	105	1%

A key finding is that 76% of failures are due to omitted required or anchor calls, not simple misorderings. Correlation between premature write actions and final success rates is strong; top-performing systems avoid such errors. Notable agent pathologies include system-specific crashes and blank output on lengthy manuals.

4. Limitations, Scalability, and Future Directions

Key constraints include:

• Bounded formal guarantee: No compliance checks beyond the trace length parameter $S$5; manual inspection required for certain ambiguities and rare failure modes.
• Assumptions: Relies on explicit tool schemas; not directly applicable for pure QA or generative free-form LLM tasks.
• String argument brittleness: Exact match requirements for arguments (e.g., addresses) present robustness challenges.

Anticipated extensions involve unbounded horizon (inductive) checking, improved sampling strategies, non-English manual support, and integration of execution log replay for semantic argument verification (Anand et al., 7 May 2026).

5. Summary of Other MANTRA Datasets

MANTRA for Astronomical Lightcurve Classification

The MANTRA dataset in astronomy comprises 4,869 transient and 71,207 non-transient lightcurves from CRTS, with labeled event types ranging from supernovae (1,723), cataclysmic variables (988), and AGN (446), among others (Neira et al., 2020). It includes an extensive set of time-series features—moment-based, magnitude-based, percentile-based, and polynomial fitting-derived—for use in standard ML classification tasks. In benchmark experiments, a Random Forest Classifier achieved 96.25% F1-score on binary classification (transient/non-transient) and 52.79% on eight-class classification, with marked disparity in per-class performance. The data are distributed as plain-text CSV and code notebooks.

MANTRA for Topological Deep Learning on Manifold Triangulations

In topological DL, “MANTRA: The Manifold Triangulations Assemblage” provides over 43,000 surface and 250,000 3-manifold triangulations, including genus/Betti/orientability labels, and supports evaluation of graph vs. higher-order (simplicial complex) neural networks (Ballester et al., 2024). Triangulations are purely combinatorial, derived from exhaustive enumeration up to 10 vertices, and distributed as JSON and PyTorch-Geometric packages. Baseline results show that simplicial-complex-based networks outperform graph-based ones on basic topological invariants but both architectures exhibit brittleness to mesh refinement.

An extended variant, as in (Schmidt et al., 7 May 2026), augments the MANTRA manifold corpus with generated triangulation refinements (Pachner/barycentric subdivisions, connected sums) and a broader class taxonomy, paired with a protocol for testing model robustness to combinatorial change. Despite in-distribution success (balanced accuracy up to 99.86%), all tested architectures fail to generalize on out-of-distribution refinements, underlining a core limitation in current topological DL approaches and establishing the benchmark as a critical testbed for developing scale-independent, topology-aware architectures.

6. Research Impact and Significance

The suite of MANTRA datasets exemplifies rigorous large-scale benchmark construction with formal guarantees, automated synthesis, and domain expansion. In LLM compliance, MANTRA exposes specific deficiencies in agent planning under global procedural constraints, providing a granular debugging interface through SMT-based trace checks. In topological machine learning, the manifold triangulation corpora and refinements codesign a resource for probing invariance and inductive bias limitations in GNNs and HOMP models, catalyzing new directions for topology-sensitive learning paradigms. In the astronomical transient domain, MANTRA establishes a baseline for standardized evaluation and feature development for time-domain classifiers.

In aggregate, MANTRA datasets function as reference points for their respective communities, structuring empirical inquiry, benchmarking, and the advancement of methods attuned to combinatorial, symbolic, and procedural structure. Each dataset is public and documented in the originating research (Anand et al., 7 May 2026, Ballester et al., 2024, Schmidt et al., 7 May 2026, Neira et al., 2020).