DeepJSONEval: Nested JSON Benchmark

• DeepJSONEval is a benchmark for assessing LLMs’ ability to extract deeply nested JSON objects from lengthy, aggregated texts across 10 diverse web domains.
• It features 2100 instances with 3–7 levels of nesting and ~17.5 properties per schema, emphasizing real-world extraction over mere syntactic validity.
• The evaluation framework uses Syntax, Key, and Strict metrics, highlighting that achieving valid JSON is easier than extracting semantically accurate, complex structures.

DeepJSONEval is a benchmark for evaluating how well LLMs extract information from long text into deeply nested JSON objects rather than merely emitting syntactically valid JSON. It is defined as a multilingual, multi-domain benchmark with 2100 instances, 10 web-oriented domains, 3–7 levels of nesting, and an average of ~17.5 properties per schema; its task formulation is explicitly text–schema–to–JSON, with each instance pairing an aggregated input text and a target JSON object conforming to a predefined schema (Zhou et al., 30 Sep 2025).

1. Scope, motivation, and problem formulation

DeepJSONEval is motivated by what the benchmark description calls an “information extraction paradox”: web content volume grows while meaningful, non-redundant information density remains low. Social media comments, repetitive news, and lengthy discussions are therefore difficult to mine directly, whereas multi-layer nested JSON can condense such material into hierarchical, semantically organized representations such as article metadata, content, and multimedia descriptors, or analogous structures in finance, healthcare, and recommendation systems (Zhou et al., 30 Sep 2025).

The benchmark is designed to measure JSON extraction rather than pure format following. Existing JSON-related benchmarks are described as largely emphasizing syntactic correctness, constrained decoding, or shallow schema adherence, often with only 1–2 levels of depth or without source text. DeepJSONEval is positioned against that background as a benchmark that evaluates whether a model can read long, information-dense text, understand field descriptions, transform values, preserve hierarchical relations, and fill a deep schema correctly. This distinction is central: a model can solve the JSON-validity problem while still failing the extraction-and-comprehension problem (Zhou et al., 30 Sep 2025).

The task is formalized over an input text space $\mathcal{X}$, a JSON schema space $\mathcal{S}$, and a JSON instance space $\mathcal{Y}$. For each instance, a schema $s \in \mathcal{S}$, an input $x \in \mathcal{X}$, and a ground-truth JSON $y^\star \in \mathcal{Y}$ are given such that $y^\star \models s$. The evaluated model implements

$$f_\theta: \mathcal{X} \times \mathcal{S} \to \mathcal{Y},$$

and the benchmark measures how close $f_\theta(x,s)$ is to $y^\star$ at several granularities (Zhou et al., 30 Sep 2025).

2. Dataset construction and benchmark composition

DeepJSONEval contains 2100 instances across the following 10 web-oriented domains: Tourist Attraction Promotion, Electronic Devices Introduction, Patient Information, Athlete Biography, Botany Encyclopedia, Financial Securities, Academic Record, Vehicle Recommendation, Movie Review, and Video Game Summary (Zhou et al., 30 Sep 2025).

Each instance consists of two parts. The first is an input text: a long, coherent summary of $\mathcal{S}$0 words synthesized from multiple web texts in a domain. The construction workflow is: crawl multiple web document fragments for a topic or entity; use DeepSeek-R1 with a prompt emphasizing multi-document aggregation, redundancy removal, faithfulness (“Do not invent facts”), and high information density; then perform a human-in-the-loop quality check to ensure semantic faithfulness and appropriateness. The second part is a target output: a complex, deeply nested JSON object conforming to a domain-specific schema tree, with required and optional properties, mixed types, arrays, and nested objects (Zhou et al., 30 Sep 2025).

The construction pipeline is described in four stages. Web Text Collection & Multi-Document Aggregation creates the dense textual input. Schema Tree Construction identifies key concepts in the domain and builds a property tree in which nodes are properties and edges are inheritance or nesting relations. Real-time Path-Value Updating Beam Search for Constrained Schema Subtree Construction extracts subtrees satisfying target depth and size bounds. Schema Generation and Ground Truth Construction converts these subtrees into JSON schemas with detailed field descriptions and then maps the aggregated texts to JSON ground truth under human-in-the-loop quality control (Zhou et al., 30 Sep 2025).

The subtree-construction stage uses a path-value function $\mathcal{S}$1 that combines node association, depth and size window rewards, and a penalty for invalid paths. The benchmark description gives the following components: $\mathcal{S}$2

$\mathcal{S}$3

$\mathcal{S}$4

$\mathcal{S}$5

$\mathcal{S}$6

and

$\mathcal{S}$7

Feasibility is defined by

$\mathcal{S}$8

This stage is the mechanism by which target schema depth and node count are controlled (Zhou et al., 30 Sep 2025).

Quality control is also explicit. Two annotators independently review each instance and check schema conformance, content faithfulness, cross-field consistency, length compliance, ambiguity level, and difficulty tag correctness, assigning Pass/Fix/Reject with fixes logged. Automatic checks include schema validation, constraint checking, text grounding probes, length checks, and leakage or bias audits using n-gram and MinHash similarity scans. Only items passing all criteria are released as Gold (Zhou et al., 30 Sep 2025).

3. Schema complexity, difficulty design, and illustrative instances

DeepJSONEval defines “deep nested structures” by the maximum number of nested object levels from the root to a leaf. The benchmark targets a depth range: 3–7, and uses difficulty labels driven mainly by that nesting depth: Medium corresponds to 3–4 levels of nesting, whereas Hard corresponds to 5–7 levels. The Hard subset is additionally characterized by multi-entity linking, cross-field dependencies, nested lists of objects, and more opportunities for type mismatch, repeated nested structures, and hallucinations (Zhou et al., 30 Sep 2025).

The schemas mix string, number, boolean, enum strings, arrays, and nested objects. The benchmark description emphasizes that success requires more than field copying. Models must locate and extract relevant spans from long text; transform values such as "4 GB" → 4 and "92%" → 92; handle nested arrays and objects; maintain hierarchical links; satisfy type constraints; and avoid hallucinating properties or values not grounded in the text (Zhou et al., 30 Sep 2025).

A Hard example from Electronic Devices Introduction includes fields such as Memory, Software.OperatingSystem, Software.PreinstalledApps, and a nested Dimensions.Sensors[*].SensorType.SensorSubtype.CalibrationData structure. The associated text fragment contains “4 GB of RAM,” “HarmonyOS 3.0,” a list of preinstalled apps, a “46.7-gram weight,” and “three sensors—calibrated at 1.05, 0.92, and 1.10.” The benchmark requires the model to extract the correct values and place them in the corresponding deep hierarchy (Zhou et al., 30 Sep 2025).

A Medium example from Movie Review uses a Ratings array whose items require Source and Score, with Source constrained to IMDb, Rotten Tomatoes, or Metacritic. The text fragment gives “8.2/10,” “92%,” and “88,” and the task requires normalization into the numeric scores 8.2, 92, and 88. The benchmark description presents this as a case where the difficulty lies less in raw depth than in multi-source aggregation and numeric normalization (Zhou et al., 30 Sep 2025).

The benchmark is therefore not only about depth in the narrow tree-theoretic sense. A plausible implication is that its effective hardness arises from the conjunction of depth, long-context extraction, type normalization, and cross-field consistency requirements.

4. Evaluation methodology and reported metrics

DeepJSONEval uses a multi-dimensional evaluation framework. The main results table reports three aggregate metrics: Syntax, Key, and Strict (Zhou et al., 30 Sep 2025).

The Syntax Score $\mathcal{S}$9 is a binary indicator of syntactic validity and schema conformance: $\mathcal{Y}$0 Here, $\mathcal{Y}$1 checks JSON parsability and $\mathcal{Y}$2 checks structural schema conformance. Model-level Syntax is reported as the mean over instances (Zhou et al., 30 Sep 2025).

The Hierarchical Key Matching Score $\mathcal{Y}$3 extends ThinkJSON’s key_matching_score to multi-layer nesting. Let $\mathcal{Y}$4 be the set of key paths in the model output and $\mathcal{Y}$5 the set of key paths in the ground truth. Then

$\mathcal{Y}$6

This score measures structural alignment and field coverage independently of exact values (Zhou et al., 30 Sep 2025).

The Strict Score $\mathcal{Y}$7 is binary exact match over the entire JSON object: $\mathcal{Y}$8 It requires every value, type, and structural detail to be exactly correct (Zhou et al., 30 Sep 2025).

This metric triad separates three different notions of success. Syntax captures parseability and coarse schema matching; Key captures hierarchical structural coverage; Strict captures full semantic correctness. One of the benchmark’s recurrent findings is that these notions diverge sharply in modern LLMs: JSON validity can be nearly solved while exact deep extraction remains difficult (Zhou et al., 30 Sep 2025).

DeepJSONEval evaluates 12 strong LLMs from the OpenCompass LLM Leaderboard, including Claude Sonnet 4, Magistral Medium 2506, DeepSeek R1 0528, Gemini 2.5 Pro, Qwen3 235B A22B, DeepSeek R1 Distill Llama 70B, Magistral Small 2506, Qwen3 30B A3B, Llama 4 Maverick, Qwen3 14B, Hunyuan A13B, and Qwen3 32B. Results are reported for the overall set and for the Medium and Hard subsets (Zhou et al., 30 Sep 2025).

5. Empirical findings, error modes, and external validity

The leaderboard results show that top models achieve very high Syntax but substantially lower Strict. Reported overall examples include Claude Sonnet 4 with Syntax 99.05, Key 90.73, Strict 57.90; Magistral Medium 2506 with 98.10, 90.36, 59.81; DeepSeek R1 0528 with 97.90, 89.57, 57.33; and Qwen3 32B with 87.81, 80.18, 50.67. The benchmark summary states that all top models easily achieve high Syntax scores (>87% overall, many >95%), while Strict scores remain only ≈48–60 overall (Zhou et al., 30 Sep 2025).

Difficulty stratification is effective. For Claude Sonnet 4, the benchmark reports Medium: Syntax 100.00, Key 94.52, Strict 69.51 and Hard: Syntax 98.61, Key 89.01, Strict 52.63. More generally, Strict declines by 17.22–37.53 percentage points from Medium to Hard, whereas Syntax degrades more modestly. This pattern is used as evidence that the Hard subset specifically stresses deep nesting and complex relationships (Zhou et al., 30 Sep 2025).

The benchmark also tests whether verbosity is confounding the scores. Reported correlations between response length and scores range from $\mathcal{Y}$9 to $s \in \mathcal{S}$0, and $s \in \mathcal{S}$1 for most models. The stated conclusion is that the metrics are not simply rewarding longer outputs (Zhou et al., 30 Sep 2025).

Domain-wise analysis shows relatively small variance. The reported key matching scores range from 0.776–0.867 for Medium difficulty and 0.474–0.540 for Hard difficulty across the 10 domains. The benchmark interprets this as evidence that no single domain dominates or is anomalously easy or hard (Zhou et al., 30 Sep 2025).

Type-wise analysis shows an asymmetry between numeric and string outputs. Performance is highest on numeric lists (0.90–1.00) and lowest on string lists (0.576–0.722). The benchmark description interprets this as evidence that primitive numeric parsing is comparatively reliable, whereas string lists remain error-prone because models truncate, hallucinate, or mis-segment items (Zhou et al., 30 Sep 2025).

The qualitative error analysis identifies several recurrent failure modes. Type mismatches include outputs such as "Memory": "4 GB" when the schema requires a number, or "Score": "92%" when the schema requires a numeric score. String extraction errors include HarmonyOS 3 instead of HarmonyOS 3.0. Schema violations and duplication include repeated copies of "Software" and "Dimensions" objects. Hallucinations include adding "personal wellness hub" as a preinstalled app when the text used that phrase only to describe the device. Omissions occur in hard cases such as missing some sensor entries under Dimensions.Sensors (Zhou et al., 30 Sep 2025).

The benchmark further reports an end-to-end external-validity study. In a crawl → preprocess → construct schema → extract JSON → query/analytics pipeline over Athlete Biography, Vehicle Recommendation, and Video Game Summary, the pipeline key matching scores are 97.36 for Claude Sonnet 4, 91.87 for DeepSeek R1 Distill Llama 70B, and 83.26 for Qwen3 32B. The correlation between pipeline key scores and DeepJSONEval key scores is

$s \in \mathcal{S}$2

The benchmark interprets this as strong external validity: improvements on DeepJSONEval track improvements in downstream web extraction (Zhou et al., 30 Sep 2025).

6. Position within structured-output evaluation, applications, and limitations

DeepJSONEval is compared against benchmarks such as StructEval, json-mode-eval, STRUCTURERAG, SoEval, FoFo, NESTFUL, JSONSchemaBench, and Schema-Guided Dialogue, and is characterized in that comparison as Multilingual, JSON Extractive, with 2100 instances, Depth: 3–7, domain labels, and support for constrained decoding. The benchmark’s explicit claim is that it bridges the gap between schema-only generation and shallow extraction by combining deep nesting, multi-domain web content, and extraction from text (Zhou et al., 30 Sep 2025).

This positioning is distinct from work on strict schema adherence such as “Think Inside the JSON: Reinforcement Strategy for Strict LLM Schema Adherence” (Agarwal et al., 18 Feb 2025). That line of work focuses on schema-constrained generation with metrics such as Rows With No Output, Rows With Valid JSON, Mean Match Percentage, and Mean Noise Percentage, and reports ThinkJSON at 62.41% mean match and 0.27% mean noise. DeepJSONEval addresses a different target: long-text comprehension and extraction into deep nested schemas, measured by Syntax, Key, and Strict rather than only field-level schema adherence (Agarwal et al., 18 Feb 2025).

In practical terms, the benchmark is framed around realistic web data mining scenarios such as news aggregation, social media analytics, and healthcare, and its schemas are presented as analogous to nested production representations used in those settings. The benchmark description also argues that practitioners can use Strict when exact correctness is required, and Key when structural fidelity is the more relevant bottleneck (Zhou et al., 30 Sep 2025).

The dataset and benchmark are open-sourced at https://github.com/GTS-AI-Infra-Lab-SotaS/DeepJSONEval, with input texts, JSON schemas, gold outputs, and evaluation scripts for syntax, key matching, and strict (Zhou et al., 30 Sep 2025).

The benchmark’s stated limitations are also specific. Its 10 domains, while broad, do not cover all web scenarios. It focuses on intra-document extraction and nesting, with limited explicit temporal reasoning, cross-document reasoning, or multimodal fields. Although 3–7 levels of nesting are substantial, the benchmark description states that there is still room for more complex relational schemas such as graphs and cross-object references. Future directions explicitly mentioned include even deeper nesting levels, dynamic schema adaptation, more domains and more complex relations, potential integration with knowledge graphs or databases, and further algorithmic efficiency improvements for schema construction (Zhou et al., 30 Sep 2025).

A persistent misconception addressed by the benchmark is that high JSON-validity scores imply robust structured extraction. DeepJSONEval’s results argue against that view: the syntactic problem is often nearly solved, but deep semantic extraction under nested schemas remains materially unsolved.