---
title: 'DeepJSONEval: Nested JSON Benchmark'
url: https://www.emergentmind.com/topics/deepjsoneval
type: topic
---

# DeepJSONEval: Nested JSON Benchmark

DeepJSONEval is a benchmark for evaluating how well large language models extract information from long text into deeply nested JSON objects rather than merely emitting syntactically valid JSON. It is defined as a multilingual, multi-domain benchmark with **2100 instances**, **10 web-oriented domains**, **3–7 levels of nesting**, and an average of **~17.5 properties per schema**; its task formulation is explicitly text–schema–to–JSON, with each instance pairing an aggregated input text and a target JSON object conforming to a predefined schema [2509.25922].

## 1. Scope, motivation, and problem formulation

DeepJSONEval is motivated by what the benchmark description calls an “information extraction paradox”: web content volume grows while meaningful, non-redundant information density remains low. Social media comments, repetitive news, and lengthy discussions are therefore difficult to mine directly, whereas multi-layer nested JSON can condense such material into hierarchical, semantically organized representations such as article metadata, content, and multimedia descriptors, or analogous structures in finance, healthcare, and recommendation systems [2509.25922].

The benchmark is designed to measure **JSON extraction** rather than pure format following. Existing JSON-related benchmarks are described as largely emphasizing syntactic correctness, constrained decoding, or shallow schema adherence, often with only **1–2 levels** of depth or without source text. DeepJSONEval is positioned against that background as a benchmark that evaluates whether a model can read long, information-dense text, understand field descriptions, transform values, preserve hierarchical relations, and fill a deep schema correctly. This distinction is central: a model can solve the JSON-validity problem while still failing the extraction-and-comprehension problem [2509.25922].

The task is formalized over an input text space \(\mathcal{X}\), a JSON schema space \(\mathcal{S}\), and a JSON instance space \(\mathcal{Y}\). For each instance, a schema \(s \in \mathcal{S}\), an input \(x \in \mathcal{X}\), and a ground-truth JSON \(y^\star \in \mathcal{Y}\) are given such that \(y^\star \models s\). The evaluated model implements
\[
f_\theta: \mathcal{X} \times \mathcal{S} \to \mathcal{Y},
\]
and the benchmark measures how close \(f_\theta(x,s)\) is to \(y^\star\) at several granularities [2509.25922].

## 2. Dataset construction and benchmark composition

DeepJSONEval contains **2100 instances** across the following **10 web-oriented domains**: **Tourist Attraction Promotion**, **Electronic Devices Introduction**, **Patient Information**, **Athlete Biography**, **Botany Encyclopedia**, **Financial Securities**, **Academic Record**, **Vehicle Recommendation**, **Movie Review**, and **Video Game Summary** [2509.25922].

Each instance consists of two parts. The first is an **input text**: a long, coherent summary of **\(\ge 1500\) words** synthesized from multiple web texts in a domain. The construction workflow is: crawl multiple web document fragments for a topic or entity; use **DeepSeek-R1** with a prompt emphasizing **multi-document aggregation**, **redundancy removal**, **faithfulness (“Do not invent facts”)**, and **high information density**; then perform a **human-in-the-loop quality check** to ensure semantic faithfulness and appropriateness. The second part is a **target output**: a complex, deeply nested JSON object conforming to a domain-specific schema tree, with required and optional properties, mixed types, arrays, and nested objects [2509.25922].

The construction pipeline is described in four stages. **Web Text Collection & Multi-Document Aggregation** creates the dense textual input. **Schema Tree Construction** identifies key concepts in the domain and builds a property tree in which nodes are properties and edges are inheritance or nesting relations. **Real-time Path-Value Updating Beam Search for Constrained Schema Subtree Construction** extracts subtrees satisfying target depth and size bounds. **Schema Generation and Ground Truth Construction** converts these subtrees into JSON schemas with detailed field descriptions and then maps the aggregated texts to JSON ground truth under human-in-the-loop quality control [2509.25922].

The subtree-construction stage uses a path-value function \(\mathrm{Val}(p \mid S)\) that combines node association, depth and size window rewards, and a penalty for invalid paths. The benchmark description gives the following components:
\[
\mathrm{Corr}(u\mid S) = \max_{v \in V_S} \mathrm{Assoc}(u,v),
\]
\[
\Delta(u\mid S) = \alpha \cdot \mathrm{Corr}(u\mid S), \quad \alpha > 0,
\]
\[
R_{\text{depth}(S\oplus p)} = 1 - \frac{\bigl|\,\mathrm{clip}(d(S\oplus p), d_{\min}, d_{\max}) - d(S\oplus p)\,\bigr|}{d_{\max} - d_{\min} + \varepsilon},
\]
\[
R_{\text{size}(S\oplus p)} = 1 - \frac{\bigl|\,\mathrm{clip}(|V_{S\oplus p}|, n_{\min}, n_{\max}) - |V_{S\oplus p}|\,\bigr|}{n_{\max} - n_{\min} + \varepsilon},
\]
\[
\mathrm{Penalty}(p\mid S) =
\mathbb{I}[u_0 \notin \mathcal{F}(S)] +
\sum_{i=1}^{\ell} \mathbb{I}[(u_{i-1}, u_i)\notin E],
\]
and
\[
\begin{aligned}
\mathrm{Val}(p\mid S)
&= \sum_{i=0}^{\ell}\gamma^{i} \,\Delta\!\left(u_i\ \middle|\ S\cup\{u_0,\ldots,u_{i-1}\}\right) \\
&\quad + \lambda_d\,R_{\text{depth}(S\oplus p)}
+ \lambda_n\,R_{\text{size}(S\oplus p)}
- \eta\,\mathrm{Penalty}(p\mid S).
\end{aligned}
\]
Feasibility is defined by
\[
\mathrm{Feas}(S\oplus p)=
\begin{cases}
0 & \text{if } d(S\oplus p) > d_{\max}\ \text{or}\ |V_{S\oplus p}| > n_{\max},\\
1 & \text{otherwise.}
\end{cases}
\]
This stage is the mechanism by which target schema depth and node count are controlled [2509.25922].

Quality control is also explicit. Two annotators independently review each instance and check schema conformance, content faithfulness, cross-field consistency, length compliance, ambiguity level, and difficulty tag correctness, assigning **Pass/Fix/Reject** with fixes logged. Automatic checks include schema validation, constraint checking, text grounding probes, length checks, and leakage or bias audits using **n-gram and MinHash** similarity scans. Only items passing all criteria are released as **Gold** [2509.25922].

## 3. Schema complexity, difficulty design, and illustrative instances

DeepJSONEval defines “deep nested structures” by the maximum number of nested object levels from the root to a leaf. The benchmark targets a **depth range: 3–7**, and uses **difficulty** labels driven mainly by that nesting depth: **Medium** corresponds to **3–4 levels of nesting**, whereas **Hard** corresponds to **5–7 levels**. The Hard subset is additionally characterized by multi-entity linking, cross-field dependencies, nested lists of objects, and more opportunities for type mismatch, repeated nested structures, and hallucinations [2509.25922].

The schemas mix `string`, `number`, `boolean`, enum strings, arrays, and nested objects. The benchmark description emphasizes that success requires more than field copying. Models must locate and extract relevant spans from long text; transform values such as `"4 GB" → 4` and `"92%" → 92`; handle nested arrays and objects; maintain hierarchical links; satisfy type constraints; and avoid hallucinating properties or values not grounded in the text [2509.25922].

A Hard example from **Electronic Devices Introduction** includes fields such as `Memory`, `Software.OperatingSystem`, `Software.PreinstalledApps`, and a nested `Dimensions.Sensors[*].SensorType.SensorSubtype.CalibrationData` structure. The associated text fragment contains “4 GB of RAM,” “HarmonyOS 3.0,” a list of preinstalled apps, a “46.7-gram weight,” and “three sensors—calibrated at 1.05, 0.92, and 1.10.” The benchmark requires the model to extract the correct values and place them in the corresponding deep hierarchy [2509.25922].

A Medium example from **Movie Review** uses a `Ratings` array whose items require `Source` and `Score`, with `Source` constrained to `IMDb`, `Rotten Tomatoes`, or `Metacritic`. The text fragment gives “8.2/10,” “92%,” and “88,” and the task requires normalization into the numeric scores `8.2`, `92`, and `88`. The benchmark description presents this as a case where the difficulty lies less in raw depth than in multi-source aggregation and numeric normalization [2509.25922].

The benchmark is therefore not only about depth in the narrow tree-theoretic sense. A plausible implication is that its effective hardness arises from the conjunction of depth, long-context extraction, type normalization, and cross-field consistency requirements.

## 4. Evaluation methodology and reported metrics

DeepJSONEval uses a **multi-dimensional evaluation framework**. The main results table reports three aggregate metrics: **Syntax**, **Key**, and **Strict** [2509.25922].

The **Syntax Score** \(S_{\text{syntax}}\) is a binary indicator of syntactic validity and schema conformance:
\[
S_{\text{syntax}} = \mathds{1}[\mathrm{valid}(\mathrm{output}) \land \mathrm{match}(\mathrm{output}, \mathrm{schema})].
\]
Here, \(\mathrm{valid}(\cdot)\) checks JSON parsability and \(\mathrm{match}(\mathrm{output}, \mathrm{schema})\) checks structural schema conformance. Model-level Syntax is reported as the mean over instances [2509.25922].

The **Hierarchical Key Matching Score** \(S_{\text{key}}\) extends ThinkJSON’s `key_matching_score` to multi-layer nesting. Let \(p_{out}\) be the set of key paths in the model output and \(p_{truth}\) the set of key paths in the ground truth. Then
\[
S_{\text{key}} = \frac{|p_{out} \cap p_{truth}|}{|p_{out} \cup p_{truth}|}.
\]
This score measures structural alignment and field coverage independently of exact values [2509.25922].

The **Strict Score** \(S_{\text{strict}}\) is binary exact match over the entire JSON object:
\[
S_{\text{strict}} = \mathds{1}[\mathrm{output} \equiv \mathrm{truth}].
\]
It requires every value, type, and structural detail to be exactly correct [2509.25922].

This metric triad separates three different notions of success. **Syntax** captures parseability and coarse schema matching; **Key** captures hierarchical structural coverage; **Strict** captures full semantic correctness. One of the benchmark’s recurrent findings is that these notions diverge sharply in modern LLMs: JSON validity can be nearly solved while exact deep extraction remains difficult [2509.25922].

DeepJSONEval evaluates **12 strong LLMs from the OpenCompass LLM Leaderboard**, including **Claude Sonnet 4**, **Magistral Medium 2506**, **DeepSeek R1 0528**, **Gemini 2.5 Pro**, **Qwen3 235B A22B**, **DeepSeek R1 Distill Llama 70B**, **Magistral Small 2506**, **Qwen3 30B A3B**, **Llama 4 Maverick**, **Qwen3 14B**, **Hunyuan A13B**, and **Qwen3 32B**. Results are reported for the overall set and for the **Medium** and **Hard** subsets [2509.25922].

## 5. Empirical findings, error modes, and external validity

The leaderboard results show that top models achieve very high **Syntax** but substantially lower **Strict**. Reported overall examples include **Claude Sonnet 4** with **Syntax 99.05**, **Key 90.73**, **Strict 57.90**; **Magistral Medium 2506** with **98.10**, **90.36**, **59.81**; **DeepSeek R1 0528** with **97.90**, **89.57**, **57.33**; and **Qwen3 32B** with **87.81**, **80.18**, **50.67**. The benchmark summary states that all top models easily achieve **high Syntax** scores (**>87% overall, many >95%**), while **Strict** scores remain only **≈48–60 overall** [2509.25922].

Difficulty stratification is effective. For **Claude Sonnet 4**, the benchmark reports **Medium: Syntax 100.00, Key 94.52, Strict 69.51** and **Hard: Syntax 98.61, Key 89.01, Strict 52.63**. More generally, **Strict** declines by **17.22–37.53 percentage points** from Medium to Hard, whereas Syntax degrades more modestly. This pattern is used as evidence that the Hard subset specifically stresses deep nesting and complex relationships [2509.25922].

The benchmark also tests whether verbosity is confounding the scores. Reported correlations between response length and scores range from \(r = -0.335\) to \(-0.040\), and \(R^2 < 0.05\) for most models. The stated conclusion is that the metrics are not simply rewarding longer outputs [2509.25922].

Domain-wise analysis shows relatively small variance. The reported **key matching scores** range from **0.776–0.867** for Medium difficulty and **0.474–0.540** for Hard difficulty across the 10 domains. The benchmark interprets this as evidence that no single domain dominates or is anomalously easy or hard [2509.25922].

Type-wise analysis shows an asymmetry between numeric and string outputs. Performance is highest on **numeric lists (0.90–1.00)** and lowest on **string lists (0.576–0.722)**. The benchmark description interprets this as evidence that primitive numeric parsing is comparatively reliable, whereas string lists remain error-prone because models truncate, hallucinate, or mis-segment items [2509.25922].

The qualitative error analysis identifies several recurrent failure modes. **Type mismatches** include outputs such as `"Memory": "4 GB"` when the schema requires a `number`, or `"Score": "92%"` when the schema requires a numeric score. **String extraction errors** include `HarmonyOS 3` instead of `HarmonyOS 3.0`. **Schema violations and duplication** include repeated copies of `"Software"` and `"Dimensions"` objects. **Hallucinations** include adding `"personal wellness hub"` as a preinstalled app when the text used that phrase only to describe the device. **Omissions** occur in hard cases such as missing some sensor entries under `Dimensions.Sensors` [2509.25922].

The benchmark further reports an end-to-end external-validity study. In a crawl → preprocess → construct schema → extract JSON → query/analytics pipeline over **Athlete Biography**, **Vehicle Recommendation**, and **Video Game Summary**, the **pipeline key matching scores** are **97.36** for **Claude Sonnet 4**, **91.87** for **DeepSeek R1 Distill Llama 70B**, and **83.26** for **Qwen3 32B**. The correlation between pipeline key scores and DeepJSONEval key scores is
\[
r = 0.987.
\]
The benchmark interprets this as strong external validity: improvements on DeepJSONEval track improvements in downstream web extraction [2509.25922].

## 6. Position within structured-output evaluation, applications, and limitations

DeepJSONEval is compared against benchmarks such as **StructEval**, **json-mode-eval**, **STRUCTURERAG**, **SoEval**, **FoFo**, **NESTFUL**, **JSONSchemaBench**, and **Schema-Guided Dialogue**, and is characterized in that comparison as **Multilingual**, **JSON Extractive**, with **2100 instances**, **Depth: 3–7**, domain labels, and support for constrained decoding. The benchmark’s explicit claim is that it bridges the gap between schema-only generation and shallow extraction by combining deep nesting, multi-domain web content, and extraction from text [2509.25922].

This positioning is distinct from work on strict schema adherence such as “Think Inside the JSON: Reinforcement Strategy for Strict LLM Schema Adherence” [2502.14905]. That line of work focuses on schema-constrained generation with metrics such as **Rows With No Output**, **Rows With Valid JSON**, **Mean Match Percentage**, and **Mean Noise Percentage**, and reports **ThinkJSON** at **62.41%** mean match and **0.27%** mean noise. DeepJSONEval addresses a different target: long-text comprehension and extraction into deep nested schemas, measured by **Syntax**, **Key**, and **Strict** rather than only field-level schema adherence [2502.14905].

In practical terms, the benchmark is framed around realistic web data mining scenarios such as **news aggregation**, **social media analytics**, and **healthcare**, and its schemas are presented as analogous to nested production representations used in those settings. The benchmark description also argues that practitioners can use **Strict** when exact correctness is required, and **Key** when structural fidelity is the more relevant bottleneck [2509.25922].

The dataset and benchmark are open-sourced at `https://github.com/GTS-AI-Infra-Lab-SotaS/DeepJSONEval`, with input texts, JSON schemas, gold outputs, and evaluation scripts for **syntax**, **key matching**, and **strict** [2509.25922].

The benchmark’s stated limitations are also specific. Its **10 domains**, while broad, do not cover all web scenarios. It focuses on intra-document extraction and nesting, with limited explicit temporal reasoning, cross-document reasoning, or multimodal fields. Although **3–7** levels of nesting are substantial, the benchmark description states that there is still room for more complex relational schemas such as graphs and cross-object references. Future directions explicitly mentioned include **even deeper** nesting levels, **dynamic schema adaptation**, more domains and more complex relations, potential integration with **knowledge graphs or databases**, and further algorithmic efficiency improvements for schema construction [2509.25922].

A persistent misconception addressed by the benchmark is that high JSON-validity scores imply robust structured extraction. DeepJSONEval’s results argue against that view: the syntactic problem is often nearly solved, but deep semantic extraction under nested schemas remains materially unsolved.

Source: https://www.emergentmind.com/topics/deepjsoneval