R3 Dataset: TRMR for Reading Comprehension

• R3 dataset is a large-scale reading comprehension benchmark that integrates explicit multi-step TRMR reasoning over DROP’s QA pairs.
• It uses a three-phase methodology—problem parsing, information retrieval, and answer derivation—to provide a complete, verifiable reasoning chain.
• The dataset enables fine-grained evaluation of parsing, retrieval, and derivation accuracies, promoting transparent and interpretable machine reading models.

The R3 dataset is a large-scale, reading comprehension benchmark in which every question–answer pair from the publicly available DROP corpus is augmented with explicit, structured representations of the full reasoning process required to obtain the answer. This formalism, termed Text Reasoning Meaning Representation (TRMR), is designed to require systems to not only produce answers but also to demonstrate explicit multi-step reasoning over unstructured text, thereby aiming to advance research on explainability and deeper language understanding in machine reading (Wang et al., 2020).

1. Text Reasoning Meaning Representation (TRMR): Formalism

TRMR defines reasoning over unstructured passages as a three-phase process, mirroring the standard approach humans use for complex question answering:

1. Problem Parsing: The question is parsed into a tree of atomic operators, each drawn from a fixed inventory (e.g., count(·), sum(·), more-select(·,·), filter(condition, set)). Each operator takes as arguments spans of the question text or results computed by other operators, forming a computation tree expressing the decomposition of the question.
2. Information Retrieval: For each argument in the parsing phase, the annotator specifies the exact span in the passage that realizes the raw information needed. This mapping is from arguments in the parse tree to character or token offsets in the passage, making explicit what textual evidence supports each reasoning step.
3. Answer Derivation: A structured, often linear description of how to combine the outputs of the previous steps to produce the final answer. This may involve sequencing operator applications or explicitly describing intermediate numeric or logical manipulations.

The overall representation is denoted $TRMR(P, Q) = (Parsing, Retrieval, Derivation)$ where $P$ is the passage and $Q$ is the question. No step is left implicit; atomic operations have fixed semantic interpretations, so the full reasoning trace can be verified or critiqued.

2. Annotation Methodology

A bespoke annotation platform was built to construct this dataset at scale:

• Step 1: Problem Parsing. Annotators are given passage ($P$), question ($Q$), and gold answer. They build the operator tree only by selecting items from the fixed inventory and choosing question spans—freeform text is not permitted.
• Step 2: Information Retrieval. Arguments from the parsing phase are automatically listed. Annotators mark the corresponding minimal spans in the passage, enforcing text-grounded evidence for each argument.
• Step 3: Answer Derivation. The system builds a first-pass derivation script by plugging the retrieved spans into the operator tree; annotators refine this as needed for completeness.

Workers are trained and must pass an examination on pre-labeled data before annotating, and are periodically re-evaluated. Validation is performed by separate panels on one third of all instances with at least two votes required for acceptance, yielding a random-check agreement rate of $95.92\%$.

3. Dataset Scale and Content

R3 encompasses all question–answer pairs from the publicly available DROP training and development sets, re-annotated with full TRMRs:

• Total pairs: Just over 60,000.
• Split: $\sim$54,000 TRMR-annotated training items, $\sim$6,000 development items.
• Operator coverage: The operator inventory captures five broad categories—Arithmetic (13 operators), Aggregate (3), Select (2), Sort (1), Filter (1)—and most questions require two or three compositional steps.
• Answer types: As with DROP, answers are either single spans from the passage or small integers.

4. Concrete Examples of TRMR Representations

Three illustrative examples demonstrate the TRMR pipeline:

Passage Excerpt	Question	Gold Answer	TRMR Parsing	Information Retrieval	Answer Derivation
“In Week 12 the Eagles kicked three field goals from beyond 40 yards,…”	“How many field goals over 40 yards were made?”	3	count(filter(“field goals”, “over 40 yards”))	“field goals” → tokens, “over 40 yards” → tokens	Filter field goals by distance ≥ 40 yards, count result → 3
“The Lions scored 21 points in the second quarter and 17 in the third; the Bears scored 14 in the third and 10 in the fourth.”	“By how many more points did the Lions outscore the Bears?”	14	more(“points by Lions”, “points by Bears”)	“points by Lions” → sum(21, 17), “points by Bears” → sum(14, 10)	Compute sums, subtract 38 – 24 = 14
“The April 12 shipment arrived; five days later it was placed on sale.”	“How many days after the shipment arrived was it placed on sale?”	5	after(“shipment arrived”, “placed on sale”)	“shipment arrived” → “April 12”, “placed on sale” → “April 17”	Compute date difference → 5

These examples show the compositional operator trees, precise span grounding, and explicit answer computation formalized in TRMR.

5. Evaluation Regime and Metrics

Unlike standard reading comprehension datasets that focus on exact match (EM) or F1 answer accuracy, the R3 benchmark requires evaluation of the entire reasoning process:

• Parsing Accuracy: Whether the predicted operator tree matches the gold parse exactly.
• Retrieval Accuracy: Percentage of argument spans correctly highlighted in the passage.
• Derivation Accuracy: Whether the stepwise script, when executed on the evidence, yields the gold answer.
• End-to-End Score: Proportion of examples correct in all three phases and answer.

No quantitative baselines or human upper-bounds for these metrics are reported in the release. The authors explicitly note that standard QA models achieve near state-of-the-art accuracy on DROP’s final answers but almost never reconstruct the full TRMR, highlighting the fundamental difference between answer prediction and explicit reasoning recovery.

6. Impact and Significance in Machine Reading Research

R3 addresses a critical shortcoming in existing reading comprehension datasets by making the reasoning chain explicit, parseable, and evaluable. The dataset:

• Enforces explainability by requiring an interpretable, linguistically grounded reasoning trace for each answer.
• Enables fine-grained analysis and benchmarking of intermediate reasoning abilities—operator selection, span retrieval, derivation—rather than pure answer accuracy.
• Provides a testbed for developing models that move beyond “black box” answer prediction to transparent, stepwise reasoning demonstration.

By setting a new standard for explainability and reasoning depth, R3 aims to catalyze progress towards genuinely interpretable and verifiable natural language understanding systems (Wang et al., 2020).