Structured Intermediate Representations (SIRs)

• Structured Intermediate Representations (SIRs) are organized intermediate objects—either explicit or latent—that impose clear structure between raw input and task execution.
• They span diverse forms such as linearized tables, scene graphs, ASTs, and differentiable latent trees, each tailored to separate complex subproblems like perception from reasoning.
• SIRs improve modularity, diagnostic clarity, and performance across domains by preserving task-relevant relationships in a transparent, inspectable format.

Structured Intermediate Representations (SIRs) are intermediate objects that impose explicit organization on computation between raw input and downstream prediction, reasoning, or execution. Across the literature, the structured object may be an inspectable artifact such as a linearized table, a scene graph, a causal chain, or a compiler graph, or a differentiable latent object such as tree-constrained marginals over dependencies. Taken together, these works suggest that the defining property of an SIR is not a single data structure, but the insertion of a structured layer that preserves task-relevant relations more explicitly than pixels, token sequences, or opaque latent vectors alone (Srivastava et al., 8 Jan 2025, Liu et al., 2017, Parmar et al., 28 Aug 2025, Hu et al., 23 Jun 2026, Reissmann et al., 2019, Webb et al., 2021).

1. Definitional scope and representational forms

The most direct form of an SIR is an explicit, human-inspectable artifact. In financial visual question answering, the intermediate representation is a linearized textual table extracted from a chart image by DEPLOT. The table preserves chart title, axis labels, categories, series labels where applicable, and value mappings, using | as a column separator and \<0x0A> as a row marker. This representation is “structured” because it retains headers, categories, and value correspondences, and “intermediate” because it stands between the chart image and downstream question answering (Srivastava et al., 8 Jan 2025).

A second form is a structured latent layer. Liu and Lapata’s document model induces a latent directed non-projective dependency tree distribution and exposes soft arc marginals $P(z_{ij}=1)$ and root marginals $P(\mathrm{root}(i)=1)$. These marginals form a latent adjacency matrix with tree constraints rather than an unconstrained attention matrix. The structure is consumed operationally by later layers that aggregate parent and child context, so the SIR is not merely diagnostic (Liu et al., 2017).

Other works occupy an intermediate position between symbolic and free-form text. In causal video question answering, the intermediate is a natural-language causal chain of the form $$[\text{Event A}] \rightarrow [\text{Event B}] \rightarrow [\text{Event C}] \rightarrow \cdots$$. In code translation, the intermediates are natural-language summaries and serialized abstract syntax trees; the AST must be flattened into text for input to an LLM. In semantic parsing, the intermediate may be reversible or lossy, but is always deterministically transformed into the final executable representation (Parmar et al., 28 Aug 2025, Tai et al., 11 Jul 2025, Herzig et al., 2021).

In robotics and visual scene understanding, the dominant SIR form is object-centric graph structure. RoBoSR defines a structured 3D scene graph $M_{sg}=(O_t,E_t)$, where nodes represent objects and edges encode relations such as on, inside, and adjacent, together with object states such as open or closed. Related work on explainable robot learning uses scene-graph-like structured image representations and learns sparse task-relevant subgraphs from a fully connected graph. In image and video understanding, visual relationship triplets $(\text{subject},\text{predicate},\text{object})$, scene graphs $G=(O,E)$, and temporally grounded human-object interactions play the same intermediate role between raw perception and reasoning (Hu et al., 23 Jun 2026, Mattes et al., 29 Jun 2026, Chiou, 2022).

Compiler research provides a further, older lineage of SIRs. GraalVM IR is a unified sea-of-nodes graph combining data flow and control flow in one structure; RVSDG is an acyclic hierarchical multigraph of nested regions; ICurry lowers declarative FlatCurry into block-structured graph-building code suitable for imperative backends. These systems are not auxiliary explanations: they are the actual representations over which optimization, lowering, or execution reasoning takes place (Webb et al., 2021, Reissmann et al., 2019, Antoy et al., 2019).

2. Structural dimensions and design choices

The literature distinguishes several orthogonal design axes. One is explicit versus latent structure. The financial VQA table is externalized, scored directly with RMS and RNSS, and passed to another model. By contrast, structured attention for text induces a soft non-projective tree distribution that is consumed internally and only converted to a hard tree for post hoc analysis (Srivastava et al., 8 Jan 2025, Liu et al., 2017).

A second axis is formal versus lightly structured text. Serialized ASTs, graph IRs, and table schemas are strongly typed or schema-constrained. Natural-language summaries and causal chains are less formal, but still exhibit fixed compositional patterns. The causal chain format is constrained by event segments, arrow-linked order, question conditioning, and a cap of at most 10 events, which makes it more structured than generic rationale text even though it remains natural language (Parmar et al., 28 Aug 2025, Tai et al., 11 Jul 2025).

A third axis is reversible versus lossy abstraction. In compositional semantic parsing, the pipeline is $x \mapsto z \mapsto y$, where $z$ is an intermediate representation and $y$ an executable form such as SPARQL or SQL. Reversible IRs preserve all information needed for deterministic reconstruction; lossy IRs deliberately remove difficult but semantically inessential details, with deterministic postprocessing or schema information restoring executability. The reported result that lossy IRs can outperform reversible ones indicates that full syntactic fidelity is not always the optimal objective for an SIR (Herzig et al., 2021).

A fourth axis is state description versus state transition modeling. Some SIRs are static summaries of content, such as chart tables or sentence/document trees. Others are explicitly dynamic. RoBoSR trains on $(SG_t,G)\rightarrow A_t$, $P(\mathrm{root}(i)=1)$0, and $P(\mathrm{root}(i)=1)$1, making the structured representation the state space in which action selection, effect prediction, and goal interpretation occur (Hu et al., 23 Jun 2026).

These distinctions show that “structured” does not imply a single favored level of formality. The code translation study is especially instructive here: ASTs are more structured than natural-language summaries, yet the strongest prompt configuration was chain-of-thought with an intermediate NL summary, not AST alone. This suggests that SIR quality depends not only on formal structure, but also on whether the structure is compatible with the downstream model’s inductive biases (Tai et al., 11 Jul 2025).

3. Operational roles across domains

Across application areas, SIRs act as modular interfaces that decouple different subproblems. The recurrent pattern is that the intermediate object absorbs one class of difficulty—perception, alignment, decomposition, or effect reasoning—so that later stages operate on a more regular substrate.

Domain	Representative SIR	Operational role
Financial VQA	Linearized chart table	separates chart reading from question answering
Document modeling	Non-projective dependency marginals	constrains attention to a globally consistent tree
Code translation	NL summary or serialized AST	guides prompt-time translation
Semantic parsing	Reversible or lossy IR	aligns target structure with natural language before compilation
Causal VideoQA	Arrow-linked causal chain	separates causal reasoning from answer selection
Robotics	Object-centric scene graph	supports precondition, effect, and goal reasoning
Compilers	Sea-of-nodes, RVSDG, ICurry	exposes dependencies and control structure for optimization or lowering

In multimodal chart reasoning, the claim is explicit: reasoning improves when the model does not have to infer both perception and logic simultaneously from pixels. DEPLOT handles title extraction, label reading, category-value correspondences, and numerical readings, while the downstream LLM reasons over the resulting table (Srivastava et al., 8 Jan 2025).

In document modeling, the structured layer replaces row-normalized pairwise attention with marginals from a globally normalized tree distribution. The downstream representation is built by aggregating along likely parent and child relations, so the induced structure functions as a structured bottleneck between contextual encoding and classification (Liu et al., 2017).

In causal video QA, the Causal Chain Extractor and Causal Chain-Driven Answerer instantiate a clean two-stage factorization: $P(\mathrm{root}(i)=1)$2 and $P(\mathrm{root}(i)=1)$3. The answerer is defined over question, chain, and options rather than over the video directly, making the causal chain the decisive intermediate state (Parmar et al., 28 Aug 2025).

In embodied reasoning, RoBoSR and related robot-learning work use scene graphs not merely for interpretation but as control-relevant state abstractions. This enables step-wise planning, causal transition modeling, and diagnosis of failures when the learned sparse subgraph includes distractors or omits key objects (Hu et al., 23 Jun 2026, Mattes et al., 29 Jun 2026).

Compiler IRs play an analogous role at a different level of abstraction. GraalVM IR unifies data and control dependencies in one graph; RVSDG replaces control-flow edges with hierarchical nodes such as $P(\mathrm{root}(i)=1)$4- and $P(\mathrm{root}(i)=1)$5-nodes; ICurry converts source-like functional logic terms into block-structured graph-building operations. In each case, the IR is designed so that later transformations operate over structurally explicit dependencies rather than rediscovering them from lower-level syntax (Webb et al., 2021, Reissmann et al., 2019, Antoy et al., 2019).

4. Construction, supervision, and induction mechanisms

SIRs are obtained by several distinct mechanisms. One common strategy is direct supervised generation of explicit structured targets. Financial VQA fine-tunes DEPLOT on 50,000 synthetic bar-chart image–table pairs; the model is trained to generate the full structured table string, including titles, headers, categories, and values. The resulting artifact is directly reusable and directly evaluable (Srivastava et al., 8 Jan 2025).

A contrasting strategy is end-task induction without structure labels. In structured document modeling, there are no dependency tree labels and no discourse annotations. The model induces tree-constrained marginals entirely from downstream supervision by embedding a differentiable non-projective parsing algorithm based on the Matrix-Tree Theorem inside the network (Liu et al., 2017).

A third strategy is deterministic transformation of gold outputs into easier targets. In compositional semantic parsing, the intermediate representation is generated from executable annotations by hand-designed transformations, and a deterministic compiler maps predicted IRs back to SPARQL or SQL. This decouples learning the structure-aligned representation from reproducing target-language syntax (Herzig et al., 2021).

Some systems use prompt-time or two-stage generation without retraining. Code translation experiments compare direct prompting, two-step pipelines in which an LLM first generates an NL summary or AST and then translates from it, and one-shot chain-of-thought prompts containing an intermediate. The critical methodological distinction is whether the intermediate is generated for each example or supplied only in a demonstration (Tai et al., 11 Jul 2025).

Other systems rely on pseudo-label generation and filtering. ChainReaction constructs 46,024 causal chains using GPT-4o, then applies programmatic validation, cross-LLM verification with Gemini 2.5, and manual video-grounded checking. The resulting chains are explicit supervised targets for both chain extraction and answering (Parmar et al., 28 Aug 2025).

Embodied systems often combine neural grounding with explicit structural extraction. RoBoSR builds the scene graph from RGB-D observations using SAM3 and a geometry-conditioned rule-based algorithm for object states and spatial relations, then trains a Qwen3-8B-based reasoner with supervised fine-tuning and reinforcement fine-tuning. SIR: Structured Image Representations for Explainable Robot Learning similarly begins from image-derived node features in a fully connected graph and learns a sparse task-relevant subgraph end-to-end (Hu et al., 23 Jun 2026, Mattes et al., 29 Jun 2026).

A further mechanism is feedback-conditioned refinement of intermediates. REFINER treats intermediate reasoning $P(\mathrm{root}(i)=1)$6 as an editable object and trains a critic to output structured feedback $P(\mathrm{root}(i)=1)$7. The generator is then trained to model $P(\mathrm{root}(i)=1)$8, refining an earlier intermediate in response to structured critique (Paul et al., 2023).

5. Evaluation and empirical effects

Because SIRs are explicit or at least partially externalized, many papers evaluate the intermediate itself rather than only the final task. In financial VQA, RNSS measures numerical interpretation accuracy and RMS measures whether numbers are attached to the correct semantic headers. The fine-tuned DEPLOT model raised RMS F1 from 50.93% to 91.06%, while RNSS remained near 98%; downstream reasoning also improved sharply, with GPT-4o moving from image-only MAPE 16.07 and RMSE 60.51 to image+$P(\mathrm{root}(i)=1)$9+Q MAPE 3.66 and RMSE 17.01, and with Qwen-7b-VL falling from MAPE 32.63 to 2.98 under the same comparison (Srivastava et al., 8 Jan 2025).

In compositional generalization, redesigning the target as an intermediate representation yielded large gains without changing model architecture. The best combinations produced a new state of the art on CFQ with a gain of +14.8 accuracy points and on the template splits of three text-to-SQL datasets with gains of +15.0 to +19.4 accuracy points. These figures are central evidence that representation design alone can alter generalization behavior substantially (Herzig et al., 2021).

In code translation, the headline result was that chain-of-thought with an intermediate NL summary performed best. Using Open Gpt4 8X7B, CoT-NL improved successful translations on CodeNet from 28.6% to 42.4% and on AVATAR from 17.6% to 24.3%, while two-step AST prompting performed very poorly at 2.6% on CodeNet. This result is often interpreted as evidence that a more formal intermediate is not automatically a better SIR if it is awkwardly serialized for the target model (Tai et al., 11 Jul 2025).

Structured latent layers also show measurable gains. Liu and Lapata report 86.9% on SNLI for structured intra-sentence attention with matrix inversion, compared with 86.2% for simple attention and 85.3% for no attention. On document classification, the strongest configuration applies structured attention at both sentence and document levels, reaching 68.6 on Yelp, 49.2 on IMDB, 82.1 on CZ Movies, and 76.5 on Debates (Liu et al., 2017).

Explainability-oriented robotics papers evaluate both task performance and analytical value. The robot-learning paper titled “SIR” reports that sparse graph policies on RoboCasa outperform image-based baselines on average with 19.5% versus 14.81% success rate, while RoBoSR reports substantial gains on GSR-bench, especially after lightweight post-training, and attributes these gains to state-space reasoning over object-centric scene graphs (Mattes et al., 29 Jun 2026, Hu et al., 23 Jun 2026).

Several works also introduce structure-specific metrics. ChainReaction proposes CauCo for causal coherence in chain generation and reports average scores of 0.89 for its generator versus 0.75 for a representative VLM baseline. Debiased scene graph generation emphasizes mean recall and non-graph mean recall rather than ordinary recall, reflecting the claim that a structured representation can be syntactically acceptable yet semantically impoverished by annotation bias (Parmar et al., 28 Aug 2025, Chiou, 2022).

6. Interpretability, limitations, and open tensions

A consistent attraction of SIRs is inspectability. Chart tables can be read directly; induced text trees can be extracted with Chu-Liu-Edmonds and compared with parser output; scene graphs reveal whether the agent is attending to the intended objects; causal chains expose the putative explanation path; compiler IRs make structural invariants explicit enough for mechanized reasoning in Isabelle/HOL or graph-based optimization passes (Srivastava et al., 8 Jan 2025, Liu et al., 2017, Mattes et al., 29 Jun 2026, Parmar et al., 28 Aug 2025, Webb et al., 2021).

Yet the literature also documents recurring limitations. One is narrow empirical scope. The financial VQA study is restricted to simple, stacked, and grouped bar charts in the finance domain and explicitly states that the model “has not been thoroughly evaluated on more complex visualizations.” The same paper contains an unresolved inconsistency in base-model RNSS, reported as both 89.67% and 86.67%, and the printed LaTeX for RMS F1 is syntactically incomplete as shown (Srivastava et al., 8 Jan 2025).

A second limitation is task-specific and sometimes weakly semantic structure. Liu and Lapata’s induced trees are unlabeled, task-specific, and only loosely aligned with standard syntax or discourse formalisms. REFINER’s intermediate reasoning traces are editable and criticable, but remain mostly textual rather than fully symbolic, and the connection between improved intermediate correctness and improved final answers is not perfect (Liu et al., 2017, Paul et al., 2023).

A third tension is that more formal structure does not necessarily help more. The code translation experiments found that ASTs were more structured but less useful than natural-language summaries in the strongest setting, and that strict two-step bottlenecks through an intermediate could be harmful. Model dependence was sharp: CoT-NL helped Open Gpt4 8X7B, slightly helped StarCoder, and harmed CodeGen (Tai et al., 11 Jul 2025).

A fourth issue is bias and grounding quality. Scene graphs can inherit reporting bias and bounded rationality from annotations, which is why positive-unlabeled reweighting and dynamic label frequency estimation were proposed for scene graph generation. Embodied systems such as RoBoSR and SIR depend heavily on accurate object detection, relation extraction, and state estimation; RoBoSR also acknowledges difficulty with highly deformable objects and fine-grained continuous states because its representation relies on discretized symbolic states (Chiou, 2022, Hu et al., 23 Jun 2026, Mattes et al., 29 Jun 2026).

Compiler-oriented SIRs expose a different trade-off: stronger invariants often require more elaborate construction and destruction procedures. RVSDG gains acyclicity, explicit hierarchy, and intrinsic SSA-like structure, but pays for them in restructuring and lowering complexity. ICurry simplifies imperative backend generation by normalizing case and let, but does so through a task-specific lowering step rather than through a universal IR (Reissmann et al., 2019, Antoy et al., 2019).

Taken together, these results suggest that SIRs are best understood not as a single technique but as a family of design choices about where to place structure in a pipeline, how explicit that structure should be, and which downstream operations it should support. The common lesson across these domains is that explicit or constrained intermediate structure can improve modularity, diagnosis, and reasoning fidelity, but only when the structure is both faithful to the task and usable by the system that consumes it.