- The paper introduces SGKR, a method that uses code dependency graphs to extract operational knowledge for multi-step data reasoning.
- It employs offline graph construction and online BFS-based retrieval to generate minimal yet contextually relevant subgraphs from code dependencies.
- Empirical results on DABSTEP-hard, ConvFinQA-hard, and FinQA-hard benchmarks demonstrate notable accuracy improvements and reduced context sizes.
Structure-Grounded Knowledge Retrieval via Code Dependencies for Multi-Step Data Reasoning
Motivation and Problem Statement
Retrieval-augmented generation (RAG) pipelines are foundational in leveraging LLMs for domain-specific data analysis. However, traditional retrieval strategies predominantly rely on lexical or embedding similarity, which provide poor proxies for the operational knowledge necessary for synthesizing multi-step reasoning solutions. For data analysis, the key knowledge is often encoded within executable code, and the functional dependencies between code modules expose critical structural signals absent from surface-level document or embedding similarity. Consequently, retrieval approaches based solely on text similarity fail to capture the interdependent computational logic required to transform inputs into outputs.
Structure-Grounded Knowledge Retrieval (SGKR)
SGKR introduces a principled retrieval strategy centered on a code dependency graph, which enforces structural coherence of domain knowledge. Each node represents a function and its associated knowledge—executable code or documentation—and edges encode function-call relationships. Domain knowledge is thus operationalized as function implementations, where dependencies articulate knowledge propagation across computation steps.
Given a query, SGKR extracts semantic input/output (I/O) tags from the question (e.g., entities, attributes, derived quantities) and maps them onto the graph. Dependency paths connecting the input and output nodes are traversed (via BFS), yielding a compact, query-specific subgraph. The code and knowledge attached to the retrieved nodes form contextual support for downstream LLMs. This approach ensures the assembled retrieval context comprises only those functions and knowledge units critical for the actual reasoning path.
Graph Construction and Retrieval Pipeline
SGKR’s pipeline consists of two phases:
- Offline Graph Construction: Code is parsed into ASTs to extract function definitions and their call dependencies. Nodes with identical functions from disparate execution traces are merged to maintain graph consistency; semantic I/O nodes are inserted to anchor input/output tags and guide retrieval.
- Online Retrieval: Upon receiving a question, semantic tags are extracted and matched to I/O nodes. BFS traverses the dependency graph, retrieving minimal, dependency-consistent paths from input to output. This restricts context size and eliminates irrelevant code snippets that often plague embedding-based retrieval. The resulting context—knowledge and function examples—is provided to the LLM or coding agent to generate programmatic solutions.
Empirical Evaluation and Results
SGKR was evaluated on three challenging, domain-centric benchmarks: DABSTEP, ConvFinQA, and FinQA, each requiring multi-step reasoning grounded in operational knowledge. Hard subsets (multi-step tasks) were isolated for rigorous testing.
Key numerical results:
- DABSTEP-hard: SGKR achieved 24.34% accuracy, a substantial improvement over both vanilla prompting (6.88%) and embedding-based RAG baselines (≤11.64%), with only 4.65 average retrieved nodes.
- ConvFinQA-hard / FinQA-hard: SGKR yielded 74.67%/65.56% accuracy, outstripping all baselines, while retrieving approx. 25% fewer nodes on average.
Ablation studies underscore the necessity of SGKR’s structural elements. Removal of query-specific subgraph retrieval, attached knowledge, or node merging led to sharp drops in solution correctness. The results robustly demonstrate that code dependency graphs offer a superior retrieval signal for complex, multi-step reasoning tasks compared to embedding-based selection.
Implications and Theoretical Considerations
SGKR’s approach challenges the conventional wisdom that semantic similarity is sufficient for effective retrieval in code-centric or knowledge-intensive domains. By leveraging the intrinsic structure of functional code dependencies, SGKR aligns retrieval relevance with the actual computational flow required to answer specific queries. This structural grounding avoids distractions from superficially similar but irrelevant functions—a limitation evident in previous embedding-based approaches.
Practical implications include:
- Compact context windows: SGKR’s retrieval yields minimal yet highly relevant context, crucial for LLMs constrained by input length.
- Task-specific knowledge chaining: The framework consistently identifies the precise set of dependencies required to solve each task, minimizing noise.
- Domain adaptation: SGKR can be ported to domains with established, reusable computational patterns, provided executable code and knowledge mapping exist.
Theoretical implications suggest a shift towards retrieval paradigms that encode procedural and dependency relationships, moving beyond “flat” semantic similarity to structure-aware retrieval capable of supporting complex reasoning pipelines.
Limitations and Future Directions
SGKR’s reliance on executable code and explicit domain knowledge raises challenges in domains lacking well-structured code bases. The construction of usable dependency graphs entails significant curation, including converting symbolic expressions to code for datasets not naturally expressed as programs. Extending SGKR to open-ended or less structured domains will require automated methods for discovery and synthesis of computational programs and associated knowledge, potentially leveraging advanced LLMs for pre-play or simulation stages.
Further research may focus on:
- Automatic graph construction: Using LLMs or agents to generate candidate reasoning programs prior to building the dependency graph.
- Expanded domain coverage: Adapting SGKR to heterogeneous tasks by integrating broader sources of executable logic and documentation.
- Reliability and validation: Reducing risk of over-trusting LLM-generated code by incorporating explicit validation and human oversight, especially in high-stakes applications.
Conclusion
SGKR advances retrieval-augmented LLM reasoning by grounding knowledge retrieval in executable code dependency structures. Empirical results across multiple benchmarks establish its superiority over embedding-based retrieval strategies, particularly for multi-step, domain-specific data analysis tasks. The structural coherence enforced by function-call dependencies enables LLMs and coding agents to reliably acquire and synthesize operational knowledge, yielding higher accuracy and more efficient context usage. Future work should address automation and scalability of graph construction, as well as the integration of validation mechanisms for mission-critical deployment.