What Happens When the Model Eats the Stack? Rethinking the Research Agenda for Data Agents to Withstand the Bitter Lesson

Abstract: The bitter lesson poses an existential question for the data systems community, whereby LLMs trained end-to-end are rapidly internalizing new capabilities that previously required carefully engineered data agents. Guided by empirical insights, we argue that as models continue to improve, many proposed system layers designed to compensate for model limitations on a given task will increasingly be subsumed by the model itself. We instead identify enduring research opportunities, which lie in supporting data agents across many queries with curated contextual information about the data environment, which we call persistent semantic context. We find that these context layers demonstrate strong promise for improving data agent performance, but they also raise significant system challenges. Thus, a key requirement for future data systems will lie in natively serving persistent semantic contexts as a first-class abstraction in order to enable capable data agents working over huge, complex knowledge corpora. Towards this vision, we outline exciting new research opportunities, including designing efficient context data structures, storage methods, compression techniques, and semantic consistency protocols, to ensure integrity and correctness of the stored contextual knowledge.

• this paper find sustained system evolution for persistent semantic context allow more accurate data analysis agonist the model’s parametric competence.
• empirical evaluation shows coding agent accuracy increased by 19 percentage points when using a simple persistent context strategies.
• data agents fail due to semantic misinterpretation, highlighting the need for reusable semantic context despite significant build and storage costs.

Empirical motivation and central thesis

The paper examines which components of data-agent systems are likely to remain valuable as foundation models acquire increasingly strong capabilities for planning, code generation, tool use, debugging, and verification. Its organizing premise is Sutton’s “bitter lesson”: methods that scale general computation tend to displace domain-specific, hand-engineered procedures. Applied to data systems, this premise raises a direct question: if general-purpose models increasingly internalize the capabilities encoded in agent scaffolds, which systems abstractions will continue to matter?

The authors’ answer is persistent semantic context. They distinguish between capabilities that can be supplied by the model’s parametric competence and knowledge that is specific to a changing data environment. The latter includes schema interpretation, authoritative-source selection, join relationships, organization-specific metric definitions, undocumented conventions, historical discoveries, and user preferences. Such knowledge is external to model parameters, changes as the environment evolves, and is repeatedly rediscovered by agents unless it is stored and reused.

The paper therefore advances two related claims. First, increasingly capable general coding agents can subsume much of the functionality provided by human-designed data-agent pipelines. Second, the persistent systems problem is not primarily the optimization of individual agent trajectories, but the construction, serving, maintenance, and validation of reusable semantic knowledge across many queries. This argument extends earlier data-agent work such as TAG (Biswal et al., 2024), while placing greater emphasis on cross-query environmental knowledge than on query-local orchestration.

Experimental design

The empirical analysis compares general coding agents with human-designed data agents across two benchmarks. TAG-Bench combines exact computation, semantic reasoning, and world knowledge over relational databases (Biswal et al., 2024). DAB evaluates multi-step analytical tasks over fragmented enterprise data (Ma et al., 21 Mar 2026). The evaluation spans three model generations—o3, GPT-5, and GPT-5.6 Sol—and uses a Codex-style coding-agent harness for the general agents.

The human-designed baselines differ by benchmark. On TAG-Bench, the authors evaluate an Agentar-Scale-SQL-inspired system based on its published SQL-generation prompt and add an execution-guided refinement step. On DAB, they use DeepEye, a multi-agent workflow system designed for multi-turn data-analysis tasks [Li_2026]. This setup permits a comparison between a relatively general agent loop and systems that encode explicit assumptions about decomposition, SQL generation, workflow construction, and repair.

The comparison is informative but not fully symmetric. The paper explicitly states that the complete Agentar-Scale-SQL inference implementation is unavailable, so its baseline is an approximation rather than a reproduction of the published system. The results should consequently be interpreted as evidence about the relative trajectory of general coding agents and a representative human-designed pipeline, not as a definitive ranking of all possible data-agent architectures.

Figure 1: Performance evolution of general coding agents and human-designed data agents on TAG-Bench and DAB across successive model generations.

General coding agents increasingly subsume task-specific pipelines

The principal empirical result is that general coding agents improve more rapidly than the human-designed data-agent baselines. With the weaker o3 model, the human-designed Agentar-inspired system performs better than the coding agent on TAG-Bench and is more token-efficient. With subsequent model generations, however, the relationship reverses. GPT-5.6 Sol’s general coding agent achieves substantially higher accuracy than the human-designed agents on both TAG-Bench and DAB while also improving token efficiency.

This reversal supports the paper’s strongest claim: as models internalize planning, code synthesis, debugging, and validation, rigid agent architectures can become liabilities because they impose human priors that restrict generalization. The authors do not claim that all scaffolding is immediately obsolete. Rather, their longitudinal comparison suggests that scaffolding designed to compensate for model weaknesses loses relative value as those weaknesses are reduced by model scaling.

On TAG-Bench, the GPT-5.6 Sol coding agent reaches accuracy comparable to an oracle baseline consisting of expert-written queries executed with the LOTUS runtime (Patel et al., 2024). On DAB, the GPT-5.6 Sol coding agent improves by more than 35 percentage points over the o3 coding agent while also achieving more than a $2\times$ improvement in token efficiency. These gains arise from a general-purpose harness without benchmark-specific task decomposition or data-specific engineering.

The implication is methodological as well as architectural. Data-agent research that evaluates only a fixed model generation may overestimate the long-term value of specialized orchestration. A system improvement that compensates for current deficiencies in planning or error recovery may be outperformed by a later model that learns those behaviors directly. Evaluations should therefore measure not only absolute performance but also how system components interact with model capability over successive generations.

Efficiency gains change the bottleneck

The paper analyzes DAB trajectories by decomposing agent turns into schema exploration, analytical computation, semantic querying, verification, and recovery. The number of turns required per query declines sharply across model generations. In particular, GPT-5.6 Sol requires approximately four times fewer turns per task than o3.

Figure 2: Turns per query for coding agents across model generations on DAB, decomposed by action type.

The reduction is concentrated in analytical and semantic querying, as well as verification and recovery. This result directly challenges the premise that future data systems will primarily be bottlenecked by “agentic speculation”—large volumes of inefficient, exploratory queries issued by increasingly autonomous agents. The measured trajectory instead indicates that improved models formulate better actions, recover from fewer failures, and reach correct answers with fewer interactions.

This finding does not imply that query volume or execution cost is irrelevant. It establishes a narrower point: the measured per-task reasoning workload is declining across the evaluated model generations, so data-system designs premised on an indefinite increase in speculative agent-issued queries may be miscalibrated. The result also shifts attention toward the residual cost that does not decline at the same rate: understanding the external data environment.

Although total turns decrease, schema exploration and profiling remain a substantial relative component of GPT-5.6 Sol’s workload. As computation, planning, and debugging become more efficient, environmental acquisition accounts for a larger fraction of the remaining effort. The authors thus identify a changing bottleneck: not how to make the model reason through more query variants, but how to provide it with accurate knowledge of the relevant data environment before and during execution.

Failure analysis: environmental knowledge dominates residual errors

The authors classify failed coding-agent trajectories into five root-cause clusters:

Cluster	Failure type
C1	Misinterpreted task meaning or invalid proxy
C2	Wrong data source, table, field, or metric
C3	Entity, join-key, or identifier mismatch
C4	Incorrect filtering, aggregation, ranking, or output logic
C5	Runtime, tooling, or command-execution failure

For GPT-5.6 Sol, more than 60% of failures arise from semantic misinterpretation. These include misunderstanding the task, selecting an invalid proxy, choosing the wrong table or metric, and mishandling entities, join keys, or identifiers. By contrast, execution and tooling failures decline as model capability improves.

The distinction between C5 and the semantic clusters is central to the paper’s argument. C5 errors are plausibly addressable through better general model competence: improved command construction, code generation, debugging, and recovery. C1–C3 errors depend on facts about a particular environment. A model can be highly capable at reasoning in the abstract while still lacking knowledge of which dataset is authoritative, whether a field represents forecast or actual revenue, how an organization defines an entity, or which identifier supports a valid join.

The implication is that further improvements to generic execution competence may yield diminishing returns on these residual failures. The paper instead motivates an external knowledge layer that records environmental semantics and makes them available across tasks. This diagnosis is consistent with the broader distinction between parametric and non-parametric memory in retrieval-augmented generation (Lewis et al., 2020), but the proposed setting is more operationally demanding: the memory must represent a live, heterogeneous, organization-specific data environment rather than a relatively static document corpus.

Persistent semantic context

Persistent semantic context is defined as an explicit, language-based representation of knowledge about a data environment. It is constructed offline, accessed during online task execution, and amortized across multiple queries. Candidate contents include schema descriptions, canonical metric definitions, authoritative data sources, entity relationships, historical query discoveries, organizational conventions, workflow guidance, and user-specific preferences.

The paper evaluates this idea on held-out DAB tasks using four context-generation strategies: Self-Curated contexts optimized for accuracy, latency, or schema knowledge, and a GEPA-generated accuracy context (Agrawal et al., 25 Jul 2025). Each context is produced offline by GPT-5.6 Sol from 12 development trajectories and then appended to the agent’s prompt during evaluation.

The results establish that even simple agent-authored context can materially affect performance. The Self-Curated Accuracy context improves accuracy by 19 percentage points over the no-context baseline. The Schema context substantially reduces turns devoted to schema exploration, demonstrating that persistent knowledge can amortize repeated environmental discovery. However, the Schema context also causes a modest accuracy decline, which the authors attribute to possible over-reliance on the curated artifact and overfitting to the small development sample.

Figure 3: Coding-agent performance and execution composition under different persistent-context strategies.

The results expose a multidimensional trade-off rather than a uniformly beneficial intervention. Context optimized for accuracy need not minimize latency; context optimized for latency need not preserve accuracy; and context optimized for schema acquisition can reduce exploration while introducing stale or overgeneralized assumptions. Persistent context should therefore not be treated as a monolithic prompt augmentation. It is better understood as a materialized semantic state with workload-dependent objectives and physical-design choices.

The reported construction costs make this distinction especially important:

Context strategy	Build time	Build cost	Context size
Self-Curated Accuracy	164.9 s	$1.09	2.52 KB
Self-Curated Latency	154.9 s	$1.13	2.25 KB
Self-Curated Schema	3,413.6 s	$9.60	162.14 KB
GEPA Accuracy	1,359.8 s	$12.16	7.55 KB

Even in the small experimental setting, schema-oriented context construction requires more than 3,400 seconds and produces a 162.14 KB artifact. The experiment uses only 12 development trajectories and 12 data environments, whereas a production enterprise may contain terabytes of data, thousands of tables, heterogeneous storage systems, evolving documentation, and long-horizon traces. The paper therefore does not present context construction as a cheap replacement for online exploration. Its result is more precise: offline construction can improve accuracy and reduce repeated exploration, but the resulting build, storage, update, and validation costs become first-class systems problems.

A database research agenda for semantic state

The proposed agenda treats persistent semantic context as a first-class data-system abstraction rather than as an incidental prompt file. This framing separates the work from conventional prompt engineering. The relevant system must construct, index, retrieve, update, compress, validate, and govern semantic artifacts whose content may be partly natural language and whose dependencies may be implicit.

Semantic consistency

The most distinctive systems problem is semantic consistency. Traditional consistency models operate over explicit objects and operations: rows, records, writes, reads, and transactions. Persistent semantic context contains natural-language statements and inferred relationships. A schema change, documentation update, revised business definition, or statistical shift can invalidate context entries whose dependencies are not explicitly represented.

The paper identifies several dimensions of this problem. Consistency scope may be user-specific, project-wide, enterprise-wide, or external. A user’s preferred data source can coexist with an enterprise canonical metric, while external regulatory guidance may impose another scope. Systems must determine which agents observe an update, how updates propagate across overlapping scopes, and how conflicting semantic views are represented.

Consistency models also require an application-dependent spectrum. Regulatory reporting may require strong semantic consistency, whereas exploratory analytics may tolerate eventual consistency. Query-triggered validation offers a lazy alternative in which context is revalidated when accessed. The appropriate choice depends on the trade-off among freshness, accuracy, availability, latency, and maintenance cost.

Maintenance methods present a further design choice. Incremental semantic maintenance could revalidate only artifacts affected by a detected change, analogous to incremental view maintenance, but semantic dependencies are often implicit and heterogeneous. Holistic regeneration avoids explicit dependency tracking but may be computationally expensive and can erase useful stable knowledge. The paper leaves open the problem of workload-aware maintenance granularity.

Representation, physical design, and compression

The paper does not prescribe a single representation. Free-form text files are simple and editable but have weak structural guarantees. Knowledge graphs expose entities and relations but require schema design and dependency maintenance. Vector indexes support semantic retrieval but may obscure provenance, freshness, and exact dependency structure. A practical system may combine these representations in the same way that database systems combine indexes, materialized views, and caches.

Physical design must distinguish query-agnostic context from query-aware context. Schema descriptions, business terminology, and organizational documentation can be reused broadly. Intermediate reasoning, execution traces, and task-specific discoveries may be useful only for particular workloads. Systems must decide what belongs in persistent storage, what should be cached, how memory should be partitioned across fast short-term representations and compact long-term storage, and how retrieval should be conditioned on query scope.

Compression is unavoidable for long-lived context layers. A mature context store may contain millions of tokens from schemas, documentation, historical queries, execution logs, and agent-generated observations. Such a corpus cannot be supplied wholesale to every model invocation. The resulting lifecycle problem includes deciding what to retain, merge, summarize, compress, or discard while preserving information relevant to future queries. Existing work on KV-cache compression addresses session-level working memory (Zweiger et al., 18 Feb 2026), but the paper emphasizes that persistent semantic memory introduces a different retention and correctness problem.

Limitations and open questions

The empirical claims are bounded by the evaluation design. The study uses two benchmarks, three model generations, and a particular coding-agent harness. The model names and benchmark environments do not establish that the same trends hold for other models, modalities, database engines, enterprise workloads, or longer-lived deployments. The Agentar comparison is approximate because the complete published inference pipeline is unavailable, and the context experiments use only 12 development trajectories. That small sample makes the observed gains informative but also creates a substantial risk of context overfitting.

The persistent-context evaluation appends generated artifacts to the prompt, so it does not isolate the benefits of semantic organization from the costs and limitations of prompt injection. A context layer implemented through retrieval, structured dependency tracking, provenance-aware storage, or model-side memory may exhibit different accuracy and latency behavior. The study also reports context size and construction cost but does not provide a complete end-to-end accounting of update, invalidation, retrieval, validation, and storage costs over a long operational horizon.

Several technical questions consequently remain open. How should semantic dependencies be represented so that a changed schema or metric definition triggers complete but selective invalidation? What correctness guarantees can be provided for model-generated context? How should conflicting user, team, enterprise, and external semantic views be reconciled? Under what conditions does retrieval outperform a compact global context, and when does additional context induce distraction or over-reliance? Finally, the paper does not establish whether persistent context can retain its accuracy advantage under continuous schema evolution and distribution shift rather than a fixed held-out evaluation.

Conclusion

The paper presents a longitudinal argument about the changing division of labor between foundation models and data systems. General coding agents increasingly outperform human-designed data-agent pipelines and require fewer turns for individual tasks, weakening the case for system layers whose main purpose is to compensate for deficiencies in generic planning, code generation, debugging, or verification.

The enduring difficulty identified by the study is environmental knowledge. Agents continue to fail because they misunderstand data semantics, select inappropriate sources, and use incorrect entity or join relationships. Persistent semantic context addresses this gap by externalizing reusable knowledge about a changing data environment. Its demonstrated benefits—most notably a 19-point accuracy improvement for an accuracy-oriented context and large reductions in schema exploration—are accompanied by substantial construction and maintenance costs.

The paper’s central systems proposal is therefore not simply to add more context to prompts, but to make semantic context a managed, auditable, physically designed, and consistency-aware data abstraction. The unresolved issue is how to provide this abstraction with database-level guarantees despite the implicit, heterogeneous, and model-generated nature of its contents.