Spider 2.0: Text-to-SQL Benchmark and Robophysical Model

• Spider 2.0 is a dual-purpose research initiative featuring an enterprise text-to-SQL benchmark that mirrors complex real-world data workflows with multi-dialect SQL generation.
• It evaluates performance using metrics like execution accuracy and success rate while highlighting challenges such as schema linking and iterative multi-step query planning.
• The platform also includes a biologically accurate spider robot that replicates arachnid active vibration sensing, enabling precise studies of biomechanical sensorimotor integration.

Spider 2.0 refers to two distinct research artifacts at the forefront of their respective fields: (1) a biologically informed spider robot for studying active vibration sensing in arachnids (Sun et al., 23 Jan 2026), and (2) an enterprise-scale text-to-SQL benchmark for evaluating autonomous code agents and LLMs in realistic data workflows (Lei et al., 2024). Both are denoted “Spider 2.0” in their research literature but embody distinct communities and technical goals. The following entry covers each artifact exhaustively.

1. Enterprise-Scale Text-to-SQL Benchmark: Design and Motivation

Spider 2.0 (Lei et al., 2024) is a comprehensive benchmark for the evaluation of models and code agents on real-world enterprise text-to-SQL (T2SQL) workflows. Unlike Spider 1.0 and contemporaneous datasets focused on single-shot semantic parsing against small toy schemas, Spider 2.0 addresses the intrinsic complexity of practical analytics pipelines:

• Schema complexity: Target databases average 812.1 columns (vs. 27 in Spider 1.0), with extensive normalized schemas, nested fields, and cross-table relationships.
• SQL dialect heterogeneity: Benchmarked agents must generate SQL in dialects such as BigQuery, Snowflake, SQLite, DuckDB, Postgres, and ClickHouse, often requiring dialect-specific operators and functions.
• Workflow complexity: Tasks frequently involve multi-step pipelines incorporating nested CTEs, DBT-style analytics, and iterative query refinement, with ground truth SQL scripts often exceeding 100 lines.
• Metadata and documentation grounding: Many problems require agents to parse project-level codebases (SQL, DBT/YAML, Markdown) and consult dialect documentation/live schema metadata to perform correct reasoning and query synthesis.

This structure reflects real analyst workflows, including metadata exploration, multi-query planning, and iterative debugging, thus bridging the gap between academic benchmarks and production enterprise data science.

2. Dataset Construction and Characteristics

Spider 2.0’s dataset is drawn from real-world sources to maximize ecological validity. Key aspects:

• Sourcing of databases: The benchmark includes 213 databases from 74 BigQuery and 54 Snowflake public sources, 30 SQLite, 40 DuckDB, 10 Postgres, and 5 ClickHouse databases.
• Problem formulation: Of 632 total tasks, 547 involve single-SQL writing and 78 are full DBT project workflows. SQL queries are collected from vendor tutorials, open-source DBT projects, and enterprise analytics forums, then meticulously rewritten (surface-level in 84.2%, semantic in 42%) to ensure originality and shield against possible data leakage.
• Prompt annotation: Tasks are paired with natural, unambiguous English instructions for both “agentic” (codebase-aware) and traditional (text→SQL) paradigms.
• Task complexity breakdown: 25.3% easy (<80 tokens), 44.2% medium (80–159 tokens), and 30.5% hard (≥160 tokens). Ground truth SQL averages 148.3 tokens and 7.1 functions per query.
• Workflow features: DBT project-level tasks comprise 12.3%; 13% require parsing external documentation; 22–19% demand metadata search across multi-schema, nested, or dynamic tables.

Spider 2.0 task resolution typically follows this workflow: codebase/schema browsing, query composition, result inspection, code/SQL editing, and iterative rerunning until the evaluation script is satisfied.

Statistic	Spider 1.0	Spider 2.0 (snow)	Spider 2.0 (overall)
Avg. DB columns	27	812.1	743.5
Problems	10,181	~632	632
SQL dialects	1	6	6

3. Evaluation Metrics and Baseline Results

Spider 2.0 employs two principal evaluation protocols:

• Execution Accuracy (EX): Used for text-to-SQL (2.0-lite and snow tracks), EX is the proportion for which the predicted SQL’s result table covers all gold columns:

$$\mathrm{EX} = \frac{1}{N}\sum_{n=1}^N \mathbf{1}(v^n,\hat v^n)$$

where $\mathbf{1}(v,\hat v)$ evaluates to 1 if all gold columns are present; focusing exclusively on “condition_cols” permits assessment under strict constraints.

• Success Rate (SR): In the agentic setting, SR is the fraction for which a model’s final artifact (string, table, or database) meets the task-specific, hand-crafted evaluation script.

Additional answer-type-dependent procedures include tolerant substring and number matching (string), table comparison with “ignore_order”, and DuckDB file content comparison (database artifacts).

Baseline results on Spider 2.0 expose a substantial drop-off in current model performance relative to their Spider 1.0 achievements:

Model/framework	Spider 1.0 EX	Spider 2.0-lite EX	Spider 2.0 SR
GPT-4o + DIN-SQL	85.3%	1.5%	5.7% (AutoEval)
GPT-4o + DAIL-SQL	86.6%	5.7%	7.3% (Reflexion)
GPT-4o + CHESS	87.2%	3.8%	7.9% (CodeR)
Spider-Agent (o1)	—	—	17.0%

Agentic frameworks (AutoEval, Reflexion, CodeR, Spider-Agent) all yield SR below 18% overall, compared with >85% EX for single-shot parsing on Spider 1.0.

4. Failure Modes and Core Technical Challenges

Empirical analysis of 300 tasks identifies common failure categories:

1. Erroneous data analysis (35.5%): Incorrect use of dialect-specific functions (10.3%), advanced aggregation/multistep CTE failures (17.7%), and misapplied SQL analytics (7.5%).
2. Schema linking errors (27.6%): Misidentification of tables (10.1%), columns (16.6%), and join keys (8.3%).
3. Dialect drift: Incorrect handling of syntax and semantics between dialects (e.g., BigQuery vs. Snowflake).
4. Nested schema confusion: Failure to flatten RECORDS/ARRAYS fields, notably in BigQuery.
5. External doc grounding: Poor translation of prose/markdown analytics rules into SQL predicates.

Fundamental challenges for model improvement highlighted by Spider 2.0 include:

• Long context modeling: Inputs routinely exceed model context window due to thousands of schema items and associated code/documentation.
• Complex metadata reasoning: Parsing and leveraging schema, code DAGs (DBT), and nested tables.
• Multi-dialect code generation: Mastery of dialect-specific logic and function libraries.
• Iterative, multi-step workflow planning: Chaining queries and edits with feedback and error recovery.
• External knowledge integration: Mapping business rules from prose Markdown/blog documentation into precise SQL logic.

5. Recommendations and Future Directions

Spider 2.0’s authors recommend several strategies to advance model and agentic capabilities:

• Agent action primitives: Design of dedicated schema/query actions (GetTables, GetTableInfo, SampleRows) to facilitate effective environment grounding.
• Retrieval-augmented prompt engineering: Selective in-context retrieval of relevant code, documentation, and schema components.
• Explicit reasoning plans: Decoupling task decomposition and SQL generation yields moderate EX improvements (~3%).
• Iterative debugging workflows: Allowing agentic models to examine query errors and refine plans (as measured, Spider-Agent performed 9 steps per task on average).
• Model architectural enhancements: Enlarged context windows and tailored retrieval for encompassing full DBT codebases and documentation in a single pass.
• Hybrid planning and synthesis: Integration of high-level pipeline/CTE planning with detailed code generation to bridge the abstraction gap.

The benchmark thus catalyzes research on scalable, robust, and contextually aware code agents for production analytics, ETL, and BI workloads.

6. Biologically Accurate Spider Robot for Active Vibration Sensing

Distinct from the software benchmark, Spider 2.0 also designates an eight-legged robophysical model intended to replicate active vibration sensing in orb-weaving spiders (Sun et al., 23 Jan 2026). Key features of this physical platform:

• Mechanical design: Eight legs (bilateral symmetry, Uloborus diversus prototypical morphology), each with four rigid segments (femur, tibia, metatarsus, tarsus) joined by compliant silicone (DragonSkin 0010, Sylgard 182) with variable joint stiffness. Segment lengths and joint ranges of motion (60–75° sagittal flexion) are dimensioned from U. diversus kinematics.
• Actuation: A central Dynamixel XM430-W350-R motor applies synchronized crouching to all eight legs using a cable–pulley tendon system, mapping motor angle to individual joint flexions.
• Sensing: Tri-axis accelerometers (ADXL326) mounted at metatarsus–tarsus joints acquire DC–500 Hz data sampled at 500 Hz, processed by low-pass IIR filtering and windowed Fourier analysis.
• Experimental validation: Baseline crouch–recover maneuvers elicit leg accelerations peaking at ±5 m/s² and dominant 3.8 Hz spectral peaks reflecting body–web resonances. Introduction of prey mass induces a superposed resonant mode (5–6 Hz), with spectral separation modulating as a function of prey-to-body mass ratio.
• Biomechanical insight: Deeper crouches elevate web tension and the baseline resonance, quantifying the mechanical gain ($$\partial A/\partial\phi \approx 0.005\ \mathrm{m/s}^2/\degree$$), thereby enabling systematic investigation into the effect of posture on vibrational sensitivity.

Spider 2.0 delivers a biologically faithful testbed for dissecting the biophysical basis of active sensing, posture-mediated coupling, and anisotropic joint mechanics in arachnid web interaction (Sun et al., 23 Jan 2026).

7. Broader Implications

Progress on both manifestations of Spider 2.0 targets open problems in their respective subfields. The T2SQL benchmark instantiates the stringent requirements of autonomous data analytics in enterprise environments and demonstrates the limitations of contemporary LLMs and code agents, revealing urgent research directions in context reasoning, execution-guided synthesis, and hybrid workflow orchestration (Lei et al., 2024). The robophysical spider system provides a tractable, empirically validated proxy for quantifying active sensing strategies in complex biological substrates, with direct application to the study of embodied sensorimotor function and the design of next-generation bioinspired robots (Sun et al., 23 Jan 2026).