Dr.Spider: Dual Domain Research Systems

• Dr.Spider is a dual-identification system featuring a diagnostic benchmark for text-to-SQL robustness and a UAV-tethered hexapod for dynamic object retrieval.
• In its text-to-SQL application, Dr.Spider applies 17 perturbation types across database schemas, NLQs, and SQL queries to expose and improve model weaknesses.
• For aerial robotics, the Dr.Spider system demonstrates rapid, robust object retrieval through a compact, tether-integrated hexapod design optimized for diverse outdoor terrains.

Dr.Spider refers to two unrelated research systems in distinctly different domains: (1) a diagnostic robustness benchmark for text-to-SQL semantic parsing, and (2) an aerially deployed tethered hexapod robot for dynamic object retrieval. Both have been recognized by the research community for their systematic design and real-world applicability. This article covers both lines under the unified name "Dr.Spider," specifying context for each.

1. Dr.Spider as a Diagnostic Evaluation Benchmark for Text-to-SQL Robustness

Dr.Spider is a comprehensive diagnostic benchmark aimed at measuring and dissecting the robustness of neural text-to-SQL models against real-world, task-specific perturbations. Built on the Spider cross-domain text-to-SQL dataset, Dr.Spider introduces realistic modifications at the levels of database schema, natural language questions (NLQs), and SQL queries to expose failure modes and drive innovation in semantic parsing systems (Chang et al., 2023, Li et al., 2024).

1.1 Benchmark Construction and Perturbation Taxonomy

Dr.Spider comprises 17 perturbation types, each operating on either the database schema, the NLQ, or the SQL:

• Database-Side (DB):

1. Schema-synonym: Replaces column names with synonyms (e.g., country→nation).
2. Schema-abbreviation: Abbreviates column names (e.g., ranking_points→rank_pts).
3. DBcontent-equivalence: Splits or re-encodes columns (e.g., age→birthyear).

• NLQ-Side:

1. Keyword-synonym, Keyword-carrier
2. Column-synonym, Column-carrier, Column-attribute, Column-value
3. Value-synonym, Multitype, Others (control).

• SQL-Side:

1. Comparison, Sort-order, NonDB-number, DB-text, DB-number

Examples are paired: for each perturbed case, the clean (original) version is included (Chang et al., 2023).

1.2 Dataset Scale and Composition

Dr.Spider contains ≈15,269 parallel example pairs, distributed over 200 databases (138 domains):

Category	#Perturbation Types	#Perturbed Samples
DB	3	5,854
NLQ	9	7,593
SQL	5	1,822

Each perturbation stresses a unique mapping (e.g., synonymy, value paraphrasing, semantic re-encoding) challenging the lexical and structural generalization ability of models (Li et al., 2024).

1.3 Perturbed NLQ Generation

Dr.Spider employs a hybrid generation process:

• Crowdsourcing (e.g., MTurk) yields manually paraphrased NLQs.
• Expert Filtering annotates and validates perturbation types.
• PLM Augmentation uses OPT-66B for in-context prompt-based paraphrase generation, further filtered by NLI models (BART-MNLI) and expert review.
• ~63% of PLM-generated NLQs are accepted; triple-annotation yields Fleiss’ κ = 0.61 consistency (Chang et al., 2023).

2. Evaluation Protocols and Metrics

The core metric for Dr.Spider is Execution Accuracy (EX):

$$\mathrm{EX} = \frac{1}{N} \sum_{i=1}^N 1\Big[ \mathrm{exec}(\text{SQL}^{pred}_i) = \mathrm{exec}(\text{SQL}^{gold}_i) \Big]$$

EX requires that the predicted SQL, when executed over the (possibly perturbed) DB, matches the gold query’s result set. Exact Match (EM) is also defined in (Chang et al., 2023), but not used in (Li et al., 2024). Test-Suite Accuracy (TS), which checks execution across multiple DB instantiations, is not used in Dr.Spider due to lack of test suites for perturbed DBs.

Robustness is measured as:

• Absolute Drop: Δ = A_clean − A_pert
• Relative Robustness Rate: R = (correct on both clean & perturbed) / (correct on clean)

These metrics quantify the susceptibility of models to structural and lexical domain shift.

3. Experimental Findings and Robustness Analyses

Dr.Spider has been adopted by multiple benchmark studies (Chang et al., 2023, Li et al., 2024). Key results and insights include:

• Macroscopic Robustness: Even state-of-the-art models, e.g., Picard, experience an average EX drop of 14% under perturbation (79.3%→65.0%), with certain phenomena (DBcontent-equivalence) causing drops >45% (Chang et al., 2023).
• Model Scale Impact: Larger PLMs (T5-base→large→3B) achieve systematically higher absolute and post-perturbation EX (57.0%→67.2%→71.7% on clean; 40.6%→52.4%→57.1% on perturbed).
• Decoder Architecture: Bottom-up (SmBop) decoders improve schema-local robustness (DB perturbations), whereas top-down (GraPPa) decoders better handle NLQ paraphrases. Entity linking boosts robustness to value-literal paraphrasing but risks over-reliance on surface matching.
• Perturbation Sensitivity: Models are most vulnerable to semantically deep DB transformations and value synonymy. Most rely on string-matching for value alignment, as exposed by value-synonym and DB-text perturbations.

The introduction of Dr.Spider has enabled fine-grained diagnostic breakdowns per perturbation type, guiding architectural and data-centric improvements.

4. Comparative Model Performance and Benchmarking

Performance on Dr.Spider has become a key reference for the semantic parsing community (Li et al., 2024). Table below summarizes EX for top models on Dr.Spider perturbation categories:

Model	DB Avg	NLQ Avg	SQL Avg	Global Avg
SMBOP	50.0	58.1	72.2	60.8
RESDSQL-3B+NSQL	59.4	69.3	83.2	71.7
ChatGPT	67.1	73.2	82.6	74.9
CodeS-1B	60.3	64.8	72.6	66.3
CodeS-3B	62.4	72.3	79.8	72.8
CodeS-7B	63.6	74.3	83.0	75.0
CodeS-15B	64.4	75.2	81.5	75.1

• CodeS-7B and 15B closely match or exceed ChatGPT+ZeroNL2SQL on global EX (75.0/75.1% vs. 74.9%) without explicit robustness-targeted training.
• On SQL-specific perturbations, specialized models with intermediate abstract forms (e.g., RESDSQL-3B+NatSQL) retain an advantage.
• DBcontent-equivalence remains the hardest, with <48% EX across systems (Li et al., 2024).

5. Recommendations and Future Directions

Dr.Spider reveals clear directions for research:

• Hybrid Decoding Architectures: Integrating both bottom-up and top-down parsers may provide unified robustness.
• Advanced Schema Linking: Moving beyond string-matching, e.g., via lexical normalization or learned fuzzy matching.
• Value Representation: Incorporating normalization for value synonyms and numeric-textual equivalence.
• Data Augmentation: Using Dr.Spider’s 17 perturbations for adversarial finetuning.
• Benchmark Expansion: Including key merges, deeper DB-content transformations, and evaluating LLMs with improved prompt robustness (Chang et al., 2023).

A plausible implication is that further model gains on Spider alone will not reflect genuine real-world robustness without parallel gains on Dr.Spider and similar challenge sets.

---

6. Dr.Spider as a UAV-Tethered Mobile Gripper (“SPIBOT”/Dr.Spider)

A distinct line of research uses the term Dr.Spider as a synonym for “SPIBOT”: a compact, tethered, six-legged robotic mobile gripper designed for aerial deployment from UAVs to autonomously retrieve objects in dynamic outdoor environments (Kang et al., 2024).

6.1 System Architecture

• Mechanical Design:
  • Hexapod robot with 6 legs; each leg has 2 DOF (coxa and tibia), 12 servos total.
  • Link topology: Metal alloy for proximal links, 3D-printed Onyx/PLA for distal segments. Exact link lengths and DH parameters not specified.
  • Compact stowage (300×268×175 mm closed) and rapid deployment (390×410×225 mm open) optimized for UAV payload profile.
• Mobility and Grasping:
  • Tripod gait (three legs lifted per step), supporting differential drive–like motion.
  • Acts as both mobile base and multi-contact gripper.
• Tether-Drone Integration:
  • Steel wire tether, winched by a Robotis MX-64AR onboard the UAV for lowering and retrieval.
  • Tethered approach decouples drone propeller downwash from the manipulation phase.

6.2 State Estimation, Sensing, and Control

• Sensor suite includes IMU (6-DOF), ultrasonic sensors, AprilTag fiducials for vision-based pose estimation, and in-line force sensing (FSR402) for contact and grip confirmation.
• State estimation is threshold-based, integrating IMU tilt, ultrasonic returns, and force feedback; no explicit Kalman filter or observer is described.
• Real-time control uses an affordance-based action selection:

$$a_t = \operatorname{argmax}_i [ g_i \alpha_i(S_t) ]$$

where $S_t$ comprises flags for mission, landing, heading alignment ($1-\tanh(C_a\theta d/D)$), approach distance ($1-\tanh(C_d|d-T_d|)$), over-object checks, and instability.

• Modes include stabilizing, aligning, approaching, fine forward approach, grasping, and retry on instability.

6.3 Experimental Performance

• Mean time from landing to successful retrieval and lift-off is ≈13 s.
• Near-100% convergence rate across diverse terrains (pontoon, grass, coastal sand w/ rubber mats); a single retry occurred due to landing error.
• Distance and heading errors converge monotonically; performance is robust to noisy UAV pose and state perturbations (Kang et al., 2024).

6.4 System Design Insights

• Dr.Spider’s (SPIBOT’s) tethered hexapod design generalizes to any drone-ground hybrid manipulation where environmental disturbance (wind, moving ship deck) precludes direct UAV interaction.
• The compact, lightweight structure with real-time affordance control enables rapid, robust adaptation to dynamic environments.
• The design is readily extensible to mobile aerial-ground constellations for tasks beyond object retrieval (e.g., perching, inspection).

---

7. Nomenclature Clarification and Disambiguation

Dr.Spider refers, in different research contexts, to:

1. A text-to-SQL robustness benchmark serving as a diagnostic standard for model evaluation under perturbations (Chang et al., 2023, Li et al., 2024).
2. A UAV-tethered hexapod manipulator (“SPIBOT”) for robust outdoor object retrieval in dynamic settings (Kang et al., 2024).

Both systems are intensively cited and distinctive in their respective communities, but share no technical or conceptual lineage beyond the name.

---

References:

• "Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustness" (Chang et al., 2023)
• "CodeS: Towards Building Open-source LLMs for Text-to-SQL" (Li et al., 2024)
• "SPIBOT: A Drone-Tethered Mobile Gripper for Robust Aerial Object Retrieval in Dynamic Environments" (Kang et al., 2024)