ShortcutsBench: Benchmarking Shortcut Reliance

• ShortcutsBench is a suite of benchmarks that exposes and quantifies shortcut-driven behavior in machine learning models across NLP, GUI automation, and multimodal tasks.
• It employs diverse evaluation strategies—including API accuracy, GUI task success, and attribution metrics—to diagnose overfitting and spurious feature exploitation.
• Empirical findings reveal significant performance drops in long-horizon planning and semantic reasoning, highlighting the need for more robust, interpretable AI systems.

ShortcutsBench is the designation for a recent wave of benchmarks designed to expose, measure, and analyze shortcut-driven behavior in machine learning systems—spanning natural language processing, multimodal reasoning, tool-using agents, GUI automation, and concept-based neuro-symbolic models. In diverse forms, ShortcutsBench frameworks rigorously quantify both a model’s reliance on spurious, surface-level solutions ("shortcuts") and its capacity for robust, semantic task-solving beyond such artifacts. This entry surveys the main instantiations of ShortcutsBench, reviewing their motivations, technical formulations, evaluation strategies, and findings.

1. Foundational Concepts: Shortcuts and Benchmarking Rationales

A "shortcut" in machine learning denotes any spurious feature, data artifact, or statistical regularity that a model can exploit to solve a given task, while bypassing the core reasoning, compositionality, or semantic understanding that the task was intended to require. Detection and mitigation of shortcut reliance are central for robust, generalizable, and interpretable systems. Several subdomains have motivated ShortcutsBench implementations, unified by these aims:

• Diagnosing Overfitting to Dataset Bias: Models frequently leverage dataset-specific quirks (token co-occurrence, output priors, UI patterns) for impressive in-distribution performance, yet generalize poorly under distribution shift.
• Tool-Augmented Agent Evaluation: As LLM-based agents become ubiquitous and interact with real APIs/GUI environments, benchmarks must determine if success reflects true autonomy and compositional planning, or merely superficial pattern matching.
• Explainability and Attribution Faithfulness: Faithful input salience, feature attribution, and concept extraction methods demand validation against ground-truth, synthetic or controlled shortcut-injected scenarios.
• Neuro-Symbolic Reasoning: In tasks connecting perception and logic (e.g., concept-based vision), it is essential to verify if models acquire intended concepts or exploit latent symmetries for correct outputs.

2. Technical Design Patterns Across Domains

API-Based Autonomous Agents

ShortcutsBench for API agents directly targets real-world API invocation and automation tasks using human-authored workflows. Specifically, a large-scale dataset was constructed by mining the public corpus of Apple Shortcuts, which provides multi-step, human-constructed sequences involving 1,414 distinct APIs from 88 apps. Each task decomposes into a natural-language trigger (user query), a gold-standard action sequence, and explicit parameter annotations including static values, output-threading, and "Ask Each Time" parameters that require runtime prompts. The system stratifies tasks by workflow complexity—p=1 (≤1 action), p=2 (2–5), p=3 (6–15), p=4 (16–30)—and by semantic category (e.g., Lifestyle, Health, Developer). Agents are assessed on:

• API Selection Accuracy: Correct choice of the next API call in a sequence, conditioning on gold action history amidst distractors.
• Parameter Filling Rates: $Acc_{spp}$ for static parameters, $Acc_{ofpa}$ for correctly using outputs of prior actions.
• Recognition of Input Need: $Acc_{afni}$ captures detection of parameters that must be set by querying the user or system at runtime.

Findings demonstrate that while high-token-count LLMs (e.g., Gemini-1.5-Pro, QWen-2-72B) approach 80–95% accuracy in isolated parameter-filling and short workflow selection, performance degrades sharply on long, compositional chains (p=4: 30–50% accuracy) and when missing values must be dynamically recognized. This exposes a bottleneck in long-horizon planning and dialogue (query disambiguation), even among state-of-the-art models (Shen et al., 2024).

GUI–Shortcut Hybrid Agents

MAS-Bench (also labeled ShortcutsBench in some references) extends the paradigm by comparing GUI-only, API-only, and hybrid agents in mobile app environments. The testbed covers 139 tasks across 11 Android applications, with all tasks solvable via GUI interaction, though some can be vastly accelerated by invoking one of 88 predefined shortcuts (including APIs, deep links, and RPA scripts). Agents are further evaluated on their ability to autonomously generate new shortcuts (replay macros, dynamic UI-anchored actions). Seven evaluation metrics are used, including success rate, mean steps, mean execution time, shortcut and GUI action counts, and shortcut success ratio.

Results indicate that hybrid agents leveraging predefined shortcuts outperform GUI-only baselines in both task completion rate and execution efficiency, and that shortcut generation quality remains a challenge: only predefined and well-engineered dynamic shortcuts consistently confer benefits, while naive macro replay is fragile (Zhao et al., 8 Sep 2025).

Salience, Attribution, and Language Modeling

ShortcutsBench protocols in text classification and language inference domains inject controlled lexical shortcuts into real or synthetic datasets (e.g., presence of particular tokens implying a label, or structured context token dependencies). The resulting setups provide ground-truth feature importance for pinpoint evaluation of input salience, and for assessing the robustness of LLMs to diverse shortcut types (lexical overlap, subsequence, constituent, negation, positional, style).

Metrics in these instantiations include:

• Precision@k and Mean Rank for input attribution faithfulness (Bastings et al., 2021).
• Accuracy Drop ($\Delta$Acc), Shortcut Bias Score, Semantic Fidelity, Internal Consistency, and Confidence for evaluating LLM performance and explanation quality on shortcut-injected versus standard test sets (Yuan et al., 2024).

Systematic findings reveal that no single salience method or prompting regime is universally optimal; models and attribution methods exhibit strong shortcut-related failure modes, particularly in complex or adversarially-structured settings.

Visual Question Answering (VQA) and Multimodal Shortcuts

The VQA-VS "ShortcutsBench" exposes shortcut reliance in VQA models, introducing out-of-distribution (OOD) test sets that each induce shifts on one of nine shortcut axes (e.g., language priors, keyword correlations, visual object biases). A robust evaluation requires model selection on IID splits, followed by metric reporting on both IID and OOD (per-shortcut), capturing accuracy, mean OOD performance, and the IID/OOD gap (Si et al., 2022).

3. Evaluation Methodologies and Core Metrics

Benchmark Domain	Main Evaluation Metrics	Shortcut Types Captured
API/Automation Agents	API Selection Acc, $Acc_{spp}$, $Acc_{ofpa}$, $Acc_{afni}$	Output chaining, parameter omission, long-horizon planning
GUI–Shortcut Hybrid Agents	Success %, Mean Steps, MET, Shortcut Call Ratio	GUI vs. API, agent-generated replay/dynamic macros
Text Salience/Attribution	Precision@k, Mean Rank	Lexical token, token-in-context, order-dependence
LLM/Language Inference	$\Delta$Acc, SBS, SFS, ICS, CFS	Lexical overlap, subsequence, constituent, negation, position, style
VQA (multimodal)	OOD Acc, mean_OOD, $\Delta$	Language, vision, multimodal priors
Neuro-Symbolic (rsbench)	Concept Alignment, Collapse, RS Count	Symbolic concept remapping, latent symmetry
Genomics (benchNGS)	F1, Precision, Recall	Mapping heuristics, read length/identity biases

Each variant injects, amplifies, or labels the potential for shortcut exploitation, then tests methods/models for generalization when shortcuts are neutralized or shifted.

4. Key Research Findings

Across domains, empirical results consistently find:

• Severe Performance Drop Under Shortcut Neutralization: E.g., LLMs trained for natural language inference experience up to 52 percentage point accuracy decreases on constituent-based adversarial cases; autonomous agents’ planning accuracy decreases up to 46% on complex, multi-step real-world workflows.
• Input Attribution and Explanation Fragility: Attribution techniques (gradient, integrated gradients, LIME) display nontrivial differences in recovering salient shortcut tokens; preferred methods depend on architecture and shortcut type, and defaults may be highly misleading (Bastings et al., 2021).
• Parameter-Filling and Information-Gap Recognition Deficits: Even the largest LLM-based agents fail to reliably detect when to prompt the user for unspecified input—a critical limitation for practical deployment (Shen et al., 2024).
• Trade-off Between In-Distribution and OOD Robustness: Head–tail splits in shortcut-specific OOD evaluation reveal that boosting performance on frequent, biased patterns often degrades rare case accuracy; model selection on IID validation is essential for honest reporting (Si et al., 2022).
• Concept Quality and Reasoning Shortcuts in Neuro-Symbolic Systems: High predictive accuracy does not guarantee the learned intermediate concepts are semantically aligned; reasoning shortcut counts (RS-count) provide a formal lower bound on the number of "semantic-free" solutions available (Bortolotti et al., 2024).

5. Framework Implementations and Extensibility

ShortcutsBench variants are distinguished by their emphasis on large-scale, real-world data, modularity, and extensibility:

• Open datasets and code bases for Apple Shortcuts mining (Shen et al., 2024), GUI task and shortcut repositories (Zhao et al., 8 Sep 2025), concept-reasoning benchmark generators (Bortolotti et al., 2024), and LLM shortcut evaluation suites (Yuan et al., 2024).
• Support for a spectrum of model classes: from black-box deep nets (BERT, LSTM, LLaMA, Gemini, GPT) to logic-constrained or concept-bottleneck neuro-symbolic models.
• Easy adaptation to new domains: Via YAML-configurable data/task generators and pluggable symbolic knowledge files (rsbench), or by mining arbitrary app automation repositories.
• Integration with standard evaluation toolchains and protocols, enabling comparative analysis over architectures, prompting regimes, and learning strategies.

6. Implications and Prospects

The ShortcutsBench paradigm has catalyzed a systematic approach to diagnosing, quantifying, and ultimately mitigating shortcut learning in AI. It has exposed longstanding weaknesses in:

• Long-horizon planning and context tracking for API agents and tool-augmented LLMs.
• Semantic and compositional generalization in vision-language and neuro-symbolic systems.
• Feature attribution faithfulness as a prerequisite for scientific or safety-critical interpretability.
• Model selection and evaluation: Demonstrating that many prior protocols (e.g., VQA-CP v2’s single-shortcut split) overestimate OOD robustness.

Ongoing challenges include improving agents’ capacity for dynamic information-gap detection, robust shortcut generation, and semantically grounded explanation. Advancements may be realized by integrating planning modules, hierarchical decomposition, causal/contrastive data augmentation, and formal RS-count–driven specification diagnostics.

ShortcutsBench is thus the contemporary reference suite for evaluating shortcut reliance and semantic robustness in intelligent systems, spanning APIs, GUIs, language, and perception-reasoning tasks (Shen et al., 2024, Zhao et al., 8 Sep 2025, Yuan et al., 2024, Bastings et al., 2021, Si et al., 2022, Bortolotti et al., 2024).