- The paper presents GLARE, a system that reframes complex global explanations as queryable databases, allowing interactive, natural language queries.
- It employs a fine-tuned LLM with SQL template parsing to achieve ≥95% query accuracy and robust performance against input perturbations.
- The architecture generalizes across datasets, offering scalable human-centered analysis for deep vision models with modular extensibility.
GLARE: Language-Based Querying for Human-Centered Global Explanations
Motivation and Problem Statement
The opacity of deep vision models remains a major barrier to their adoption in high-stakes domains, especially where granular understanding of the model's decision logic across diverse inputs and classes is required. Global explanation methods, which summarize the behavior of classifiers via logic rules or concept structures over entire datasets, contribute essential XAI capabilities. However, their complex and monolithic nature makes them impractical for human-driven exploration. The paper introduces GLARE, an LLM-mediated interface that reframes global explanations as queryable databases, accessed through natural language queries. This system enables users to investigate aggregate model behavior efficiently and interactively, bypassing the cognitive overload inherent in static explanation artifacts.
System Architecture and Methodology
GLARE leverages a local-to-global explanation framework that aggregates local Minimal Sufficient Explanations (MSXs) into a global Disjunctive Normal Form (DNF) structure. These logic-based explanations are ingested into a relational database, facilitating precise analytical queries. The interface employs a fine-tuned LLM (Gemma 2-9B), trained via QLoRA and SQL-fence loss masking, as a semantic parser that maps user queries to SQL templates specialized for explanation retrieval. The fence-masking strategy focuses learning solely on the SQL tokens, promoting generalizable query structure acquisition and mitigating overfitting to dataset-specific entity names. The system parses, validates, and executes SQL queries, returning structured statistics, supporting evidence images, and visualizations aligned with the user's information needs.
The template taxonomy comprises 24 query types spanning object-class relationships, sophisticated counterfactuals, and set-based analytical tasks. Template extensibility allows the system’s coverage to expand efficiently with new types, requiring only synthetic training data generation—no manual annotation.
Empirical Evaluation and Numerical Results
GLARE is evaluated using synthetic queries over an ADE20K-derived database and tested for robustness and generalization. Three primary axes are examined:
- In-Distribution Query Accuracy: Fine-tuned Gemma 2 models (2B–27B) achieve result-match rates ≥95% (95.2% for 9B, 95.4% for 2B) and near-perfect structural metrics (fence detection, SQL parsing, execution). Adding partial matches (Jaccard > 0.5) raises effective accuracy above 97%. Untrained base models and regex baselines fail to achieve comparable accuracy, confirming the necessity of task-specific fine-tuning.
- Robustness to Input Perturbations: GLARE demonstrates high resilience to spelling errors (+31pp advantage over regex), synonym substitution (zero degradation), verbose padding, word drop, and grammar variations. Consistency across paraphrased queries reaches 94.1%, with 100% adherence to domain sanity constraints.
- Out-of-Distribution Generalization: On unseen query phrasings and novel SQL constructs, GLARE maintains 99.3% execution rate but result-match accuracy drops to 35% (45.1% for phrasing, 19.5% for novel constructs). The system gracefully degrades to the closest supported template for unsupported syntax, ensuring syntactic correctness.
- Cross-Dataset Transfer: Without retraining, the model achieves up to 90.6% result-match accuracy on Pascal VOC (Gemma 2 27B), a dataset with a disjoint object vocabulary. This substantiates the learned SQL structure’s ability to generalize given a new entity name list, confirming architecture-agnostic, schema-centric transferability.
Model comparisons show that both Gemma and Qwen families (≥7B parameters) saturate accuracy; robustness scores average >80% across all perturbation types. For certain complex query types (e.g., combos, images_with_exact_count), accuracy is lower, highlighting compositional limitations related to multi-way joins and HAVING clauses.
Qualitative Analysis
GLARE's query pipeline is exemplified by the response to "In living_room, what objects appear with sculpture?". The interface maps the question to a SQL co-occurrence query, retrieves statistical relationships (e.g., wall: 80%, sofa: 60% co-occurrence), and presents visual evidence aligning objects with segmentation maps. This yields a multi-modal explanation combining fluent textual summaries, numerical statistics, and grounded visualizations.
Theoretical and Practical Implications
GLARE’s approach transforms the interaction paradigm in global XAI, supporting iterative, intent-driven explanation seeking. Anchoring query generation in formal SQL templates enforces correctness and enables compositional generality. The architecture ensures that model structure (SQL grammar and relational algebra) is decoupled from vocabulary, achieving zero-shot dataset transfer. The fine-tuning pipeline is extensible: supporting new analytical constructs only requires expanding the template taxonomy and regenerating synthetic training, facilitating scalable coverage of emerging explanation needs.
Practically, GLARE democratizes access to global explanations for human users, lowering the barrier for non-programmers and supporting dynamic hypothesis-driven exploration. Theoretically, it demonstrates that neuro-symbolic interfaces, when anchored in deterministic intermediate representations, can mediate between opaque models and transparent, actionable insight.
Future Directions
Unresolved challenges include limited compositional generalization for untrained SQL constructs and the need for further extensibility in template taxonomy. Future work will focus on expanding coverage to more analytical forms (e.g., nested queries, advanced aggregations), integrating richer visualizations, and investigating direct deployment in real-world XAI workflows across domains.
Conclusion
GLARE introduces a novel interface for querying global explanations of vision models via natural language, mediating with an LLM-constrained SQL parser over a structured explanation database. Empirical results validate high accuracy, robustness, and cross-domain generalization for fine-tuned models, setting a new standard for accessibility and compositional expressiveness in human-centered XAI. The architecture’s modularity and extensibility position it as a practical framework for evolving explanation needs in AI systems.
For further details and comprehensive data, refer to "GLARE: A Natural Language Interface for Querying Global Explanations" (2606.19735).