SAKE: Software Architectural Knowledge Evaluation Benchmark for Large Language Models

Abstract: LLMs are increasingly used as assistants across the software development lifecycle, yet their ability to reason about software architecture remains largely unmeasured. Architectural decision-making depends on quality attribute trade-offs, design patterns, and system-level constraints, none of which are exercised by benchmarks that target syntactic or algorithmic tasks. We introduce SAKE (Software Architectural Knowledge Evaluation), a standardized and reproducible benchmark for assessing software architectural knowledge in LLMs. SAKE comprises 2154 expert-curated multiple-choice questions, each with four options, stratified across eight architectural categories and four context-length levels. We evaluate 11 proprietary and open-weight models in zero-shot and five-shot settings. Overall accuracy is high, but performance varies markedly across categories, revealing competency gaps in areas central to professional practice. SAKE, its evaluation scripts, and all results are released as open source to give the community a baseline for tracking architectural reasoning in LLMs.

• The paper introduces SAKE, a novel benchmark using 2154 peer-reviewed multiple-choice questions to rigorously evaluate software architectural knowledge in LLMs.
• It employs a multi-phase methodology that categorizes architectural knowledge into eight domains and stratifies question complexity by context length.
• Results reveal high overall accuracy but notable weaknesses in reasoning-intensive tasks and architectural trade-offs, guiding future LLM deployments and research.

SAKE: A Benchmark for Evaluating Software Architectural Knowledge in LLMs

The paper "SAKE: Software Architectural Knowledge Evaluation Benchmark for LLMs" (2606.29520) introduces SAKE, a rigorously validated, standardized, and open-source benchmark aimed at probing the depth and breadth of software architectural knowledge in LLMs. SAKE addresses the field's acute lack of targeted diagnostic tools for evaluating architectural reasoning, as compared to the plethora of benchmarks that exist for code synthesis, syntactic reasoning, and general language understanding.

Motivation and Domain Context

Software architecture is a core discipline requiring mastery of quality attribute trade-offs, design patterns, architectural tactics, and system-level constraints. Unlike low-level design, architectural decision-making is characterized by high-level abstractions, multi-dimensional trade-offs, and a knowledge base that is both broad and fragmented. While LLMs are increasingly adopted as assistants throughout the software lifecycle—including for architectural design—their capabilities in this domain had not been systematically evaluated due to the absence of domain-specific benchmarks. Prior studies highlighted not only the lack of baselines but also the confounding conflation of syntactic/code-level benchmarks with higher-order architectural reasoning tasks.

Benchmark Construction and Methodology

SAKE is meticulously constructed through a multi-phase process:

1. Domain-Grounded Taxonomy: The benchmark's eight knowledge categories derive from canonical literature—"Software Architecture in Practice" (Bass et al.) and the "Gang of Four Design Patterns"—spanning system-level reasoning (e.g., quality attributes, architectural styles) and design pattern knowledge.
2. Curated Question Pool: 2154 multiple-choice questions, each with four options, are expertly authored and subjected to a dual blind peer review for content validity and inter-rater agreement. Questions span eight categories and four context-length bins.
3. Question Complexity Stratification: Beyond category, each question is binned by context length (quartiles on word count), enabling analysis of model performance as the requirement for contextual integration increases.

   Figure 1: Overview of the SAKE benchmark methodology, from taxonomy definition through dual-review question curation to LLM evaluation across multiple context-length strata.

Experimental Protocol

Eleven LLMs are evaluated, spanning proprietary (e.g., GPT-5.4, Claude Opus/Sonnet/Haiku, Gemini 3, Grok, Deepseek) and open-weight (Qwen3 235B, Mistral Small 4) models. Both zero-shot and five-shot prompting settings are considered; in the latter, category-specific in-context examples are prepended to the prompt. Precision is ensured by constraining outputs and deploying robust answer extraction pipelines via regular expressions.

Performance metrics are reported globally (overall accuracy), per-category, and per-context bin, providing multidimensional diagnostic resolution.

Key Findings

Aggregate Performance

• All models achieve high overall accuracy (minimum 89.31%, maximum 94.23%), with top performance by Claude Opus 4.6 (five-shot: 94.23%) and GPT 5.4 (zero-shot: 93.95%).
• The performance differential among frontier models is marginal; cost and open-source availability, not accuracy, become primary discriminators for deployment.

Category-Level Competence

• Quality Attributes (QA) and Creational Patterns (CP) are the most tractable categories, with mean accuracies exceeding 97%.
• Architectural Solutions (ASO) consistently emerges as the most challenging category (mean accuracy 88–89%), indicating that trade-off-centric, reasoning-heavy questions induce most errors.
• Questions on Quantum Computing (QC) also exhibit lower accuracy, especially in zero-shot settings, reflecting the novelty and technical demands of the category.
• Inter-model variance is highest in difficult categories, with lightweight models (e.g., GPT 5.4 Nano, Deepseek) showing pronounced performance degradation relative to state-of-the-art models.

Context Length Effects

• On general recall-oriented categories, longer context leads to increased accuracy.
• In contrast, for complex reasoning categories (notably ASO), longer context correlates with reduced performance. The likely etiology is the greater density of semantically plausible distractors, emphasizing the challenge of multi-faceted architectural trade-off reasoning in LLM architectures.
• The effect of five-shot prompting is model-dependent: large models make effective use of extended context, while performance in smaller/efficiency-optimized models often degrades with increased prompt length.

Implications

• High aggregate accuracy masks substantial domain-specific competency gaps; practitioners must consult per-category results to assess LLM trustworthiness for specific architectural tasks.
• Adding prompt context is not uniformly beneficial—benefits are seen for recall, but harms accrue for reasoning-intensive tasks. This nuance has significant real-world consequences: longer, more detailed prompts do not guarantee improved architectural reasoning.
• Open-source, cost-efficient models (e.g., Qwen 3 235B) offer practical accuracy at a dramatically reduced cost, and are thus candidates for privacy- or resource-constrained settings.

Limitations and Threats to Validity

• The multiple-choice format, while standardized and automatable, abstracts away from the full complexity and open-endedness of real-world architectural analysis. Real decision-making often requires synthesis, trade-off articulation, and defense of rationale not captured by fixed-choice questions.
• The validation pool, while dual-reviewed, is limited to the author team; further community review is encouraged.
• LLM sensitivity to prompt formulation remains an open issue, though uniform templates partially mitigate this.

Future Directions

• The benchmark provides a dynamic, reproducible foundation for tracking emergent model capabilities and targeting fine-grained weaknesses within the architectural knowledge domain.
• Unexplored LLM architectures and prompt engineering strategies (e.g., retrieval-augmented generation, chain-of-thought reasoning) may target the reasoning deficits illuminated by SAKE.
• Expansion of the benchmark to richer modalities (e.g., diagrammatic input, open-ended rationale generation) is feasible and desired as LLM capabilities mature.
• Community adoption and iterative refinement are encouraged via the open-source release, supporting comparative studies as model architectures evolve.

Conclusion

SAKE represents a principled, open, and comprehensive framework for evaluating LLM competency in software architectural knowledge—a domain previously unaddressed by existing benchmarks. The results demonstrate that while aggregate knowledge recall is near-saturated among leading models, significant domain-specific reasoning deficits persist, particularly in architectural trade-off evaluation. The benchmark's granular structure and transparency position it as a community standard for both tracking and advancing LLM architectural reasoning, supporting both responsible deployment and targeted research in software architecture and AI.

References to all major results, model families, and prior literature can be found in the original paper's bibliography (2606.29520).