CAKE: Cloud Architecture Knowledge Evaluation of Large Language Models

Abstract: In today's software architecture, LLMs serve as software architecture co-pilots. However, no benchmark currently exists to evaluate LLMs' actual understanding of cloud-native software architecture. For this reason we present a benchmark called CAKE, which consists of 188 expert-validated questions covering four cognitive levels of Bloom's revised taxonomy -- recall, analyze, design, and implement -- and five cloud-native topics. Evaluation is conducted on 22 model configurations (0.5B--70B parameters) across four LLM families, using three-run majority voting for multiple-choice questions (MCQs) and LLM-as-a-judge scoring for free-responses (FR). Based on this evaluation, four notable findings were identified. First, MCQ accuracy plateaus above 3B parameters, with the best model reaching 99.2\%. Second, free-response scores scale steadily across all cognitive levels. Third, the two formats capture different facets of knowledge, as the MCQ accuracy approaches a ceiling while free-responses continue to differentiate models. Finally, reasoning augmentation (+think) improves free-response quality, while tool augmentation (+tool) degrades performance for small models. These results suggest that the evaluation format fundamentally shapes how we measure architectural knowledge in LLMs.

• The paper introduces CAKE, a benchmark evaluating cloud-native architectural knowledge in LLMs through structured cognitive levels like recall, analyze, design, and implement.
• It employs dual-format evaluation using MCQs and free-response questions to capture both factual recall and generative reasoning across 22 LLM configurations.
• The study reveals that while MCQ accuracy saturates above 3B parameters, free-response scores and structured reasoning expose nuanced differences in model performance.

CAKE: A Rigorous Benchmark for Cloud-Native Software Architecture Understanding in LLMs

Introduction

The CAKE benchmark addresses a significant evaluation gap in the field of LLMs as applied to software engineering: robust assessment of architectural knowledge within the cloud-native domain. By situating its evaluation across distinct cognitive levels derived from Bloom’s revised taxonomy, CAKE offers a structured framework for measuring not just factual recall but also analyze, design, and implement capabilities. The benchmark enables comparative analysis across 22 LLM configurations, spanning leading open and proprietary models and incorporating both base and reasoning/tool-augmented variants. The detailed annotation protocol and format-specific evaluation pipeline provide a high-confidence substrate for assessing both the breadth and depth of architectural competence.

CAKE’s contributions center on methodical construction and validation of the benchmark, empirical findings from rigorous large-scale evaluation, and transparent public release of the dataset and evaluation pipeline.

Benchmark Design and Validation

CAKE is rooted in expert-driven concept selection and alignment with industrial cloud architecture standards (e.g., AWS Well-Architected Framework, Kubernetes Enhancement Proposals). The question set is stratified along four cognitive levels—recall, analyze, design, and implement—and five core knowledge areas: architectural patterns, quality attributes, decomposition strategies, cloud deployment, and technical debt. The final version comprises 130 MCQs and 58 free-response (FR) items after expert-based filtering and formatting correction.

Figure 1: Question distribution over cloud-native topics and Bloom-levels, demonstrating comprehensive coverage and density.

Expert annotation yields high average scores for clarity and correctness (means 4.72 and 4.63, respectively), with negligible impact of flagged ambiguity/typo issues on model accuracy. The CAKE-Core subset further tightens quality assurance by excluding items with any issues below rigorously set thresholds.

Figure 2: Expert ratings across core quality axes demonstrate narrow variance and high reliability, supporting the benchmark’s robustness.

Evaluation Protocol

MCQs are resolved through majority voting over three randomized runs per item—reducing position bias and quantifying model “conviction” in answer selection. This conviction correlates strongly with response reliability: unanimous model answers achieve markedly higher accuracy than split-majority cases, providing a valuable in-situ confidence metric.

Free-response grading leverages a single deterministic LLM-as-judge (DeepSeek-R1:32B), following rubric-based evaluation on a 0–5 scale. This methodology ensures reproducibility and fairness, although it acknowledges limitations from the judge model’s own biases and LLM alignment phenomena.

Empirical Results

MCQ Performance Characteristics

MCQ accuracy increases monotonically with model size but saturates rapidly above 3B parameters, with the best models achieving 99.2% accuracy. Notably, model scaling effects are almost extinguished above this threshold, and augmentation strategies such as chain-of-thought (+think) or tool use (+tool) yield at best modest improvements and at worst pronounced degradation for inadequately sized models.

Figure 3: Benchmark and CAKE-Core MCQ accuracy are tightly aligned, indicating consistency and validity of quality filtering.

Free-Response Scoring

Free-response results are more discriminative. Unlike MCQ accuracy, FR scores exhibit a roughly linear scaling with parameter count and continue to differentiate performance across the full spectrum of model capacities, with top-performing local models (Mistral 14B) approaching premium proprietary models (GPT-5-Mini) in capability.

Figure 4: Free-response judge scores maintain differentiation across model sizes, even beyond MCQ saturation points.

Mistral models dominate among open-weight configurations, indicating that model architecture and training data, not just parameter count, are central to the acquisition of procedural and generative architectural knowledge.

Augmentation Effects

Structured reasoning (+think) consistently lifts FR scores, especially below the MCQ ceiling, but its utility diminishes with size, and for MCQ can destabilize answer selection through over-elaboration. Tool augmentation (+tool) degrades both MCQ and FR results on models with insufficient parameter count, leading to tool misuse or degenerate reasoning behavior. The practical implication is a clear minimum viable model size (approximately 8B parameters) for effective agentic augmentation in this domain.

Figure 5: +think and +tool effects are size-dependent, with +think aiding FR most for small models and +tool generally detrimental below 8B.

Fine-Grained Trends

Topic-level breakdown exposes early limitations in smaller models. Sub-1B LLMs exhibit sharp deficits in pattern recognition and deployment tasks, with near-random performance on complex subjects—strengths and weaknesses that attenuate with model scale. Unanimity (conviction) in MCQ response also emerges as a strong, model-scale-dependent confidence signal.

Figure 6: Topic-level scores illustrate systematic deficiencies in small models that converge only at higher parameter counts.

Expert–model alignment analysis shows that human-rated question difficulty does not correlate with model accuracy, reinforcing the need for empirically grounded benchmarks rather than expert intuition alone.

Figure 7: No statistically significant association exists between expert difficulty ratings or annotation flags and model scoring outcomes.

Theoretical and Practical Implications

The study demonstrates that MCQ-only benchmarks are insufficient proxies for architectural competence in LLMs, particularly in procedural, generative, or high-level reasoning tasks—it is free-response evaluation that continues to stratify models as scaling laws flatten for MCQ. This fundamentally influences how LLMs should be validated and selected for architectural co-pilot applications.

The empirical separation between multiple-choice and free-response results suggests that commonly used LLM leaderboards and general knowledge benchmarks may systematically overestimate the readiness of mid-tier models for architectural co-design assistance in practice. The “conviction” metric offers practitioners an actionable rule for confidence-based filtering at the inference level.

From the developer’s perspective, mid-range open-weight models (Mistral 14B) now approach closed-source, high-cost APIs in performance for demanding generative tasks, creating viable alternatives for organizations under resource or privacy constraints. For educational and training scenarios, CAKE’s cognitive stratification enables precise mapping between LLM assistance level and required human oversight.

Limitations and Future Directions

The work openly acknowledges several limitations, most notably the exclusion of a minor subset of implementation-level MCQs due to formatting defects, use of a single judge model for FR scoring, and minor annotation consistency issues. Iterative refinement is planned to address these.

Further research directions include expansion into classical patterns, improved distractor construction, more diverse and multilingual evaluation settings, and meta-evaluation of judge model robustness.

Conclusion

CAKE inaugurates meaningful evaluation of LLMs for architectural knowledge in cloud-native software, substantiating the need for cognitive-level and format-diverse testing. Critical findings are as follows: MCQ accuracy saturates above 3B parameters; free-response evaluation continues to distinguish model capabilities across all sizes; +think reasoning augmentation is usually beneficial but tool-augmentation requires high capacity; and practitioner-facing metrics such as conviction offer valuable reliability signals.

CAKE establishes a new baseline for the rigorous study of LLMs as software architecture assistants and should serve as an indispensable tool for future research in trustworthy, knowledge-driven AI systems for software engineering.