- The paper introduces MÖVE, a benchmark evaluating 39 language models on German public-sector summarization, question answering, topic extraction, hallucination, sustainability, transparency, and political values.
- The paper finds that no single model leads across all tasks, model size is only a moderate performance predictor, and energy use varies 63-fold—from 0.647 Wh to 40.6 Wh per query.
- The paper validates the benchmark through confidence intervals, judge-reliability tests, dataset and prompt sensitivity analyses, and energy comparisons, showing that rankings should often be interpreted as performance tiers rather than exact orderings.
MÖVE (Modelle für die Öffentliche Verwaltung Evaluieren) is a benchmark developed at Bundesdruckerei GmbH for evaluating LLMs in the context of German public administration. It evaluates 39 models across two complementary dimensions—performance and governance—using ten German-language datasets, several of which were constructed specifically for the benchmark. Beyond reporting model rankings, the authors subject the benchmark itself to a methodological self-evaluation covering statistical precision, LLM judge reliability, dataset impact, prompt sensitivity, and energy-estimate validity. Results are maintained as a living leaderboard (2606.13111).
Motivation and positioning
The paper is motivated by the observation that model selection in the German public sector is largely ad hoc, driven by media coverage or country-of-origin heuristics, while existing benchmarks are English-centric, US-centric in content, and focused exclusively on task performance. Multilingual benchmarks such as Global MMLU address language but not domain: translated US civics questions do not reflect German administrative work. Government-specific benchmarks exist for other national contexts (PubHealthBench for UK public health, CitizenQuery-UK, MSGABench for Chinese government affairs), but none combine a non-English language, public-administration domain data, and governance criteria. A cited meta-analysis concludes that no existing benchmark meets public-sector requirements. The authors argue that for public institutions, governance is not an add-on but a core requirement, citing Article 53 of the EU AI Act, sustainability considerations in procurement, hallucination risk on legal documents, and constitutional value alignment.
Framework design
MÖVE evaluates models along seven criteria. Performance criteria cover summarization, question answering (QA), and topic extraction. Governance criteria cover hallucination, sustainability, transparency, and politics/values. The design was validated through stakeholder sessions with ministries and more than 20 practitioner events; four target groups (decision-makers, domain experts, IT/security stakeholders, civil society) shaped the criteria and reporting logic.
The evaluation setup is fully automated for performance criteria: task definition, dataset, prompt, and metrics, with aggregation at setup and task levels. Governance evaluation is heterogeneous: hallucination reuses QA faithfulness scores, sustainability tracks resource usage during task execution, transparency is assessed through document analysis of model cards against EU AI Act requirements, and politics/values uses dedicated datasets.
Datasets and models
Ten datasets are used, combining four existing resources (Eur-Lex-Sum, Swiss Leading Decision Summarization, German-QuAD, Wahl-O-Mat) with six internally constructed goldstandard (manually curated) and silverstandard (curated from public sources) datasets, including KIKC Summary/QA/Topics built from ministry-representative documents and German Ministry Publications scraped from a federal ministry portal (source deliberately omitted to reduce contamination risk). Sample sizes for existing datasets were chosen via stability analysis of metric convergence.
The 39 evaluated models span open-weight systems (Llama, Mistral, Gemma, Qwen, DeepSeek, Phi, Apertus, EuroLLM, Teuken, SmolLM, SauerkrautLM fine-tunes) plus GPT-4o and GPT-4o Mini via API. Open-weight models run on up to four NVIDIA A100s with vLLM or ollama; safety filters are disabled where configurable. Documents exceeding context windows are truncated from the end with reserved output capacity (512–2,048 tokens; 8,192 for reasoning models); no chunking or map–reduce strategies are used, which implicitly penalizes short-context models.
Metrics
Metrics fall into three categories: classical (ROUGE, BLEU, exact match, reported but excluded from composite scores), embedding-based (BERTScore, SemScore using jina-embeddings-v2-base-de, Semantic Topic Match), and LLM-as-a-judge (Factual Correctness, Faithfulness, Noise Sensitivity via RAGAS with GPT-4o Mini as judge; Topic Adherence). Task composites average these metrics equally. Two task-specific additions are notable: a German Summary Proportion metric (FastText-based language identification calibrated on 200 annotated summaries) penalizes models producing non-German output, and Topic Adherence includes word-count and topic-count penalty factors to down-weight verbose outputs.
Summarization: Mistral Large 3 leads (0.780), followed by GPT-4o Mini and Phi-4 (both 0.770)—two of the top three are small models (≤20B parameters). Small models span nearly the entire performance range (mean 0.646 vs. 0.757 for large models), with parameter count correlating only moderately (Spearman ρ = 0.45). Language adherence varies sharply: Smollm 1.7B produces only 39.6% German summaries despite German prompts, while four models exceed 99%. Dataset-level rank correlations range from ρ = 0.46 (Swiss Leading Decisions vs. Ministry Publications) to 0.83, indicating that aggregated rankings obscure dataset-specific strengths.
Question answering: Mistral Large 2.1 leads (0.704), with large models dominating five of the top ten slots; the small-model summarization leaders drop to ranks 10–11. FAQ-LAW proves hardest (mean 0.594), with long legal contexts degrading factual correctness—for example, Qwen3 4B's Factual Correctness falls from 0.557 on KIKC QA to 0.291 on FAQ-LAW. The paper notes that apparent context-size effects are confounded with model generation: ≤8k-context models underperform even on short-context datasets.
Topic extraction: Gemma2 9B (a 9B-parameter model) leads (0.671), with Google models taking three of the top five positions. Output length ratio correlates strongly negatively with score (ρ = −0.88): bottom performers emit outputs 20–60× longer than expected, while Gemma models average a length ratio of 1.26 versus 10.67 for others. Verbosity is task-specific rather than general—Mistral Large 2.1 is verbose on topic extraction but concise on QA—suggesting format-compliance failures rather than global instruction-following deficits.
Across all three tasks, top performers differ systematically, supporting the paper's central claim that no single model dominates and size alone is a poor predictor of quality.
Governance results
Hallucination: Measured via aggregated RAGAS Faithfulness across QA datasets, scores range from 0.20 (Smollm2 1.7B) to 0.83 (Mixtral 8x22B). Large models cluster tightly at high faithfulness (mean 0.81); small models average 0.61 with extreme variance. FAQ-LAW yields substantially lower faithfulness (0.54) than the other QA datasets (both 0.76). The authors explicitly caution that strict literal matching by the judge makes this a coarse indicator rather than a precise hallucination measure.
Sustainability: Using EcoLogits estimates, per-query energy varies by a factor of 63 (0.647 Wh for Teuken 7B to 40.6 Wh for DeepSeek R1). Summarization is the most energy-intensive task. Pareto analysis shows steep trade-offs: Mistral Large 3 requires 57.8 Wh versus GPT-4o Mini's 2.12 Wh for a performance gain of only 0.010. Reasoning models incur substantial overhead—Qwen3 4B consumes 14.3× the energy of similarly sized Gemma2 2B—with thinking tokens comprising up to 88.9% of output. Energy estimates for GPT-4o and GPT-4o Mini rest entirely on EcoLogits' default parameter assumptions and should be interpreted cautiously.
Transparency: A 21-question Transparency Matrix derived from the GPAI Code of Practice Model Documentation Form was scored by two human raters plus an automated Claude Sonnet 4.6 agent (27.4% divergence rate, resolved manually). Scores range from 18/42 to 38/42 (Apertus 70B, designed for AI Act compliance). Three domains are almost universally opaque: Compute & Energy (mean 11.5%, with 84.6% of models disclosing nothing about energy consumption), Training & Data (48.4%), and Use & Deployment (54.3%, dragged down by 97.4% of models failing to describe suitable AI-system categories). Notably, GPAI Code of Practice signatories score only marginally higher than non-signatories overall (27.6 vs. 26.3), with their advantage concentrated in downstream deployment information rather than the upstream disclosures (training data, compute) the regulation targets—a finding with direct procurement implications: signature status should not be treated as a proxy for documentation depth.
Politics and values: On party-position classification over 4,788 Wahl-O-Mat positions from 64 parties across four federal elections, the best accuracy is only 0.671 (shared by DeepSeek R1 and GPT-4o Mini), indicating the task remains difficult. For constitutional values, eight Grundgesetz-derived values probed under five scenario framings show no model generating clearly negative stance; European models attain the highest median stance scores under the GottBERT metric, though the authors flag the regional samples as small and unbalanced and note the possibility that positive scores reflect surface-level positivity bias rather than genuine alignment.
Benchmark self-evaluation
This section distinguishes MÖVE methodologically. Key findings:
- Precision: Hierarchical bootstrap confidence intervals show summarization rankings are precise (median CI width 0.015; the top model stands alone statistically), whereas QA has a 16-model indistinguishable top tier and topic extraction a 7-model tier. Equal dataset weighting preserves gold-standard influence but inflates CIs; sample-size-proportional weighting would shrink top tiers dramatically (QA: 16→4) at the cost of diluting domain-specific signal.
- Judge reliability: Within-judge CI widths are small (0.009–0.021), but inter-judge agreement mostly falls below conventional thresholds (Krippendorff's α = 0.42–0.68), with between-judge variance 2–4× within-judge variance—systematic, not stochastic. Qualitative analysis shows judges differ in strictness (e.g., GPT-4o Mini demands explicit causal support for faithfulness). No self-scoring bias was detected. The authors note these agreement levels overlap with typical human inter-annotator agreement (α = 0.4–0.7) on comparable subjective tasks.
- Internal dataset impact: Removing internal datasets shifts ranks measurably (ρ = 0.914 summarization, 0.877 QA; 77% and 59% of models show statistically significant score changes respectively), validating their contribution, though contamination resistance is precautionary rather than empirically demonstrated.
- Prompt sensitivity: Across three prompt variants on Eur-Lex-Sum, more specific prompts raise absolute scores uniformly, but composite-score rank stability is moderate-to-strong (Kendall's τ = 0.683), with shifts concentrated among mid-range models.
- Energy validity: CodeCarbon measurements agree with EcoLogits within an order of magnitude (median ratio 0.9×), but EcoLogits ignores prefill cost and underestimates energy for input-dominated workloads (ratios up to 9.1× for topic extraction on ~19,000-word documents).
Limitations and open questions
The paper is candid about constraints. The hallucination criterion covers only context-grounded QA faithfulness, not summarization or abstention behavior, and inherits the judge's strictness bias. Inter-judge reliability below thresholds means most LLM-judged rankings should be read as tiers, not orderings. Internal gold/silver datasets are small (e.g., KIKC Summary has 40 documents), conflating domain effects with sampling noise. The contamination argument for private data is precautionary. Prompt-sensitivity analysis covers one dataset and one task. Regional value-alignment conclusions rest on unbalanced samples. Whether verbose low-scoring models fail on content or merely format cannot be determined from F1-based metrics alone, and whether a hypothetical high-performing-but-verbose QA model would be unfairly penalized remains unresolved since none exists in the sample.
Conclusion
MÖVE provides a domain-grounded, bilingual-aware, multi-criteria evaluation framework for a stakeholder community poorly served by existing benchmarks, and its self-evaluation offers a template for uncertainty-aware benchmark reporting. Its principal empirical contributions are the demonstration that task-specific evaluation is necessary (top models differ across all three tasks), that energy consumption varies far more than quality (63-fold), that Code of Practice signatures do not indicate documentation depth, and that LLM-judge disagreement is systematic and comparable in magnitude to human annotator disagreement. Open questions include how quantization affects scores, how to evaluate multi-turn and agentic administrative workflows, and whether the observed regional differences in value alignment persist with larger samples and matured methodology.