Mil-Bench-5K: Military LLM Evaluation

• Mil-Bench-5K is a military evaluation dataset comprising 5,000 SME-verified Q/A pairs from doctrinal, logistics, and command publications.
• It employs a rigorous three-stage curation process involving AI summarization, machine classification, and expert review to ensure data quality.
• Evaluation protocols use precise statistical metrics to compare models, with EdgeRunner 20B showing significant wins in medium to high reasoning tasks.

Mil-Bench-5K is a standardized evaluation dataset and benchmark designed to assess performance of LLMs on general military knowledge tasks. Developed and described in the context of EdgeRunner 20B, a military-optimized LLM, Mil-Bench-5K encompasses 5,000 question–answer (Q/A) pairs curated from doctrinal publications across diverse military domains. The benchmark provides a rigorous, statistically-significant measure for comparing LLM effectiveness on military-specific information retrieval and reasoning, with particular emphasis on settings relevant to edge-deployed, air-gapped environments (Fitzgerald et al., 30 Oct 2025).

1. Dataset Properties and Curation

Mil-Bench-5K is designated as a "Silver" data resource, characterized by subject matter expert (SME) verification atop AI-assisted Q/A generation. The 5,000 Q/A pairs are drawn from an equal mix of four domains: U.S. Army publications, Joint Staff doctrine, Department of Defense (DoD) directives, and Logistics documentation (≈1,250 pairs per domain).

The curation pipeline consists of:

• Stage 1: Source documents are segmented and summarized by an LLM.
• Stage 2: Diverse Q/A pairs are generated from each chunk.
• Stage 3: Machine classification (PASS/FIX/FAIL) prioritizes quality; "FIX" items are rewritten, "FAIL" cases are discarded.
• SME Verification: A panel of three U.S. Army officers (combined 45+ years service, 20+ in special missions) reviews all pairs for doctrinal correctness and realism.
• De-duplication: Both exact-match and fuzzy-search procedures are used to exclude overlap with the fine-tuning corpus, preserving test integrity.

Covered topics include doctrine definitions, planning processes, sustainment and logistics, command and control, equipment characteristics, and force deployment procedures. Illustrative sample prompts are:

• "According to Army Doctrine Reference Publication 4-0, what is the definition of ‘sustainment’ and its three principal elements?"
• "List the five steps of the Joint Operations Planning Process (JOPP) and briefly describe the purpose of each."
• "What is the primary function of a Forward Arming and Refueling Point (FARP) in maneuver operations?"

2. Evaluation Protocol and Metrics

Evaluation employs the Inspect 0.3.130 framework (UK AI Security Institute). Model outputs are generated in a single pass with no supplementary system or developer prompt. Scoring is performed with the model_graded_qa method, adjudicated by a W8A8-quantized Atla Selene 1 model (Llama 3.3 70B-based), awarding full, partial, or zero credit (Correct = 1.0, Partially correct = 0.5, Incorrect = 0.0).

The primary outcome metric is accuracy:

$$\text{Accuracy} = \frac{ \sum_{i=1}^N \mathbf{1}[\text{grade}_i=\text{C}] + \frac{1}{2} \sum_{i=1}^N \mathbf{1}[\text{grade}_i=\text{P}] }{N}$$

The error rate is $1 - \text{Accuracy}$. Precision is further quantified by the standard error for a proportion $p$:

$SE(p) = \sqrt{ \frac{p(1-p)}{N} }$

and the 95% confidence interval is $p \pm 1.96 \cdot SE(p)$. Significance is evaluated using a two-sided test (paired bootstrap or permutation) under the null hypothesis of equal mean score, with threshold $\alpha = 0.05$.

3. Model Comparisons and Performance Outcomes

Relative performance results for Mil-Bench-5K are summarized in the following table (reported as percentage error relative to GPT-5 at medium-reasoning):

Model	Reasoning	Relative Error (%)	SE (%)	p-value	Outcome
GPT-5	low	4.14	±1.78	—	baseline
EdgeRunner 20B	low	9.83	±1.87	0.002	loss
GPT-5	med	0.00	±1.73	—	baseline
EdgeRunner 20B	med	–5.41	±1.78	0.003	win
GPT-5	high	–1.25	±1.70	—	baseline
EdgeRunner 20B	high	–6.29	±1.77	0.005	win

GPT-5 outperforms EdgeRunner 20B in the low-reasoning setting, but EdgeRunner 20B achieves statistically significant wins in the medium- and high-reasoning settings. The pre-fine-tuned gpt-oss-20b model was not separately evaluated on Mil-Bench-5K in this study.

4. Error Analysis and Failure Modes

Errors from both GPT-5 and EdgeRunner 20B are contextually clustered. In low-reasoning scenarios, models often omit multi-step doctrinal details (e.g., conflating logistics subfunctions), and errors frequently surface on items requiring synthesis across multiple paragraphs of doctrine. Partial-credit answers—those marked "P"—typically involve correct terminology but incomplete enumerations or incorrect sequencing.

These systematic error types indicate that surface-level knowledge injection suffices for single-step factual recall, while multi-step retrieval or synthesis tasks expose current LLM limitations on nuanced doctrine.

5. Hyperparameter Influence and Training Regimes

Several hyperparameter choices and training protocols were empirically optimized for Mil-Bench-5K performance:

• Reasoning Effort: Medium chain-of-thought prompts maximize the balance between accuracy and throughput; high reasoning occasionally degrades performance on recall items due to over-elaboration.
• Learning Rate: $1\times10^{-6}$ produced stable gains without overfitting.
• Batch Size: Large global batches ($1,\!024$–$1,\!536$ examples) enhance in-domain knowledge injection.
• Chat Template: The "alpaca" chat template better preserves reasoning behavior than the default gpt-oss template.
• Synthetic Reasoning Traces: Incorporation of such traces into training data diminished factual recall results, suggesting that simple Q/A construction is optimal for doctrine memorization.

A plausible implication is that factual military knowledge is best encoded in straightforward Q/A formats rather than via elaborate reasoning scaffolds for the specific use case of Mil-Bench-5K.

6. Implications for Edge-Hosted LLM Deployment

The design and evaluation of Mil-Bench-5K address requirements unique to military and sensitive environments:

• Security and Data Sovereignty: Edge-hosted models eliminate the need for external API calls, reducing exposure risk.
• Cost Predictability: Fixed deployment obviates per-token cloud costs, supporting agentic or persistent monitoring applications with stable budgeting.
• Redundancy and Resilience: Air-gapped system capability is preserved, ensuring continued operation in communications-contested or disconnected environments.
• Performance Parity: Domain-specific fine-tuning enables smaller models such as EdgeRunner 20B (20B parameters) to achieve parity or outperform larger models like GPT-5 on Mil-Bench-5K in medium/high-reasoning settings, demonstrating that optimization of training protocols and datasets can decrease resource requirements for specialized uses.

Collectively, Mil-Bench-5K establishes an empirical standard for evaluating LLMs on military knowledge and guides development of secure, domain-optimized LLMs for deployment in edge and mission-critical contexts (Fitzgerald et al., 30 Oct 2025).