CAME-Bench: Civil Aviation Maintenance Benchmark

• CAME-Bench is a specialized evaluation suite designed to assess embedding models and LLMs across maintenance tasks such as fault diagnosis, bilingual terminology alignment, and procedural reasoning.
• It organizes evaluations into seven distinct task families, covering challenges from semantic retrieval to structured fault-tree reasoning based on realistic aviation maintenance workflows.
• Empirical results indicate that while embedding models excel in semantic similarity, LLMs struggle with complex reasoning and factual precision in safety-critical maintenance contexts.

Searching arXiv for the benchmark to ground the article in the primary source and related naming context. arxiv_search({"query":"CAME-Bench civil aviation maintenance benchmark CAMB", "max_results": 5, "sort_by": "submittedDate"}) CAME-Bench, expanded as the Civil Aviation Maintenance Benchmark, is a domain-specific evaluation suite for embedding models and LLMs in civil aviation maintenance. As instantiated by the CAMB paper, it is described as a purpose-built, industrial-grade benchmark designed to measure both embedding models and general-purpose LLMs on a knowledge-intensive and safety-critical domain characterized by maintenance procedures, troubleshooting, terminology alignment, procedural lookup, and multi-step causal reasoning (Zhang et al., 28 Aug 2025). The benchmark addresses a gap in evaluation practice by providing a standardized tool for assessing model performance in civil aviation maintenance, while also supporting analysis of deficiencies in domain knowledge, retrieval behavior, and complex reasoning (Zhang et al., 28 Aug 2025).

1. Definition and scope

CAME-Bench is organized around seven core task families, each instantiated both as an embedding evaluation and as an LLM-QA evaluation (Zhang et al., 28 Aug 2025). The task taxonomy comprises Alignment Bilingual Terminology, Aircraft Fault System Localization, Aircraft Text Chapter Localization, Civil Aviation Maintenance Multiple-Choice, Fault Description ↔ FIM Manual Matching, Civil Aviation Maintenance Open QA, and Fault-Tree-Structured QA (Zhang et al., 28 Aug 2025).

The benchmark design reflects typical maintenance workflows. These include terminology alignment, fault identification, procedural lookup, decision-making, and multi-hop causal reasoning (Zhang et al., 28 Aug 2025). Its topical coverage spans aerodynamics, electromechanical control, materials science, communications, and avionics subsystems (Zhang et al., 28 Aug 2025). This suggests that the benchmark is intended not merely as a generic question-answering suite, but as a workflow-aligned evaluation environment in which retrieval quality, knowledge localization, and structured reasoning are all assessed under domain constraints.

The CAMB paper presents CAME-Bench as serving a dual purpose: it measures LLM capabilities within civil aviation maintenance and identifies specific gaps in domain knowledge and complex reasoning, thereby establishing a basis for targeted improvement such as domain-specific fine-tuning, RAG optimization, and specialized prompt engineering (Zhang et al., 28 Aug 2025).

2. Task taxonomy and benchmark structure

The seven task families define the operational structure of CAME-Bench (Zhang et al., 28 Aug 2025). Each family pairs an embedding-oriented formulation with an LLM-oriented formulation, so that semantic retrieval quality and end-task reasoning quality can be compared within the same domain.

Task family	Embedding evaluation	LLM evaluation
Alignment Bilingual Terminology	Bitext Mining	Translation
Aircraft Fault System Localization	Vector Recall/Rerank	Classification
Aircraft Text Chapter Localization	Clustering	Classification
Civil Aviation Maintenance Multiple-Choice	Rerank	Multiple-Choice QA
Fault Description ↔ FIM Manual Matching	Recall	Pair Classification
Civil Aviation Maintenance Open QA	Retrieval/Rerank	Open-ended QA
Fault-Tree-Structured QA	Retrieval/Rerank	Multi-step Reasoning QA

The task selection is explicitly tied to maintenance practice (Zhang et al., 28 Aug 2025). Alignment tasks evaluate bilingual terminology mapping. System localization and chapter localization test whether models can place a fault description or text segment into the appropriate maintenance subsystem or document structure. Multiple-choice tasks measure professional knowledge and procedural recall. FIM matching targets correspondence between observed fault descriptions and fault isolation manual entries. Open QA and fault-tree QA extend beyond retrieval into explanation and causal reasoning (Zhang et al., 28 Aug 2025).

The benchmark therefore spans several model functions that are often conflated in general-purpose evaluation: semantic matching, structured retrieval, categorical localization, answer generation, and multi-step reasoning. A plausible implication is that CAME-Bench enables error analysis at a finer granularity than single-score aviation benchmarks, because failure can be localized to retrieval, classification, bilingual alignment, or reasoning stages.

3. Data sources and dataset composition

CAME-Bench aggregates seven datasets derived from document sources that include aviation maintenance textbooks, ATA chapters, Boeing 737NG materials, FAA regulations, bilingual aligned corpora, FIM and TSM manuals, real fault cases, and exam Q&A and multiple-choice banks (Zhang et al., 28 Aug 2025). The benchmark is thus assembled from both instructional and operational materials.

The reported task-specific dataset sizes are as follows (Zhang et al., 28 Aug 2025):

Dataset or task slice	Size	Median length
Chinese-English aligned	1 336 samples	3
Categorize (fault → system)	100	15
Cluster (chapter localization)	613 chunks	286 tokens
Sentence pairing	5 984 pairs	20
Open QA	202 Q&A pairs	263
Multiple-choice	7 969 professional + 995 open questions	—
Fault-tree	50 structured cases	33 nodes

The question taxonomies include troubleshooting reasoning, procedural lookup, system diagnostics, fault-cause tracing, and bilingual term translation (Zhang et al., 28 Aug 2025). Real fault cases are drawn from B737 and A320 historical logs, while the document basis includes fault isolation manuals and troubleshooting manuals (Zhang et al., 28 Aug 2025). This composition is significant because it combines canonical reference knowledge with operationally grounded cases. The benchmark is therefore positioned between educational assessment and maintenance-support evaluation.

The paper also provides representative task instances. Examples include translating “keel beam” to “龙骨梁,” identifying the installation location of a DC power control component, localizing a kerosene odor event to the fuel system, matching “autothrottle arm problem” to a description about the arm light on the MCP, explaining radar antenna-interface burn marks as a cause of no echo, and producing up to five next probable causes in a historical B737 radar fault sequence (Zhang et al., 28 Aug 2025). These examples show that the benchmark ranges from lexical precision to diagnosis-oriented reasoning.

4. Metrics and scoring methodology

CAME-Bench uses standard information-retrieval and classification metrics together with custom multi-tier scoring for open-ended reasoning (Zhang et al., 28 Aug 2025). The benchmark assumes a test set of size $N$ and defines the following metrics (Zhang et al., 28 Aug 2025):

• Accuracy:

$$\mathrm{Acc} = \frac{1}{N}\sum_{i=1}^N \mathbb{I}(\hat y_i = y_i)$$

• Precision, Recall, and $F_1$:

$$P=\frac{TP}{TP+FP},\quad R=\frac{TP}{TP+FN},\quad F_1=\frac{2PR}{P+R}$$

• V-measure for clustering, with homogeneity $h$ and completeness $c$:

$\mathrm{V} = 2\frac{h\,c}{h + c}$

• BLEU for translation, with a correct translation defined by $\mathrm{BLEU}>0.05$
• nDCG@k for retrieval and reranking:

$$\mathrm{DCG}_k = \sum_{i=1}^k \frac{2^{\mathrm{rel}_i}-1}{\log_2(i+1)},\quad \mathrm{nDCG}_k=\frac{\mathrm{DCG}_k}{\mathrm{IDCG}_k}$$

For open-ended QA and fault-tree reasoning, human or GPT-4o judges assign 0, 1, or 2 points corresponding to no match, partial match, and complete match against gold answers (Zhang et al., 28 Aug 2025). This scoring design is a distinctive feature of the benchmark. It acknowledges that maintenance reasoning often yields partially correct causal or procedural answers that are not well captured by exact match.

The use of BLEU for bilingual terminology and nDCG@k for retrieval-related tasks situates CAME-Bench within established IR and MT evaluation practice, while the 0/1/2 judgment scale adapts evaluation to safety-critical reasoning tasks where graded correctness is more appropriate (Zhang et al., 28 Aug 2025). At the same time, the paper notes that such human-judged scoring introduces evaluation noise and requires domain experts or LLM-as-judge pipelines (Zhang et al., 28 Aug 2025).

5. Empirical results

The benchmark evaluates eight embedding models and thirteen LLMs (Zhang et al., 28 Aug 2025). For embeddings, the evaluated models are Conan, gte-large-zh, m3e-large, BGE, gte-Qwen2 (1.5B and 7B), and Qwen3-Embed (4B and 8B) (Zhang et al., 28 Aug 2025). Mean task scores range from 43.4 for m3e-large to 66.3 for Qwen3-8B, and the paper reports that larger models above 1B parameters show positively correlated gains (Zhang et al., 28 Aug 2025).

For embeddings, BitextMining $F_1$ peaks at 81–82% for Qwen3, whereas classification for system localization remains low at 18–33%, which the paper interprets as reflecting semantic-versus-factual embedding gaps (Zhang et al., 28 Aug 2025). The reported retrieval-versus-reranker trade-off shows, for example, that Qwen3-8B retrieves a more diverse corpus with a decrease of 4% nDCG, while Reranker-Text gains 8% (Zhang et al., 28 Aug 2025). These observations indicate that semantic proximity alone is insufficient for accurate maintenance fact localization.

Thirteen LLMs are compared, including OpenAI-o3-mini, Claude-Opus-4, Deepseek, six Qwen variants, and Kimi-K2 (Zhang et al., 28 Aug 2025). Across six metrics, the benchmark reports the following peak scores (Zhang et al., 28 Aug 2025):

Metric	Peak score	Model
Translation	71.1	Kimi-K2
System Localization	84	Kimi-K2
Chapter Localization	94.9	Qwen3-235B-Instruct
FIM Match	86.9	Deepseek
Open QA	61.9	Qwen3-235B-Instruct
Fault-Tree Reasoning	58	Qwen3-235B-Instruct

For multiple-choice QA, non-thinking Qwen3-235B-Instruct achieves 68.9% overall, which is reported as 3.2 percentage points above the next best system (Zhang et al., 28 Aug 2025). Thinking mode yields modest gains of at most 1.2 percentage points on smaller models and negligible gains on the 235B model (Zhang et al., 28 Aug 2025). Domain variation is also pronounced: A320 ME/AV tasks reach up to 82.5% accuracy, whereas HAWKER800XP remains at or below 60%, highlighting platform-specific knowledge gaps (Zhang et al., 28 Aug 2025).

The general findings are that embedding models excel at semantic similarity but under-retrieve domain facts, LLMs plateau at approximately 60–70% accuracy for multiple-choice tasks, and deep reasoning tasks such as fault-tree analysis achieve only 30–58% partial or complete matches (Zhang et al., 28 Aug 2025). The paper further states that thought-chain prompts exhibit Test-Time Scaling Law trends but remain constrained by knowledge omissions and prompt length limits (Zhang et al., 28 Aug 2025).

6. Interpretation, applications, and limitations

The empirical profile of CAME-Bench indicates a separation between semantic competence and maintenance-grade factual or procedural competence (Zhang et al., 28 Aug 2025). Embedding systems can score strongly on semantic similarity tasks while remaining weak on system localization and other factual retrieval tasks. Likewise, LLMs can attain moderate performance on multiple-choice questions yet remain substantially weaker on structured reasoning tasks such as fault-tree analysis (Zhang et al., 28 Aug 2025). This suggests that benchmark success in this domain depends on domain grounding and retrieval architecture as much as on general language-model scale.

The paper identifies several applications for CAME-Bench. These include benchmarking domain-adapted embeddings to improve RAG pipelines in maintenance assistants, evaluating fine-tuned LLMs or hybrid KG-LLM systems for fault diagnosis and procedure retrieval, and guiding curriculum development in aviation maintenance training by highlighting weak knowledge areas (Zhang et al., 28 Aug 2025). Because tasks are aligned with maintenance workflows, the benchmark can support both systems research and operational tool development.

The limitations stated in the paper are specific and consequential. Coverage is limited to documented faults in B737 and A320 materials and existing exam QAs, so rare or emergent failure modes may not be represented (Zhang et al., 28 Aug 2025). Human-judged 0/1/2 scoring introduces evaluation noise and requires either domain experts or LLM-as-judge workflows (Zhang et al., 28 Aug 2025). In addition, LLM token limits from 1 024 to 32 768 can truncate deep-reasoning chains or long procedural texts (Zhang et al., 28 Aug 2025). These constraints imply that CAME-Bench should be understood as a rigorous but still bounded representation of the maintenance domain rather than an exhaustive operational surrogate.

7. Recommendations and research directions

The CAMB paper offers several recommendations for practitioners and future benchmark development (Zhang et al., 28 Aug 2025). First, it recommends domain-specific fine-tuning through incorporation of FIM and TSM corpora and bilingual manuals into continual pre-training of embeddings and LLM adapters, with the goal of enriching factual coverage (Zhang et al., 28 Aug 2025). Second, it recommends hybrid retrieval architectures that combine dense retrieval with structured knowledge graphs for multi-hop fault tracing beyond flat-text RAG (Zhang et al., 28 Aug 2025). Third, it recommends prompt engineering via dynamic chain-of-thought with chunk-wise context windows and short, targeted sub-task prompts for fault-tree reasoning (Zhang et al., 28 Aug 2025).

Two further directions concern benchmark expansion and evaluation methodology. The paper recommends extending CAME-Bench to additional airframes such as A330 and B777, broader system categories including hydraulics and flight controls, and multimodal inputs such as diagrams and sensor logs (Zhang et al., 28 Aug 2025). It also recommends refining LLM-as-judge workflows, including GPT-4o consistency checks, to reduce dependence on human annotation for open QA and reasoning tasks (Zhang et al., 28 Aug 2025).

Taken together, these recommendations position CAME-Bench as an end-to-end evaluation infrastructure for diagnosing domain knowledge and reasoning deficits in both embedding models and LLMs (Zhang et al., 28 Aug 2025). A plausible implication is that its long-term value lies not only in ranking models, but in decomposing the interaction among retrieval quality, platform-specific knowledge, bilingual terminology handling, and fault-oriented reasoning within a safety-critical engineering domain.