Flames Benchmark Overview

• Flames Benchmark is an overloaded term defining several distinct evaluation systems across Chinese LLM value alignment, wildfire monitoring, finance, and combustion research.
• The Chinese LLM value-alignment version uses 2,251 manually designed prompts to assess fairness, safety, morality, legality, and data protection with metrics like the harmless rate and score.
• Physical benchmarks like FlameVQA evaluate multimodal wildfire reasoning using paired RGB–thermal data and multiple-choice questions to support safety-critical decision-making.

Searching arXiv for papers using the term “Flames Benchmark” and closely related “FLAMES/FLAME benchmark” usages. “Flames Benchmark” is not a single canonical research artifact across arXiv. The label is used, or invoked, in several distinct ways: as the name of a Chinese value-alignment benchmark for LLMs called Flames (Huang et al., 2023); as shorthand for a physically grounded wildfire reasoning benchmark, FlameVQA, built on FLAME 3 (Habibpour et al., 25 Jun 2026); as a Chinese financial LLM evaluation system, FLAME, with certification and scenario sub-benchmarks (Guo et al., 3 Jan 2025); and, in combustion and reactive-flow research, as a generic description for benchmark flame configurations rather than a uniquely titled benchmark suite (Lee et al., 2023). The term therefore requires domain-specific disambiguation.

1. Nomenclature and scope

The most direct benchmark named Flames in the supplied literature is the Chinese LLM value-alignment benchmark introduced in “Flames: Benchmarking Value Alignment of LLMs in Chinese” (Huang et al., 2023). In parallel, the wildfire paper “FlameVQA” explicitly presents itself as a strong “flames benchmark” for multimodal reasoning in UAV wildfire monitoring, but its formal benchmark name is FlameVQA, not Flames (Habibpour et al., 25 Jun 2026). Other papers use closely related names—FLAME or FLAMES—for evaluation systems, methods, or frameworks in finance, math reasoning, methane detection, or event-based learning, with different benchmark semantics (Guo et al., 3 Jan 2025).

This naming overlap is not merely stylistic. In some papers, the capitalized form denotes a benchmark; in others, it denotes a method evaluated on some other benchmark. For example, the methane-detection paper “FLAME: Physics-Guided Neural Operators for Onboard Satellite Methane Detection in Hyperspectral Imagery” states that FLAME is the proposed method, while the benchmark context is STARCOP (Heo et al., 1 Jun 2026). Likewise, the event-based learning paper “FLAMES: A Hybrid Spiking-State Space Model for Adaptive Memory Retention in Event-Based Learning” states that FLAMES is not a new benchmark introduced by the paper, but a model evaluated on Long Range Arena, HAR-DVS, and CeleX-HAR (Chakraborty et al., 2 Apr 2025). A plausible implication is that “Flames Benchmark” functions more as an overloaded label than as a stable bibliographic entity.

2. Flames as a Chinese value-alignment benchmark

In its most explicit benchmark sense, Flames is a benchmark for evaluating the value alignment of LLMs in Chinese (Huang et al., 2023). The acronym is given as Fairness, Legality, d*ata protection, **Morality, **Safety. The benchmark is designed to probe “deeper” alignment rather than superficial refusal behavior, with emphasis on adversarial prompts, implicit malice, fine-grained annotation, and Chinese-specific moral concepts such as **harmony, **benevolence, **courtesy, and **Zhongyong* (Huang et al., 2023).

The prompt dataset, Flames-prompts, contains 2,251 manually designed prompts. The reported dimension counts are 590 for Fairness, 779 for Safety, 522 for Morality, 118 for Legality, and 242 for Data Protection, with an overall average prompt length of 85.92 tokens (Huang et al., 2023). For public evaluation, the paper states that 1,000 prompts are randomly selected and released, while the rest are retained for future evaluation. The benchmark authors prompted 17 mainstream LLMs and collected approximately 22.9K annotated responses, with each response labeled by 2 annotators, and disagreements adjudicated by a third annotator (Huang et al., 2023).

Flames reports two main metrics. The harmless rate is the proportion of responses scored 3 on a dimension, and the harmless score is the average normalized score on that dimension (Huang et al., 2023). The paper further introduces Flames-scorer, a benchmark-specific automatic evaluator. Its best reported test accuracy is 79.5% with an InternLM-Chat-7B backbone, compared with 61.3% for the strongest GPT-4-as-judge setting reported in the paper (Huang et al., 2023). The benchmark’s central empirical message is that all evaluated models perform relatively poorly, especially on Safety and Fairness, and that Flames is substantially harder than earlier Chinese safety benchmarks.

3. FlameVQA as a physically grounded wildfire benchmark

In wildfire monitoring, the strongest benchmark that the supplied material explicitly characterizes as a “flames benchmark” is FlameVQA (Habibpour et al., 25 Jun 2026). FlameVQA is a multiple-choice visual question answering benchmark for UAV-based wildfire intelligence built on the FLAME 3 computer vision subset, with each sample containing an RGB image, a color-mapped thermal JPEG, and a radiometric thermal TIFF (Habibpour et al., 25 Jun 2026). The benchmark summary reports approximately $\sim 6{,}100$ paired RGB–thermal frames from Sycan Marsh, Willamette Valley, and Shoetank, and defines 34 multiple-choice questions per image.

The question set spans six operational capability groups: Presence/Detection, Classification, Distribution/Segmentation, Localization/Direction, Cross-Modal Reasoning, and Flight Planning (Habibpour et al., 25 Jun 2026). This structure is central to FlameVQA’s identity: it is not a fire detector, but a benchmark for physically grounded, safety-critical reasoning in wildfire scenes where visible and thermal evidence may disagree. The paper’s annotation pipeline combines MLLM-assisted annotation, deterministic thermal rules, metadata-derived labels, cross-question consistency constraints, and human auditing (Habibpour et al., 25 Jun 2026).

The benchmark reports a targeted human evaluation on a Willamette Valley subset comprising 12,733 evaluated question–answer pairs, chosen because of dense smoke, mixed fuels, and ambiguity between active fire and residual heat (Habibpour et al., 25 Jun 2026). Overall agreement between the automated labeling pipeline and a human expert is 70.78\%. Per-question agreement varies sharply, from 36.12% on PD5 (“Is active fire present despite heavy smoke or visual occlusion?”) to 90.98% on CMR3 (“Is fire visibility consistent across RGB and thermal?”) (Habibpour et al., 25 Jun 2026). Baseline model evaluation on the same subset uses LLaVA-1.6-7B and Qwen3-VL-8B-Instruct, with reported overall accuracies of 36.23% and 48.65% respectively; both models do relatively well on explicit cross-modal reasoning, but remain weak on coverage estimation and heavy-smoke presence detection (Habibpour et al., 25 Jun 2026). This suggests that FlameVQA is deliberately discriminative rather than merely descriptive.

4. Other benchmark systems using the FLAME/FLAMES name

Outside value alignment and wildfire monitoring, the FLAME/FLAMES naming family also appears in benchmark systems for finance and synthetic math-reasoning evaluation. FLAME, in “Financial Large-LLM Assessment and Metrics Evaluation,” is a Chinese financial LLM evaluation system with two core benchmarks: FLAME-Cer and FLAME-Sce (Guo et al., 3 Jan 2025). FLAME-Cer covers 14 financial certifications with approximately 16,000 manually reviewed questions, while FLAME-Sce contains 10 primary business scenarios, 21 secondary scenarios, and nearly 100 tertiary application tasks, with more than 5,000 evaluation questions (Guo et al., 3 Jan 2025). The paper evaluates 6 representative LLMs and reports Baichuan4-Finance as the best overall performer, with 93.62% average accuracy on FLAME-Cer and 84.15% average usability rate on FLAME-Sce (Guo et al., 3 Jan 2025).

The math-reasoning paper “FLAMES: Improving LLM Math Reasoning via a Fine-Grained Analysis of the Data Synthesis Pipeline” uses FLAMES primarily as a framework and evaluation protocol for synthetic math data, rather than as a static benchmark dataset (Seegmiller et al., 22 Aug 2025). It systematically studies 10 existing data synthesis strategies, argues that higher coverage can matter more than stricter filtering under a fixed budget, and produces the FLAMES dataset as a blended synthetic corpus (Seegmiller et al., 22 Aug 2025). The headline result reported in the supplied material is that Qwen2.5-Math-7B fine-tuned on the FLAMES dataset achieves 81.4% on MATH (Seegmiller et al., 22 Aug 2025). In this usage, “FLAMES benchmark” is technically incomplete; the paper presents a benchmarking framework for data-synthesis pipelines.

By contrast, not every paper with the name FLAME defines a benchmark. The methane-detection paper explicitly states that FLAME is the proposed method, not the dataset or evaluation suite, and that its benchmark context is STARCOP (Heo et al., 1 Jun 2026). This distinction is essential in any encyclopedia treatment of the term.

5. “Flame benchmark” in combustion and reactive-flow research

In combustion literature, the phrase “flame benchmark” usually denotes a benchmark configuration rather than a benchmark suite named Flames. Several supplied papers are best understood in that sense. The DNS study “Turbulent burning velocity and thermo-diffusive instability of premixed flames” provides a controlled benchmark-quality matrix for lean hydrogen-air premixed flames propagating in forced turbulence in a box, with outputs centered on turbulent burning velocity $U_T$, transport formulation, and the laminar instability threshold $\Lambda_n$ (Lee et al., 2023). The paper reports that crossing $\Lambda=\Lambda_n$ increases $\overline{U_T}/S_L$ from 1.05 to 1.64 in weak turbulence, but has negligible effect in moderately turbulent flames with $\mathrm{Ka}\ge 3.4$ (Lee et al., 2023).

A different benchmark genre appears in computer-vision work on combustion imagery. “Segmentation of Industrial Burner Flames” evaluates Global Thresholding, Region Growing, SVM, RF, MLP, U-Net, and DeepLabV3+ on a relabeled subset of a public industrial burner flame dataset, with 160 training images and 40 test images of size 552 × 552 (Landgraf et al., 2023). The best reported model is DeepLabV3+ (RN18-I) with 93.2% IoU, while the fastest method is Global Thresholding at 0.1 ms/image on CPU (Landgraf et al., 2023). Here, “flame benchmark” means a supervised image-segmentation benchmark over flame/background masks.

Reactive-flow computing papers use still another benchmark idiom. The GPU/ML framework paper “An integrated framework for accelerating reactive flow simulation using GPU and machine learning models” evaluates on two turbulent flame benchmarks: a reactive Taylor–Green vortex interacting with an H$_2$-air diffusion flame and the Cambridge stratified burner SWB5 methane-air flame (Mao et al., 2023). The supercritical DeepFlame paper uses a supercritical reactive TGV benchmark and a 127-injector rocket engine as extreme-scale application benchmarks (Guo et al., 26 Aug 2025). The swirling-premixed LES paper validates a GPU-ANN approach on the XJTU swirling premixed methane-air flame and Cambridge SWB7 (zhang et al., 2024). The Cambridge SWB5 paper uses that flame as a benchmark for ANN-accelerated LES fidelity (zhang et al., 2023). Experimental combustion papers add further benchmark-style resources: global flame geometry and scaling laws for methane and hydrogen jet flames (Maffei et al., 16 Mar 2026), mode transitions and traveling-wave dynamics in a cavity-stabilized circular Hele-Shaw burner (Nie et al., 9 Apr 2026), Rayleigh–Taylor unstable flames as a DNS reference for flame-speed models (Hicks, 2015), and 1-D laminar premixed H$_2$/air flames under elevated pressure for real-fluid modeling in HiPrFlame (Zhang et al., 8 Sep 2025). Collectively, these works show that in combustion research the relevant unit is usually a benchmark case, not a benchmark named Flames.

6. Misidentification, exclusion, and editorial disambiguation

A recurring misconception is that any paper containing “Flame,” “FLAME,” or “FLAMES” defines a benchmark. Several supplied cases directly contradict that assumption. The methane paper states that FLAME is the proposed method and that the benchmark is STARCOP (Heo et al., 1 Jun 2026). The event-based learning paper states that FLAMES is not a new benchmark introduced by the paper, but a hybrid spiking/state-space model evaluated on existing datasets such as Long Range Arena, HAR-DVS, and CeleX-HAR (Chakraborty et al., 2 Apr 2025). The smart-contract security paper similarly presents FLAMES as a method for synthesizing runtime guards, with a multi-part evaluation suite rather than a separately named benchmark artifact (Eshghie et al., 24 Oct 2025).

The supplied record for arXiv (Rajoli et al., 2024) is a further disambiguation case. Although its header in the prompt describes a flame-detection framework called FlameFinder, the detailed note states that the provided document is actually an IEEE LaTeX template titled “How to Use the IEEEtran LaTeX Templates,” and therefore should be excluded from any flame-benchmark corpus (Rajoli et al., 2024). This is not a minor clerical issue: it illustrates that the phrase “Flames Benchmark” is vulnerable to bibliographic false positives.

The most defensible encyclopedia interpretation is therefore taxonomic. In current arXiv usage, “Flames Benchmark” is an overloaded expression covering at least four patterns: a named benchmark for Chinese LLM value alignment, a wildfire VQA benchmark for UAV thermal reasoning, benchmark systems in adjacent LLM domains such as finance and synthetic math reasoning, and a broad combustion convention in which “flame benchmark” denotes a canonical case rather than a branded benchmark suite (Huang et al., 2023). This suggests that any serious citation of the term should specify the underlying domain and the formal resource name.