---
title: Flames Benchmark Overview
url: https://www.emergentmind.com/topics/flames-benchmark
type: topic
---

# Flames Benchmark Overview

Searching arXiv for recent papers using the term “Flames Benchmark” and closely related “FLAMES/FLAME benchmark” usages.
“Flames Benchmark” is not a single canonical research artifact across arXiv. The label is used, or invoked, in several distinct ways: as the name of a Chinese value-alignment benchmark for large language models called **Flames** [2311.06899]; as shorthand for a physically grounded wildfire reasoning benchmark, **FlameVQA**, built on FLAME 3 [2606.27128]; as a Chinese financial LLM evaluation system, **FLAME**, with certification and scenario sub-benchmarks [2501.06211]; and, in combustion and reactive-flow research, as a generic description for benchmark flame configurations rather than a uniquely titled benchmark suite [2302.11899]. The term therefore requires domain-specific disambiguation.

## 1. Nomenclature and scope

The most direct benchmark named **Flames** in the supplied literature is the Chinese LLM value-alignment benchmark introduced in “Flames: Benchmarking Value Alignment of LLMs in Chinese” [2311.06899]. In parallel, the wildfire paper “FlameVQA” explicitly presents itself as a strong “flames benchmark” for multimodal reasoning in UAV wildfire monitoring, but its formal benchmark name is **FlameVQA**, not Flames [2606.27128]. Other papers use closely related names—**FLAME** or **FLAMES**—for evaluation systems, methods, or frameworks in finance, math reasoning, methane detection, or event-based learning, with different benchmark semantics [2501.06211].

This naming overlap is not merely stylistic. In some papers, the capitalized form denotes a **benchmark**; in others, it denotes a **method** evaluated on some other benchmark. For example, the methane-detection paper “FLAME: Physics-Guided Neural Operators for Onboard Satellite Methane Detection in Hyperspectral Imagery” states that **FLAME is the proposed method**, while the benchmark context is **STARCOP** [2606.01577]. Likewise, the event-based learning paper “FLAMES: A Hybrid Spiking-State Space Model for Adaptive Memory Retention in Event-Based Learning” states that **FLAMES is not a new benchmark introduced by the paper**, but a model evaluated on Long Range Arena, HAR-DVS, and CeleX-HAR [2504.01257]. A plausible implication is that “Flames Benchmark” functions more as an overloaded label than as a stable bibliographic entity.

## 2. Flames as a Chinese value-alignment benchmark

In its most explicit benchmark sense, **Flames** is a benchmark for evaluating the **value alignment** of LLMs in **Chinese** [2311.06899]. The acronym is given as **F**airness, **L**egality, d**a**ta prot**e**ction, **M**orality, **S**afety. The benchmark is designed to probe “deeper” alignment rather than superficial refusal behavior, with emphasis on adversarial prompts, implicit malice, fine-grained annotation, and Chinese-specific moral concepts such as **harmony**, **benevolence**, **courtesy**, and **Zhongyong** [2311.06899].

The prompt dataset, **Flames-prompts**, contains **2,251 manually designed prompts**. The reported dimension counts are **590** for Fairness, **779** for Safety, **522** for Morality, **118** for Legality, and **242** for Data Protection, with an overall average prompt length of **85.92 tokens** [2311.06899]. For public evaluation, the paper states that **1,000 prompts** are randomly selected and released, while the rest are retained for future evaluation. The benchmark authors prompted **17 mainstream LLMs** and collected approximately **22.9K annotated responses**, with each response labeled by **2 annotators**, and disagreements adjudicated by a **third annotator** [2311.06899].

Flames reports two main metrics. The **harmless rate** is the proportion of responses scored **3** on a dimension, and the **harmless score** is the average normalized score on that dimension [2311.06899]. The paper further introduces **Flames-scorer**, a benchmark-specific automatic evaluator. Its best reported test accuracy is **79.5%** with an **InternLM-Chat-7B** backbone, compared with **61.3%** for the strongest GPT-4-as-judge setting reported in the paper [2311.06899]. The benchmark’s central empirical message is that all evaluated models perform relatively poorly, especially on **Safety** and **Fairness**, and that Flames is substantially harder than earlier Chinese safety benchmarks.

## 3. FlameVQA as a physically grounded wildfire benchmark

In wildfire monitoring, the strongest benchmark that the supplied material explicitly characterizes as a “flames benchmark” is **FlameVQA** [2606.27128]. FlameVQA is a **multiple-choice visual question answering benchmark for UAV-based wildfire intelligence** built on the **FLAME 3 computer vision subset**, with each sample containing **an RGB image, a color-mapped thermal JPEG, and a radiometric thermal TIFF** [2606.27128]. The benchmark summary reports approximately **\(\sim 6{,}100\)** paired RGB–thermal frames from **Sycan Marsh, Willamette Valley, and Shoetank**, and defines **34 multiple-choice questions per image**.

The question set spans **six operational capability groups**: **Presence/Detection**, **Classification**, **Distribution/Segmentation**, **Localization/Direction**, **Cross-Modal Reasoning**, and **Flight Planning** [2606.27128]. This structure is central to FlameVQA’s identity: it is not a fire detector, but a benchmark for physically grounded, safety-critical reasoning in wildfire scenes where visible and thermal evidence may disagree. The paper’s annotation pipeline combines **MLLM-assisted annotation, deterministic thermal rules, metadata-derived labels, cross-question consistency constraints, and human auditing** [2606.27128].

The benchmark reports a targeted human evaluation on a **Willamette Valley subset** comprising **12,733 evaluated question–answer pairs**, chosen because of dense smoke, mixed fuels, and ambiguity between active fire and residual heat [2606.27128]. Overall agreement between the automated labeling pipeline and a human expert is **70.78\%**. Per-question agreement varies sharply, from **36.12%** on **PD5** (“Is active fire present despite heavy smoke or visual occlusion?”) to **90.98%** on **CMR3** (“Is fire visibility consistent across RGB and thermal?”) [2606.27128]. Baseline model evaluation on the same subset uses **LLaVA-1.6-7B** and **Qwen3-VL-8B-Instruct**, with reported overall accuracies of **36.23%** and **48.65%** respectively; both models do relatively well on explicit cross-modal reasoning, but remain weak on coverage estimation and heavy-smoke presence detection [2606.27128]. This suggests that FlameVQA is deliberately discriminative rather than merely descriptive.

## 4. Other benchmark systems using the FLAME/FLAMES name

Outside value alignment and wildfire monitoring, the FLAME/FLAMES naming family also appears in benchmark systems for finance and synthetic math-reasoning evaluation. **FLAME**, in “Financial Large-Language Model Assessment and Metrics Evaluation,” is a Chinese financial LLM evaluation system with two core benchmarks: **FLAME-Cer** and **FLAME-Sce** [2501.06211]. **FLAME-Cer** covers **14** financial certifications with approximately **16,000** manually reviewed questions, while **FLAME-Sce** contains **10** primary business scenarios, **21** secondary scenarios, and nearly **100** tertiary application tasks, with more than **5,000** evaluation questions [2501.06211]. The paper evaluates **6 representative LLMs** and reports **Baichuan4-Finance** as the best overall performer, with **93.62%** average accuracy on FLAME-Cer and **84.15%** average usability rate on FLAME-Sce [2501.06211].

The math-reasoning paper “FLAMES: Improving LLM Math Reasoning via a Fine-Grained Analysis of the Data Synthesis Pipeline” uses **FLAMES** primarily as a **framework and evaluation protocol** for synthetic math data, rather than as a static benchmark dataset [2508.16514]. It systematically studies **10 existing data synthesis strategies**, argues that higher **coverage** can matter more than stricter filtering under a fixed budget, and produces the **FLAMES dataset** as a blended synthetic corpus [2508.16514]. The headline result reported in the supplied material is that **Qwen2.5-Math-7B** fine-tuned on the FLAMES dataset achieves **81.4% on MATH** [2508.16514]. In this usage, “FLAMES benchmark” is technically incomplete; the paper presents a benchmarking framework for data-synthesis pipelines.

By contrast, not every paper with the name FLAME defines a benchmark. The methane-detection paper explicitly states that **FLAME is the proposed method, not the dataset or evaluation suite**, and that its benchmark context is **STARCOP** [2606.01577]. This distinction is essential in any encyclopedia treatment of the term.

## 5. “Flame benchmark” in combustion and reactive-flow research

In combustion literature, the phrase “flame benchmark” usually denotes a **benchmark configuration** rather than a benchmark suite named Flames. Several supplied papers are best understood in that sense. The DNS study “Turbulent burning velocity and thermo-diffusive instability of premixed flames” provides a controlled benchmark-quality matrix for **lean hydrogen-air premixed flames propagating in forced turbulence in a box**, with outputs centered on turbulent burning velocity \(U_T\), transport formulation, and the laminar instability threshold \(\Lambda_n\) [2302.11899]. The paper reports that crossing \(\Lambda=\Lambda_n\) increases \(\overline{U_T}/S_L\) from **1.05** to **1.64** in weak turbulence, but has negligible effect in moderately turbulent flames with \(\mathrm{Ka}\ge 3.4\) [2302.11899].

A different benchmark genre appears in computer-vision work on combustion imagery. “Segmentation of Industrial Burner Flames” evaluates **Global Thresholding, Region Growing, SVM, RF, MLP, U-Net, and DeepLabV3+** on a relabeled subset of a public industrial burner flame dataset, with **160 training images** and **40 test images** of size **552 × 552** [2306.14789]. The best reported model is **DeepLabV3+ (RN18-I)** with **93.2% IoU**, while the fastest method is **Global Thresholding** at **0.1 ms/image** on CPU [2306.14789]. Here, “flame benchmark” means a supervised image-segmentation benchmark over flame/background masks.

Reactive-flow computing papers use still another benchmark idiom. The GPU/ML framework paper “An integrated framework for accelerating reactive flow simulation using GPU and machine learning models” evaluates on two turbulent flame benchmarks: a **reactive Taylor–Green vortex interacting with an H\(_2\)-air diffusion flame** and the **Cambridge stratified burner SWB5** methane-air flame [2312.13513]. The supercritical DeepFlame paper uses a **supercritical reactive TGV benchmark** and a **127-injector rocket engine** as extreme-scale application benchmarks [2508.18969]. The swirling-premixed LES paper validates a GPU-ANN approach on the **XJTU swirling premixed methane-air flame** and **Cambridge SWB7** [2402.18858]. The Cambridge SWB5 paper uses that flame as a benchmark for ANN-accelerated LES fidelity [2312.05800]. Experimental combustion papers add further benchmark-style resources: **global flame geometry and scaling laws** for methane and hydrogen jet flames [2603.15592], **mode transitions and traveling-wave dynamics** in a cavity-stabilized circular Hele-Shaw burner [2604.08035], **Rayleigh–Taylor unstable flames** as a DNS reference for flame-speed models [1505.00025], and **1-D laminar premixed H\(_2\)/air flames under elevated pressure** for real-fluid modeling in HiPrFlame [2509.06717]. Collectively, these works show that in combustion research the relevant unit is usually a benchmark **case**, not a benchmark named Flames.

## 6. Misidentification, exclusion, and editorial disambiguation

A recurring misconception is that any paper containing “Flame,” “FLAME,” or “FLAMES” defines a benchmark. Several supplied cases directly contradict that assumption. The methane paper states that **FLAME is the proposed method** and that the benchmark is **STARCOP** [2606.01577]. The event-based learning paper states that **FLAMES is not a new benchmark introduced by the paper**, but a hybrid spiking/state-space model evaluated on existing datasets such as **Long Range Arena**, **HAR-DVS**, and **CeleX-HAR** [2504.01257]. The smart-contract security paper similarly presents **FLAMES** as a method for synthesizing runtime guards, with a multi-part evaluation suite rather than a separately named benchmark artifact [2510.21401].

The supplied record for arXiv **2404.06653** is a further disambiguation case. Although its header in the prompt describes a flame-detection framework called **FlameFinder**, the detailed note states that the provided document is actually an IEEE LaTeX template titled “How to Use the IEEEtran LaTeX Templates,” and therefore should be excluded from any flame-benchmark corpus [2404.06653]. This is not a minor clerical issue: it illustrates that the phrase “Flames Benchmark” is vulnerable to bibliographic false positives.

The most defensible encyclopedia interpretation is therefore taxonomic. In current arXiv usage, **“Flames Benchmark” is an overloaded expression** covering at least four patterns: a named benchmark for Chinese LLM value alignment, a wildfire VQA benchmark for UAV thermal reasoning, benchmark systems in adjacent LLM domains such as finance and synthetic math reasoning, and a broad combustion convention in which “flame benchmark” denotes a canonical case rather than a branded benchmark suite [2311.06899]. This suggests that any serious citation of the term should specify the underlying domain and the formal resource name.

Source: https://www.emergentmind.com/topics/flames-benchmark