CogRail: Railway Cognitive Intrusion Benchmark

• CogRail is a benchmark for cognitive intrusion perception in railway systems, shifting from reactive object detection to proactive cognitive reasoning.
• It integrates curated open-source datasets with detailed question-answer annotations to support spatio-temporal threat analysis.
• Joint fine-tuning of visual-language models on CogRail improves accuracy in position, movement, and threat assessments, demonstrating performance gains over zero-shot evaluations.

Searching arXiv for the specified topic and closely related benchmark/framework papers. arxiv_search query: "CogRail Benchmarking VLMs in Cognitive Intrusion Perception for Intelligent Railway Transportation Systems" max_results: 5 arxiv_search query: "RailVQA benchmark framework efficient interpretable visual cognition automatic train operation" max_results: 5 CogRail is a benchmark for cognitive intrusion perception in intelligent railway transportation systems that shifts railway intrusion perception from reactive object-detection within fixed danger zones to proactive “cognitive” reasoning that anticipates latent intrusion risks via joint spatial, semantic, and motion understanding. It integrates curated open-source datasets with cognitively driven question-answer annotations to support spatio-temporal reasoning and prediction, and it is paired with a systematic evaluation and adaptation pipeline for state-of-the-art visual-LLMs (VLMs) in a safety-critical railway setting (Tian et al., 14 Jan 2026).

1. Problem framing and objectives

CogRail is motivated by a limitation of existing railway intrusion perception systems: most focus narrowly on object classification within fixed visual scopes and apply rule-based heuristics to determine intrusion status, often overlooking targets that pose latent intrusion risks. The benchmark therefore targets accurate and early perception of potential intrusion targets, with particular emphasis on cognition of spatial context and temporal dynamics for the object of interest (OOI) (Tian et al., 14 Jan 2026).

The benchmark’s stated objective is to support safety-critical railway scenarios by providing a unified testbed for spatio-temporal reasoning, enabling evaluation and adaptation of foundation VLMs beyond simple classification. In this formulation, the task is not merely to identify an object, but to determine where it is relative to railway infrastructure, how it is moving, and whether the integrated scene semantics imply a safe, potential, or serious threat (Tian et al., 14 Jan 2026).

A common misconception is that railway intrusion perception is reducible to zone-based object detection. CogRail explicitly rejects that reduction by formalizing intrusion perception as a cognitive reasoning problem with coupled spatial, motion, and threat judgments. This suggests that the benchmark is designed not only as an evaluation suite, but also as a stress test for whether general-purpose multimodal foundation models can be specialized for railway safety reasoning under constrained visual evidence.

2. Benchmark construction and annotation design

CogRail is built from two publicly released datasets used as backbones: RailSem19, which provides semantic rail-scene images, and MRSI, which provides multimodal video frames (Tian et al., 14 Jan 2026). The benchmark is divided into two subsets:

Subset	Images	Distinct OOIs
Cog-MRSI	347	614
Cog-RailSem19	495	1,536

The scenarios span daytime and infrared frames, varied backgrounds including straight vs. curved tracks, ballast zones, and surroundings, and OOI instances including pedestrians, cars, trucks, bicycles, animals, and stones. To enrich foreground diversity in RailSem19, approximately 2,000 LVIS-extracted instances are composited with geometric and depth-guided placement (Tian et al., 14 Jan 2026).

The annotation pipeline is explicitly structured for spatio-temporal reasoning. Static infrastructure, including tracks and ballast, is annotated with polygon masks. Dynamic OOIs are annotated with axis-aligned bounding boxes and instance IDs. Each OOI receives four aligned labels: semantic class, spatial position tag (RailPos), motion state (RailMove), and overall threat level (RailThreat) (Tian et al., 14 Jan 2026).

The question-answer annotation scheme is similarly designed for instruction-following multimodal models. Seed templates are hand-crafted for each subtask, such as “Which area is person 1 located in?”, then expanded via GPT-4 paraphrasing while forcing consistency with ground-truth attributes. The final instruction-response corpus is serialized as JSON with entries of the form {image_id, visual_prompt, text_instruction, answer}, thereby supporting both multimodal prompting and instruction tuning (Tian et al., 14 Jan 2026).

3. Formal task definition

CogRail formalizes three core tasks over an image frame $I$, a set of visual hits $H$ with indices $i \in H$, and visual features $x_i$ for hit $i$. The model $f$ takes $(I, i, \text{prompt}) \to y_i$ (Tian et al., 14 Jan 2026).

Position Perception (RailPos) classifies the relative spatial relationship of OOI $i$ with respect to railway infrastructure into one of three classes,

$$Y_{\text{pos}}=\{\text{outside}, \text{ballast}, \text{track}\}.$$

The input space is given as $I \in \mathbb{R}^{H \times W \times 3}$ with a hit index $H$0, and the output is $H$1. The perception function is

$H$2

with cross-entropy loss

$H$3

Movement Prediction (RailMove) predicts the current movement state of OOI $H$4 as one of

$H$5

Although single frames are used, the annotation embeds latent temporal context inferred from short video clips. A virtual temporal window is defined as

$H$6

with

$H$7

and loss

$H$8

Threat Analysis (RailThreat) integrates spatial and motion cues plus scene semantics to assign each OOI a threat level

$H$9

The classification rule is

$i \in H$0

with loss

$i \in H$1

This task decomposition is central to CogRail’s design. It separates perception into interpretable subtasks while preserving their dependency structure. A plausible implication is that the benchmark treats threat estimation as a downstream synthesis problem rather than an isolated label prediction problem.

4. Joint fine-tuning framework

CogRail couples the benchmark with a joint fine-tuning framework for multi-task learning. A single backbone VLM, exemplified by Qwen2.5-VL, processes $i \in H$2, where $i \in H$3 is a shared embedding. Task-specific heads $i \in H$4 then operate on $i \in H$5 to produce logits for each subtask. This enables parameter sharing of the vision encoder and language decoder while capturing inter-task dependencies (Tian et al., 14 Jan 2026).

The overall multi-task objective is

$i \in H$6

A typical weighting choice is

$i \in H$7

for balanced training, though the weights can be tuned to emphasize underperforming tasks (Tian et al., 14 Jan 2026).

For efficient adaptation, the framework uses Low-Rank Adaptation (LoRA). For each linear weight matrix $i \in H$8 in the VLM, a low-rank update is inserted:

$i \in H$9

where $x_i$0, $x_i$1, and $x_i$2. Only $x_i$3 are learned by minimizing

$x_i$4

The training strategy uses a 4:1 train:test split for both subsets, class-balanced resampling with probability

$x_i$5

prompt paraphrasing to mitigate over-fitting to fixed question templates, learning rate $x_i$6, LoRA rank $x_i$7, batch size $x_i$8, and $x_i$9 training epochs (Tian et al., 14 Jan 2026).

In the summary formulation of the work, CogRail establishes a structured benchmark and adaptation framework, termed RailGPT, to reveal and partially overcome the cognitive reasoning limitations of current VLMs in safety-critical railway intrusion perception (Tian et al., 14 Jan 2026).

5. Evaluation protocol and empirical findings

CogRail evaluates models using per-class Precision and Recall and overall macro F1, defined as the harmonic mean of average precision and recall. Threat assessment additionally reports F1 per threat level. The evaluated baseline and state-of-the-art VLMs are Qwen2-VL, Qwen2.5-VL, LLaMA-3.2-Vision, Yi-VL, and LLaVA-1.6, all in the approximate range of 6–11B parameters (Tian et al., 14 Jan 2026).

Under zero-shot evaluation, averaged over Type-I/II prompts and the two subsets, the best reported results are 48.87% for RailPos with Qwen2.5-VL, 45.10% for RailMove with Qwen2-VL, and 52.35% for RailThreat with Qwen2.5-VL. The mean overall accuracy is approximately 45%, described as well below safe-operation standards (Tian et al., 14 Jan 2026). The paper correspondingly concludes that current large-scale multimodal models struggle with the complex spatial-temporal reasoning required by the cognitive intrusion perception task.

Individual fine-tuning yields substantial gains. For RailPos on Cog-MRSI with Type-I prompts, average F1 increases from 29.90% to 37.37% (+7.47 points). For RailMove on Cog-MRSI with Type-II prompts, average F1 increases from 36.93% to 47.03% (+10.10 points). For RailThreat on Cog-RailSem19 with Type-II prompts, average F1 increases from 43.06% to 61.66% (+18.60 points). The best single-task post-finetune results reported are 55.15% F1 for LLaVA-1.6 on RailMove (Type-II, MRSI) and 68.63% for LLaMA-3.2-Vision on RailThreat (Type-II, RailSem19) (Tian et al., 14 Jan 2026).

Joint fine-tuning outperforms both zero-shot and individual fine-tuning in most cases: 17 of 20 task/prompt/dataset configurations show joint-train greater than both alternatives. Representative results include Cog-MRSI RailThreat (Type-I), where LLaVA-1.6 reaches 59.66% F1 (+6.95 versus individual); Cog-MRSI RailThreat (Type-II), where LLaMA-3.2-Vision reaches 58.26% (+7.51); Cog-RailSem19 RailThreat (Type-I), where Qwen2-VL reaches 75.31% (+12.63); and Cog-RailSem19 RailThreat (Type-II), where Qwen2.5-VL reaches 76.11% (+9.77) (Tian et al., 14 Jan 2026).

These findings support two conclusions stated in the source material. First, off-the-shelf VLMs lack explicit modeling of track topology and motion dynamics, leading to erratic threat judgments. Second, multi-task adaptation yields consistent, often large, improvements over single-task fine-tuning by exploiting cross-task synergies, such as better spatial cues improving threat scoring. Jointly trained models achieve F1 greater than 75% on threat assessment in RailSem19, demonstrating practical viability for early warning, and the use of disentangled prompts and modular task heads provides “audit trails” of position→motion→threat reasoning (Tian et al., 14 Jan 2026).

6. Subsequent extensions, implications, and limitations

A later related work, RailVQA, explicitly frames its contribution in terms of CogRail requirements for cognitive rail systems. RailVQA-bench is described as the first VQA benchmark for cab-view visual cognition in automatic train operation (ATO), comprising 20,000 single-frame and 1,168 video based QA pairs, and RailVQA-CoM is presented as a collaborative large-small model framework with a transparent three-module architecture and adaptive temporal sampling (Zhang et al., 28 Mar 2026). Its structured Chain-of-Thought output is

$i$0

which separates perception, logical or physical inference, planning, and final answer.

RailVQA’s “CogRail Implications” identify three properties as central to CogRail systems: safe, efficient, and interpretable. In this account, structured CoT reasoning improves hazard anticipation for dynamic intrusions and out-of-distribution anomalies; a small-model front-end combined with adaptive sampling yields 24%–36% latency reduction and greater than 2× LMM throughput; and explicit perceiving–reasoning–planning outputs enable post-hoc audit, regulatory compliance, and operator trust (Zhang et al., 28 Mar 2026). This suggests that CogRail has already begun to function as a broader design target for railway multimodal cognition rather than only as a benchmark name.

The limitations and future directions articulated around CogRail are consistent across these works. The original benchmark proposes incorporating explicit temporal sequences through multi-frame or video inputs, extending beyond monocular images to lightweight sensor fusion such as LiDAR or multi-view, exploring self-supervised pretraining on spatio-temporal railway data, and investigating long-horizon causal reasoning and counterfactual queries such as “If this object continues, will it intrude in 5s?” (Tian et al., 14 Jan 2026). RailVQA adds that current reliance on monocular RGB data limits depth estimation, that LMM inference still requires GPU compute and therefore demands edge-device resource budgeting and fail-safe modes, and that defensive prompting alleviates but does not eliminate hallucination and out-of-distribution risk (Zhang et al., 28 Mar 2026).

Within this trajectory, CogRail occupies a specific position: it establishes a structured benchmark and adaptation framework for railway cognitive intrusion perception, demonstrates that zero-shot VLM performance remains weak under single-frame spatio-temporal reasoning, and shows that structured multi-task fine-tuning can materially improve both accuracy and interpretability in a safety-critical domain (Tian et al., 14 Jan 2026).