Highway-Rail Crossing Incident Data

• Highway-Rail Grade Crossing Incident Data is an official FRA record comprising 66 structured and unstructured subfields for documenting crossing incidents.
• Automated extraction using a two-stage VLM/LLM pipeline enhances transcription accuracy by aggregating multiple samples to reduce errors.
• Real-time processing with grouped QA strategies addresses layout complexity and field ambiguity, ensuring timely and reliable data capture.

The Highway-Rail Grade Crossing Incident Data (Form 57) is an official record maintained by the U.S. Federal Railroad Administration (FRA) to document incidents at highway-rail grade crossings. Form 57 encapsulates approximately 66 distinct subfields, spanning structured elements such as checkboxes and choice lists to unstructured free-text components. The accurate and timely population of this form is critical for real-time situational awareness but has historically been delayed due to reliance on official investigations. Recent research has focused on automating the extraction and population of these data fields from unstructured news sources, with particular emphasis on real-time systems that can operate robustly despite the challenges posed by noisy, semantically complex, and visually irregular input modalities (Lim et al., 27 Dec 2025).

1. Form 57 Structure and Representation

Form 57 exemplifies complex document layouts: its fields include a mixture of free-text regions, unordered choice lists, multiple checkboxes, and diverse units of measure, without a regular, tabular structure. To facilitate machine-readable extraction, the scanned form image is modeled as a set of visual regions $\{x_1, \ldots, x_n\}$, where each $x_i$ is derived from region proposal features via a vision backbone of a vision-LLM (VLM). Each subfield $s$—such as "6.Time of Accident: textbox" or "6.Time of Accident: AM/PM checkbox"—is treated as a unique extraction target.

To mitigate omission and hallucination errors typical of a single-pass transcription, a sample-aggregation framework is applied: $N$ independent VLM transcriptions $T_1, \ldots, T_N$ are generated and each is validated against a rigid, human-centric JSON schema codified in the system (see Appendix A.2 of (Lim et al., 27 Dec 2025)). Only valid samples are used. Aggregated results for each subfield $s$ with choice set $C_s$ are computed via majority vote: $$p_{\mathrm{agg}}(k\mid s) = \frac{1}{N}\sum_{i=1}^N \mathbf{1}[T_i(s)=k] \quad \text{for } k\in C_s,$$ with the final value assigned as

$$T_{\mathrm{final}}(s) = \arg\max_{k\in C_s} p_{\mathrm{agg}}(k\mid s).$$

This aggregation substantially reduces hallucination and improves subfield accuracy.

2. Question Generation and Grouped QA Pipeline

Upon obtaining $T_{\mathrm{final}}$, each subfield is programmatically transformed into a succinct natural language question $x_i$0. For example, the subfield "15. Direction: single-choice among {Northbound, Southbound,…}" is mapped to the question "What direction was the highway vehicle traveling?" Valid answer sets are injected automatically to constrain the LLM.

Field ambiguity, particularly for temporally or spatially proximate subfields (e.g., time, location, casualty detail), is addressed through semantic grouping. Questions are clustered into groups $x_i$1 using a hybrid metric: 2D form layout proximity and embedding-based linguistic similarity (cosine similarity of LLM-derived embeddings, with a threshold $x_i$2; spatial adjacency threshold $x_i$3 ensures fields in close visual proximity cluster). Each group is presented to the LLM in a batch QA prompt mirroring the form's structure. This grouping increases answer coverage at the cost of only a marginal decrease in per-field precision.

3. News Source Retrieval and Data Alignment

Candidate articles are identified by querying each FRA record with spatiotemporal filters (state, county, city, date $x_i$4 7 days) across multiple news scraping and parsing engines (newspaper3k, trafilatura, readability, goose3). Initial retrieval yielded 1,707 articles for California records, of which approximately two-thirds were irrelevant or generic. Rigorous filtering retained 266 article–record pairs with clear incident specificity.

To quantify data retrievability, 50 pairs underwent fine-grained annotation: annotators categorized each field as "answerable" or "no info." Inter-annotator agreement, measured by Cohen’s $x_i$5, reached 0.82, indicating high schema reliability for assessing information presence in noisy news text.

4. Evaluation Framework and Performance Results

Measurement criteria include precision $x_i$6, recall $x_i$7, F1-score $x_i$8, per-subfield accuracy (using fuzzy match rules for free text and digit tolerances), and overall coverage $x_i$9. Coverage is defined as the proportion of answerable fields attempted—i.e., those for which the model supplies a non-"Unknown" answer.

Key results (using the Phi-4 and Gemini 2.5 Flash backends) are summarized below. Table 1 demonstrates average key information extraction (KIE) errors across 66 fields for various strategies:

Pipeline	Model	Strategies	Avg. Errors
AWS Textract	–	–	25.0
o4-mini	sample only	(\S1)	4.75±1.78
o4-mini	schema only	(\S2)	2.75±0.82
o4-mini	both	(\S1+\S2)	1.25±0.43
Gemini 2.5 Flash	schema only		1.25±0.02
Gemini 2.5 Flash	both	(\S1+\S2)	1.00±0.01

Grouped QA outperforms alternative approaches in operational coverage, reaching 0.95 coverage and 0.67±0.01 accuracy (Phi-4), reflecting a strategic trade-off optimal for timely situational awareness demands.

5. System Architecture and Operational Characteristics

The overall architecture operates in a near-real-time pipeline:

1. Continuous news scraping yields raw HTML.
2. Multiple parsers extract and clean article text.
3. Retrieved articles are indexed against new FRA records.
4. The scanned Form 57 image is parsed as visual regions by a VLM, producing a JSON-based field transcription and grouping assignments.
5. For each group, LLM-based QA extracts field values from relevant articles.
6. Final output is a fully populated incident form draft typically generated in $s$02.5 seconds per incident (on a single V100 GPU and TPU-backed LLM), with major latency attributable to VLM transcription ($s$11 s) and grouped LLM QA ($s$21.2 s).

6. Error Analysis, Limitations, and Prospective Work

Analysis reveals elevated error rates for granular equipment-related fields (“17. Equipment” at 0.56 accuracy, “24. Type of Equipment Consist” at 0.48), attributed to the scarcity of precise FRA code reporting in journalistic sources. Numeric fields (e.g., casualty counts, speeds) exhibit instability due to evolving reporting in live articles, while narrative divergence in free-text entries (notably "Accident Description") introduces hallucination risk.

The system is constrained by the relatively low rate of news reporting: 154 articles were found for 3,962 California records. Extending the input domain to encompass social media or local bulletins could ameliorate this coverage gap. National deployment demands increased source diversity and supports multilingual contexts, and fusing data from multiple articles per incident presents further aggregation and consistency challenges. Incorporating semi-automated, human-in-the-loop verification could reduce propagation of extraction errors prior to official data release (Lim et al., 27 Dec 2025).

7. Summary and Research Implications

The integration of a two-stage VLM/LLM pipeline—featuring robust JSON schema conversion by sample aggregation and layout- as well as semantics-aware grouped QA—provides markedly improved coverage and competitive accuracy for real-time population of Highway-Rail Grade Crossing Incident Data (Form 57) from noisy, open-domain news sources. This research establishes practical methodologies and performance baselines for real-world, high-value structured data extraction under challenging multimodal input regimes, offering a template for future extensions involving larger geographic regions and broader data modalities (Lim et al., 27 Dec 2025).