- The paper introduces HATS, a traffic scene graph generation framework that selects ego-path-relevant entities and predicts each hazard’s mechanism, side, and severity using panoptic perception, depth, and accident-knowledge priors.
- HATS substantially outperforms generic scene-graph baselines, reaching 54.87 mAP@3 for hazard prioritization, 62.79 R@50 and 76.13 mR@50 on SGDet, and 95.03% F1 for entity relevance classification.
- The paper finds that ego-path entity selection is essential, with its removal causing over-40-point R@1 drops, while knowledge-graph and depth cues each improve reasoning about visually ambiguous collision risks.
The paper addresses a deficiency in scene graph generation (SGG) for driving: existing methods either model generic spatial predicates over foreground objects or panoptic relations over all scene entities, neither of which captures safety-relevant, ego-centric interactions. The authors formalize a new task, Traffic Scene Graph Generation (TSGG), in which a model must identify entities relevant to the ego path, and for each prominent hazard predict three ego-centric attributes: the effect mechanism (8 classes, e.g., sideswipe, head-on, cross-traffic conflict), the relative side (left/front/right), and a severity level (4 levels). The output graph color-codes entities by severity and annotates each hazard with its predicted mechanism and relative location, providing an immediately interpretable attention-allocation cue for drivers or downstream planners.
The HATS framework
The proposed HATS model has two branches. The main scene graph branch comprises three modules. First, a panoptic segmentation (PS) module based on a ResNet50 Mask2Former produces non-overlapping instance masks for all entities. Second, an Ego-path Related Entities Selection (ERES) module extracts a path representation from the mask-feature map via masked pooling, then applies multi-head cross-attention with the path token as query and entity embeddings as keys/values. Relevance logits are predicted from a fused feature combining entity content, a path-conditioned context summary, and a gated path signal, and only relevant candidates proceed to relation prediction. Third, the TSGG module builds a gated pair descriptor per ego–entity pair that fuses six representations: ego-path, ego-vehicle, visual (RGB plus disparity), semantic, geometric (path overlap and proximity), and knowledge graph embedding (KGE). Dedicated heads classify mechanism (via cosine similarity to mechanism prototypes in the KGE space with a learned temperature), side, and severity, where severity additionally attends over seven severity-related KG node groups (CAIS, VAIS, MAIS, DAMSEV, CONSEQ, TREATMENT, ROLLINITYP) as structured priors.
Traffic accident knowledge graph and embedding
The auxiliary branch constructs a traffic accident knowledge graph from NHTSA field crash investigation data through a four-stage pipeline: node extraction (16,039 nodes across 26 types), schema-driven relation wiring (122,263 edges across 25 types), causal enrichment (31,104 causal edges with provenance properties distinguishing actor/victim reports), and bridging-node alignment to Cityscapes categories and the mechanism taxonomy. The resulting graph contains 153,488 triplets, split 8:1:1 for KGE training.
The KGE module extends StarE in three ways: literal-aware node initialization that fuses categorical and numerical node properties with ID embeddings; FiLM-based qualifier modulation, where edge properties (e.g., source=actor) adaptively scale and shift relation-aware messages during message passing; and a transformer-based triplet scorer over a two-token head–relation sequence. Trained with a 1-to-N filtered multi-label objective in both head and tail directions, it substantially outperforms vanilla StarE: object prediction MRR improves from 38.66 to 88.31, subject prediction H@10 reaches 99.80, and triplet prediction H@10 reaches 99.79. These near-ceiling retrieval scores validate the KG as a reliable prior-knowledge source, although the authors do not test the KGE on held-out external benchmarks, so generalization beyond this graph remains unquantified.
Experimental results
Because no suitable benchmark exists for TSGG, the authors annotate 820 Cityscapes images with relation labels and evaluate on 10 tasks spanning 5 perspectives, using 5-fold cross-validation for hyperparameter selection. The learning-curve experiment shows mechanism-classification accuracy rising from 0.437±0.016 at 5% of the training set to 0.684±0.005 at 80%, with shrinking variance; the still-rising trend indicates the annotated dataset is a limiting factor rather than model capacity.
On hazards prioritization (mAP@K, MRR@K, NDCG@K at K=3,5,10), HATS consistently exceeds CFHP, with the largest gains at low K — e.g., mAP@3 of 54.87 versus 31.66 and NDCG@3 of 64.19 versus 37.94 — implying that the most critical hazards are ranked near the top, which matters for time-critical alerting. On SGDet, HATS reports 62.79/76.13 R@50/mR@50 versus 29.68/9.38 for MOTIFS and 29.60/5.23 for VCTREE; the order-of-magnitude mR advantage reflects the explicit removal of the dominant "irrelevant" predicate and the decomposition of prediction into well-defined subtasks, directly countering the long-tail bias the authors identify as disqualifying generic SGG in traffic scenes. On entity relevance classification, HATS attains 96.96% precision, 94.07% recall, F1 of 95.03%, and AUC of 97.20%, versus F1 of at most 61.61% among baselines; on prominence classification it achieves F1 of 90.89% and AUC of 92.17%, exceeding both the SGG baselines and the specialized CFHP method.
Ablation findings
The ablation study yields the paper's strongest structural claim: removing ERES alone causes more than a 40-point drop in R@1 across severity, side, and mechanism prediction, and HATS w/o ERES+KGE+Depth performs nearly identically to HATS w/o ERES. This indicates ERES is a prerequisite for the other components to contribute — without it, irrelevant panoptic queries (sky, distant parked cars) flood the TSGG heads with unstructured context regardless of feature richness. Removing KGE or depth cues each costs roughly 30–38 points of R@1 on severity and mechanism, confirming that structured accident priors resolve visually ambiguous cases (e.g., an oncoming vehicle in the opposite lane implying head-on risk) and that disparity-derived geometry resolves lateral-offset ambiguity (adjacent-lane sidewipe versus a parked car behind a barrier). A notable negative interaction is reported: HATS w/o KGE+Depth slightly underperforms HATS w/o Depth, suggesting KGE priors without geometric grounding inject conflicting semantic bias — evidence that the components are jointly, not independently, necessary.
Limitations and open questions
The paper concedes several constraints. Evaluation rests on 820 annotated images from Cityscapes, a single urban-driving dataset, and the learning curve shows performance has not saturated, so absolute numbers should be read as lower bounds dependent on annotation scale. The KGE is validated only intrinsically on the constructed graph. The disparity branch assumes depth availability at inference, and the mechanism taxonomy is fixed at eight classes derived from NHTSA coding, which may not cover all interaction types. The authors leave open extension to video-based temporal reasoning, stronger segmentation backbones, and broader driving environments; whether the accident-prior mechanism transfers across geographic regions with different crash statistics is not examined.
Conclusion
HATS formulates TSGG as an ego-centric hazard-reasoning task and demonstrates that combining panoptic perception, path-aware candidate filtering, multi-cue pair descriptors, and KGE-grounded accident priors yields large gains over generic SGG and specialized hazards-prioritization baselines across prioritization, classification, and graph-detection metrics. The ablation evidence that candidate filtering gates the utility of knowledge and depth cues is arguably the paper's most transferable finding for multi-component scene-understanding architectures.