Frequency-guided Multi-level Reasoning for Scene Graph Generation in Video

Abstract: Video Scene Graph Generation aims to obtain structured semantic representations of objects and their relationships in videos for high-level understanding. However, existing methods still have limitations in handling long-tail distributions. This paper proposes the Frequency-guided Relational Multi-level Reasoning (FReMuRe) model, which enhances the modeling ability of long-tail relationships from a mechanism perspective. We introduce relation-specific branches to deal gradient conflicts, yielding more balanced and tail-aware learning. And we design a frequency-aware dual-branch predicate embedding network to model high-frequency and low-frequency relationships separately and improve the recall rate of tail classes through gated fusion. Meanwhile, we propose two types of interchangeable relation classification heads: Bayesian Head for uncertainty estimation and new Gaussian Mixture Model Head to enhance intra-class diversity. Experimental results show that FReMuRe significantly improves the recall rate of long-tail relationships and overall reasoning robustness on the Action Genome dataset.

• The paper introduces FReMuRe, a frequency-guided, decoupled framework that balances head and tail predicates to mitigate gradient conflicts.
• It employs a dual-branch predicate embedding generator with Bayesian and GMM-Plus heads to robustly model uncertainty and multimodal relation features.
• Experimental results on the Action Genome dataset demonstrate state-of-the-art improvements in mR@K metrics, significantly boosting rare predicate detection.

Frequency-guided Multi-level Reasoning for Scene Graph Generation in Video

Introduction and Motivation

This work addresses the Video Scene Graph Generation (SGG) problem, which aims to produce structured semantic representations of objects and their interactions within video streams. The dynamic and long-tailed nature of relationship distributions in real-world video poses several challenges: dominant head classes bias learning, tail-class relations are poorly represented, and annotation noise and visual ambiguity introduce considerable uncertainty. The FReMuRe model introduced in this paper directly targets these deficiencies by proposing a frequency-guided and decoupled relational reasoning framework.

The motivation for this architectural decision is illustrated by the observation that models with a monolithic backbone suffer from gradient conflicts, leading to diminished performance on rare predicates. By reallocating model capacity to relation-specific pathways, FReMuRe decouples the learning process such that the head-class gradient dominance is mitigated, facilitating balanced and tail-aware learning.

Figure 1: Shared backbones in conventional SGG induce gradient conflicts and bias toward head relations, which FReMuRe addresses via relation-specific branches.

FReMuRe Architecture

FReMuRe is a comprehensive framework that integrates object detection, temporal alignment, local and global relational modules, and dual classification strategies to achieve robust video scene graph reasoning under long-tail distributions. The model design is modular, incorporating three core components: a frequency-guided mechanism, a dual-branch predicate embedding generator, and interchangeable Bayesian or GMM-Plus heads for classification.

Figure 2: Overview of FReMuRe: object detection, temporal alignment, three-way relation decoupling, frequency-guided fusion, and Bayesian/GMM-Plus heads for robust, uncertainty-aware relation reasoning.

Frequency-guided Mechanism

Long-tail priors are encoded by explicitly modeling predicate frequency. The input relation features are adaptively recalibrated using a learnable gate that inverts the statistical prior, amplifying features of tail relations in feature space rather than merely reweighting the loss. This structural debiasing enacts self-information-based modulation, multiplying rare predicate activation—an architectural rather than optimization-level correction.

Dual-branch Predicate Embedding Generator (DPEG)

DPEG decouples the representation pathway for high-frequency and low-frequency predicates via a dual-branch Transformer. For each object pair, one branch specializes in robust representation of head predicates, while the alternative branch, aided by a frequency-aware gating mechanism, models rare tail predicates. The final representation fuses these branches, stabilizing learning through careful initialization and LayerNorm, minimizing destructive interference between gradients associated with divergent relation subpopulations.

Classification Heads: Bayesian and GMM-Plus

To counteract annotation uncertainty and further enhance tail-class modeling:

• Bayesian Head: Outputs mean and variance for each relation, using Monte Carlo sampling over the learned distribution to estimate probabilities, capturing aleatoric and epistemic uncertainties.
• GMM-Plus Head: Extends standard GMMs to the relation classification context, implementing multiple components per relation, with variance regularization explicitly preventing collapse during few-shot learning. Frequency-aware regularization allows robust multi-modal modeling, critical for rare or visually ambiguous relations.

Experimental Evaluation

FReMuRe was evaluated on the Action Genome dataset, which presents a challenging benchmark due to its exhaustive object and relation annotation and severe long-tail distribution for predicates. Multiple tasks were considered: predicate classification (PREDCLS), scene graph classification (SGCLS), and full scene graph detection (SGDET).

Quantitative Results: FReMuRe sets new benchmarks in mean Recall@K (mR@K), the principal metric for unbiased long-tail evaluation. On PREDCLS, SGCLS, and SGDET, FReMuRe, especially with the GMM-Plus head, consistently surpassed prior methods, notably TEMPURA, STTran-TPI, and FloCoDe, establishing a new state-of-the-art for both overall and tail-class recall. Notably, the SGDET mR@50 increased to 22.5 with GMM-Plus, representing a significant boost in capturing diverse, rare relationships.

Qualitative Analysis: The generated scene graphs from FReMuRe exhibit a higher fidelity to ground truth, particularly in tail relations. Error analysis indicates fewer misclassifications of rare predicates, supporting the efficacy of the decoupled, frequency-guided approach.

Figure 3: Example scene graphs: ground truth, predictions by FReMuRe, and incorrect relationships highlighted, demonstrating improved tail-class reasoning.

Ablation and Component Analysis

Ablation studies demonstrate the indispensability of each core module. Removal of the dual-branch architectural decoupling, the frequency-guided feature gating, or either classification head each caused dramatic declines in mR@K across all tasks. The largest degradation was observed when eliminating the decoupling mechanism, confirming that the mitigation of feature and gradient interference is pivotal for learning robust representations of tail predicates.

Implications and Future Directions

This research introduces a mechanism-driven approach to video SGG under long-tail constraints, in contrast to prior reliance on loss re-weighting or resampling. By structurally decoupling relation reasoning and infusing frequency priors at the feature and classification levels, FReMuRe establishes theoretical and empirical advances. The Bayesian and GMM-Plus heads further demonstrate that explicit modeling of uncertainty and multi-modality can be synergistically combined with architectural debiasing for robust generalization.

Practically, FReMuRe's capacity for accurate and consistent reasoning over rare relationships enhances its applicability to video understanding pipelines in complex real-world scenarios (e.g., surveillance, robotics, and human-object interaction analysis). Future research directions may include adaptation to extremely imbalanced settings, extension to open-vocabulary predicates, and transfer to other domains such as industrial inspection or medical video analysis. Integration with larger-scale vision-language pretraining frameworks warrants further exploration for generalizable long-tail scene graph reasoning in video.

Conclusion

FReMuRe provides a frequency-guided, multi-level, and modular solution for video SGG under long-tail distributions. Its effectiveness is substantiated through state-of-the-art quantitative results, substantial qualitative fidelity, and robust performance on rare predicates. The architectural insights, particularly the decoupling and frequency-aware feature modulation, offer a principled path for future models aiming to reconcile both common and rare relational representations within video data (2604.17298).