- The paper introduces a novel low-light crowd counting approach by recasting counting as feature-level reflectance re-calibration using multi-modal hyper-graph fusion.
- It leverages complementary RGB, depth, and edge data through MMHGF and deformable rectangular sparse attention to enhance feature reliability under severe illumination degradation.
- Empirical evaluations on synthetic and real benchmarks demonstrate significant MAE/MSE reductions, confirming its robustness in challenging low-light environments.
Multi-Modal Hyper-Graph Fusion for Robust Low-Light Crowd Counting
Problem Analysis and Motivation
Crowd counting in visually degraded, low-light environments has been largely understudied relative to mainstream, well-illuminated scenarios. Conventional approaches relying solely on RGB modalities or sequential enhance-then-count pipelines suffer from severe underexposure, sensor noise, loss of contrast, and structural detail—all resulting in unreliable predictions and amplified errors across stages. The underlying issue is not merely restoring the pixel-level appearance but recovering robust, illumination-invariant crowd representations at the feature level. Addressing this, the paper introduces a paradigm shift anchored in Retinex theory, where crowd counting is reframed as a reflectance re-calibration problem utilizing geometrically and structurally invariant priors.
Benchmark Construction
Recognizing the deficiency of dedicated low-light benchmarks, the paper proposes three datasets: SHA_Dark and SHB_Dark (synthetic), created by controlled gamma transformation and exposure reduction of the standard ShanghaiTech datasets [45], and LC-Crowd (real-world), comprising diverse night-time crowd images with substantial scale and illumination variability. Each sample includes RGB, depth (estimated via Depth Anything V2 [43]), Canny edge maps, and head-point annotations. The benchmarks facilitate standardized evaluation and enable systematic exploration of cross-modal fusion strategies in severe low-light conditions.
Methodological Contributions
Reflectance Re-Calibration via Multi-Modal Fusion
The approach operationalizes Retinex theory in the feature domain. Under severe low-light, the RGB feature's illumination-related component is degraded, weakening overall representation. Depth and edge modalities, which are less sensitive to illumination, act as complementary anchors. The feature representation is enhanced by incorporating these auxiliary signals, yielding more reliable cues for dense prediction.
Multi-Modal Hyper-Graph Fusion Module
Conventional pairwise graph fusion and self-attention are vulnerable to local cue degradation under darkness. The Multi-Modal Hyper-Graph Fusion (MMHGF) module formalizes RGB, depth, and edge features as nodes in a hyper-graph, enabling group-wise aggregation across mutually consistent neighbors via dynamically constructed hyperedges. Each spatial token aggregates top-k similarity neighbors within foreground/background groupings, tolerating local failure and maximizing cross-modal calibration. This high-order reasoning robustly refines crowd representations beyond fragile pairwise fusion.
Foreground crowd information is inherently sparse in nighttime scenes, rendering dense attention computation wasteful and noise-prone. The Deformable Rectangular Sparse Attention (DRSA) module adaptively selects informative anchors in rectangular windows via granularity filtering and applies deformable attention focused only on those anchors. Sparse, data-dependent sampling is performed via bilinear grid sampling, and attention aggregation eschews explicit query-key similarity matrices. The decoupling of granularity prediction (supervised via ground-truth window counts) from differentiable attention preserves computational adaptivity without sacrificing optimization stability.
LCNet Architecture
The Low-Light Counting Network (LCNet) integrates MMHGF and DRSA into a unified, multi-modal framework. Hierarchical encoding of RGB, depth, and edge modalities is followed by token-level hyper-graph fusion, sparsity-driven attention refinement, and point-based crowd localization. The joint optimization objective encompasses classification, localization, and granularity filtering loss components.
Empirical Evaluation
Quantitative Results
Results across SHA_Dark, SHB_Dark, and LC-Crowd show that LCNet achieves superior performance against established RGB-only and multi-modal baselines, with MAE/MSE reductions (e.g., 4.7%/8.8% vs. second-best on SHA_Dark, and 20.6%/25.0% on SHB_Dark). LCNet outperforms both enhance-then-count pipelines and state-of-the-art multi-modal fusion architectures on RGBT-CC-Dark, demonstrating consistent suppression of large estimation errors and enhanced robustness to illumination degradation.
Ablation Analyses
- Modality Complementarity: RGB-only models exhibit highest errors; sequentially adding depth and edge modalities substantially improves accuracy, and the joint use achieves best MAE and NAE.
- Fusion Strategy: Hyper-graph fusion with foreground/background grouping outperforms simple concatenation, gated fusion, cross-attention, and pairwise GCN.
- Attention Mechanism: DRSA demonstrates lower MAE and MSE than rectangular, deformable, and dilated attention baselines.
- Fusion Position: Value-branch fusion optimizes feature enrichment, while excessive redundancy is avoided by proper placement within the FPN and sparse decoder stages.
Qualitative Analysis
Visualization of results shows accurate localization of head points and reliable crowd estimation in both dense and sparse scenes under diverse low-light conditions. DRSA efficiently allocates anchors to foreground regions, enabling computation concentration and reducing redundancy.
Computational Efficiency
LCNet maintains competitive parameter efficiency relative to baselines (22.59M vs. 20.9M for PET), though current inference speed is limited by non-fused, data-dependent sparse sampling operations. The paper explicitly positions DRSA as computation-adaptive, leaving runtime optimization for future work.
Theoretical and Practical Implications
The paper highlights the necessity of robust multi-modal calibration in visually challenging environments, moving beyond naïve pixel-level enhancement. Hyper-graph fusion is established as a principled mechanism for group-wise aggregation, mitigating the unreliability of any single modality. Sparse, deformable attention paves the way for efficiency in dense prediction tasks under extreme sparsity and noise.
Practically, the method is highly relevant for intelligent surveillance, public safety, and urban management applications in nighttime or poorly lit scenarios, where sensory exclusivity (e.g., thermal, infrared) is impractical. The benchmark suite enables standardized evaluation and promotes further exploration of modality complementarity.
Future Directions
The current runtime bottleneck within sparse sampling calls for kernel-level optimization, potentially leveraging fused GPU primitives for further acceleration. Extension to broader adverse conditions (e.g., weather degradation), integration with hardware-based sensory modalities, and exploration of more adaptive loss formulations are promising avenues. Generalization of hyper-graph fusion to other dense prediction tasks (e.g., segmentation, pose estimation) is theoretically attractive.
Conclusion
By reframing low-light crowd counting as feature-level reflectance re-calibration and operationalizing cross-modal high-order fusion via hyper-graphs, the LCNet architecture demonstrates robust, generalizable performance gains across synthetic and real-world benchmarks. The integration of adaptive sparse attention further enhances computational focus. The paper establishes a foundation for principled multi-modal reasoning under extreme illumination degradation, with substantial implications for real-world deployment and future research in adverse visual environments.
(2606.18566)