TimeMM: Time-as-Operator Spectral Filtering for Dynamic Multimodal Recommendation

Abstract: Multimodal recommendation improves user modeling by integrating collaborative signals with heterogeneous item content. In real applications, user interests evolve over time and exhibit nonstationary dynamics, where different preference factors change at different rates. This challenge is amplified in multimodal settings because visual and textual cues can dominate decisions under different temporal regimes. Despite strong progress, most multimodal recommenders still rely on static interaction graphs or coarse temporal heuristics, which limits their ability to model continuous preference evolution with fine-grained temporal adaptation. To address these limitations, we propose TimeMM, a time-conditioned spectral filtering framework for dynamic multimodal recommendation. TimeMM instantiates Time-as-Operator by mapping interaction recency to a family of parametric temporal kernels that reweight edges on the user--item graph, producing component-specific representations without explicit eigendecomposition. To capture non-stationary interests, we introduce Adaptive Spectral Filtering that mixes the operator bank according to temporal context, yielding prediction-specific effective spectral responses. To account for modality-specific temporal sensitivity, we further propose Spectral-Aware Modality Routing that calibrates visual and textual contributions conditioned on the same temporal context. Finally, a ranking-space Spectral Diversity Regularization encourages complementary expert behaviors and prevents filter-bank collapse. Extensive experiments on real-world benchmarks demonstrate that TimeMM consistently outperforms state-of-the-art multimodal recommenders while maintaining linear-time scalability.

• The paper introduces a time-conditioned spectral filtering framework that dynamically adjusts graph operators to capture user interest drift.
• It employs adaptive multi-scale operator mixtures and spectral-aware modality routing, achieving 3–10% performance gains over state-of-the-art benchmarks.
• Empirical and ablation studies confirm that leveraging temporal structure and operator diversity is critical for robust, scalable recommendations.

TimeMM: A Time-Conditioned Spectral Approach for Dynamic Multimodal Recommendation

Problem Setting and Motivation

Multimodal recommender systems integrate collaborative signals with item content from heterogeneous modalities (e.g., text, vision) to mitigate interaction sparsity and improve preference understanding. However, classical multimodal recommenders generally operate on static user–item graphs, resulting in temporal agnosticism that fails to model user interest drift and shifting modality relevance. User behaviors are non-stationary: visual preferences may shift rapidly with trends while semantic interests evolve more slowly. This motivates temporal dynamics not as an auxiliary side feature but as a primary axis modulating graph propagation.

Prior work on spectral graph modeling enables frequency-aware preference decomposition, separating transient high-frequency from enduring low-frequency signals. However, these methods typically rely on explicit global spectral decompositions over static graphs. This makes them ineffective for non-stationary and fine-grained temporal adaptation and introduces computational bottlenecks. There exists also a strong modality–time entanglement: modality relevance is regime-dependent (e.g., visuals dominate in recency-driven trends).

TimeMM Framework and Technical Contributions

TimeMM addresses the core limitations of previous approaches with a dynamic, time-conditioned spectral filtering framework for multimodal recommendation. The architecture is summarized below:

Figure 1: Architecture of TimeMM featuring a temporal-kernel-weighted operator bank, adaptive spectral filtering, and spectral-aware modality routing with a diversity regularizer.

Time-as-Operator Spectral Filtering: The foundation is replacing the static propagation operator with a temporal-kernel-weighted bank. For each user–item edge, a recency-aware temporal kernel (logistic-exponential decay) defines edge weights parameterized by a temporal scale $\tau$. By discretizing $\tau$ to obtain a spectrum of propagation scales, TimeMM efficiently instantiates a family of graph operators, each corresponding to a spectral filter emphasizing different temporal ranges. Unlike explicit eigendecomposition methods, this architecture has linear time and memory complexity and is directly compatible with sparse graph computation.

Adaptive Spectral Filtering: Static spectral filters are suboptimal in the presence of non-stationary, heterogeneous user trajectories. TimeMM learns to adaptively fuse the multi-scale operator outputs using context-conditioned mixture weights informed by user-specific and item-specific temporal patterns (e.g., activity span, recency histograms). This selects the appropriate spectral expert for each user–item prediction.

Spectral-Aware Modality Routing: Given that modality relevance is context-dependent, TimeMM jointly learns modality fusion weights conditioned on each user’s temporal state and the active operator bank, enabling dynamic calibration of visual and textual contributions per decision.

Spectral Diversity Regularization: Without explicit constraints, multi-scale operators and expert mixtures are susceptible to degeneracy (filter collapse). TimeMM regularizes the ranking-space preference margins between spectral experts, enforcing behavioral diversity and preserving the interpretability and utility of the expert bank.

Empirical Evaluation and Core Findings

TimeMM is evaluated on four established Amazon benchmarks and a large-scale industrial dataset. It is compared with leading interaction-only models, strong multimodal graph baselines (MMGCN, DualGNN, MMIL), advanced alignment and intention-aware recommenders (AlignRec, SLMRec), and static spectral models (SMORE, FITMM). All evaluation employs leave-one-out temporal splits and negative sampling.

Key Results:

• TimeMM consistently achieves the top performance across all datasets and all metrics (Recall@K, NDCG@K), typically exhibiting relative gains of 3–10% over the strongest baseline in each setting. The improvements generalize from academic data to industry-scale distributions.
• Direct ablation confirms that removing the Time-as-Operator (TO) kernel bank is most detrimental, with notable drops in Recall@20 and NDCG@20. Multi-scale (MS) and gated fusion (GF) are also critical, while spectral diversity regularization (DR) provides a smaller but measurable boost.

  Figure 2: Ablation results on three datasets quantifying performance drops for each principal module removal, highlighting the dominant contribution from time-conditioned operator kernels (TO).

• Timestamp perturbation analysis demonstrates that destroying temporal order (shuffling) substantially reduces performance, confirming that TimeMM’s advantage derives from exploiting meaningful time structure rather than spurious correlations.
• Analysis by user temporal-span shows benefits are maximal for long-span users: as user activity histories become more temporally diverse, the adaptive operator mixture substantially outperforms static approaches, confirming TimeMM’s advantage in capturing drift and nonstationary regimes.

Operator Bank Behavior and Interpretability

TimeMM's operator bank learns a monotonic, interpretable short-to-long smoothing hierarchy, as demonstrated by energy-decay diagnostics. For each user–positive-item pair, the multi-scale distance consistently decreases from short- to long-scale operators, with the clearest monotonicity in long-span user regimes. This supports that operators do not collapse but specialize across the temporal spectrum.

Figure 3: Energy-decay profiles reveal systematic attenuation as operator scale increases, with sharpest separation for long-span users.

At the user level, operator-mixing weights exhibit a structured, non-uniform distribution. Most users allocate the dominant mass to short-horizon or long-horizon experts, with mid-horizon often acting as a prior. Mixture selectivity is neither degenerate (hard assignment) nor naive (uniform averaging), enabling user-adaptive filter selection.

Figure 4: Empirical distribution of fusion weights indicates structured and soft selectivity in operator mixture allocations across users.

Spectral-Aware Modality Fusion and Practical Insights

TimeMM captures modality–span correlations. Empirical routing weights demonstrate that users with short temporal spans upweight vision channels, while long-span users shift preference to semantic (text) signals. The model automatically learns these patterns in a data-driven way, reflecting the theoretical expectation that visual cues are more relevant to transient, recency-driven preferences, while textual signals are more appropriate over extended histories.

Figure 5: The learned modality mixtures show monotonic decay in vision and an increase in text as user span increases, with the ID channel being stable.

Theoretical and Practical Implications

TimeMM formalizes a polynomial spectral filtering perspective in which time-conditioned kernels modulate the underlying graph geometry, acting as low-pass filters over varying temporal supports. Adaptive mixture weights select the effective smoothing regime per prediction. This avoids explicit SVD or eigendecomposition and is amenable to industrial-scale deployment. Practically, TimeMM achieves strong generalization and robustness across both benchmark and real-world scenarios, with empirical evidence for interpretability and adaptation to user heterogeneity.

Theoretically, TimeMM bridges dynamic graph learning and multimodal temporal integration, suggesting applications beyond recommender systems, including dynamic knowledge graphs, social network behavior analysis, and evolving user modeling in LLM-driven interfaces. The modular design enables future integration with agentic architectures, meta-learning, and self-supervised objectives.

Conclusion

TimeMM introduces a time-conditioned, spectral filtering methodology for dynamic multimodal recommendation that integrates multi-scale temporal modeling, adaptive operator mixtures, and spectral-aware modality routing with diversity regularization. The approach yields state-of-the-art performance on both academic and industrial datasets, robustly models user drift, and produces interpretable operator and modality adaptation. The framework establishes a scalable paradigm for temporal graph learning suited for broader applications in nonstationary, heterogeneous user–item domains (2604.26247).