Content-Driven Micro-Video Recommendation

• Content-driven micro-video recommendation is a domain that uses multimodal analysis and user behavior to tailor short video recommendations.
• It leverages techniques such as explicit multimodal embeddings, segment-based attention, and hybrid models to address data sparsity and cold-start issues.
• Real-world implementations show improved ranking metrics and explainability, offering scalable, real-time solutions for video recommendation.

Content-driven micro-video recommendation refers to computational methods for ranking or selecting short-form videos for users based on the intrinsic content of the videos and, increasingly, the joint modeling of user interests, contextual factors, and behavioral feedback. Approaches have evolved from leveraging explicit multimodal representations—such as visual, audio, and textual features—to architectures that implicitly distill content signals from user engagement patterns at various granularities. The focus on micro-videos (typically <60 seconds, seen on platforms like TikTok and Kuaishou) introduces challenges and opportunities distinct from conventional long-form video recommendation, including extreme data sparsity, high interaction velocity, and the need for real-time, explainable, and scalable models.

1. Content Modeling Paradigms

Content modeling in micro-video recommendation is dominated by three central paradigms: explicit multimodal embedding, implicit content inference from aggregated user behavior, and hybridization with collaborative filtering signals.

• Explicit Multimodal Embedding involves extracting handcrafted or pre-trained features from modalities such as video frames, audio tracks, and text metadata. For instance, in MMGCN, pre-extracted ResNet features (visual), BERT/Word2Vec (textual), and audio CNNs (acoustic) are combined via modality-specific graph convolutional networks, and then fused to form holistic item representations (Najafabadi, 29 Jun 2025).
• Implicit Content Modeling via User Feedback is exemplified by the Segment Content Aware Model via User Engagement Feedback (SCAM): the model never observes video pixels or text directly, but instead divides each video into duration-based segments and represents each segment via historic user engagement metrics—such as average watch ratio or likes—which are then embedded and processed by a Transformer-like architecture. The segment sequence is contextually self-attended, with the resulting hidden states used to infer fine-grained content signals (Feng et al., 2 Apr 2025).
• Unified Multimodal Space approaches, such as DreamUMM, project both user histories and item multimodal features into a shared latent space $\mathcal{Z}$, motivated by the Platonic Representation Hypothesis (cross-modality convergence). User interests are modeled as points in this space by aggregating (with preference-weighting) the multimodal embeddings of historically engaged videos (Lin et al., 2024).
• Hybrid Models such as MHCR, MicroLens-E2E, and concept-aware GNNs (e.g., CONDE) explicitly fuse both collaborative and content-driven signals, with strategies like hypergraph-based message passing, contrastive alignment across views/modalities, and multi-phase representation refinement to combat over-smoothing and exploit content where collaborative signals are insufficient (Lyu et al., 2024, Ni et al., 2023, Liu et al., 2021).

2. Model Architectures and Learning Objectives

A variety of architectural motifs have emerged:

• Segment-Based Attention Networks: SCAM models the user's browsing process over temporal video segments. Each segment $m_i$ is encoded as

$$\mathbf{s}_i = \mathbf{E}_{\text{pos}}[i] + \mathbf{E}_{\text{vid}}[v] + \mathbf{E}_{\text{usr}}[u] + \mathbf{E}_{\text{fb}}[\text{feedback}_i] + \mathbf{p}_i,$$

where $\mathbf{p}_i$ is (learned or sinusoidal) positional encoding. A multi-layer, multi-head self-attention stack computes

$$\alpha_{ij} = \frac{\exp(\mathbf{h}_i^\top \mathbf{h}_j / \sqrt{d})}{\sum_{k=1}^M \exp(\mathbf{h}_i^\top \mathbf{h}_k / \sqrt{d})},\quad \mathbf{z}_i = \sum_j \alpha_{ij} \mathbf{h}_j.$$

The final watch time is reconstructed as $\hat T = \sum_i p_i d_i$ with a composite loss including cross-entropy, Huber, and ordinal regularization enforcing monotonic decay of $p_i$ (Feng et al., 2 Apr 2025).

• Tripartite/Hypergraph GNNs: CONDE builds a tripartite user–video–concept graph, successively propagating concepts $\to$ videos $\to$ users and back, followed by user-centric neighborhood denoising and preference refinement phases (Liu et al., 2021). MHCR constructs a multi-view system—user–item connection (LightGCN), KNN item-item graphs per modality, and high-order hypergraphs for modality-specific clustering of users/items—while enforcing InfoNCE contrastive alignment between modalities and between graph/hypergraph embeddings (Lyu et al., 2024).
• Shared Latent Space and Product-of-Experts Fusion: In content-matching recommendation (e.g., micro-video $\to$ music), the CMVAE model learns modality-specific Gaussian posteriors for visual and textual descriptions, fusing them via a product-of-experts mechanism

$$\mu_v = \frac{\mu_{v_v}/\sigma_{v_v}^2 + \mu_{v_t}/\sigma_{v_t}^2}{1/\sigma_{v_v}^2 + 1/\sigma_{v_t}^2}$$

to form robust video latent variables, cross-generating video $\leftrightarrow$ music and ranking pairs with a bi-directional hinge loss (Yi et al., 2021).

• Content-Centric Knowledge Graphs with Attention: Advanced models build heterogeneous, relation-aware KGs encompassing users, videos, multimodal attributes, and user-side auxiliary entities (demographics, tags). TransR-style relation-specific projections, neighbor- and layer-level attention, and multi-layer message passing yield user and item representations for dot-product scoring, with joint KG loss and BPR ranking (Lim et al., 2022).

3. Datasets, Benchmarks, and Evaluation Metrics

Progress in content-driven micro-video recommendation is closely tied to the availability of large-scale, multi-modal benchmarks.

• MicroLens provides up to 1 billion user–item interactions over 1 million+ micro-videos, with raw text, cover images, audio, and full-length video modalities. The dataset supports both collaborative and content-driven models, facilitating item cold-start and modality ablation studies (Ni et al., 2023).
• TT-150k contains approximately 3,000 background music clips and 150,000 micro-videos, with labels derived from actual TikTok user assignments, enabling rigorous content-based video–music matching evaluation (Yi et al., 2021).
• Key metrics include HR@K, NDCG@K for ranking; MAE and XAUC for watch time regression (Feng et al., 2 Apr 2025); AUC, NDCG@5, MAP@5 for tripartite GNNs (Liu et al., 2021); and Recall@K for retrieval (with popularity debiasing in content-recommendation settings).

Empirical findings demonstrate the necessity for end-to-end content learning: frozen visual features alone underperform compared to models that fine-tune video encoders with user–item signals (e.g., VideoMAE with SASRec). Content-driven methods often yield superior relative gains for cold-start items or users, with up to 3× HR@10 improvement in the lowest-popularity decile (Ni et al., 2023, Lyu et al., 2024).

4. Addressing Duration, Cold-Start, Sparsity, and Explainability

• Duration Bias and Sequential Dependence: By reconstructing total watch time via segment-level continuation probabilities ($\sum_i p_i d_i$), SCAM avoids the spurious association between video length and engagement, as the model cannot maximize $\hat T$ merely by preferring longer videos. Self-attention enables non-autoregressive, global-context inference, overcoming the exposure bias of sequential models (Feng et al., 2 Apr 2025).
• Cold-Start Mitigation: MHCR counters interaction sparsity and cold-start with multi-view contrastive self-supervision and hypergraph augmentation, so that content-alone (e.g., text/image/video encoders) provides meaningful embeddings in the absence of user history. Quantitatively, it improves recall by up to 5% in the lowest-history cohort over leading multimodal baselines (Lyu et al., 2024).
• Long-Tail and Semantic Explainability: Models leveraging explicit textual/concept graphs (e.g., CONDE) improve long-tail recommendation by propagating semantic attributes from head to tail items (propagating concept nodes across low-frequency items), with AUC uplift from ~0.65 to 0.69 for rare items (Liu et al., 2021). Attention weights in segment-based models provide a mechanism for content tagging or key moment extraction (Feng et al., 2 Apr 2025).
• Explainable and Efficient Serving: Real-time user embedding construction via closed-form weighted sums (DreamUMM) enables efficient serving architecture—achieving 4 ms per request on commodity hardware, and supporting high-QPS deployments (Lin et al., 2024).

5. Extensions, Variants, and Open Challenges

Current and prospective extensions include:

• Multimodal and Cross-Modal Fusion: Enriching segment embeddings with Video-LLaMA/CLIP features within the SCAM framework, or explicitly incorporating audio, speech, and OCR-extracted metadata as concepts or hyperedges in GNN-based models (Feng et al., 2 Apr 2025, Liu et al., 2021, Lyu et al., 2024).
• Personalization Dynamics: Modeling the temporal evolution of user interest, e.g., by stacking user-level RNNs over segment Transformers or learning user drift in the multimodal latent space (Feng et al., 2 Apr 2025, Lin et al., 2024).
• Hierarchical and Multitask Objectives: Defining coarse-to-fine hierarchies over segments or scenes, and introducing auxiliary objectives such as like/comment prediction alongside watch time (Feng et al., 2 Apr 2025).
• Scalability and Adaptation: Employing neighbor-sampling or scalable graph processing for million-node graphs, and emphasizing continual learning for dynamic update in high-velocity data regimes (Lim et al., 2022, Najafabadi, 29 Jun 2025).
• Open Problems: Lightweight on-device content encoders remain a bottleneck for low-latency serving (Ni et al., 2023). Transfer to cross-domain settings—i.e., deploying pretrained video recommenders across platforms—is an ongoing challenge. With foundation recommenders, there is interest in training with universal content+interaction pretraining corpora for broad generalization (Ni et al., 2023).

6. Comparative Performance and Representative Results

Empirical benchmarks highlight the impact of content-driven strategies:

Model / Dataset	Key Metric	Baseline	Content Model	Relative Gain
SCAM / KuaiRec	MAE	D2Q: 4.8880	SCAM: 4.4906	↓8.1%
	XAUC	D2Q: 0.5874	SCAM: 0.6318	+4.4 pts
CONDE / Micro-Video	AUC	GAT: 0.7345	CONDE: 0.7952	+6.1%
MMGCN / Kwai	F1	XGBoost: 0.368	MMGCN: 0.574	+55.9%
MHCR / MicroLens-100K	Recall@10	MGCN: 0.0717	MHCR: 0.0798	+11.3%
DreamUMM / A/B (Play)	Play count ↑	-	+0.273–0.867%	p<0.05/0.01

Notes:

• End-to-end content models consistently outperform frozen-feature or ID-only models, especially in cold-start regimes (Ni et al., 2023, Lyu et al., 2024).
• Hybrid losses (binary CE + pairwise ranking + regularization) are the standard; segment ordering loss further improves monotonicity in watch probabilities (Feng et al., 2 Apr 2025).
• Incorporating user engagement feedback as a content proxy (SCAM) achieves scalable and parallelizable inference (Feng et al., 2 Apr 2025).

7. Synthesis and Prospects

Content-driven micro-video recommendation systems integrate rich multi-modal item features and user-centric behavioral signals, deploying advanced architectures such as attention-based segment models, concept-enhanced tripartite GNNs, and hypergraph-contrastive fusions. These systems are empirically validated at scale, with particular efficacy in addressing cold-start, sparsity, and explainability requirements. The state of the art continues to evolve toward compositional, end-to-end, and self-supervised learning architectures that leverage both explicit and implicit content signals, with scalability and robustness paramount for real-world deployment (Feng et al., 2 Apr 2025, Lin et al., 2024, Ni et al., 2023, Lyu et al., 2024, Liu et al., 2021, Yi et al., 2021, Najafabadi, 29 Jun 2025, Liu, 2022, Lim et al., 2022).