Fine-Grained Time-Text Alignment

• Fine-grained time-text semantic alignment is the process of matching detailed temporal media units to specific textual semantics, preserving order, duration, and causality.
• It leverages structured representations such as scene graphs and dense similarity maps to achieve token-level localization and robust performance in tasks like video-text evaluation and speech translation.
• The approach addresses limitations of global similarity methods by enhancing evaluation protocols and training signals across diverse modalities, including cross-lingual settings.

Fine-grained time-text semantic alignment is the problem of matching textual semantics to temporally localized evidence at the granularity required by the task: atomic prompt constraints in generated video, token-by-clip correspondences in temporal language grounding, frame-level speech tokens in code-switch speech translation, or temporally matched demonstrations in cross-lingual in-context learning. Across the recent literature, the defining contrast is with coarse global similarity. In text-to-video evaluation, for example, CLIPScore computes a cosine similarity between a global text embedding and a global video embedding, but it does not inspect entity-level, relation-level, or temporally ordered properties such as whether one event occurs before another or whether a physics-constrained outcome is visually realized (Guan et al., 21 Mar 2025). Fine-grained approaches instead preserve temporal order, duration, concurrency, causality, and compositional structure, and they operationalize alignment with scene graphs, similarity maps, token-level routing, hierarchical preferences, or retrieval spaces specialized for temporal semantics (Guan et al., 21 Mar 2025, Kim et al., 4 Apr 2025, Wang et al., 2022, Han et al., 2022, Gao et al., 9 Nov 2025, Bajpai et al., 2024).

1. Conceptual scope across modalities

The term has converged on a family of closely related problems rather than a single task. In continuous video-text modeling, it denotes the ability to match textual descriptions to the correct portions of a video as events unfold, preserving both temporal order and compositional meaning at a granular level (Kim et al., 4 Apr 2025). In weakly supervised temporal language grounding, it denotes token-by-clip correspondences, where an alignment map $A \in \mathbb{R}^{T \times L}$ measures the semantic relevance between video clip $t$ and token $l$ (Wang et al., 2022). In long-video instructional alignment, it denotes deciding whether a sentence is visually alignable and, if so, localizing its temporal extent through a sentence-time similarity matrix $\hat{A} \in \mathbb{R}^{K \times T}$ (Han et al., 2022). In code-switch speech translation, it denotes aligning frame-level speech tokens to the correct target-side textual semantics despite rapid language alternation (Gao et al., 9 Nov 2025). In cross-lingual temporal reasoning, it denotes aligning a low-resource temporal query with highly compatible in-context exemplars that preserve the same temporal semantics (Bajpai et al., 2024).

Setting	Temporal-text unit	Representative operationalization
Text-to-video evaluation (Guan et al., 21 Mar 2025)	Atomic prompt predicates	Semantic scene graph and atomic yes/no verification
Multi-event video-text models (Kim et al., 4 Apr 2025)	Event-caption sequence	Hierarchical preference over disrupted negatives
Weakly supervised TLG (Wang et al., 2022)	Clip-token relevance	Alignment map $A \in \mathbb{R}^{T \times L}$
Long-video alignment (Han et al., 2022)	Sentence-time similarity	Joint transformer and alignability prediction
Code-switch speech translation (Gao et al., 9 Nov 2025)	Frame/token to target text	Token-level MoE routing and decoder attention
Cross-lingual temporal reasoning (Bajpai et al., 2024)	Query-demonstration compatibility	In-language similarity labels transferred cross-lingually

A unifying property is that temporal semantics are not reduced to a single pooled representation. The aligned units differ by domain—clips, frames, events, sentences, or questions—but the methodological aim is comparable: preserve the temporal microstructure that global embeddings or coarse bag-level objectives tend to erase.

2. Representational primitives and alignment operators

One line of work makes fine-grained alignment explicit through structured decomposition. ETVA parses a prompt into a semantic scene graph using an Element Extractor, a Graph Builder, and a Graph Traverser. Entities, attributes, and relations become typed nodes, with grammar constraints such as attribute $\rightarrow$ entity edges, entity $\leftrightarrow$ relation $\leftrightarrow$ entity connectivity, and disallowed relation-relation or attribute-attribute adjacency. Temporal tokens such as “then,” “after,” and “while” are preserved as labels on relation nodes or edges, enabling downstream questions about order, duration, concurrency, camera motion, and physics-sensitive outcomes. The resulting atomic yes/no questions include existence, action, material, spatial, number, shape, color, camera, physics, and other categories (Guan et al., 21 Mar 2025).

A second line encodes alignment as a dense similarity surface. FSAN constructs a fine-grained cross-modal semantic alignment map by first encoding text tokens and video clips, then applying an Iterative Cross-Modal Interaction Module with bidirectional cross-modal attention, inner-modal self-attention, residual connections, layer normalization, and feed-forward sublayers. After $K=6$ layers, projected and normalized video and text features yield

$$A = \hat{X}\hat{Y}^{\top}, \qquad A_{t,l} = \langle \hat{x}_t, \hat{y}_l \rangle,$$

so that grounding can be performed directly on top of $t$0 rather than through a proposal set (Wang et al., 2022). The same representational principle appears in long-video alignment, where TAN forms a sentence-time similarity matrix from cosine similarities between transformed sentence embeddings and transformed video embeddings after a multimodal transformer over concatenated video and text tokens (Han et al., 2022).

In speech-text alignment under code-switching, the dominant primitive is token-level routing into language-specific semantic subspaces. The MoE speech projector in the code-switch speech translation model receives frame-level representations from a pretrained Whisper encoder, routes each time step to top-$t$1 experts, and produces

$t$2

with experts organized in language-specific groups and top-$t$3 routing set to $t$4 in experiments (Gao et al., 9 Nov 2025). Because routing is per token, language identity can change from one frame sequence to the next without requiring a shared projector to absorb the full cross-lingual semantic gap.

Cross-lingual temporal reasoning uses yet another operator: similarity transfer. CLiTSSA constructs cross-language query pairs labeled by expected in-language semantic similarity, where the label is computed in the low-resource monolingual space after translating the resource-rich query. The retriever is then fine-tuned so that direct cross-lingual retrieval reflects the temporal compatibility encoded by those in-language labels (Bajpai et al., 2024). This does not introduce a separate temporal module; time sensitivity is encoded through temporal tasks and temporal similarity labels.

3. Supervision regimes and optimization strategies

The literature spans weak supervision, noisy supervision, preference learning, multi-stage curricula, and pure inference-time evaluation.

In weakly supervised temporal language grounding, FSAN combines a triplet-style hinge loss on video-sentence compatibility with two alignment-map regularizers. The video-level matching loss encourages a positive sentence to score above a randomly sampled negative sentence. The inner-sample loss enhances per-token alignment inside likely positive segments, while the outer-sample loss suppresses complementary clips by penalizing their weakest token alignment. The total objective is

$t$5

with $t$6 in the reported configuration (Wang et al., 2022).

In long-video instructional alignment, TAN begins with an InfoNCE-like temporal correspondence loss based on noisy ASR spans, then applies a co-training stage in which a multimodal transformer and a dual encoder provide complementary similarity matrices. Pseudo-timestamps are inferred by sliding-window argmax, updated by IoU-based agreement, and converted into pseudo alignability labels by thresholding an agreement-weighted score $t$7. An EMA with momentum $t$8 stabilizes pseudo-label generation, and the Stage 2 objective is $t$9 (Han et al., 2022).

In multi-event video-text models, VideoComp combines standard dual-encoder contrastive alignment with a hierarchical pairwise preference loss. The contrastive term is symmetric InfoNCE over video-to-text and text-to-video retrieval. The preference term enforces that the positive pair outranks disrupted negatives and that less-disrupted negatives outrank more-disrupted negatives. In the paper’s implementation, the hinge uses zero margin, the weights are uniform, and the total fine-tuning objective is

$l$0

with $l$1 set to $l$2 in fine-tuning for ActivityNet-Comp (Kim et al., 4 Apr 2025). Pretraining extends the same logic to concatenated short video-caption pairs, with stack size $l$3 giving the strongest gains over $l$4.

In code-switch speech translation, the central strategy is staged specialization. Stage 1 performs per-language ASR alignment pretraining with sequence-to-sequence cross-entropy and no CTC. Stage 2 initializes language-specific expert groups from the Stage-1 projectors and adds a language-specific routing loss and an intra-group load balancing loss. Stage 3 transitions from ASR to monolingual speech translation with

$l$5

while keeping routing regularization active. Stage 4 transitions from monolingual ST to CS-ST with

$l$6

and omits the routing regularizers because token-level language labels are unavailable for CS audio (Gao et al., 9 Nov 2025).

ETVA represents a contrasting design point: it is purely inference-time and model-agnostic. No training is required. A multi-agent system decomposes a prompt into atomic verification questions, an auxiliary LLM externalizes commonsense knowledge relevant to those questions, and a video LLM answers them through video understanding, general reflection, and conclusion stages. Each answer is binary, and the final score is the mean of the binary outcomes:

$l$7

This makes evaluation itself an alignment pipeline (Guan et al., 21 Mar 2025).

4. Benchmarks and evaluation protocols

The recent literature has placed unusual emphasis on evaluation design, largely because coarse metrics often hide temporal failure modes.

ETVA introduces ETVABench, a benchmark specifically designed for text-to-video alignment evaluation, with 2,000 prompts and 12,000 atomic yes/no questions across 10 categories: existence, action, material, spatial, number, shape, color, camera, physics, and other. Temporal-relations coverage is ensured by including 510 human motion prompts from GAIA, 160 physics-aware prompts from VideoPhy and PhyGenBench, and 100 temporal coherence prompts from ChronoMagic-Bench. A compact ETVABench-105 subset preserves the category distribution for resource-constrained evaluation. Five human annotators labeled 1,575 model-generated videos on ETVABench-105 using a 0–5 Likert text-alignment score with 0.5 increments and binary answers to atomic questions, with majority voting used for ground truth. Automatic metrics are evaluated with Kendall’s $l$8 and Spearman’s $l$9 (Guan et al., 21 Mar 2025).

VideoComp introduces two compositional benchmarks derived from dense video captioning corpora: ActivityNet-Comp and YouCook2-Comp. Positives are constructed by sorting event captions chronologically and concatenating them into a single paragraph after LLM-temporal structuring. Negatives are produced by temporal reordering, action word replacement, segment-level mismatch, and combined disruptions. Automatic validation requires token-level precision and recall of at least $\hat{A} \in \mathbb{R}^{K \times T}$0 against the original. The benchmark contains approximately 17k triplets for ActivityNet-Comp and 4.5k for YouCook2-Comp; the detailed counts include 16,080 ActivityNet-Comp training triplets and 835 validation triplets, and 3,356 YouCook2-Comp training triplets and 1,167 validation triplets (Kim et al., 4 Apr 2025).

FSAN and TAN are evaluated on more established temporal grounding resources. FSAN reports results on ActivityNet-Captions and DiDeMo. ActivityNet-Captions uses Recall@k at IoU $\hat{A} \in \mathbb{R}^{K \times T}$1, while DiDeMo uses Rank@1, Rank@5, and mIoU (Wang et al., 2022). TAN introduces HTM-Align, a manually curated 10-hour subset of HowTo100M containing 80 videos with sparse temporal descriptions. In HTM-Align, 49K sentences were inspected and 13K were aligned; the evaluation uses Recall@1 in a pointing-game formulation and alignability ROC-AUC (Han et al., 2022).

For speech and multilingual reasoning, benchmarks reflect different temporal granularities. The code-switch speech translation work uses Common Voice for ASR in English, Spanish, and Chinese, Fisher for English-Spanish CS-ST to English, and NTUML2021 for Chinese-English CS-ST to English, with BLEU and COMET for ST and WER for the preliminary study (Gao et al., 9 Nov 2025). CLiTSSA introduces mTEMPREASON by translating TEMPREASON into Romanian, German, and French, with three task levels: L1 (time-time), L2 (time-event), and L3 (event-event). The dataset spans time ranges from 1014 to 2023 and includes 400,000/4,000/4,000 train/dev/test examples for L1, 16,017/5,521/5,397 for L2, and 13,014/4,437/4,426 for L3. Translation Success Rate is 99.00% for Romanian, 97.57% for German, and 97.79% for French; Back-Translation BLEU-3 is 51.12, 56.47, and 43.63, respectively (Bajpai et al., 2024).

5. Empirical findings and recurrent failure modes

The strongest quantitative evidence for the value of fine-grained alignment comes from evaluation correlation and targeted ablations. ETVA achieves a Spearman’s correlation coefficient of 58.47 on ETVABench-105, with Kendall’s $\hat{A} \in \mathbb{R}^{K \times T}$2 of 47.16, while the best prior metric, VideoScore, reaches 31.0 and 23.7. ETVA leads in all ten categories, with especially strong category-level correlations in Camera (44.2 $\hat{A} \in \mathbb{R}^{K \times T}$3), Physics (60.4 $\hat{A} \in \mathbb{R}^{K \times T}$4), Action (46.6 $\hat{A} \in \mathbb{R}^{K \times T}$5), and Spatial (66.8 $\hat{A} \in \mathbb{R}^{K \times T}$6). Ablations attribute substantial gains to both components: multi-agent scene graph question generation improves by +14.67 $\hat{A} \in \mathbb{R}^{K \times T}$7 and +12.12 $\hat{A} \in \mathbb{R}^{K \times T}$8 over vanilla in-context question generation, and the knowledge-augmented, multi-stage QA raises answer accuracy to 89.27%, a gain of +26.20% over direct video LLM answering (Guan et al., 21 Mar 2025).

VideoComp shows that temporally informed training substantially improves compositional sensitivity without collapsing retrieval performance. On ActivityNet-Comp, VidCLIP-base zero-shot reaches 52.0 on temporal reorder, 62.1 on action replacement, 58.4 on segment mismatch, and 18.9 comprehensive accuracy. CompLoss improves these to 65.4, 73.1, 65.3, and 31.2, while CompPretrain + CompLoss reaches 68.2, 75.4, 68.0, and 35.0. On YouCook2-Comp, the corresponding comprehensive score rises from 21.4 to 28.4. Standard fine-tuning yields the best positive-pair retrieval on ActivityNet-Comp, but compositional training largely preserves retrieval while adding fine-grained sensitivity (Kim et al., 4 Apr 2025).

Weakly supervised temporal grounding results point to the utility of dense alignment maps. FSAN reports, on ActivityNet-Captions, R@1 of 78.45 at IoU 0.1, 55.11 at IoU 0.3, and 29.43 at IoU 0.5, with R@5 of 92.59, 76.79, and 63.32. On DiDeMo, it achieves Rank@1 of 19.40, Rank@5 of 57.85, and mIoU of 31.92. Removing the alignment-based grounding module or the losses on $\hat{A} \in \mathbb{R}^{K \times T}$9 degrades performance, and removing cross-modal attention or inner-modal attention reduces mIoU by 3.42% and 1.92%, respectively (Wang et al., 2022).

Long-window joint modeling plus denoising also yields large gains. On HTM-Align, CLIP obtains 16.8 R@1 and 71.7 ROC-AUC, MIL-NCE obtains 31.3 and 73.1, TAN Stage 1 obtains 45.8 and 73.0, and TAN Stage 1 plus Stage 2 co-training with $A \in \mathbb{R}^{T \times L}$0 reaches 49.4 and 82.4. The same model yields state-of-the-art zero-shot text-video retrieval on YouCook2, with 20.1/45.5/59.5/7 for R@1/R@5/R@10/MedR, and state-of-the-art performance on weakly supervised Breakfast-Action segmentation after fine-tuning (Han et al., 2022).

In code-switch speech translation, explicit subspace routing improves both alignment and end-task performance. On Fisher CS split, the proposed model reaches 37.13 BLEU and 77.74 COMET versus 33.78 and 75.35 for LLaST. On NTUML2021 CS split, it reaches 37.48 and 81.69 versus 32.53 and 78.92. The preliminary study reports that replacing a shared projector with language-specific expert projectors reduces WER from 7.99 to 7.70 in English and from 4.46 to 4.41 in Spanish. Expert routing accuracy on Common Voice is high, with Top-1 English 99.99% and Chinese 95.29%, and Top-3 English 99.98% and Chinese 92.27% (Gao et al., 9 Nov 2025).

In cross-lingual temporal reasoning, CLiTSSA shows that multilingual LLMs reward in-language aligned exemplars. For L1 on LLaMA3-8B, selecting examples by in-language similarity rather than direct cross-lingual similarity improves Romanian from 33.65/10.15 to 43.48/18.65 in F1/EM, German from 56.63/35.45 to 64.77/46.35, and French from 46.62/22.05 to 60.50/37.17. Relative to X-InSTA, CLiTSSA raises mean F1 by 5.32 in French, 1.62 in German, and 1.43 in Romanian across tasks, with $A \in \mathbb{R}^{T \times L}$1 by Mann-Whitney U (Bajpai et al., 2024).

Across domains, recurrent failure modes are strikingly similar. In text-to-video generation they include incorrect ordering, duration misestimation, motion inconsistency, causal implausibility, and partial transitions (Guan et al., 21 Mar 2025). In multi-event video-text modeling they include temporal reorder sensitivity, segment mismatch, subtle verb changes, and residual difficulty under combined disruptions (Kim et al., 4 Apr 2025). In grounding, failures arise from ambiguous boundaries, generic tokens, or fixed clip resolution (Wang et al., 2022). In code-switch speech translation, ambiguous frames, borrowings, heavy accent, and the absence of explicit routing supervision in Stage 4 can misroute tokens (Gao et al., 9 Nov 2025). In cross-lingual temporal reasoning, improved retrieval does not correct errors caused by incorrect factual knowledge in the base LLM, and implicit temporality in L3 remains difficult (Bajpai et al., 2024).

6. Limitations, open problems, and likely directions

Several bottlenecks recur across otherwise different formulations. One is perceptual fidelity under long temporal horizons. ETVA notes reliance on video LLM capability and identifies long-range temporal reasoning in very long or multi-shot videos as a limitation; proposed directions include explicit temporal localization, event segmentation, and symbolic temporal logic such as Allen’s interval relations (Guan et al., 21 Mar 2025). VideoComp similarly notes that uniformly sampling 16 frames can miss subtle temporal cues in long videos and points to explicit temporal token-level cross-attention as a future extension (Kim et al., 4 Apr 2025). FSAN observes that scan time grows with $A \in \mathbb{R}^{T \times L}$2 for long videos and suggests coarse-to-fine search, multi-scale pooling, or hierarchical ICIM (Wang et al., 2022). TAN still ties localization window length to ASR duration, so variable-length segment prediction remains open (Han et al., 2022).

A second bottleneck is supervision quality. Noisy ASR spans motivate TAN’s co-training denoising (Han et al., 2022). Weak video-level grounding labels motivate FSAN’s alignment-map regularization (Wang et al., 2022). Dense temporal annotation scarcity motivates VideoComp’s concatenative pretraining over short pairs (Kim et al., 4 Apr 2025). CS-ST scarcity motivates multi-stage transfer from monolingual ASR and ST into code-switch translation (Gao et al., 9 Nov 2025). Low-resource temporal reasoning motivates CLiTSSA’s transfer of in-language semantic structure into the cross-lingual retrieval space (Bajpai et al., 2024). This suggests that fine-grained alignment is as much a data-construction problem as a model-design problem.

A third bottleneck is the absence of explicit temporal structure in some systems. CLiTSSA explicitly states that its implementation uses $A \in \mathbb{R}^{T \times L}$3 and $A \in \mathbb{R}^{T \times L}$4 in the notional combination $A \in \mathbb{R}^{T \times L}$5, meaning that time sensitivity is induced through temporal data and labels rather than a dedicated temporal scorer (Bajpai et al., 2024). VideoComp proposes learned non-uniform weights and non-zero margins as future work (Kim et al., 4 Apr 2025). ETVA suggests broader knowledge grounding through curated physics knowledge or simulation priors (Guan et al., 21 Mar 2025). These proposals imply a broader trend toward hybrid systems in which semantic similarity is augmented by explicit models of temporal relations, physical constraints, or discourse structure.

Finally, the literature increasingly treats alignment not only as an evaluation target but also as a training signal. ETVA proposes using atomic correctness as supervision for reward modeling, RLHF, or multi-task training, with heavier weighting on camera, physics, and action order (Guan et al., 21 Mar 2025). VideoComp shows that hierarchical preferences can preserve retrieval strength while teaching models “how much wrong” each negative is (Kim et al., 4 Apr 2025). The code-switch speech translation work shows that token-level routing plus attention can align temporal speech units to textual semantics under rapid intra-utterance language switches (Gao et al., 9 Nov 2025). A plausible implication is that future systems will combine dense local alignment representations with task-specific structural priors, using evaluation frameworks that can diagnose temporal errors at the same granularity at which models are trained.