Explicit Timestamp-modal Aligned (ETA)

• ETA is a multimodal design principle that explicitly couples semantic content with temporal coordinates, ensuring precise 'when' and 'what' alignment.
• It is exemplified by In-Sync in ASR, where word-level timestamp tokens are generated in a single pass to achieve integrated transcription and timing.
• Lightweight strategies like timestamp embedding regularization and reduced teacher forcing boost temporal consistency and improve metrics such as WER and AAS.

Explicit Timestamp-modal Aligned (ETA) denotes a family of multimodal designs in which temporal reference is made explicit inside the model or alignment pipeline, rather than being deferred to post hoc alignment tools, implicit positional encodings, or heuristic recovery. In automatic speech recognition, “In-Sync” realizes ETA by integrating word-level timestamp prediction directly into a speech-aware LLM, so that transcription and timing are generated in a single pass (Fan et al., 14 Apr 2026). Related work applies the same principle in text temporal ordering, long-video summarization, clinical timeline reconstruction, speech reasoning, autonomous driving, long-video understanding, and text-to-audio generation through timex embeddings, timestamp tokens, retrieved structured anchors, time-aware sensor fusion, or explicit temporal conditioning (Goyal et al., 2019, Patel et al., 11 Apr 2026, Kumar et al., 14 May 2026, Jeong et al., 19 Mar 2026, Song et al., 2024, Yuan et al., 11 Sep 2025, Xie et al., 2024).

1. Formal scope of ETA

A compact formalization appears in long-video summarization. Let a long video $V$ span $[0,T]$, let a predicted summary be $S=\{s_k\}_{k=1}^K$ with $$s_k=(\tau_k^{\mathrm{start}},\tau_k^{\mathrm{end}},d_k)$$, and let ground truth be $G=\{g_i\}_{i=1}^M$ with $$g_i=(\sigma_i^{\mathrm{start}},\sigma_i^{\mathrm{end}},r_i,h_i)$$. ETA is then judged by temporal overlap and semantic alignment. LVSum expresses this with an IoU term over time intervals, a similarity term $\mathrm{Sim}(d_k,h_i)$, and a combined score of the form

$$\mathrm{ETA}(S,G) =\frac{1}{K}\sum_{k=1}^K \max_i\Bigl[\alpha\,\mathrm{IoU}(s_k,g_i)+\beta\,\mathrm{Sim}(d_k,h_i)\Bigr].$$

LVSum does not optimize this score directly, but evaluates the same underlying desiderata through rank-correlation of importance over time, Content Relevance, and Modality Coherence (Patel et al., 11 Apr 2026).

In textual temporal ordering, the same pattern appears without audio or video. A timex is encoded from characters by a BiLSTM, average-pooled into an embedding $e$, and then concatenated to token-level inputs; the same timex embedding is also projected to the event it modifies through a dependency-parse rule such as $\text{nmod:tmod}$. This makes the temporal signal explicit in the event-ordering model instead of leaving it latent in lexical context alone (Goyal et al., 2019).

Taken together, these formulations suggest that ETA is best understood as an explicit coupling between semantic content and temporal coordinates. The exact mechanism varies by domain, but the shared objective is stable: the system should represent not only what occurred, but when it occurred, and should preserve that coupling through inference.

2. In-Sync as an ETA architecture for ASR

“In-Sync” instantiates ETA in speech recognition by extending a speech-aware LLM to emit timestamp tokens directly alongside transcript tokens (Fan et al., 14 Apr 2026). The input is raw audio $[0,T]$0. A frozen 10-layer Conformer encoder $[0,T]$1 maps the audio to acoustic features

$[0,T]$2

A trainable Task-Aware Speech Adapter, implemented as an MLP, prepends a task indicator token for ASR versus SRWT and projects the acoustic sequence into the LLM embedding space: $[0,T]$3

Decoding uses a frozen Granite-3.3-8B-Instruct backbone with LoRA adapters. At decoding step $[0,T]$4, the model performs cross-attention over $[0,T]$5 and past tokens $[0,T]$6 to produce a hidden state $[0,T]$7. A shared output projection is used for both lexical and temporal generation: $[0,T]$8 with $[0,T]$9. Here $S=\{s_k\}_{k=1}^K$0 timestamp tokens represent 10 ms bins up to 60 s.

Timestamp prediction is not handled by a separate regression head. Instead, it is treated as generation over the timestamp sub-vocabulary. If a word token $S=\{s_k\}_{k=1}^K$1 has just been produced, the next decoding position may emit a timestamp token. Let $S=\{s_k\}_{k=1}^K$2 be the hidden state of that timestamp slot and let

$S=\{s_k\}_{k=1}^K$3

where the rows of $S=\{s_k\}_{k=1}^K$4 are a subset of $S=\{s_k\}_{k=1}^K$5. The discrete prediction $S=\{s_k\}_{k=1}^K$6 yields an end time

$S=\{s_k\}_{k=1}^K$7

An optional soft estimate is

$S=\{s_k\}_{k=1}^K$8

A notable design decision is the absence of any additional gating beyond the task-indicator token. The report states that alignment emerges through the LLM’s cross-attention to acoustic embeddings. This makes ETA here model-internal: timing is not recovered by an external aligner after transcription, but generated as part of the same autoregressive sequence.

3. Objectives and lightweight optimization strategies

Training alternates evenly between two tasks. With probability $S=\{s_k\}_{k=1}^K$9, the model performs standard ASR with words only; with probability $$s_k=(\tau_k^{\mathrm{start}},\tau_k^{\mathrm{end}},d_k)$$0, it performs SRWT with interleaved words and timestamps. In both cases, optimization is based on next-token cross-entropy: $$s_k=(\tau_k^{\mathrm{start}},\tau_k^{\mathrm{end}},d_k)$$1 where $$s_k=(\tau_k^{\mathrm{start}},\tau_k^{\mathrm{end}},d_k)$$2 is the output-sequence length for the selected task (Fan et al., 14 Apr 2026).

A second term regularizes the geometry of timestamp embeddings. Let $$s_k=(\tau_k^{\mathrm{start}},\tau_k^{\mathrm{end}},d_k)$$3 denote the normalized timestamp-token embeddings. Their cosine-similarity matrix is

$$s_k=(\tau_k^{\mathrm{start}},\tau_k^{\mathrm{end}},d_k)$$4

This is matched to a Gaussian kernel

$$s_k=(\tau_k^{\mathrm{start}},\tau_k^{\mathrm{end}},d_k)$$5

through

$$s_k=(\tau_k^{\mathrm{start}},\tau_k^{\mathrm{end}},d_k)$$6

The batch loss is

$$s_k=(\tau_k^{\mathrm{start}},\tau_k^{\mathrm{end}},d_k)$$7

with $$s_k=(\tau_k^{\mathrm{start}},\tau_k^{\mathrm{end}},d_k)$$8. No separate $$s_k=(\tau_k^{\mathrm{start}},\tau_k^{\mathrm{end}},d_k)$$9 for an explicit timestamp-regression term is used, because timestamp-token generation is already part of $G=\{g_i\}_{i=1}^M$0.

The report identifies three lightweight strategies. Speech Length Augmentation concatenates two utterances, shifts the second utterance’s timestamps by the duration of the first, and thereby extends the effective timestamp range. Timestamp Embedding Regularization is applied at every step through $G=\{g_i\}_{i=1}^M$1. Reduced Teacher Forcing perturbs the prefix during SRWT training: with probability $G=\{g_i\}_{i=1}^M$2, each ground-truth timestamp token in the prefix is replaced by a smaller timestamp token sampled uniformly between 0 ms and the true time. The stated purpose is to break over-reliance on perfect past timestamps.

These strategies are lightweight in the sense used by the report: they modify training behavior without introducing a heavy external alignment stage, an auxiliary Viterbi procedure, or an HMM pass. A plausible implication is that ETA can be improved substantially through representation shaping and autoregressive robustness, not only through architectural expansion.

4. Experimental regime and reported performance

The training corpora are LibriSpeech, CommonVoice, AMI-IHM, and VoxPopuli, with MFA-aligned timestamps. Evaluation uses LS-C, LS-O, CV, AMI, VOXP, MLS, TIMIT, and Buckeye; MFA is used for missing labels, while TIMIT and Buckeye rely on human annotation. The audio encoder is the frozen 10-layer Conformer, the adapter is a 2-layer MLP, and the language backbone is Granite-3.3-8B with frozen weights plus LoRA of rank 32 and $G=\{g_i\}_{i=1}^M$3. Training uses batch size $G=\{g_i\}_{i=1}^M$4, 400 k steps, AdamW, peak learning rate $G=\{g_i\}_{i=1}^M$5, 1 k-step warmup, and adapter downsample rate 5 (Fan et al., 14 Apr 2026).

Ablation results averaged over eight evaluation sets are reported as follows:

Model Variant	WER (%)	AAS (ms)
ASR-only Baseline	6.55	–
Mixed ASR + SRWT (no tricks)	6.71	41.66
+ Length Augmentation	6.60	41.41
+ + Timestamp Reg.	6.34	37.23
+ + Reduced Teacher Forcing	6.47	35.89

The report attributes the largest joint gain to Timestamp Regularization, with $G=\{g_i\}_{i=1}^M$6 and $G=\{g_i\}_{i=1}^M$7 ms. Reduced Teacher Forcing yields the best timestamp accuracy, with $G=\{g_i\}_{i=1}^M$8 ms, while keeping the WER gap minimal. The conclusion summarizes the resulting regime as producing model-internal word-level timestamps with AAS down to approximately 36 ms and WER approximately $G=\{g_i\}_{i=1}^M$9.

Relative to external baselines, the report states that In-Sync outperforms Qwen-Audio on WER and outperforms CrisperWhisper on average AAS, while using a single pass and no external aligner. It also states that the computational overhead is minimal: there are no extra Viterbi or HMM passes, and real-time feasibility is tied to the same decoder latency as Granite-speech plus negligible token-generation overhead. Within the ETA framing, this is significant because the temporal signal is produced as part of the ASR decode itself rather than as a downstream repair step.

5. ETA across modalities and tasks

The same design logic recurs outside ASR. In temporal information extraction from text, a character-level timex encoder reaches $$g_i=(\sigma_i^{\mathrm{start}},\sigma_i^{\mathrm{end}},r_i,h_i)$$0 intrinsic timex-ordering accuracy, compared with $$g_i=(\sigma_i^{\mathrm{start}},\sigma_i^{\mathrm{end}},r_i,h_i)$$1 for ELMo + token-BiLSTM; on MATRES, gains from adding timex embeddings are small and statistically weak because only approximately $$g_i=(\sigma_i^{\mathrm{start}},\sigma_i^{\mathrm{end}},r_i,h_i)$$2 of examples contain distinct, explicit timexes, but on a distantly supervised dataset the learned timex embeddings improve accuracy by about $$g_i=(\sigma_i^{\mathrm{start}},\sigma_i^{\mathrm{end}},r_i,h_i)$$3 points over systems without them (Goyal et al., 2019). In clinical timeline reconstruction, ETA is formulated as a graph-based three-step process with central anchor extraction, relative placement, and absolute timestamp calibration against retrieved structured EHR rows. On the i2m4 benchmark of 20 discharge summaries, AULTC increases for all evaluated models under ETA; examples given include GLM5 from $$g_i=(\sigma_i^{\mathrm{start}},\sigma_i^{\mathrm{end}},r_i,h_i)$$4 and DeepSeek V3.2 from $$g_i=(\sigma_i^{\mathrm{start}},\sigma_i^{\mathrm{end}},r_i,h_i)$$5, while event match rate remains essentially unchanged. The same study reports that $$g_i=(\sigma_i^{\mathrm{start}},\sigma_i^{\mathrm{end}},r_i,h_i)$$6 of text-derived events are entirely absent from tabular records, which directly motivates multimodal alignment rather than unimodal replacement (Kumar et al., 14 May 2026).

Speech reasoning and audio generation provide two further variants. Timestamp-grounded speech reasoning trains a Large Audio-LLM in two stages: Supervised Timestamp Alignment on 268 K examples, followed by Group Relative Policy Optimization with rewards for answer correctness and concise timestamp use. On four benchmarks, the reported ETA configuration improves over zero-shot CoT and reasoning SFT without timestamps; for example, MMAR rises from $$g_i=(\sigma_i^{\mathrm{start}},\sigma_i^{\mathrm{end}},r_i,h_i)$$7 under only STA to $$g_i=(\sigma_i^{\mathrm{start}},\sigma_i^{\mathrm{end}},r_i,h_i)$$8 under STA + GRPO, AIR-Bench submetrics rise to $$g_i=(\sigma_i^{\mathrm{start}},\sigma_i^{\mathrm{end}},r_i,h_i)$$9, $\mathrm{Sim}(d_k,h_i)$0, and $\mathrm{Sim}(d_k,h_i)$1, and MELD reaches $\mathrm{Sim}(d_k,h_i)$2. The same work reports broader region exploration, higher audiology verification, and higher consistency (Jeong et al., 19 Mar 2026). In text-to-audio generation, PicoAudio uses a one-hot timestamp matrix $\mathrm{Sim}(d_k,h_i)$3 at 40 ms resolution and injects it into a diffusion backbone via FiLM and cross-attention. On single- and multi-event tasks, full PicoAudio reports segment-level $\mathrm{Sim}(d_k,h_i)$4 of $\mathrm{Sim}(d_k,h_i)$5, FAD of $\mathrm{Sim}(d_k,h_i)$6, $\mathrm{Sim}(d_k,h_i)$7 of $\mathrm{Sim}(d_k,h_i)$8, MOS$\mathrm{Sim}(d_k,h_i)$9 of $$\mathrm{ETA}(S,G) =\frac{1}{K}\sum_{k=1}^K \max_i\Bigl[\alpha\,\mathrm{IoU}(s_k,g_i)+\beta\,\mathrm{Sim}(d_k,h_i)\Bigr].$$0, and MOS$$\mathrm{ETA}(S,G) =\frac{1}{K}\sum_{k=1}^K \max_i\Bigl[\alpha\,\mathrm{IoU}(s_k,g_i)+\beta\,\mathrm{Sim}(d_k,h_i)\Bigr].$$1 of $$\mathrm{ETA}(S,G) =\frac{1}{K}\sum_{k=1}^K \max_i\Bigl[\alpha\,\mathrm{IoU}(s_k,g_i)+\beta\,\mathrm{Sim}(d_k,h_i)\Bigr].$$2, substantially ahead of the reported baselines (Xie et al., 2024).

Video and sensor-fusion settings emphasize the same temporal explicitness under different constraints. LVSum provides 72 long-form videos across 13 domains, each with up to 10 human annotations and a 15% summary-length budget; its evaluation shows that current MLLMs achieve at most about $$\mathrm{ETA}(S,G) =\frac{1}{K}\sum_{k=1}^K \max_i\Bigl[\alpha\,\mathrm{IoU}(s_k,g_i)+\beta\,\mathrm{Sim}(d_k,h_i)\Bigr].$$3 of human-level Kendall’s $$\mathrm{ETA}(S,G) =\frac{1}{K}\sum_{k=1}^K \max_i\Bigl[\alpha\,\mathrm{IoU}(s_k,g_i)+\beta\,\mathrm{Sim}(d_k,h_i)\Bigr].$$4, often exhibit over-coverage, and can understand “what happened” while still struggling with “when” (Patel et al., 11 Apr 2026). DATE addresses long-video understanding by interleaving frame embeddings with learnable timestamp tokens and selecting query-relevant frames through Temporal-Aware Similarity Sampling; on a 7B model with 256 frames, Video-MME rises from $$\mathrm{ETA}(S,G) =\frac{1}{K}\sum_{k=1}^K \max_i\Bigl[\alpha\,\mathrm{IoU}(s_k,g_i)+\beta\,\mathrm{Sim}(d_k,h_i)\Bigr].$$5 to $$\mathrm{ETA}(S,G) =\frac{1}{K}\sum_{k=1}^K \max_i\Bigl[\alpha\,\mathrm{IoU}(s_k,g_i)+\beta\,\mathrm{Sim}(d_k,h_i)\Bigr].$$6, LongVideoBench from $$\mathrm{ETA}(S,G) =\frac{1}{K}\sum_{k=1}^K \max_i\Bigl[\alpha\,\mathrm{IoU}(s_k,g_i)+\beta\,\mathrm{Sim}(d_k,h_i)\Bigr].$$7 to $$\mathrm{ETA}(S,G) =\frac{1}{K}\sum_{k=1}^K \max_i\Bigl[\alpha\,\mathrm{IoU}(s_k,g_i)+\beta\,\mathrm{Sim}(d_k,h_i)\Bigr].$$8, and LVBench from $$\mathrm{ETA}(S,G) =\frac{1}{K}\sum_{k=1}^K \max_i\Bigl[\alpha\,\mathrm{IoU}(s_k,g_i)+\beta\,\mathrm{Sim}(d_k,h_i)\Bigr].$$9 to $e$0 (Yuan et al., 11 Sep 2025). In autonomous driving, TimeAlign treats camera time as reference, predicts LiDAR BEV features at the camera timestamp from historical sweeps, and aligns predicted and observed LiDAR features under camera guidance. Under a one-frame lag, reported mAP improves from $e$1 to $e$2 for Car, $e$3 to $e$4 for Truck, $e$5 to $e$6 for Bus, and $e$7 to $e$8 for Pedestrian (Song et al., 2024).

These cases suggest that ETA is not domain-specific. The explicit temporal object may be a timestamp token, a timex embedding, an interval, an EHR row time, a sensor capture time, or a binary activity matrix, but the methodological role is analogous: temporal anchoring is made first-class and is optimized jointly with semantic fidelity.

6. Misconceptions, limitations, and likely directions

ETA should not be conflated with any single implementation pattern. In-Sync does not use an explicit timestamp-regression loss separate from next-token generation; timestamps are part of the decoded vocabulary. DATE carries absolute time through learnable timestamp embeddings while removing absolute-time indices from the baseline’s MRoPE scheme. PicoAudio enforces temporal placement through an explicit conditioning matrix injected by FiLM and cross-attention. TimeAlign does not regress $e$9 directly, but treats it as an implicit variable and learns spatial sampling offsets in BEV space (Fan et al., 14 Apr 2026, Yuan et al., 11 Sep 2025, Xie et al., 2024, Song et al., 2024). A common misconception is therefore that ETA necessarily means a dedicated scalar time head; the literature shows a wider design space.

The limitations are equally consistent across domains. In-Sync predicts only end-of-word timestamps and does not model start times or explicit silence; the same report notes that regularization and reduced teacher forcing can interact poorly if combined naively. In temporal ordering from text, gains are modest when explicit timex coverage is low, and future work is identified in jointly learning event-timex links, extending the grammar to durations and composite expressions, and incorporating global constraints over the temporal graph. In long-video summarization, recurring failure modes include over-coverage, temporal misalignment, and cross-modal hallucination, and one model is reported to suffer modality interference when combining frames and transcript. In clinical alignment, the framework is explicitly generalized to other multimodal, time-sensitive domains such as imaging timelines, financial news with market data, and security logs with system-event records. In controllable audio generation, remaining challenges include scaling beyond 3–4 simultaneous events, handling overlapping events with controlled overlap ratios, and extending to non-temporal relations such as changing loudness or spatial cues (Fan et al., 14 Apr 2026, Goyal et al., 2019, Patel et al., 11 Apr 2026, Kumar et al., 14 May 2026, Xie et al., 2024).

Across these works, the dominant research direction is not merely to add more modalities, but to make their temporal relations explicit, trainable, and evaluable. This suggests that ETA is less a narrow module than a design criterion for multimodal systems: semantic competence alone is insufficient when downstream use depends on precise temporal grounding.