Time Series Captioning

• Time series captioning is the automated generation of natural-language descriptions that capture temporal trends, anomalies, and contextual events in numeric and multimodal data.
• Approaches range from rule-based templates and programmatic methods to advanced neural architectures like contrastive and LLM-guided models for robust domain adaptation.
• Evaluation methods combine linguistic metrics with numeric precision measures, driving improved interpretability for applications in industrial monitoring, healthcare, and beyond.

Time series captioning is the automated generation of natural-language descriptions that summarize salient temporal patterns, anomalies, and contextual phenomena in numeric (and sometimes visual or audio-visual) time series data. This task is foundational for interpretable analytics in science, industry, and engineering, enabling the integration of quantitative signals with human-readable, domain-adaptive narrative. Recent developments span from scalar sensor streams and financial series to multivariate monitoring data, time-series images, and event-rich video, reflecting a rapid expansion in model architectures, captioning paradigms, domain adaptation strategies, and benchmark methodologies.

1. Problem Definition and Formalization

The core objective of time series captioning is to learn or specify a mapping from a time series $x$ (univariate, multivariate, or even audio-visual temporal sequence) to a textual caption $y$ describing its salient structure. For univariate series, $x \in \mathbb{R}^n$ and $y$ is a natural language description. The canonical probabilistic formulation is to train a model to approximate $p_\theta(y|x)$, often by minimizing negative log-likelihood over a dataset of paired examples: $L(\theta) = -\sum_i \log p_\theta(y_i|x_i)$ Captions can be of varying granularity: point-level (per time step), fragment-level (contiguous segments), or instance-level (whole series). Generation objectives may emphasize semantic fidelity to patterns such as trend, extremum, nonlinearity, periodicity, domain-aware signals, or context-specific events, with requirements differing by application (Dohi et al., 2024, Ge et al., 5 May 2025).

2. Captioning Paradigms and Model Architectures

Several modeling paradigms have emerged:

• Template-Driven and Rule-Based Approaches: Early and synthetic-caption methods pair a time series with template-generated text based on predefined shape classes (e.g., "Rising", "Falling", "Spiky") chosen via explicit signal-processing or simulation. Synthetic datasets like SUSHI are constructed this way (Dohi et al., 2024).
• Programmatic and Truth-Conditional Captioning: The TRUCE architecture learns a set of modular neural programs, each detecting patterns such as "peak in the middle" or "increase at end." A truth-conditional prior ensures only patterns with high response scores for a given input $x$ are selected, and the decoder generates a caption based solely on the selected program embedding, improving factual alignment over soft attention models (Jhamtani et al., 2021).
• Contrastive Learning and Multimodal Models: Recent work exploits large transformer encoders for both signals and text (e.g., Informer for signals, T5 for text), aligning them in a joint latent space via contrastive (InfoNCE) objectives (Dohi et al., 2024). Decoding architectures often autoregressively generate captions conditioned on learned signal embeddings via dedicated bridge layers.
• LLMs and Dual-Stream Systems: DualSG and TimeCAP use LLMs as semantic guides or contextualizers; captions are produced by dedicated encoder–decoder modules or by prompting large-scale LLMs (e.g., GPT-4) with serializations of the time series, then used as additional semantic features to steer forecasts or event predictions (Ding et al., 29 Jul 2025, Lee et al., 17 Feb 2025).
• Retrieval-Based Captioning for Domain Awareness: TADACap uses a retrieval-plus-in-context method, pairing a CLIP-like vision encoder for line-plot images with an LLM that translates generic, domain-agnostic captions into domain-specific forms by reference to a curated set of in-domain captioned exemplars, enabling adaptation without retraining (Fons et al., 15 Apr 2025).
• Synthetic Data and Denoising: TSLM applies in-context LLM generation to build large synthetic time series–caption sets, followed by cross-modal dense retrieval filtering to denoise and retain only semantically faithful pairs, before supervised multimodal encoder-decoder training (Trabelsi et al., 3 Jan 2025).
• Time-Aware Structured Captioning for Video: TimeChat-Captioner extends the paradigm to dense, time-stamped, six-dimensional structured captions aligned with video scenes (Events, Background, Camera, Editing, Dialogue, Acoustics), mapping temporally localized segments to compositional script-like narratives (Yao et al., 9 Feb 2026).

3. Datasets, Data Construction, and Annotation Paradigms

A variety of benchmark datasets underline the diversity and scale of time series captioning:

Dataset	Input Type	Coverage	Caption Level
TACO	Real univariate	Industrial/IoT domains	Instance
SUSHI	Synthetic univariate	All shape classes	Instance
CaTS-Bench	Multimodal (series + img + metadata)	11 real domains	Instance
TimeChatCap-42K	Audio-visual video	Movies, trailers	Scene/fragment

Construction strategies include (1) forward synthesis via simulation and pairwise templates (Dohi et al., 2024), (2) backward extraction from real data using signal-processing heuristics to assign class labels (Dohi et al., 2024), (3) LLM-based or crowdsourced annotation at scale (Zhou et al., 25 Sep 2025), (4) in-context LLM bootstrapping (Trabelsi et al., 3 Jan 2025), and (5) DPP-based diverse retrieval for minimal expert labeling in new domains (Fons et al., 15 Apr 2025). Caption granularity spans point, fragment, and instance, each presenting distinct annotation burdens and descriptive richness (Ge et al., 5 May 2025).

4. Domain Generalization and Adaptation

Generalization to unseen domains and semantic adaptation is a central challenge:

• Contrastive pretraining on large, diverse datasets (TACO) yields models that transfer with minor degradation to out-of-domain (OOD) settings, significantly surpassing nearest-neighbor or naive baselines in both BLEU and semantic similarity metrics (Dohi et al., 2024).
• Retrieval-based frameworks such as TADACap-diverse ensure coverage of domain-specific semantics by dynamically selecting a diverse, representative subset of annotated exemplars using DPP, reducing expert annotation costs (Fons et al., 15 Apr 2025).
• Prompting-based models, either through explicit in-context example engineering or modular LLM prompting, have demonstrated robust transfer across domains, conditional on selection of informative exemplars and adaptation to domain phraseology (Trabelsi et al., 3 Jan 2025, Lee et al., 17 Feb 2025).

Ablation studies show explicit semantic streams or caption-guided fusion modules consistently improve forecasting or event detection accuracy by 4–11% over purely numerical methods, and adaptation to new domains can be achieved rapidly with small, expert-authored annotation pools (Ding et al., 29 Jul 2025, Fons et al., 15 Apr 2025).

5. Evaluation Methodologies and Metrics

Evaluation encompasses both linguistic similarity and domain-specific semantic or numeric fidelity:

Metric	Description
BLEU, METEOR, ROUGE_L	Standard n-gram similarity and longest common subsequence
CIDEr, SPICE	Consensus-based and scene-graph/tuple semantic similarity
BERTScore, Sentence-BERT	Embedding-based token/sentence similarity
Numeric Score, Statistical Inference Accuracy	Precision of numeric/statistical mentions <br> (e.g., correct mean/max within 5%)
SodaM	Structural and timestamped alignment in dense video captioning

Benchmarks such as CaTS-Bench introduce numeric-oriented metrics, matching extracted numbers in output and reference captions with specified tolerance, and scoring accuracy and recall proportionally (Zhou et al., 25 Sep 2025). Human indistinguishability analyses confirm that LLM-generated captions, when properly prompted and verified, are not reliably distinguishable from expert-written text at statistically significant levels.

6. Representative Applications and Case Studies

Time series captioning is now integral to:

• Interpretable Decision Support: Sensor status summaries in IoT, anomaly reports in industrial monitoring, or summarization of physiological time series for clinical insight (Dohi et al., 2024, Ding et al., 29 Jul 2025).
• Forecasting and Reasoning: Semantic guidance modules in forecasting frameworks leverage captions to enhance long-horizon multivariate prediction and capture rare transition phenomena (Ding et al., 29 Jul 2025).
• Domain-Aware Communication: TADACap's retrieval-plus-in-context method supports rapid adaptation for finance, healthcare, or scientific visualization, producing field-specific narratives tailored to practitioner vocabulary (Fons et al., 15 Apr 2025).
• Text-to-Series Generation: Reverse mapping from human-written descriptions to high-resolution synthetic time series for data augmentation, simulation, and scenario analysis (Ge et al., 5 May 2025).
• Video Event Narrative: Structured dense-captioning for temporally grounded video analytics integrates audio-visual feature tracking with six-dimensional narrative decomposition (Yao et al., 9 Feb 2026).

7. Limitations and Future Research Directions

Current limitations arise from:

• Coverage: Restrictive pattern spaces (e.g., only basic trends or few compositional modules), limiting the expressivity of generated captions (Jhamtani et al., 2021).
• Numeric Precision: Even leading models often hallucinate or misreport exact statistics, highlighting the need for programmatic or execution-augmented captioners (Zhou et al., 25 Sep 2025).
• Semantic Fusion: Many VLMs underutilize visual features when captioning image-based or plot-based series, defaulting to text priors (Zhou et al., 25 Sep 2025).
• Data Scarcity: Domain-specific, high-quality annotated data remains a bottleneck; synthetic generation and bootstrapped denoising are promising but imperfect (Trabelsi et al., 3 Jan 2025).

Ongoing research is exploring:

• Automatic threshold selection and expanded pattern taxonomies for richer structural capture (Dohi et al., 2024).
• Hierarchical, multivariate, and multiscale captioning, moving beyond scalar or short-fragment narratives (Dohi et al., 2024, Ge et al., 5 May 2025).
• Code-execution or PAL modules to guarantee numeric statement fidelity (Zhou et al., 25 Sep 2025).
• Unified multimodal architectures capable of arbitrary-length input/output, dense time-aware structuring, and joint video–audio–numeric segmentation (Yao et al., 9 Feb 2026, Ge et al., 5 May 2025).

Cumulatively, the field of time series captioning is advancing toward interpretable, domain-robust, multimodal integration, where model outputs can be faithfully aligned with factual content and flexibly adapted to both underlying data semantics and user domain requirements.