AniCaption: Future Image Captioning

• AniCaption is a future-oriented captioning approach that predicts unseen image descriptions from a sparse sequence of observed images.
• It employs a commonsense-augmented prompt-learning framework with VinVL, ROI features, and a Graph Attention Network for coherent concept integration.
• Empirical results on a repurposed VIST benchmark demonstrate improved captioning metrics over baselines, emphasizing selective forecasted concept retrieval.

In the captioning literature, AniCaption is explicitly formalized in A-CAP as anticipation captioning: given a sparsely temporally-ordered sequence of images, the model generates a caption for an unseen future image or oracle image, rather than describing a visible input image (Vo et al., 2023). The task shifts captioning from retrospective description to future-oriented inference, and therefore sits between image captioning, visual storytelling, and forecasting. Its central question is not merely what is present now, but what caption is justified by the partial visual evidence accumulated so far.

1. Formal task and conceptual scope

A-CAP defines the input as a temporally ordered set of images $I_1, I_2, \dots, I_k$, where the ordering is preserved but the sequence is sparse rather than video-dense. The target is the caption $C_{k+1}$ of an unseen future image $I_{k+1}$, formalized as

$C_{k+1} = Caption(I_1, I_2, \dots, I_k).$

The model does not output captions for the observed images and does not generate the oracle image itself; it generates only the future caption (Vo et al., 2023).

This definition distinguishes AniCaption from several adjacent problems. Standard image captioning describes a visible image. Visual storytelling produces a coherent narrative over an observed sequence. Video captioning typically uses dense temporal input and describes visible events. Future prediction often targets future frames or action labels. AniCaption instead takes sparse ordered images as input and produces free-form text for an unseen future state (Vo et al., 2023).

The broader sequential-captioning context is important. DeepDiary argues that captioning sequential visual content is not just “single-image captioning repeated many times,” because temporal continuity, caption stability, reduced redundancy, event segmentation, and coherence across neighboring frames become first-class objectives in ordered visual streams (Fan et al., 2016). AniCaption inherits that sequential setting, but redirects it from stream summarization toward future caption inference.

2. A-CAP architecture

A-CAP implements AniCaption as a commonsense-augmented prompt-learning framework built on a frozen VinVL captioning model. The pipeline combines visual features, detected concepts, retrieved commonsense concepts, graph reasoning, and prompt-like conditioning of the caption generator (Vo et al., 2023).

For visual input, A-CAP extracts ROI features from the $k$ observed images using Faster R-CNN trained on COCO. These ROI features are projected by VinVL’s embedding layer to $1 \times 768$ vectors, and their mean defines a global visual context

$$\bar{\mathbf{f}} = \frac{1}{N}\sum_{i=1}^{N}\mathbf{f}_i.$$

This context is not used as a standalone caption condition; it also guides concept selection (Vo et al., 2023).

Concept detection is performed with Clarifai, taking the top 10 concepts per image. Future-oriented concepts are then retrieved from ConceptNet by taking 2-hop neighbors of the detected concepts. Because raw retrieval often yields more than 400 candidates, A-CAP ranks candidate forecasted concepts by relevance to the visual context using a pretrained LLM: $$\rho_{c_i^f} = f_{\rm head}(f_{\rm enc}([\bar{\mathbf{f}}; \mathbf{c}_i^f])).$$ The top $M$ forecasted concepts are retained, with the main experiments using $M=60$ (Vo et al., 2023).

Detected and forecasted concepts are organized into a knowledge graph. Nodes correspond to concepts; edges connect concepts that are related in ConceptNet, and temporal order is encoded by linking related concepts across adjacent observed images. Each concept is embedded with pretrained BERT, concatenated with the visual context, and processed by a Graph Attention Network. The graph layer is written as

$C_{k+1}$0

with two graph layers in practice (Vo et al., 2023).

The resulting enriched concept embeddings replace VinVL’s ordinary object-tag prompt. During training, the decoder sees the ground-truth future caption tokens plus the graph-conditioned concepts and ROI features; during inference, the word positions are replaced by $C_{k+1}$1 [MASK] tokens, and VinVL decodes the anticipated caption conditioned on the enriched concept prompt and the observed-image features (Vo et al., 2023).

3. Dataset construction and evaluation protocol

Because no benchmark was designed specifically for AniCaption, A-CAP repurposes the Visual Storytelling Dataset (VIST). Each original five-image, five-sentence sample is converted so that the first four images are the observed input, the fifth image is the oracle image, and the fifth sentence is the target future caption. The resulting split sizes remain 4,098 for training, 4,988 for validation, and 5,050 for testing (Vo et al., 2023).

The paper checks whether VIST is a plausible proxy for future captioning. Using SentenceTransformer embeddings, it finds that similarity between $C_{k+1}$2 and preceding sentences is strictly monotonic increasing in 72.69% of cases, with 10.32% showing only one violation and 4.4% not complying with monotonicity. Using a pretrained BERT next-sentence check, 77.34% of samples satisfy the next-sentence condition for all transitions, 17.78% have only one violation, and 0.06% never satisfy it (Vo et al., 2023).

Evaluation is split between accuracy and descriptiveness. Accuracy uses BLEU-1, BLEU-4, CIDEr, SPICE, CLIPScore, and RefCLIPScore. Descriptiveness is evaluated through a CLIP-based self-retrieval setup reporting R@1, R@5, and R@10 for retrieval of the oracle image from the generated caption (Vo et al., 2023). The paper explicitly notes that AniCaption is open-domain and admits multiple plausible futures, so exact-reference metrics are only partial indicators.

4. Empirical results and ablations

On the customized VIST benchmark, A-CAP outperforms the compared baselines. Its reported scores are BLEU-1 37.2, BLEU-4 6.9, CIDEr 4.7, SPICE 20.1, CLIPScore 65.2, RefCLIPScore 70.2, R@1 8.7, R@5 18.9, and R@10 31.5 (Vo et al., 2023). These exceed both standard VinVL and the AREL + BART storytelling baseline, and they also exceed the reported VinVL + Oracle image baseline on the same metrics.

The ablations isolate the role of commonsense reasoning. Removing the GNN drops performance to BLEU-1 34.8, BLEU-4 5.2, CIDEr 3.7, SPICE 14.5, CLIPScore 38.2, RefCLIPScore 47.3, R@1 3.6, R@5 8.7, and R@10 15.4. Removing image-context concatenation while retaining the graph gives BLEU-1 36.1, BLEU-4 6.2, CIDEr 4.2, SPICE 13.9, CLIPScore 39.8, RefCLIPScore 46.9, R@1 4.1, R@5 9.5, and R@10 16.1 (Vo et al., 2023). Both components are therefore functionally important, with the graph providing the larger lift.

The number of retained forecasted concepts is also highly sensitive. With $C_{k+1}$3, performance collapses to SPICE 5.8, CLIPScore 15.3, RefCLIPScore 12.1, R@1 1.1, R@5 3.7, and R@10 7.6. With $C_{k+1}$4, the model reaches its best reported results. With $C_{k+1}$5, meaning no forecasted concepts, performance falls back to SPICE 14.2, CLIPScore 43.1, RefCLIPScore 44.7, R@1 1.9, R@5 7.3, and R@10 11.2 (Vo et al., 2023). The paper’s conclusion is that commonsense helps only when selection is selective rather than exhaustive.

5. Relation to adjacent captioning paradigms

AniCaption belongs to a broader rethinking of captioning beyond single-image maximum-likelihood decoding. DeepDiary shows that temporally ordered visual data benefit from explicit sequence-level reasoning such as temporal smoothing, activity summarization, and event segmentation; it uses a chain-structured MRF to stabilize captions across egocentric image streams (Fan et al., 2016). That work targets stream coherence, whereas AniCaption targets future inference.

IC$C_{k+1}$6 addresses a different failure mode: the tendency of captioners to generate a single “best” caption that is “informationally impoverished.” Its solution is a two-stage process in which a committee of diverse candidate captions is sampled and then summarized into one richer consensus description (Chan et al., 2023). In relation to AniCaption, IC$C_{k+1}$7 is relevant because it treats caption distributions as multimodal and leverages diversity rather than collapsing immediately to one local mode. A plausible implication is that future AniCaption systems could combine future reasoning with multi-hypothesis caption aggregation.

These neighboring lines define a useful conceptual triangle. DeepDiary emphasizes temporal coherence over observed sequences. IC$C_{k+1}$8 emphasizes multi-viewpoint aggregation for one image. AniCaption emphasizes prediction of an unseen future caption from sparse evidence (Vo et al., 2023).

6. Terminological reuse and later extensions

The label is not unique across later work. The AnyCap Project defines AnyCapModel, AnyCapDataset, and AnyCapEval as a unified framework for controllable omni-modal captioning across image, video, and audio, with user instructions and explicit separation of content accuracy from stylistic fidelity (Ren et al., 17 Jul 2025). This is a different problem from anticipation captioning: it focuses on instruction-following refinement of captions for visible inputs.

A further reuse appears in AniMatrix, where AniCaption names a production-variable inference module for anime video generation. There, AniCaption is not a future-captioning task at all; it is a specialized anime caption model that infers structured production variables—Style, Motion, Camera, and VFX—from clips and rewrites them into directorial directives used for generator conditioning (Team, 5 May 2026). The shared label therefore covers at least two distinct research objects: anticipation captioning in A-CAP and production-oriented supervision in AniMatrix.

7. Limitations and open research directions

A-CAP identifies several limitations directly. The model depends heavily on the initial concept detector: missing concepts such as “falling” can cause incorrect future captions. It is also highly sensitive to the number and quality of retrieved forecasted concepts. The authors state that they did not pursue a fully trainable version because training collapsed when trying to unfreeze everything, attributing this to the small size of the adapted VIST benchmark relative to VinVL’s original pretraining scale (Vo et al., 2023).

The benchmark itself is only a proxy for AniCaption. VIST was designed for storytelling, not future prediction, and the paper shows examples where the ground-truth future makes an abrupt scene jump, producing low automatic scores even when the predicted continuation is commonsensically plausible. Evaluation remains open-domain and underconstrained; the paper explicitly notes that human evaluation could help but is costly and subjective (Vo et al., 2023).

The paper’s own forward-looking agenda is specific: it calls for a purpose-built AniCaption dataset covering near future, far future, abnormal/non-commonsense thinking, and rationale/explanation, and for concept selection to be learned jointly during training rather than handled by a fixed top-$C_{k+1}$9 heuristic (Vo et al., 2023). This suggests that AniCaption, as presently defined, is best understood as an early but technically clear formulation of future-oriented captioning rather than a solved problem.