NN-Caption: Neural Captioning Systems

• NN-Caption is a technique that uses encoder–decoder architectures to convert visual or acoustic inputs into natural language captions.
• It employs methods like attention, concept injection, and external memory to refine outputs and ensure semantic alignment.
• Various benchmarks and metrics, such as BLEU and CIDEr, are used to evaluate its robustness, efficiency, and control in caption generation.

Searching arXiv for recent and foundational work relevant to “NN-Caption”. NN-Caption denotes neural-network-based caption generation systems that map perceptual inputs to natural-language descriptions. In the cited literature, the term covers canonical image captioning pipelines, audio captioning, domain-specific systems for medical and news media, controllable caption generators for out-of-context scenarios, and evaluation frameworks that score caption correctness at semantic or token granularity; it also appears as the proper name of an LLM-guided neural architecture search pipeline that generates runnable captioning models under a strict Net API (Wu et al., 2019, Jesani et al., 7 Dec 2025). Across these settings, the recurrent design pattern is an encoder–decoder architecture in which visual or acoustic features are transformed into a conditional LLM, then refined through attention, concept injection, external memory, alignment control, or post-hoc confidence estimation.

1. Task definitions and problem variants

NN-Caption is most commonly formulated as conditional sequence generation: given an image or audio input, generate a fluent caption that is semantically faithful to the input. In image captioning, the task is treated as a fundamental problem in visual understanding, while in audio captioning it is explicitly defined as automatically generating natural-language sentences that describe an audio scene (Tan et al., 2019, Wu et al., 2019).

Several task variants extend this basic formulation. Audio captioning is distinguished from audio event detection or classification because it must summarize how multiple, possibly overlapping sound events relate to each other, often in temporal order, and may include who is making a sound, how it sounds, and where it occurs (Wu et al., 2019). Zero-shot novel object captioning requires a model to mention object words that never appear in the paired training sentences and are not seen in any unpaired text, using only an external detector as the source of novel-object knowledge (Wu et al., 2018). News image captioning differs from generic image captioning because captions must connect the image to accompanying article context, often through named entities and article-derived facts that are not visually obvious (Qu et al., 2023). Out-of-context caption generation, motivated by cheapfakes, conditions generation on an image plus a compact textual context of named entities and their types in order to produce realistic news-style captions whose semantics can be steered toward particular people, places, organizations, dates, or events (Singh et al., 2024).

These task definitions show that NN-Caption is not restricted to surface description. It is repeatedly framed as a grounded language problem in which the model must decide which information is visually or acoustically observable, which information should be imported from auxiliary context, and how that information should be serialized into a sentence.

2. Architectural patterns

The dominant NN-Caption architecture is the encoder–decoder. A strong baseline form uses a CNN encoder and an autoregressive decoder such as an LSTM, GRU, or Transformer (Tan et al., 2019, Wang et al., 2018). One core design question is where multimodal fusion should occur. A systematic comparison of GRU-based captioners found that image features can either be injected into the RNN dynamics or merged after the RNN in a post-RNN multimodal layer; the reported conclusion is that it is not especially detrimental to performance whether one architecture is used or another, while the merge architecture allows the RNN hidden state vector to shrink in size by up to four times and trains faster with fewer parameters (Tanti et al., 2017).

Alternative decoders replace recurrence with convolution. "CNN+CNN: Convolutional Decoders for Image Captioning" uses a VGG-16 vision module, a causal 1D convolutional language decoder with GLUs, and visual attention over a $14 \times 14$ spatial grid, obtaining a model that is around 3 times faster than NIC during training time and achieves comparable BLEU and METEOR together with higher CIDEr than similar LSTM-based models (Wang et al., 2018). A different line augments recurrent decoders rather than replacing them: one study stacks a GRU after each LSTM in both attention and language pathways of a Top-Down captioner and reports BLEU-4 up to 0.406 and CIDEr 1.249 on MS COCO 2014 validation (Tan et al., 2019).

Attribute- and module-centric architectures explicitly decompose captioning into subtasks. AIC-AB Net combines adaptive spatial attention, a visual sentinel, and synchronously injected text attributes in a ResNet-152 plus LSTM encoder–decoder; its attribute extractor is a VGG-16-based classifier over the 1000 most frequent nouns and adjectives (Tu et al., 2023). "Image Captioning with Compositional Neural Module Networks" builds a hierarchical framework around Faster R-CNN regions, an Attention/Visual/Semantic LSTM trio, stacked Noisy-Or object detection, and specialized modules for color, count, size, spatial relation, and semantic relation (Tian et al., 2020). In medical captioning, concept-aware attention is implemented by first predicting refined UMLS concepts with Swin-V2 and then conditioning a BEiT+BioBART captioner on those concepts (Nguyen et al., 2024).

A further architectural theme is controllability through alignment signals. Noise-aware Captioning learns an alignment-level-controllable captioner by discretizing CLIP image–text similarity into $K$ bins, embedding the alignment level, and conditioning caption generation on that control variable; at inference, setting a high alignment level steers the model toward more descriptive and distinctive captions (Kang et al., 2022). This suggests that NN-Caption increasingly treats conditioning variables—alignment levels, attributes, concepts, or entity chains—not as auxiliary metadata but as first-class control signals in generation.

3. Corpora, benchmarks, and data regimes

NN-Caption research is organized around specialized corpora rather than a single universal benchmark. The datasets differ not only in modality but also in annotation protocol, caption length, and the degree of external context made available to the model.

Setting	Corpus or benchmark	Distinctive property
Audio captioning	AudioCaption hospital subset (Wu et al., 2019)	3710 clips, 3 Mandarin captions per 10 s clip, English translations
Medical captioning	ROCOv2 / ImageCLEFmedical Caption 2024 (Nguyen et al., 2024)	70,108 train, 9,972 validation, 17,237 test images
Detailed captioning	DetailCaps-100 and DetailCaps-4870 (Dong et al., 2024)	Long-form references, ≈176 and ≈122 words on average
Humorous captioning	BoketeDB (Yoshida et al., 2018)	999,571 image–caption pairs over 70,981 images with star counts
News captioning	GoodNews and NYTimes800k (Qu et al., 2023)	Image plus article context, face and entity grounding
Out-of-context news	COSMOS (Singh et al., 2024)	~200k images and ~450k captions for cheapfake-oriented generation
Caption-driven scanpaths	CapMIT1003 (Zanca et al., 2024)	1,003 images, 4,573 image–subject observations, 27,865 clicks
Zero-shot novel objects	Held-out MSCOCO protocol (Wu et al., 2018)	Eight novel object categories removed from caption training

The released AudioCaption data illustrate the importance of annotation design. Each 10-second clip receives three free-form Mandarin captions through a two-step process that elicits sounds heard, sound owner, attributes, and location before the annotator writes a sentence; all captions are then translated to English via Baidu Translation, and both languages are processed through the same tokenization pipeline (Wu et al., 2019). DetailCaps-100 and DetailCaps-4870, by contrast, are explicitly built for long-form “detail caption” evaluation and contain high-recall descriptions of objects, attributes, and relations rather than short gist sentences (Dong et al., 2024).

In domain-specific settings, the dataset often dictates the captioning problem itself. ROCOv2 couples images with a refined subset of UMLS 2022 AB concepts for medical concept detection and caption prediction (Nguyen et al., 2024). GoodNews and NYTimes800k couple images to articles, making retrieval of image-relevant sentences and named-entity grounding central to model design (Qu et al., 2023). COSMOS is used to generate realistic out-of-context captions rather than randomly mismatched negatives, because random caption swaps are argued to be linguistically biased and unrepresentative of real out-of-context scenarios (Singh et al., 2024).

4. Evaluation and the problem of caption correctness

NN-Caption evaluation has moved from corpus-level n-gram overlap toward more explicit tests of semantic adequacy, per-caption reliability, and hallucination control. Standard metrics remain BLEU, METEOR, ROUGE-L, CIDEr, and SPICE, and these appear across MS COCO, medical, news, and audio settings (Tan et al., 2019, Nguyen et al., 2024).

However, several studies show that standard overlap metrics can be misleading. In AudioCaption, BLEU scores are similar for Mandarin and English, and the model’s BLEU is near human BLEU under the paper’s reference protocol, yet human usefulness ratings show a clear gap: the model averages 1.89 versus humans 2.88 on a 1–4 scale, with a two-tailed t-test $p < .001$ (Wu et al., 2019). For long-form captions, "Benchmarking and Improving Detail Image Caption" argues that BLEU, ROUGE, CIDEr, and METEOR are brittle because short references miss fine-grained semantics and model-based metrics such as CLIPScore suffer truncation and length bias (Dong et al., 2024).

Two response strategies are especially notable. CAPTURE evaluates detailed captions by extracting objects, attributes, and relations with a textual scene-graph parser, then matching them through exact, synonym, and soft stages. On average across DetailCaps-100 and DetailCaps-4870, CAPTURE attains Pearson $\rho = 0.505$, compared with 0.437 for SPICE, 0.402 for METEOR, 0.262 for BLEU, and 0.102 for CIDEr, and it also leads on pairwise ranking fidelity (Dong et al., 2024). LCEval, by contrast, learns a neural composite metric from lexical, semantic, and syntactic features using a one-hidden-layer MLP whose output is the probability that a caption is human-like; on COMPOSITE it reaches Kendall’s $\tau = 0.418$, exceeding SPICE at 0.349, CIDEr-D at 0.342, and BLEU-4 at 0.213 (Sharif et al., 2020).

A third line scores correctness directly at the token level. Token-Level Confidence aggregates token probabilities or learned token-confidence estimates over words or sequences to judge image–caption consistency and rerank beam outputs. On Winoground, algebraic TLC achieves 27.0% image score and 17.5% group score on OFA-Large, surpassing previous state of the art; on SVO-Probes it improves verb understanding from 81.23% to 89.47%; and on MS COCO it reduces object hallucination rates by roughly 30% relative over the original model, while setting a new state of the art on CHAIR measures on the Karpathy test split (Petryk et al., 2023).

Taken together, these results indicate that NN-Caption evaluation is no longer reducible to a single corpus-level similarity score. The field increasingly separates fluency, semantic coverage, grounding fidelity, and per-token trustworthiness.

5. Grounding, control, and robustness

A defining trend in NN-Caption is the addition of explicit grounding mechanisms that constrain what the decoder may say. In medical image captioning, a high-precision Swin-V2 concept detector predicts UMLS-aligned concepts, thresholds them, and injects them into a BEiT+BioBART captioner; concept integration and post-processing jointly improve caption prediction, with the best reported test BERTScore of 0.5794 and a ninth-place competition ranking (Nguyen et al., 2024). In zero-shot novel object captioning, the Decoupled Novel Object Captioner fully decouples the language sequence model from object descriptions: a Sequence Model with the Placeholder emits <PL> tokens, and a key-value object memory built from Faster R-CNN detections fills those placeholders with detector labels, achieving $F1_{\text{avg}} = 57.92\%$ on held-out MSCOCO novel categories without extra text (Wu et al., 2018).

News-oriented systems similarly isolate grounding subproblems. "Visually-Aware Context Modeling for News Image Captioning" introduces a face-naming module, CLIP sentence retrieval, and CoLaM, a contrastive training objective with a frozen language-model backbone. The full BART-large system reaches CIDEr 71.96 on GoodNews and 71.65 on NYTimes800k, improving over the previous state of the art without external data by +7.97 and +5.80 CIDEr, respectively (Qu et al., 2023). NewsCaption conditions a Transformer captioner on CLIP image features, DETR object proposals, a relational object graph, and named entities together with their types encoded via GPT-2 BPE embeddings; on COSMOS validation it improves over the Transformer baseline by +6.2 BLEU-4, +2.96 CIDEr, +8.5 ROUGE, and +7.3 METEOR (Singh et al., 2024).

Noise robustness can also be framed as conditioning. Noise-aware Captioning uses CLIP cosine similarity to bucket web-crawled image–text pairs into alignment levels and trains a captioner on the full noisy corpus with alignment-level control. Under CC3M pretraining, a high alignment setting at inference yields zero-shot MS COCO CIDEr 62.04 and CLIPScore 66.70, outperforming vanilla training, filtering, and loss-weighting baselines (Kang et al., 2022). This suggests that many recent NN-Caption systems address robustness not by discarding uncertain data or context, but by making uncertainty and alignment explicit in the generative interface.

6. Automation, attention beyond decoding, and future directions

The most explicit attempt to automate NN-Caption design is the pipeline named "NN-Caption" (Jesani et al., 7 Dec 2025). It uses DeepSeek-R1-0528-Qwen3-8B to synthesize PyTorch image-captioning models by composing LEMUR classification backbones with LSTM, GRU, or Transformer decoders under a strict Net API. Prompt assembly includes a baseline captioning model, $K$ candidate component snippets from classification backbones, and detailed implementation rules; generated code is parsed, repaired, trained for 3 epochs on MS COCO with batch size 32, AdamW, and CrossEntropyLoss(ignore_index=0, label_smoothing=0.1), then scored with BLEU-4. Across 357 schema-valid archived models, over half trained successfully and produced meaningful captions, the best generated model reached BLEU-4 = 0.1192 within three epochs, and a controlled prompt ablation found a lower success rate when 10 snippets were provided rather than 5 (Jesani et al., 7 Dec 2025).

Attention modeling is also expanding beyond decoder internals toward human exploration priors. CapMIT1003 augments MIT1003 with click-contingent explorations during caption writing, and NevaClip predicts scanpaths by maximizing alignment between CLIP text embeddings of captions and CLIP vision embeddings of dynamically foveated images. On CapMIT1003, NevaClip with the correct caption attains the best reported ScanPath Plausibility score, 0.35 (0.18); on MIT1003 free-viewing it reaches 0.50 (0.17), tying the best condition (Zanca et al., 2024). This suggests a possible future NN-Caption direction in which fixation policies, foveated encoders, and caption decoders are optimized jointly rather than treated as separate components.

The future-work agendas in the cited papers are consistent. Audio captioning points to attention, transformers, and hierarchical temporal modeling to better capture co-occurrence and temporal order (Wu et al., 2019). Medical captioning points to uncertainty-aware concept weights, ontology-guided constraints, and LLM-augmented decoders (Nguyen et al., 2024). Detailed image captioning points to stronger scene-graph parsers, co-reference resolution, and region–text grounding for attributes and relations (Dong et al., 2024). News captioning identifies adaptive weighting of article and image context, and context modules for non-visually grounded information, as open problems (Qu et al., 2023). In aggregate, these directions imply that NN-Caption is evolving from a generic encoder–decoder paradigm into a family of controlled, grounded, and increasingly self-designing systems whose main research frontier is not sentence fluency alone, but reliable alignment between what is perceived, what is conditioned, and what is said.