Code-Grounded Caption Generation

• Code-grounded caption generation is a set of methods using code artifacts to produce natural language captions with verifiable, explicit semantic structures.
• Techniques include converting images to structured Python code and verbalizing formal program representations like SQL and Android source code.
• Empirical evaluations show improved factual consistency and retrieval performance, driven by self-consistency checks and execution-based corrections.

Code-grounded caption generation denotes a family of captioning methods in which natural-language descriptions are generated, refined, or supervised with direct reference to formal code representations rather than unconstrained text alone. In current arXiv literature, the term spans two principal regimes. One regime uses code as a grounding substrate for visual understanding: images are converted into structured Python snippets or executable reconstruction programs, and captions are then derived from those code artifacts or corrected by them (Wang et al., 2024, Guan et al., 11 Mar 2026). The other regime treats code itself as the semantic source to be verbalized, as in SQL2Text, privacy-caption generation from Android source code, and source-code summarization (Al-Lawati et al., 6 Jan 2025, Jain et al., 9 Jan 2026, Chen et al., 2018). Across these regimes, the common technical premise is that code supplies a more explicit semantic scaffold for alignment, verification, and controllability than free-form natural language.

1. Conceptual scope and task variants

The literature represented here instantiates several distinct but related notions of grounding. In multimodal data construction, code is introduced as a structured representation of visual content. World to Code (W2C) organizes image descriptions into a Python-style schema containing a global caption, object-level captions, OCR strings, and bounding boxes, while CodePercept constructs image-caption-code triplets in which executable Python code reproduces the image and serves as the factual basis for the caption (Wang et al., 2024, Guan et al., 11 Mar 2026). In code-to-text generation, the direction is reversed: the caption is a natural-language rendering of formal or program semantics, such as a SQL query, a privacy-relevant Android behavior, or the purpose of a source-code file (Al-Lawati et al., 6 Jan 2025, Jain et al., 9 Jan 2026, Chen et al., 2018).

Formulation	Representative work	Grounding object
Visual captioning grounded by code	W2C (Wang et al., 2024); CodePercept (Guan et al., 11 Mar 2026)	Python snippets; executable matplotlib code
Captioning natural language from code	Semantic Captioning (Al-Lawati et al., 6 Jan 2025); PCapGen (Jain et al., 9 Jan 2026); ARNet (Chen et al., 2018)	SQL; PDG-derived Java slices; source-code sequences
Adjacent captioning-system generation	NN-Caption (Jesani et al., 7 Dec 2025)	Strict Net API and generated model code

These variants differ in what counts as the primary semantic authority. In W2C, the authority remains the image, but the image is re-expressed as structured code. In CodePercept, executable code is treated as ground truth for caption construction. In SQL2Text and privacy-caption generation, the authority is the formal program itself. This distinction matters because the verification mechanisms, error modes, and evaluation protocols vary accordingly.

2. Visual captioning through structured or executable code

W2C presents a multi-modal data construction pipeline that converts a raw image set $D_{\text{raw}}$ into a structured code dataset $D_{W2C}$ through four stages: Visual-Concept Extraction, Self-Instructed Information Extraction, Consistency Filtering, and Structured Formatting. The pipeline first generates a simple caption $o_g$ and a long caption $o_d$, applies noun-phrase extraction and GroundingDINO to obtain concepts $C=\{c_i\}$ and boxes $B=\{b_i\}$, then queries the VLM twice per region for a compositional caption $o_{\text{desc}}(c_i)$ and OCR output $o_{\text{ocr}}(c_i)$. The final representation is folded into a Python snippet in which the global caption becomes a class doc-string and each object is stored as a dictionary with "concept", "caption", "text", and "bbox" fields; duplicate concepts may be grouped into lists (Wang et al., 2024).

A key feature of W2C is that generation and validation are both performed by the VLM itself. Identical concepts are merged and checked by a counting prompt, while beam candidates for region captions are re-ranked by asking whether each extracted sub-concept is a valid and visible visual concept. This self-instructed loop is designed to reduce inconsistency without external caption/OCR specialists or stronger VLM APIs. The paper explicitly argues that code parsing ability presents better cross-modal equivalence than the commonly used detail caption ability, and its downstream evaluations are organized to test that claim (Wang et al., 2024).

CodePercept generalizes the visual-code-grounding idea further by constructing ICC-1M, a dataset of approximately $1{,}000{,}000$ $(\text{image}, \text{caption}, \text{code})$ triplets spanning chart-style STEM visuals, geometric diagrams, and physical schematics. The code is executable Python using matplotlib and is intended to reproduce the image when run. Dataset construction combines three synthesis pipelines—Image Reproduction, Image Diversity, and Solid Geometry—followed by three automatic filters $D_{W2C}$0, $D_{W2C}$1, and $D_{W2C}$2, plus human spot-checks on a random $D_{W2C}$3. After filtering, the average code execution success rate exceeds $D_{W2C}$4 (Guan et al., 11 Mar 2026).

In CodePercept, the caption is not merely paired with code; it is corrected by code analysis. A draft caption is first generated from the image, the executable code is then run and traced to extract exact primitives and facts, and a second decoding step performs a factual correction. The paper states that natural-language-only captions frequently hallucinate counts, positions, colors, and relationships, whereas executable code is a formal, unambiguous specification of plotted elements. The reported result is a captioning regime with zero uncertain quantifiers and exact numeric statements in place of vague terms such as “several” (Guan et al., 11 Mar 2026).

3. Captioning semantics from formal and program representations

Al-Lawati et al. define semantic captioning for SQL2Text as the task of generating a natural-language question $D_{W2C}$5 that is semantically equivalent to a SQL query $D_{W2C}$6. Their formulation emphasizes an asymmetry with Text2SQL: many utterances can map to one canonical SQL query, but SQL2Text must map one SQL query to potentially infinitely many paraphrases. The paper identifies three immediate difficulties: diversity of valid paraphrases, the lack of large SQL$D_{W2C}$7NL corpora, and the need to avoid hallucinated logic. To address this, it repurposes CoSQL, Spider, and SParC through an iterative in-context-learning loop using GPT-4o with Generation, Feedback, and Refinement steps, yielding three utterances per SQL query and improving AlignScore by approximately $D_{W2C}$8 on average after refinement; human vetting over $D_{W2C}$9 samples produced Fleiss’s $o_g$0 before inconsistent items were dropped (Al-Lawati et al., 6 Jan 2025).

PCapGen addresses a different code-to-text problem: automatic generation of privacy captions from Android source code. Its pipeline begins with a catalog of privacy-related Android APIs, identifies call sites through static analysis, computes backward and forward program slices via a Program Dependence Graph, assembles the slices into Java snippets of up to $o_g$1–$o_g$2 lines, and serializes each snippet as a JSON record containing the app name, a behavior identifier, and the code body. Caption generation then proceeds in two prompt stages: a coarse-grained “Brief Caption” describing what type of personal information is accessed or transmitted and for what high-level purpose, and a fine-grained “Expanded Description” that enumerates the exact data element, source, processing or transmission behavior, and stated purpose or user benefit (Jain et al., 9 Jan 2026).

An earlier code-captioning line is represented by ARNet, which augments an encoder-decoder caption generator with an Auto-Reconstructor Network that reconstructs the previous decoder hidden state from the current one. The model is evaluated not only on image captioning but also on source code captioning using HabeasCorpus, a dataset of $o_g$3 Java source-code files paired with a single summarizing comment and split into $o_g$4 training, $o_g$5 validation, and $o_g$6 testing examples. In this setting, the code is treated as the source sequence to be summarized, and the contribution lies in regularizing recurrent transition dynamics to improve the generated description (Chen et al., 2018).

4. Representations, filters, and structural priors

A central design choice in code-grounded caption generation is the choice of intermediate representation. W2C makes that representation explicit as

$o_g$7

with $o_g$8. This schema is compact but information-bearing: it binds concept identity, attribute-rich captioning, OCR, and localization in a single object record. Its caption re-ranking stage assigns

$o_g$9

and selects the beam with maximal score. The method therefore uses code-like structure together with explicit self-consistency checks rather than relying on formatting alone (Wang et al., 2024).

In SQL2Text, the relevant structural prior is the Abstract Syntax Tree. A SQL query is parsed into a graph $o_d$0 whose nodes correspond to keywords, functions, columns, tables, and operators, and whose edges encode parent-child relations. Demonstration retrieval is then driven by a two-layer GCN encoder,

$o_d$1

followed by graph-level pooling and cosine similarity $o_d$2 for AST-ICL-TOP selection. Here, code grounding is not execution-based but structure-based: the captioning model is steered by graph proximity in the formal program space rather than lexical overlap alone (Al-Lawati et al., 6 Jan 2025).

PCapGen uses program analysis as its structural prior. The backward slice identifies where sensitive data originates, and the forward slice identifies how the data propagates, for example whether it is stored, transmitted, or logged. The assembled snippet preserves commented declarations, relevant control statements, and data-flow operations. This means the LLM is not prompted on an entire source file, but on a program slice already filtered for privacy-relevant semantics. The grounding signal therefore resides in static-analysis retrieval and context assembly rather than in end-to-end neural perception (Jain et al., 9 Jan 2026).

ARNet introduces a different kind of structural constraint: temporal reconstruction inside the decoder. Its additional loss

$o_d$3

is combined with the standard captioning loss as $o_d$4. In source-code captioning, this regularizer is intended to make each current hidden state carry more information from the previous one, thereby improving long-range dependency modeling and narrowing the discrepancy between teacher-forced training and free-running inference (Chen et al., 2018).

CodePercept adds yet another grounding primitive: executable tracing. The code $o_d$5 is executed, a tracer $o_d$6 records plotted elements, and exact facts extracted from execution are used to surgically correct a draft caption. This mechanism is stronger than textual self-consistency because it can verify geometric primitives, colors, coordinates, and counts through execution. The paper’s joint supervised fine-tuning objective optimizes caption and code generation together with $o_d$7 by default, and its second training stage refines code generation using Group Rel. Policy Opt. with $o_d$8, $o_d$9, and $C=\{c_i\}$0 rollouts per sample (Guan et al., 11 Mar 2026).

5. Empirical behavior and benchmark outcomes

W2C reports consistent downstream gains across both question answering and grounding. On LLaVA-NeXT-7B, replacing ShareGPT4V with W2C improves accuracy on $C=\{c_i\}$1 visual question answering benchmarks, including MMT-Bench from $C=\{c_i\}$2 to $C=\{c_i\}$3. On LLaVA-NeXT-13B, the average lift is $C=\{c_i\}$4–$C=\{c_i\}$5 across ScienceQA, ChartQA, and DocVQA. In visual grounding, the average IoU gains are $C=\{c_i\}$6 for LLaVA-1.5-7B, $C=\{c_i\}$7 for LLaVA-1.5-13B, $C=\{c_i\}$8 for LLaVA-NeXT-7B, and $C=\{c_i\}$9 for LLaVA-NeXT-13B. The few-shot “code-parsing” versus “detail-caption” comparison without images is especially diagnostic: for Qwen-1.5-14B, GQA 2-shot rises from $B=\{b_i\}$0 to $B=\{b_i\}$1 for the 7B setting and from $B=\{b_i\}$2 to $B=\{b_i\}$3 for the 13B setting, while MME 4-shot rises from $B=\{b_i\}$4 to $B=\{b_i\}$5 and from $B=\{b_i\}$6 to $B=\{b_i\}$7, respectively (Wang et al., 2024).

CodePercept evaluates directly on STEM2Code-Eval, a benchmark of $B=\{b_i\}$8 manually curated images with ground-truth code. On the 4B scale, Qwen3-VL-4B-Instruct obtains Image Score $B=\{b_i\}$9, Code Score $o_{\text{desc}}(c_i)$0, Exec Rate $o_{\text{desc}}(c_i)$1, and Avg $o_{\text{desc}}(c_i)$2, whereas CodePercept-4B-R1 reaches $o_{\text{desc}}(c_i)$3, $o_{\text{desc}}(c_i)$4, $o_{\text{desc}}(c_i)$5, and $o_{\text{desc}}(c_i)$6. On the 8B scale, Avg rises from $o_{\text{desc}}(c_i)$7 for Qwen3-VL-8B-Instruct to $o_{\text{desc}}(c_i)$8 for CodePercept-8B-R1, with Exec Rate improving from $o_{\text{desc}}(c_i)$9 to $o_{\text{ocr}}(c_i)$0. On the 32B scale, Qwen3-VL-32B-Instruct scores Avg $o_{\text{ocr}}(c_i)$1, while CodePercept-32B-R1 reaches $o_{\text{ocr}}(c_i)$2 with $o_{\text{ocr}}(c_i)$3 execution rate. The reported improvements are $o_{\text{ocr}}(c_i)$4 to $o_{\text{ocr}}(c_i)$5 points in Avg over baselines of the same size (Guan et al., 11 Mar 2026).

In SQL2Text, graph-aware in-context learning produces the largest gains for smaller models. Under the $o_{\text{ocr}}(c_i)$6 demo setting, GPT-J improves from Random-2 BLEU $o_{\text{ocr}}(c_i)$7 to AST-ICL-TOP-2 BLEU $o_{\text{ocr}}(c_i)$8, a $o_{\text{ocr}}(c_i)$9 relative increase. Mistral improves from $1{,}000{,}000$0 to $1{,}000{,}000$1 $1{,}000{,}000$2, and CodeLlama from $1{,}000{,}000$3 to $1{,}000{,}000$4 $1{,}000{,}000$5. GPT-4 benefits less dramatically, moving from zero-shot BLEU $1{,}000{,}000$6 to BM25-2 $1{,}000{,}000$7 and AST-ICL-TOP-2 $1{,}000{,}000$8. The paper also reports that AST-ICL-TOP significantly outperforms Random and BM25 across metrics according to t-tests, and that nested and aggregate queries receive the largest relative BLEU improvements (Al-Lawati et al., 6 Jan 2025).

PCapGen compares favorably against a zero-shot LLM baseline that prompts directly on the full file without slicing and against PriGen. On approximately $1{,}000{,}000$9 code slices from $(\text{image}, \text{caption}, \text{code})$0 real-world open-source Android apps, BLEU-4 improves from $(\text{image}, \text{caption}, \text{code})$1 to $(\text{image}, \text{caption}, \text{code})$2, ROUGE-L from $(\text{image}, \text{caption}, \text{code})$3 to $(\text{image}, \text{caption}, \text{code})$4, Precision from $(\text{image}, \text{caption}, \text{code})$5 to $(\text{image}, \text{caption}, \text{code})$6, Recall from $(\text{image}, \text{caption}, \text{code})$7 to $(\text{image}, \text{caption}, \text{code})$8, and F1 from $(\text{image}, \text{caption}, \text{code})$9 to $D_{W2C}$00. Human evaluators prefer PCapGen captions $D_{W2C}$01 of the time, and GPT-4 used as judge prefers them $D_{W2C}$02 of the time. All improvements are reported as statistically significant under the Wilcoxon signed-rank test with $D_{W2C}$03 (Jain et al., 9 Jan 2026).

ARNet reports that on the HabeasCorpus test split, the attentive encoder-decoder baseline achieves BLEU-4 $D_{W2C}$04, METEOR $D_{W2C}$05, and ROUGE-L $D_{W2C}$06, whereas attentive ARNet reaches BLEU-4 $D_{W2C}$07, METEOR $D_{W2C}$08, and ROUGE-L $D_{W2C}$09. The stated relative improvement in BLEU-4 over the attentive baseline is approximately $D_{W2C}$10. The model also reduces the train-test discrepancy measures $D_{W2C}$11 from $D_{W2C}$12 to $D_{W2C}$13 and $D_{W2C}$14 from $D_{W2C}$15 to $D_{W2C}$16, supporting the claim that reconstructing past states with present ones mitigates drift between training and inference (Chen et al., 2018).

6. Misconceptions, limitations, and adjacent directions

A common misconception is that code grounding is equivalent to rendering a caption in code-like syntax. The literature does not support that simplification. W2C attributes its gains to a combination of structured formatting and two explicit self-consistency filters, and its ablations show that applying only counting or only re-ranking yields partial gains, whereas both together deliver the largest downstream improvements. This suggests that the utility of code formatting depends on accompanying validation machinery rather than on serialization alone (Wang et al., 2024).

A second misconception is that structural retrieval helps all LLMs equally. SQL2Text shows a more qualified picture: AST-aware demonstration selection strongly benefits GPT-J, Mistral, and CodeLlama, but GPT-4 improves less dramatically under in-context learning. The same paper also notes that the dataset remains modest at roughly $D_{W2C}$17 SQL queries and that real-world SQL corpora can be noisy, with excessively inconsistent examples discarded. PCapGen likewise identifies concrete operational limits: large apps can exceed LLM context limits and require chunking, static analysis can miss reflection or native code, and the privacy API catalog must be maintained for new Android releases (Al-Lawati et al., 6 Jan 2025, Jain et al., 9 Jan 2026).

For executable visual grounding, the main strength is determinism, but the domain assumptions are stronger. CodePercept focuses on STEM visuals for which reconstruction code can be generated and executed, and its benchmark evaluates image understanding through executable code generation rather than problem-solving accuracy. A plausible implication is that domains lacking faithful reconstruction code may not benefit from the same degree of factual correction, even if the general principle of structured grounding remains useful (Guan et al., 11 Mar 2026).

An adjacent but distinct research direction uses code not to ground captions, but to synthesize captioning systems themselves. NN-Caption prompts DeepSeek-R1-0528-Qwen3-8B to generate runnable image-captioning models under a strict Net API, then parses, fixes, repairs, trains, and evaluates the resulting code inside the LEMUR framework. With $D_{W2C}$18 input snippets, approximately $D_{W2C}$19 of generated models train successfully; with $D_{W2C}$20 snippets, the success rate drops to approximately $D_{W2C}$21. The best BLEU-4 reported after $D_{W2C}$22 epochs on MS COCO is $D_{W2C}$23 for the C5C-RESNETLSTM family, exceeding a baseline ResNet+LSTM score of $D_{W2C}$24 at the same epoch budget. This line is adjacent rather than identical to code-grounded caption generation proper, because the code artifact specifies the captioner rather than the caption content (Jesani et al., 7 Dec 2025).

Taken together, these works define code-grounded caption generation as a technically heterogeneous but increasingly coherent area. Some methods use code as a latent or explicit semantic representation for images, some verbalize formal programs directly, and some treat executable traces or program slices as the basis for factual correction. The recurring result is not that code universally replaces language, but that when code carries verifiable structure—bounding boxes, AST relations, PDG slices, or executable drawing primitives—it can materially improve factual fidelity, retrieval quality, and evaluation rigor across captioning tasks.