---
title: 'LLMShot: Prompting and Retrieval Methods'
url: https://www.emergentmind.com/topics/llmshot
type: topic
---

# LLMShot: Prompting and Retrieval Methods

Searching arXiv for recent papers related to “LLMShot” and adjacent few-shot LAG/ICL methods.
LLMShot denotes a family of methods in which large language models are used in few-shot or zero-shot regimes, typically without parameter updates, by converting task-specific evidence into promptable context or by selecting a small set of demonstrations that steer prediction. Across the literature, the term is used both narrowly, as the name of a snapshot-testing framework for semantic UI-diff analysis, and more broadly, as a paradigm for solving multimodal, multilingual, software-engineering, and reasoning tasks through retrieval, demonstration selection, structured prompting, or teacher-assisted distillation [2507.10062]. The unifying pattern is that competence is elicited at inference time through prompt construction, shot selection, or lightweight external memory rather than supervised end-task training, although the exact operationalization varies substantially across domains [2305.19821].

## 1. Definition and conceptual scope

In its broadest sense, LLMShot refers to performing tasks with large language models using only a handful of in-context examples or auxiliary prompt context, without parameter updates, to generalize to new tasks, modalities, or languages [2403.16512]. This framing includes standard in-context learning, cross-lingual in-context learning, retrieval-augmented prompting, multimodal caption-mediated prompting, and shot-selection strategies that choose demonstrations rather than sampling them randomly [2412.12852].

Several papers explicitly cast disparate systems into this paradigm. LMCap is described as a “prototypical LLMShot method” for multilingual image captioning because it converts visual input into retrieved captions and then prompts a multilingual language model with fixed demonstrations [2305.19821]. Work on shot-level video editing formulates an “LLMShot pipeline” in which each shot is mapped to a unified textual representation, enabling selection, ordering, and classification entirely in the language domain [2505.12237]. Research on low-resource languages defines LLMShot as few-shot in-context learning and cross-lingual in-context learning for 25 low-resource and 7 relatively higher-resource languages [2403.16512]. In software engineering, selective shot learning for code explanation studies how few-shot example selection can materially alter code-explanation quality across open-source Code-LLMs [2412.12852].

A narrower use of the term appears in “LLMShot: Reducing snapshot testing maintenance via LLMs,” where LLMShot is the name of a vision-based framework for automatically analyzing snapshot test failures through hierarchical classification of UI changes [2507.10062]. That work is not merely an instance of the broader paradigm; it fixes the term as a proper system name for semantic snapshot-test triage.

This suggests that “LLMShot” is best understood as an umbrella label for prompt-centric, shot-centric, or retrieval-centric use of LLMs, rather than as a single algorithm. A plausible implication is that the literature converges less on a specific architecture than on a design principle: move task adaptation into prompt assembly, external retrieval, or demonstration choice.

## 2. Core operational patterns

The broad literature organizes LLMShot around a small number of recurring mechanisms. One is few-shot prompting with curated demonstrations. Another is retrieval-augmented prompting, where external text or parallel examples are selected and inserted into the prompt. A third is caption- or storyboard-mediated multimodal reduction, in which raw perceptual input is first converted into text so that the LLM operates entirely in its native token space. A fourth is teacher-assisted amortization, where an LLM is used offline to generate data or scores, and a smaller model absorbs that supervision [2412.09807].

In low-resource language learning, the central mechanism is in-context learning with semantically relevant exemplars and, when needed, query alignment via parallel sentence pairs that align input semantics between a low-resource target language and a high-resource source language [2403.16512]. In code explanation, selective shot learning ranks candidate demonstrations by token overlap, CodeBERT cosine similarity, or entity overlap derived from UniversalNER, with the selected top-$k$ pairs prepended to the prompt [2412.12852]. In fairness-aware classification, prompt assembly is formalized as $p = C(\tau, \eta, \kappa, \pi)$, where $\tau$ is the task instruction, $\eta$ the demonstrations, $\kappa$ the query instance, and $\pi$ a fairness policy expressed as a natural-language rule [2402.18502].

In multimodal settings, the pattern is often “textualization before reasoning.” LMCap retrieves similar captions using multilingual CLIP and supplies those captions, plus fixed few-shot demonstrations, to XGLM; the generation model never directly processes the image [2305.19821]. The shot-level video editing framework similarly maps each shot to a structured row $x_s = U(ID_s, \tau_s, size_s, angle_s, motion_s, desc_s, subs_s)$, replacing raw frames with a Markdown-like textual abstraction [2505.12237]. In few-shot VQA, one branch generates a caption and then asks the frozen LLM to answer the question from caption plus question, while another injects visual embeddings directly into the LLM’s token space through a trainable adapter [2403.11317].

The common claim across these settings is not that one interface universally dominates, but that prompting design and example selection frequently dominate end performance. For few-shot VQA, the winner depends critically on example selection: zero-shot favors caption-based prompting, whereas few-shot with joint question+image retrieval can favor embedding-based conditioning [2403.11317]. For low-resource ICL, semantic relevance of exemplars is central, and label alignment can harm performance while query alignment is often safer [2403.16512]. For code explanation, intelligently selected demonstrations improve BLEU, ROUGE-L, and METEOR relative to zero-shot and random few-shot [2412.12852].

## 3. Retrieval, selection, and alignment

Shot selection is a major technical axis of LLMShot systems. The literature distinguishes random selection, similarity-based retrieval, diversity-based selection, demographic-aware strategies, syntax-aware entity matching, and cross-lingual alignment.

In demographic fairness for few-shot NLP classification, six main strategies are studied: zero-shot, random, similarity, diversity, within, and representative/stratified; two hybrids, within + similarity and within + diversity, are also reported in ablations [2311.08472]. Similarity uses SBERT all-mpnet-base-v2 and cosine similarity to select the top-$k$ nearest neighbors for each test instance. Diversity runs KMeans with $K = k$ and picks a centroid-nearest representative per cluster. Within restricts the shot pool to the test instance’s demographic group, whereas representative/stratified balances groups globally in the fixed shot set [2311.08472]. The paper concludes that similarity is the most reliable trade-off on average, while within is often the least effective for both F1 and fairness [2311.08472].

In fairness-aware income classification, retrieval is implemented with text-embedding-ada-002 and Chroma over 46,621 non-test Adult instances, and $k = 20$ retrieved demonstrations are concatenated with a fairness rule $\pi$ [2402.18502]. The retrieval itself is similarity-based; fairness is imposed via prompt instructions such as “Ensure Demographic Parity on the basis of ‘gender’” or “Ensure fair results on basis of ‘gender’” [2402.18502].

In code explanation, selection methods are more syntax-aware. Token-based selection computes Jaccard similarity over preprocessed code tokens,
$$
Selection\_token = \frac{|tokens_{target} \cap tokens_{candidate}|}{|tokens_{target} \cup tokens_{candidate}|},
$$
while embedding-based selection uses CodeBERT cosine similarity, $Selection\_{semantic}(q,d) = cos(\vec{q}, \vec{d})$ [2412.12852]. The proposed SSL\_ner extracts programming entities such as function, library, class, data structure, algorithm, data type, variable, and value, then scores overlap by
$$
s_{e_i}(q,d) = \frac{|e_i(q) \cap e_i(d)|}{|e_i(q) \cup e_i(d)|},
$$
and
$$
sim_{ne}(q,d) = \sum_{i=1}^{|E|} w_{e_i} \cdot s_{e_i}(q,d),
$$
with $w_{e_i} = 0$ for ‘data type’, ‘variable’, ‘value’ and $w_{e_i} = 1$ otherwise [2412.12852]. This yields an interpretable and customizable retrieval signal keyed to programming entities.

In low-resource language ICL, retrieval is tied to alignment. Query alignment selects the top-$k$ parallel sentence pairs from $D_{para}$ whose target side is most similar to the target query, then inserts pairs such as “In $L_{tgt}$, $s^{src}_1$ means $s^{tgt}_1$” into the prompt [2403.16512]. The predicted label is chosen by
$$
c_{pred} = \arg\max_c f(X_{icl} + X_{align} + q_{tgt} + c),
$$
with source-only labels generally outperforming target-only labels or label alignment [2403.16512].

A related retrieval regime appears in multilingual image captioning. LMCap encodes the image and datastore captions with M-CLIP, retrieves top-$K$ captions by inner product using FAISS IndexFlatIP, and re-ranks generated captions by image–text similarity:
$$
s(i,j) = e_i \cdot e_{t_j},
$$
and
$$
\hat{h}^* = \arg\max_m s(i,\hat{h}_m).
$$
The default configuration is $K = 4$ retrieved captions, $N = 3$ shots, beam size $= 3$, and $c = 3$ candidates [2305.19821].

## 4. Textual interfaces for multimodal tasks

A major strand of LLMShot research proceeds by replacing direct multimodal fusion with textual surrogates. This is not a single method but a recurrent interface choice.

LMCap is “image-blind” at generation time: XGLM never sees pixels and conditions only on retrieved captions and the few-shot template [2305.19821]. Its prompt template is:
> “I am an intelligent image captioning bot. Similar images have the following captions: \<caption 1\> \<caption 2\> \<caption 3\> \<caption 4\>. A creative short caption I can generate to describe this image in \<language\> is:”

Each prompt begins with $N = 3$ fixed demonstrations showing how four retrieved captions map to a gold caption in the target language [2305.19821]. The model uses M-CLIP retrieval, XGLM-2.9B generation, beam search with beam $= 3$, and image–text re-ranking, all in inference-only mode on a single NVIDIA V100S 32GB GPU [2305.19821].

Few-shot VQA with frozen LLMs studies two text–vision interfaces under controlled conditions. Caption-to-text first generates a caption from visual tokens and then concatenates caption and question as text; embedding-to-LLM maps image features to $k$ visual tokens and prepends them to the question embeddings [2403.11317]. The frozen LLM is Flan-T5 XL, the image encoder is CLIP ViT-G, and the main system uses a non-linear MLP adapter:
$$
h = \phi(W_1 x + b_1), \quad f(x) = reshape(W_2 h + b_2) \in \mathbb{R}^{k \times d_t},
$$
with $k = 20$, $d_v = 1280$, and $d_t = 2048$ [2403.11317]. Zero-shot accuracy favors caption-based prompting, 45.9 versus 41.4, but with joint question+image retrieval the embedding-based interface reaches 52.4 at 4-shot compared with 50.8 for caption-based [2403.11317].

In video editing, L-Storyboard converts each segmented shot into a language object. The mapping is formalized as
$$
f(s) = U(ID_s, \tau_s, h_{shot}(F_s), p_{vlm}(F_s), t_{asr}(A_s)),
$$
where ShotTransformer extracts shot attributes, MiniCPM-o generates content descriptions, and Whisper ASR produces subtitles with timestamps [2505.12237]. The resulting row contains shot size, angle, motion, content description, and subtitles; raw visuals are replaced with textual abstractions for privacy and interpretability [2505.12237]. This interface supports Shot Attributes Classification, Next Shot Selection, and Shot Sequence Ordering [2505.12237].

These systems support a general claim that LLMShot often works by constructing a task-specific textual bottleneck. This suggests that the textual bottleneck is not merely an implementation detail but a recurring abstraction layer that makes LLM reasoning possible over modalities the base model does not natively encode.

## 5. Empirical results across domains

The empirical record is heterogeneous because tasks, metrics, and objectives differ, but several representative results define the state of the paradigm.

On XM3600, LMCap is competitive with fully supervised multilingual captioning despite using no supervised caption training. CIDEr on English, Spanish, Hindi, and Chinese is reported as 0.452, 0.329, 0.132, and 0.221, respectively; LMCap+ with a CCS-augmented datastore reaches 0.526, 0.326, 0.078, and 0.251 [2305.19821]. Supervised baselines trained on machine-translated COCO/CC3M-35L include BB+CC at 0.584, 0.425, 0.197, 0.202 and BB at 0.297, 0.194, 0.098, 0.087 [2305.19821]. LMCap outperforms all supervised approaches on Chinese, 0.221 versus best 0.202, and captions the full XM3600 set in about 45 minutes per language, while supervised baselines used about 5000 TPU hours [2305.19821].

In few-shot VQA on the VQAv2 validation split of 214K image–question–answer triplets, caption-based zero-shot with Flan-T5 XL reaches 45.9 accuracy, significantly above embedding-based zero-shot at 41.4 [2403.11317]. Under joint question+image similarity retrieval, embedding-based prompting improves to 50.5, 51.8, and 52.4 at 1-, 2-, and 4-shot, slightly above caption-based 50.0, 50.3, and 50.8 [2403.11317]. With question-only retrieval, caption-based remains stronger, such as 47.6 versus 40.6 at 1-shot and 48.5 versus 47.3 at 4-shot [2403.11317].

In shot-level video editing, Qwen3:32B with audio+video reaches 51.63% on AVE for Next Shot Selection, surpassing contrastive baselines at approximately 37.5–41.4% [2505.12237]. For Shot Sequence Ordering, traditional MViT attains Top-1 accuracy 32.04% and Mean KTD 1.345, while StoryFlow obtains Top-1 29.82% and Mean KTD 1.205, which is the lowest KTD among evaluated methods and narrows the gap to the best video model without training [2505.12237].

In code explanation, selective shot learning consistently improves automatic metrics. On CoNaLa with Llama2-Coder-7B, BLEU rises from 0.292 in zero-shot to 0.364 with random few-shot, 0.393 with Selection\_token, 0.405 with Selection\_semantic, and 0.408 with SSL\_ner; ROUGE-L rises from 0.298 to 0.419, and METEOR from 0.236 to 0.386 across the same progression [2412.12852]. For CodeUp-13B, BLEU reaches 0.412 with SSL\_ner, compared with 0.382 for Selection\_token and 0.402 for Selection\_semantic [2412.12852]. On TLC, gains are often larger for function-level Java intents, such as “property” with StarCoder: BLEU 0.467, 0.544, and 0.558 for Selection\_token, Selection\_semantic, and SSL\_ner, respectively [2412.12852].

In simultaneous machine translation, open-source LLMs can be competitive zero-shot. On TED-TST-2023 en-de, the prompt-based system reports BLEU 22.13, AL 1360.59, and LAAL 2089.16, compared with NAIST at BLEU 21.39 and TRANSLLAMA at 19.36 [2406.13476]. On AMBIEVAL, a technical domain benchmark, the same system reaches BLEU 42.60, above NAIST at 39.80, SEAMLESSSTREAMING at 29.76, and TRANSLLAMA at 32.43 [2406.13476]. Background information improves TED-TST-2024 BLEU by +5.62 for en-de, +2.81 for en-fr, +3.88 for en-it, +3.77 for en-es, and +2.76 for en-ru [2406.13476].

In low-resource language learning, few-shot ICL improves over zero-shot in most cases, label alignment underperforms in about 88.46% of runs with roughly $-20\%$ Weighted F1, and query alignment outperforms no alignment in 56.25% of runs with roughly $+10\%$ Weighted F1 [2403.16512]. Source-only label configuration consistently outperforms target-only labels and label alignment in most languages [2403.16512].

In MCQA distillation, DeBERTa-v3-base finetuned directly on 5-shot examples obtains 28.9% average accuracy on MMLU, while JSON-generated data with soft-label distillation reaches 39.3%, a gain of 10.4 points [2412.09807]. Category scores are 32.5 for STEM, 43.2 for Social, 44.3 for Humanities, and 40.6 for Other [2412.09807].

In semantic snapshot-test analysis, gemma3:12b reaches Hit Rate 82.35%, Recall 84.21%, Precision 66.67%, and F1 74.42%, while gemma3:4b yields Hit Rate 76.47%, Recall 78.95%, Precision 57.69%, and F1 66.67% [2507.10062]. The 12B variant also reduces pixel\_diff error from 0.095 ± 0.161 to 0.059 ± 0.048 relative to the 4B model [2507.10062].

## 6. Fairness, control, and failure modes

Fairness is a distinct subtheme in LLMShot research because the few-shot setting constrains standard interventions such as finetuning or direct parameter regularization.

“Few-Shot Fairness” studies fairness-aware classification on UCI Adult using fairness rules written directly into the prompt. The paper formalizes the input as $\hat{y}^i = L(C(\tau, \eta^i, \kappa(\lambda^i), \pi))$ and evaluates Demographic Parity, Equal Opportunity, Equalized Odds, Overall Accuracy Equality, Treatment Equality, Causal Discrimination, Fairness Through Unawareness, and Generic Fairness [2402.18502]. GPT-4 is the most reliable model under this protocol. In zero-shot, adding fairness rules improves accuracy from 0.76 to 0.79 with $\pi_A$ and raises F1 from 0.75 to 0.79; in few-shot, accuracy remains around 0.72–0.73 while several fairness gaps decrease [2402.18502]. Gemini and LLaMA-2-70B are less reliable, with Gemini often degrading when fairness prompts are added [2402.18502].

A complementary line asks how shot selection itself affects fairness. “Selecting Shots for Demographic Fairness in Few-Shot Learning with Large Language Models” uses a modified Equal Opportunity score, 1-GAP, based on worst-pair TPR differences across demographic groups [2311.08472]. Across models and datasets, moving from zero-shot to random 10-shot improves F1 in about 70% of cases but worsens fairness in about 60% of cases [2311.08472]. Similarity-based selection yields the best average F1 and a reasonable fairness profile, while within-group matching is often ineffective [2311.08472]. On HateXplain, mean macro-F1 and 1-GAP across models are 45.8 and 86.6 for zero-shot, 49.6 and 78.9 for random 10-shot, and 52.1 and 77.5 for similarity [2311.08472].

Failure modes are recurrent across domains. LMCap’s image-blindness means mismatched or noisy retrieved captions can cause errors; domain shift and datastore bias are explicit limitations [2305.19821]. In few-shot VQA, caption-based systems can hallucinate or omit crucial details, while embedding-based systems become unstable when in-context examples are poorly matched [2403.11317]. In low-resource ICL, label alignment can distort the output distribution, especially with weak label-token representations and unfamiliar scripts [2403.16512]. In code explanation, UniversalNER outputs were not formally validated, so extraction errors can affect selection [2412.12852]. In snapshot analysis, prompt-only selective ignore is unreliable: under Ignore From Analysis, hit rate falls to 18.75%, recall to 28.57%, precision to 9.09%, and ignore compliance is only 31.25% [2507.10062].

These results clarify a common misconception: adding shots is not inherently beneficial. In several settings, relevance, alignment, and formatting matter more than shot count, and inappropriate shots can reduce robustness, fairness, or accuracy.

## 7. System design variants and future directions

Although LLMShot methods often emphasize training-free inference, the literature also includes amortized variants in which the LLM is used once to create supervision for a smaller deployable model. In MCQA, the teacher LLM generates approximately 1,024 synthetic items per subject from 5 seed examples, scores answer choices with next-token logits over letters, and provides soft targets
$$
p_t(c) = \frac{\exp(\tilde{z}_c)}{\sum_{c'} \exp(\tilde{z}_{c'})},
$$
which are used to train DeBERTa-v3-base via
$$
L_{distill} = CE(p_t, p_s) = - \sum_c p_t(c)\log p_s(c)
$$
[2412.09807]. This variant extends LLMShot beyond pure prompting into offline data generation and distillation.

Another direction is structured stabilization for divergent tasks. StoryFlow addresses output instability in shot sequence ordering by sweeping temperatures $T = \{0.0, 0.2, \ldots, 2.0\}$, generating candidate orderings and narratives, and then using an LLM evaluator to choose the most coherent story [2505.12237]. This converts divergent generation into a convergent selection mechanism and can be interpreted as temperature-swept ensembling plus prompt-defined re-ranking [2505.12237].

The multilingual and low-resource literature points toward broader language coverage through better cross-lingual alignment, stronger multilingual datastores, and semantically relevant query alignment rather than explicit label mapping [2305.19821]. The simultaneous translation literature highlights response priming, assistant-side partial target conditioning, and lightweight background JSON as practical prompt controls for streaming behavior and terminology adherence [2406.13476]. The code-explanation literature points toward hybrid selection functions that combine entity-aware, token-level, and embedding-level signals, although the combined score is presented only as future work [2412.12852].

The snapshot-testing interpretation of LLMShot suggests a different future agenda: stronger controllability for selective filtering, potentially through region masks, programmatic detectors, or fine-tuning, since prompt-only ignore controls were insufficiently reliable [2507.10062].

Taken together, the literature positions LLMShot as a prompt-engineering and retrieval-engineering paradigm that spans multimodal generation, multilingual processing, software engineering, fairness-aware prediction, and efficient student-model training. The most stable conclusions are that semantically relevant shots matter more than random shots, that textual abstraction is often an effective multimodal interface, and that selection policy can dominate performance, fairness, and robustness [2403.11317].

Source: https://www.emergentmind.com/topics/llmshot