TagGPT: Zero-Shot Multimodal Tagging

• TagGPT is a modular LLM-driven tagging paradigm that uses prompt engineering to generate and refine tags from textual and multimodal inputs.
• It employs a two-stage architecture where GPT models generate candidate tags and sentence embeddings filter them for semantic coherence.
• The system demonstrates robust precision across various domains including social media, open government data, and graph-language applications.

Searching arXiv for the relevant TagGPT papers and closely related work. TagGPT denotes a class of LLM-centered tagging systems that use prompt-based inference to generate or assign tags with little or no task-specific supervision. In the literature provided here, the term appears most directly in the zero-shot multimodal tagging framework introduced in "TagGPT: LLMs are Zero-shot Multimodal Taggers" (Li et al., 2023), and it is also used in a deployment-oriented sense for open government data portals through lessons drawn from the TAGIFY prototype in "TAGIFY: LLM-powered Tagging Interface for Improved Data Findability on OGD portals" (Kliimask et al., 2024). Across these formulations, the common pattern is a modular pipeline in which an LLM produces candidate tags from textual clues, auxiliary semantic encoders or translation services refine the output, and the resulting tags are used to improve findability, distribution, or metadata quality.

1. Origin and scope of the term

In the 2023 formulation, TagGPT is a "fully automated system capable of tag extraction and multimodal tagging in a completely zero-shot fashion" (Li et al., 2023). Its central claim is that, through elaborate prompt engineering, LLMs can "extract and reason about proper tags given textual clues of multimodal data," including OCR, ASR, titles, and related metadata (Li et al., 2023). The system is presented as an end-to-end architecture for both constructing a tag vocabulary and tagging unseen items without further training.

A second usage emerges from the TAGIFY study on open government data portals, which describes how a "TagGPT"-style service can be built from the prototype’s architecture, prompt design, evaluation procedure, and deployment lessons (Kliimask et al., 2024). In that setting, the emphasis is not multimodal social-media content but dataset metadata enrichment, bilingual tag generation, and integration into dataset-submission workflows.

This suggests that "TagGPT" functions less as a single immutable implementation than as a design pattern: an LLM-driven tagging stack with modular components for tag proposal, filtering or matching, and operational integration. A plausible implication is that the term now spans both domain-specific tagging systems and a broader family of prompt-based tagging services.

2. Core architecture

The original TagGPT architecture consists of two loosely coupled modules: "Tagging System Construction" and "Zero-shot Multimodal Tagger" (Li et al., 2023). Both depend on two off-the-shelf foundation components: a pre-trained LLM, instantiated in the paper with GPT-3.5 "gpt-3.5-turbo," and a sentence-embedding model, instantiated with SimCSE (Li et al., 2023). The overall pipeline is described as raw video, image, or text inputs flowing through unimodal extraction steps such as OCR, ASR, and object detections; these textual clues are then passed to the LLM for candidate-tag generation, after which SimCSE supports filtering, semantic clustering, and final tag matching (Li et al., 2023).

The OGD-oriented Tagify/TagGPT architecture is more explicitly service-oriented. A front-end implemented in React calls a FastAPI back-end via POST, sending a JSON payload that contains only the first ten rows of a CSV dataset, including the header, together with query parameters for the number of requested tags and the model choice from {"gpt-3.5-turbo", "gpt-4"} (Kliimask et al., 2024). The back-end assembles a two-part prompt, invokes chat.completions.create(...), receives a comma-separated list of English tags, and then pipes those tags into DeepL through its Python SDK to produce Estonian equivalents before returning both lists as JSON (Kliimask et al., 2024).

The two systems share a low-coupling philosophy. In the multimodal TagGPT paper, the framework is deliberately modular, so GPT-3.5 and SimCSE can be replaced by other instruction-tuned LLMs or sentence encoders (Li et al., 2023). In the OGD setting, the same modularity appears as separation among ingestion, prompt assembly, generation, translation, UI feedback, and later evaluation or feedback loops (Kliimask et al., 2024).

3. Prompt engineering and tag generation

Prompt engineering is central to both variants. In the multimodal TagGPT system, prompt templates instruct the LLM to infer "reasonable interest tags" or domain-specific alternatives such as "recipe tags" from textual clues extracted from data items (Li et al., 2023). A generative prompt for Kuaishou is described in plain text as: "You are a video-interest-tag robot. Given Title={title}, Category={cat}, OCR={ocr}, ASR={asr}, please list reasonable interest tags (each ≥2 words, comma-separated)" (Li et al., 2023). The output is a list of candidate tags, repeated over many corpus items to form a large multiset of raw candidates.

In the TAGIFY-derived service, the prompting strategy is formalized with a system prompt and a user prompt over a CSV prefix. Let

$$D = [h; v_1; \ldots; v_k] \in \Sigma^{(k+1)\times m}$$

denote the header row and the first $k=9$ data rows, and let $n$ be the requested number of tags with $3 \le n \le 10$ (Kliimask et al., 2024). The system prompt specifies the LLM’s role as an expert data curator and requires "exactly $n$ distinct, general yet relevant tags in English," with output restricted to a comma-separated list and "no extra commentary" (Kliimask et al., 2024). The user prompt contains the CSV snippet itself.

Post-processing in the OGD pipeline is simple but operationally important: the reply is split on commas, whitespace is trimmed, and duplicates are removed if necessary (Kliimask et al., 2024). In the multimodal TagGPT system, prompt outputs are subsequently refined by embedding-based filtering and matching rather than by pure string cleanup (Li et al., 2023).

A notable difference is that the multimodal system uses prompt engineering not only for tag generation but also for tag-set construction and for a selective tagging mode in which the LLM chooses from a provided candidate list (Li et al., 2023). By contrast, the OGD pipeline described in TAGIFY focuses on direct generation of English tags followed by machine translation into Estonian (Kliimask et al., 2024).

4. Tag vocabulary construction and semantic matching

A defining contribution of the 2023 TagGPT system is that it constructs a fixed vocabulary of high-quality tags from large amounts of multimodal data before tagging new instances (Li et al., 2023). After candidate-tag generation, it applies frequency-based filtering. If $\mathrm{freq}(t_i)$ is the number of times a tag $t_i$ is generated, tags with frequency below a minimum $f_{\min}$ or above a maximum $f_{\max}$ are removed to prune both spurious one-offs and overly generic or popular tags:

$$\mathcal{T}_{\mathrm{freq}}=\{\,t_i : f_{\min} \le \mathrm{freq}(t_i)\le f_{\max}\,\}.$$

The paper emphasizes the pruning of both tails, although it does not specify exact values for $k=9$0 and $k=9$1 in the experiments (Li et al., 2023).

The remaining tags are embedded with SimCSE to obtain vectors $k=9$2, and pairwise cosine similarity

$k=9$3

is used for semantic clustering or fusion (Li et al., 2023). If $k=9$4, with $k=9$5 in the reported experiments, the tags are treated as semantically redundant and merged into a canonical form (Li et al., 2023). The output is a consolidated tag set $k=9$6.

For unseen items, TagGPT offers two zero-shot taggers. In the generative method, the LLM first produces candidate tags $k=9$7 from clue text $k=9$8, and these candidates are semantically matched against the fixed tag set (Li et al., 2023). The matching score is defined as

$k=9$9

and a tag is selected if the score exceeds threshold $n$0 (Li et al., 2023). In the selective method, cosine similarities are computed directly between the clue text embedding and each tag embedding, the top-$n$1 tags form a candidate pool, and the LLM is prompted to choose the most appropriate tags from that list (Li et al., 2023).

The OGD-oriented service does not include a comparable embedding-based vocabulary-construction stage in its prototype description. Instead, it relies on direct LLM generation and translation (Kliimask et al., 2024). However, the deployment guidance recommends maintaining a ground-truth tagging corpus, measuring precision and recall continuously, refining prompts periodically, and potentially integrating "a lightweight supervised tag-prediction model" from correction data (Kliimask et al., 2024). This suggests a path from direct generation toward a more structured tagging system.

5. Evaluation protocols and reported results

The multimodal TagGPT paper evaluates both the structure of the constructed tag system and the performance of the taggers themselves (Li et al., 2023). The datasets are Kuaishou, with 222 K items containing caption, category, OCR, and ASR, and Food.com, with 5.4 K recipe videos containing dish name, description, and ASR (Li et al., 2023). For the tagging system as a whole, the evaluation uses popularity, practicality, and uniformity; for the taggers, a held-out 100-video test set is assessed with Precision, Recall, and F1 based on human-judged correctness (Li et al., 2023).

The reported quantitative results favor the generative method over the selective method in both domains. On Kuaishou, the selective tagger achieves $n$2, while the generative tagger reaches $n$3 (Li et al., 2023). On Food.com, the selective method reports $n$4 and the generative method $n$5 (Li et al., 2023). The paper also states that generative tagging is typically about two F1 points better than selective tagging because it can recover more fine-grained tags (Li et al., 2023).

For the tag system itself, TagGPT is reported to have "a much flatter popularity curve" than user-supplied hashtags, with far fewer singleton tags and far fewer extremely popular ones (Li et al., 2023). Average tags per item are higher than the baseline, but within-item redundancy is 0% compared with 3–6% for human tags, and global redundancy at $n$6 is 0% versus 0.97–2.85% for human tags (Li et al., 2023).

The TAGIFY prototype was evaluated differently. Its initial user study used two sample datasets and approximately 22 participants, emphasizing human acceptability judgments rather than automatic precision or recall (Kliimask et al., 2024). Relevance of generated tags received a mean rating of about 4.4/5, with lower scores observed when tags became overly specific (Kliimask et al., 2024). About 74% of participants found that adjusting the "number of tags" improved relevance, mostly slightly, and the best trade-off was $n$7–$n$8 in the majority of cases (Kliimask et al., 2024). In model comparison, 65% judged GPT-4 outputs more relevant than GPT-3.5-turbo outputs, and native speakers rated Estonian translation accuracy at 4–5/5 (Kliimask et al., 2024). Usefulness was also high, with 81% indicating they would consider integrating the tool into their workflow, while user-friendliness received a mean rating of 3.9/5 (Kliimask et al., 2024).

The same study recommends a more formal evaluation regime for a production TagGPT service. With a ground-truth tag set $n$9 and generated tags $3 \le n \le 10$0, it defines

$3 \le n \le 10$1

and an $3 \le n \le 10$2 score balancing the two (Kliimask et al., 2024). The proposed procedure is to assemble a corpus of datasets with human-curated tag sets, generate tags with fixed $3 \le n \le 10$3, compute per-item metrics, and report macro or micro averages (Kliimask et al., 2024).

6. Deployment settings and application domains

The application domain of the original TagGPT system is multimodal content distribution on Internet platforms, including search and recommendation settings in which tags are pivotal for content organization (Li et al., 2023). Its inputs can include video, image, and text data, mediated through textual clues such as OCR and ASR (Li et al., 2023). The method is presented as applicable to "various modalities of data in modern social media" and as showing strong generalization across applications (Li et al., 2023).

The TAGIFY-derived service addresses a different operational problem: improving data findability and accessibility on open government data portals by automating dataset tagging in English and Estonian (Kliimask et al., 2024). The study motivates this by noting that, on the Estonian Open Data Portal, 11% of datasets had no associated tags and 26% had only one tag assigned (Kliimask et al., 2024). The deployment recommendations therefore emphasize direct embedding into the dataset-submission UI, in-context validation by publishers, advanced controls for model and tag-count selection without re-uploading, multilingual outputs, and feedback loops that store corrections to improve future performance (Kliimask et al., 2024).

A further extension appears in the TAGLAS report, which frames text-attributed graph datasets as a substrate for building and evaluating graph-LLMs and explicitly presents a guide to using TAGLAS for "building and evaluating TagGPT, a graph–language foundation model" (Feng et al., 2024). TAGLAS collects and integrates more than 23 datasets with a unified node and edge text feature format, standardized loading and task APIs, text-to-embedding conversion, graph-to-text conversion, and evaluation utilities (Feng et al., 2024). In that guide, tasks span node classification, link prediction, graph-level prediction, and graph question answering (Feng et al., 2024).

This suggests that the TagGPT concept is portable across modalities so long as the input can be converted into textual clues or text-like attributes. In the provided sources, these clues range from OCR and ASR to CSV headers and rows, and even to node and edge text in graphs (Li et al., 2023, Kliimask et al., 2024, Feng et al., 2024).

7. Modularity, limitations, and future directions

The original TagGPT paper emphasizes modularity and replaceability. The LLM can be swapped for another instruction-tuned model, and SimCSE can be replaced by another open-source sentence-embedding model, provided it yields semantically meaningful cosine similarities (Li et al., 2023). Prompt templates and thresholds are treated as external hyperparameters, and no backpropagation or fine-tuning is required (Li et al., 2023). This architecture underwrites the paper’s characterization of TagGPT as a low-coupling system (Li et al., 2023).

Its stated limitations are also architectural. Performance is tied to LLM quality; input-length caps in current LLM APIs can truncate long videos or texts; and privacy concerns arise when sensitive data are sent to a public API (Li et al., 2023). Future work proposed there includes tighter integration of multimodal features into prompts, adaptive threshold tuning, and on-device LLM embeddings for privacy (Li et al., 2023).

The TAGIFY study adds deployment-specific limitations and prospective improvements. User suggestions focused on avoiding re-uploading when changing parameters, supporting additional file formats such as JSON, XLS(X), and XML, and integrating the system directly into portals rather than exposing it as a standalone web widget (Kliimask et al., 2024). The paper’s deployment agenda also includes maintaining a ground-truth tagging corpus, monitoring model drift or domain gaps through online Precision and Recall, periodic prompt refinement, and feedback loops that convert user edits into future improvements (Kliimask et al., 2024).

The TAGLAS guide contributes a benchmarking and systems perspective. It standardizes evaluation utilities, supports conversion of tasks into QA format, and recommends practices such as subgraph sampling, mixed training across domains, curriculum learning from small to large graphs, and caching embeddings when heavy LLMs are used (Feng et al., 2024). A plausible implication is that such infrastructure could make TagGPT-style systems easier to compare across domains that differ substantially from social-media tagging.

Taken together, the sources portray TagGPT as a modular, zero-shot or near-zero-shot tagging paradigm rather than a single fixed product. In one form, it builds a high-quality multimodal tag vocabulary and assigns tags through generative or selective semantic matching (Li et al., 2023). In another, it acts as an LLM-powered metadata assistant for open-data publishers, with bilingual output and user-in-the-loop validation (Kliimask et al., 2024). Related graph-language tooling indicates that the same general pattern can be extended to text-attributed graphs and multi-task evaluation settings (Feng et al., 2024).