Convergent Token-based Model Inversion (TMI-C)

• The paper introduces TMI-C, a token-conditioned inversion method that systematically optimizes each token to enhance the reconstruction of private visual training data.
• TMI-C employs a convergent per-token update schedule that minimizes gradient conflicts by thoroughly optimizing each token before moving to the next.
• Empirical results indicate that while TMI-C improves token-level accuracy, sequence-based methods still achieve higher coherence and reconstruction performance.

Convergent Token-based Model Inversion (TMI-C) is a model inversion strategy for vision-LLMs (VLMs) that reconstructs private visual training data by optimizing an image generator’s latent variable against a target output sequence one token position at a time, with repeated optimization for each token before advancing to the next. It was introduced together with Token-based Model Inversion (TMI), Sequence-based Model Inversion (SMI), and Sequence-based Model Inversion with Adaptive Token Weighting (SMI-AW) in a study that examined, for the first time, whether VLMs leak private visual training data and showed that inversion attacks are feasible even on publicly released VLMs (Nguyen et al., 6 Aug 2025).

1. Concept and problem setting

Model inversion (MI) attacks reconstruct private training data from trained neural networks. Prior work emphasized conventional unimodal DNNs, typically by maximizing the likelihood of a single class label through iterative optimization in an image generator’s latent space. TMI-C arises because VLMs do not return a single class label; they generate a sequence $(y_1, y_2, \ldots, y_m)$, and each token $y_i$ is produced conditioned on previous tokens $y_{<i}$ (Nguyen et al., 6 Aug 2025).

The VLM setting introduces three constraints that are explicit in the formulation of TMI-C. First, the output is a sequence rather than a single label. Second, generation is sequential and conditional, so token objectives are not independent. Third, private visual information resides mainly in the LLM and projector, as vision encoders are often frozen. TMI-C is designed to more faithfully mirror this token-wise conditional generation process by encouraging per-token convergence before moving to later positions (Nguyen et al., 6 Aug 2025).

A plausible implication is that TMI-C should be understood as a token-conditioned adaptation of classical latent-space MI to the token-based generative nature of VLMs, rather than as a direct reuse of class-conditioned inversion.

2. Optimization procedure

The baseline token-based method, TMI, updates the image generator’s latent vector $w$ once per token, per inversion iteration, and cycles over tokens several times in $N$ total steps. The motivation for TMI-C is the stated weakness of that schedule: because only one update per token per iteration is performed, updates made for later tokens may overwrite gradients relevant for early tokens, failing to ensure that earlier tokens are correctly induced by the image, especially under autoregressive dependence (Nguyen et al., 6 Aug 2025).

TMI-C modifies the update schedule. For a VLM $M$, image generator $G$, input prompt $t$, and target output sequence $y = (y_1, \ldots, y_m)$, it sets $K = N/m$, initializes $y_i$0 randomly, and then, for each token position $y_i$1, performs $y_i$2 updates of $y_i$3 against the token-specific inversion loss before moving to token $y_i$4. In the notation given for the method, the optimized loss at step $y_i$5 is

$y_i$6

and the latent update is

$y_i$7

with learning rate $y_i$8 (Nguyen et al., 6 Aug 2025).

The defining feature is therefore not a new model architecture, but a convergent per-token optimization schedule. Earlier tokens are reinforced or corrected before the optimization context is changed by later-token objectives. The accompanying discussion also states a potential downside: once $y_i$9 is updated for $y_{<i}$0, prior tokens may still be affected unless the representation is already very stable (Nguyen et al., 6 Aug 2025).

3. Objective functions

TMI-C is defined at the level of optimization schedule, not a single mandatory loss. The formulation explicitly allows multiple inversion losses for the token objective (Nguyen et al., 6 Aug 2025).

With cross-entropy loss, the token objective is

$y_{<i}$1

With max-margin logit loss,

$y_{<i}$2

where $y_{<i}$3 is the predicted logit for token $y_{<i}$4.

With logit maximization with representation regularization,

$y_{<i}$5

where $y_{<i}$6 is the penultimate activation vector for token $y_{<i}$7, and $y_{<i}$8 is a regularization vector from public data (Nguyen et al., 6 Aug 2025).

These objectives share the same token-wise conditioning structure, but they differ in what is being directly optimized: conditional token probability, token-vs-competitor logit margin, or target-token logit plus representation regularization.

4. Relation to adjacent inversion strategies

TMI-C was introduced as one point in a larger design space. TMI performs one update per token and repeats the cycle across the sequence. Its stated weakness is the lack of guaranteed convergence for each token, together with the possibility that earlier token objectives are overwritten by subsequent updates. TMI-C retains the token-wise decomposition but allocates $y_{<i}$9 updates to each position before proceeding, thereby better modeling sequential conditional generation and solving each per-token subproblem more thoroughly (Nguyen et al., 6 Aug 2025).

SMI departs from token-wise scheduling and computes the loss for all tokens at once: $w$0 A single update to $w$1 is then based on the global aggregate gradient. The stated advantage is that it considers joint sequence likelihood and can ensure coherence and consistency across the full output sequence (Nguyen et al., 6 Aug 2025).

SMI-AW further modifies SMI by dynamically weighting per-token losses: $w$2 with adjustment if all tokens are confident. The explicit purpose is to focus optimization on low-confidence, mistaken tokens and amplify helpful gradient signals (Nguyen et al., 6 Aug 2025).

This comparison clarifies a central limitation of TMI-C. It improves faithfulness to autoregressive token dependence relative to naïve token-wise cycling, but it still does not optimize the full sequence jointly. The study explicitly associates the strongest empirical performance with sequence-based methods rather than with token-based methods (Nguyen et al., 6 Aug 2025).

5. Empirical behavior in VLM inversion

On the FaceScrub dataset using LLaVA-1.6-7B, the reproduced results report the following values for several inversion strategies and losses (Nguyen et al., 6 Aug 2025).

Method and loss	AttAcc_M / AttAcc_D Top-1	$w$3 / $w$4
TMI, CE	37.78% / 17.71%	0.89 / 147.35
TMI-C, CE	21.77% / 6.39%	1.09 / 636.50
TMI, LOM	44.34% / 21.77%	0.85 / 141.87
TMI-C, LOM	31.16% / 9.32%	1.02 / 457.49
SMI, LOM	59.17% / 33.47%	0.75 / 140.83
SMI-AW, LOM	59.25% / 36.98%	0.73 / 135.90

The same experimental summary states that sequence-based methods, particularly SMI-AW combined with a logit-maximization loss based on vocabulary representation, can achieve competitive reconstruction and outperform token-based methods in attack accuracy and visual similarity. It also reports token-to-token match rates over 95% in the best SMI-AW case (Nguyen et al., 6 Aug 2025).

The analysis attached to these results attributes the relative weakness of token-based strategies, including TMI-C, to difficulty in maintaining global sequence coherence. Because token positions interact through sequential dependencies, optimizing one token at a time may fail to jointly optimize full sequence accuracy. Sequence-based methods, especially SMI-AW, instead compute simultaneous gradients over the whole sequence and adaptively weight difficult tokens (Nguyen et al., 6 Aug 2025).

The privacy implications are explicit. All strategies, including TMI-C, can reconstruct training images with significant accuracy, even from public VLMs. Human evaluation of reconstructed images yields an attack accuracy of $w$5, which is presented as evidence of severe model inversion threats in VLMs. A plausible implication is that frozen vision encoders do not eliminate leakage, because the study places the relevant private visual information mainly in the LLM and projector. The paper further frames this as a practical concern for VLM deployment in applications such as healthcare and finance (Nguyen et al., 6 Aug 2025).

6. Broader usage and adjacent inversion literature

In the supplied literature, the label TMI-C also appears in decoder-only language-model inversion, but with a different operationalization. In hidden-state inversion, the method is framed as a continuous embedding-space optimization in which a soft proxy is driven toward the leaked target without hard-token projection during the search, and a token is committed only once, at the end of the inner loop. This design exposes rank trajectories of the ground-truth token, per-position loss curves, and a discrete loss measured at commit time; it also enables evaluation by cumulative discrete loss. On 10-token C4 prompts, the exact-match rate rises from $w$6 to $w$7 and mean similarity reaches $w$8 as the candidate window is widened. The same work contrasts this with SIPIT, where per-step hard projection is faster, at approximately $w$9 s/token, whereas the continuous approach takes $N$0–$N$1 s/token depending on configuration (Słowikowski et al., 1 Jul 2026).

That hidden-state study also reports a categorical asymmetry in failure modes: space-prefixed, high-frequency function words in dense regions of the embedding matrix dominate failures, while content-bearing tokens are recovered almost perfectly. It concludes that last-layer hidden states of GPT-2 are as sensitive as the original text (Słowikowski et al., 1 Jul 2026).

Adjacent work on prompt inversion from next-token distributions places TMI-C in a wider family of token-level inversion attacks. One study shows that next-token probabilities can recover prompt tokens with BLEU $N$2, token-level F1 $N$3, and exact recovery of $N$4 of prompts on Llama-2 7b, using a pretrained encoder-decoder inverter and, under limited-access settings, a parallelizable binary search algorithm for logit recovery (Morris et al., 2023). Another study, PILS, uses sequences of compressed next-token distributions over multiple generation steps and reports $N$5–$N$6 times higher exact recovery rates across test sets, in one case increasing recovery from $N$7 to $N$8; it argues that next-token probabilities are a considerably more vulnerable attack surface for inversion attacks than previously known (Nazir et al., 20 Jun 2025).

Taken together, these results place TMI-C within a broader inversion landscape in which token-wise convergence, delayed commitment, and sequence-level information all serve the same end: recovering latent training or prompt information from model outputs or internal representations. The most direct lesson from the VLM setting is narrower and more specific: TMI-C captures autoregressive token dependence more faithfully than naïve token-wise inversion, but the strongest reported attacks on VLMs are sequence-based rather than token-based (Nguyen et al., 6 Aug 2025).