PTP: Inverting Language Models by Learning to Predict Backwards
This presentation explores a breakthrough approach to reconstructing the prompts that produced a language model's output. By training an inverse model to predict backwards through synthetic reversed sequences, PTP achieves near-exact prompt reconstruction in a black-box setting, with profound implications for model security, auditing, and our understanding of the many-to-one mapping between prompts and responses.Script
Given only a language model's output, can you recover the exact prompt that generated it? This paper introduces a method that achieves 64% exact match reconstruction by training a model to predict backwards, treating inversion as a mirror image of forward generation.
The core challenge is that generation is non-injective: many different prompts can produce nearly identical responses. Traditional inversion methods either require access to model internals like logits and hidden states, or rely on pretrained encoder-decoder architectures that struggle with exact token recovery.
PTP takes a radically different approach. The authors seed forward generation with every token in the vocabulary, sample diverse outputs using top-k and temperature, then reverse each sequence at the token level to create a synthetic training corpus. This requires no real data, just query access to the forward model.
On Qwen3 Chat, PTP achieves 64.77% exact match, compared to 25% for the next-best baseline. Token-level metrics see dramatic gains, while semantic measures like BERT F1 remain high, confirming that reconstructed prompts induce functionally equivalent responses even when lexically distinct.
The inverse model transfers across architectures and even to closed models like GPT-4. Token-level fidelity requires tokenizer alignment, but semantic reconstruction remains robust under mismatch, revealing that the method captures intent rather than surface form alone.
PTP opens a new frontier for auditing language models in deployment. By learning to predict backwards, we gain a reusable tool for prompt security, adversarial elicitation, and understanding the full equivalence class of inputs that drive model behavior. Learn more and create your own research videos at EmergentMind.com.
