Pseudo Token Injection in Neural Models

• Pseudo token injection is a technique that inserts synthetic, non-linguistic tokens into neural processing pipelines to serve as control signals, structural metadata, or security shields.
• It strategically integrates these tokens at input, intermediate, or latent stages, enabling methods like cross-attention and bottlenecking to enhance model robustness and efficiency.
• Empirical studies report performance improvements in semantic alignment and notable reductions in adversarial attack success, with some techniques offering up to 9.2× defense enhancement.

Pseudo token injection refers to a broad family of techniques in which externally-injected, non-linguistic token identifiers—often special tokens, synthetic embeddings, or carefully-constructed codepoints—are inserted into the input, hidden state, or intermediate layers of neural models. These “pseudo tokens” are not naturally occurring units of language but serve as latent control signals, information bottlenecks, structural metadata, security shields, or adversarial attack vectors. Pseudo token injection is now central to diverse fields: robust sentence embedding, meta-learning with neural processes, knowledge graph completion, prompt-injection defense, adversarial alignment attacks, and tokenizer manipulation. Its mechanisms, effects, and defenses are empirically and theoretically characterized in contemporary literature.

1. Formal Definitions and Core Mechanisms

Pseudo tokens are tokens introduced into a neural model’s processing pipeline that do not correspond to genuine text subwords or linguistic content. They may be:

• Special tokens (e.g., <SEP>, <|assistant|>, <|endoftext|>): Used for metadata demarcation or instruction hierarchy (Zhu et al., 11 Oct 2025, Zhou et al., 2024, Kariyappa et al., 25 May 2025).
• Learned parametric embeddings unattached to vocabulary: Injected as bottlenecks, structure-aware features, or security devices (Yang et al., 8 Sep 2025, Chen et al., 10 Jul 2025, Lara-Rangel et al., 19 Apr 2025, Tan et al., 2022).
• Adversarial token sequences: Crafted via manipulation of token structure, e.g., character prefixes, incomplete byte bigrams, or eos token spamming (Jang et al., 2024, Schulz et al., 9 Jun 2025, Yu et al., 2024).

Injection occurs either:

• At the input level: as additional tokens inserted before, inside, or after legitimate user text (Chen et al., 10 Jul 2025, Tan et al., 2022, Kariyappa et al., 25 May 2025, Schulz et al., 9 Jun 2025, Jang et al., 2024).
• In intermediate representations: as additive signals or embedding substitutions at each layer (Kariyappa et al., 25 May 2025).
• Within the latent semantic or structural spaces: as cross-attention bottlenecks or entity markers (Yang et al., 8 Sep 2025, Lara-Rangel et al., 19 Apr 2025).
• By adversarial perturbation of token sequences: to confuse tokenization or alignment classifiers (Yu et al., 2024, Jang et al., 2024, Schulz et al., 9 Jun 2025).

2. Pseudo Token Injection in Model Architectures and Representation Learning

2.1 Sentence Embedding: PT-BERT

PT-BERT introduces a learnable pseudo-token embedding matrix $P \in \mathbb{R}^{m \times d}$, independent of the BERT vocabulary. Each input sentence is projected (via cross-attention) into a fixed-length pseudo-token sequence, forcing every sentence to produce $m$ pseudo embeddings regardless of original length or syntax (Tan et al., 2022). The model then aggregates these via a secondary attention step. This forces the model to focus on underlying semantics, not superficial features, and supports rigorous contrastive learning with substantially improved alignment and uniformity metrics compared to baseline methods.

2.2 Latent Bottlenecking in Neural Processes

Transformer neural processes (TNPs) adopt pseudo-token injection via a parameter tensor $Z = \{z_j\}_{j=1}^m$ that absorbs context set representations by induced set attention (Lara-Rangel et al., 19 Apr 2025). Attending from $Z$ to the context $C$ creates a bottleneck that dramatically reduces computational complexity ($O(nm)$ vs $O(n^2)$ when $m \ll n$), while offering a tunable trade-off between expressivity and efficiency. Query points then attend into $Z$, yielding predictions conditioned on compressed but information-rich context codes.

2.3 Structure Injection in Knowledge Graph Completion

SLiNT utilizes structure-guided pseudo-neighbors retrieved from external knowledge graphs. Pseudo-neighborhoods are gathered by nearest-neighbor search and fused by multi-head attention. These structure-enriched vectors replace standard token embeddings at specific slots via gradient-decoupled dual injection (GDDI), leveraging LoRA modules for parameter-efficient adaptation without updating base LLM weights (Yang et al., 8 Sep 2025).

3. Prompt Injection Attacks and Security

Pseudo-token injection is pivotal in both prompt-injection attacks and defenses for LLMs.

3.1 Attacks: Virtual Context and MetaBreak

• Virtual Context (VC) exploits injection of a chat delimiter or <SEP> token inside the user prompt. This causes the LLM to treat attacker-supplied content (often “Sure, here is…”) as part of its own prior generation, bypassing standard alignment filters. Across broad benchmarks and model families, VC increases jailbreak attack success rates by 40–55 percentage points for multiple attack classes (Zhou et al., 2024).
• MetaBreak exploits four primitives—response injection, turn masking, input segmentation, and embedding mimicry—using special tokens to forge or segment dialogue structure so as to evade both alignment and moderator-based defenses. If special tokens are sanitized, semantically similar regular tokens are substituted based on $L_2$ distance in embedding space, maintaining attack effectiveness even after sanitization (Zhu et al., 11 Oct 2025). MetaBreak outperforms leading prompt engineering attacks under content moderation by 11.6–34.8 percentage points and is synergistic with other jailbreak techniques.

3.2 Adversarial Input Tokenization and Hidden Boundary Shifts

• Context segmentation via end-of-sequence (eos) injection: Appending “inconspicuous” eos pseudo-tokens ($<|endoftext|>$) to user prompts shifts the LLM's internal representation in the hidden space, moving both harmful and benign queries across the refusal boundary with no change in surface semantics. Attack success rates for standard jailbreak strategies increase by 12–33 percentage points, and even eos-only attacks achieve up to 70% success on some models (Yu et al., 2024).
• Token-level injection (TokenBreak): Inserting single-character prefixes creates new token boundaries, exploiting the vulnerability of BPE and WordPiece models to left-to-right greedy segmentation with SOE or “##” identifiers. This causes classifiers that would detect malicious input to misclassify via altered tokenization. BPE- and WordPiece-based classifiers see an average vulnerability of 55.6%, compared to 0% for Unigram models. The attack is entirely defeated by preprocessing through a Unigram tokenizer and remapping tokens to the model's vocabulary (Schulz et al., 9 Jun 2025).

3.3 Defense: DefensiveTokens and Layer-wise Injection

• DefensiveTokens: Special pseudo-tokens with optimized embeddings are prepended to the model input when robust injection defense is desired. Their embeddings are learned (hedged against injected and clean data) while keeping model weights fixed. On benchmarks, only 5 DefensiveTokens reduce attack success rates from ~50% to 0.5% without significant utility drop. Omission of DefensiveTokens yields original model behavior, allowing on-demand security (Chen et al., 10 Jul 2025).
• Augmented Intermediate Representations (AIR): Layer-specific injection of instruction hierarchy pseudo-tokens in the hidden state of each transformer layer maintains a robust privilege signal throughout all processing steps, providing up to 9.2× improvement over input-level pseudo-token injection on adversarial prompt attacks (Kariyappa et al., 25 May 2025).

4. Tokenization Fragility, Incomplete Tokens, and Improbable Bigrams

Pseudo-token injection also encompasses adversarial construction of token pairs or sequences that are rare, incomplete, or undecodable:

• Improbable bigrams of incomplete tokens: In byte-level BPE tokenization, incomplete tokens are single tokens consisting of stray bytes not decodable as standalone UTF-8 codepoints. Adversarial construction of bigrams wherein a prefix and suffix incomplete token, when joined, form a valid but extremely rare Unicode character causes hallucination rates of up to 79% in models like EXAONE-3.0 and 43% in Llama-3.1, compared to 0–26% for complete-token baselines. Pre-segmentation of prompts at Unicode character boundaries (thus avoiding incomplete tokens) reduces hallucination rates by over 90% in most models (Jang et al., 2024).

5. Methodologies, Empirical Results, and Defenses

Comparative Table: Representative Pseudo Token Injection Mechanisms

Mechanism	Core Function	Empirical Finding
PT-BERT (semantic learning)	Fixed-length pseudo-token attention	+1.5pt STS score vs SimCSE (Tan et al., 2022)
ISANP (TNP bottleneck)	Induced set attention via pseudo-tokens	Matches full-attention NPs, $O(nm)$
Virtual Context (attack)	Insert <SEP> + “answer prefix”	+40–55pp jailbreak success (Zhou et al., 2024)
DefensiveTokens (defense)	Prepend learned embeddings	ASR falls from 51% → 0.5% (Chen et al., 10 Jul 2025)
AIR (hierarchy defense)	Layer-wise privilege embeddings	1.6–9.2× lower attack SR (Kariyappa et al., 25 May 2025)
TokenBreak (tokenization)	Prefixes to break greedy segmenters	BPE/WP classifiers: 55.6% vulnerable
Improbable bigrams	Incomplete token pairs	Up to 79% hallucination (Jang et al., 2024)

Mitigation strategies are task- and threat-model dependent:

• Input pre-filtering (remove or canonicalize pseudo-tokens such as eos, special headers) (Yu et al., 2024, Zhu et al., 11 Oct 2025).
• Tokenizer conversion to Unigram with mapping (Schulz et al., 9 Jun 2025).
• Robust design of chat templates to prevent unauthorized metadata exploitation (Zhu et al., 11 Oct 2025).
• Layer-wise signal reinforcement (AIR) (Kariyappa et al., 25 May 2025).
• Defensive token optimization with prompt tuning (Chen et al., 10 Jul 2025).
• Alignment training on pseudo-token-augmented data (Yu et al., 2024).

6. Open Challenges and Future Directions

Persistent open problems include: designing chat templates and token vocabularies that are robust against embedding-based semantic mimicry; achieving defense against both prompt-level and hidden-state pseudo-token attacks without sacrificing user utility; and establishing formal criteria and diagnostics for geometric robustness in the embedding and activation manifolds.

Major research trajectories now emphasize hybrid input/intermediate-layer defenses, adversarial regularization at concept boundaries, and tokenizer engineering for hallucination prevention. A key insight is that any token with a distinct embedding can, in principle, serve as a stealth vector; thus, holistic alignment must account for the entire token geometry and metadata, beyond linguistic units.

7. Broader Significance

Pseudo token injection constitutes both a substrate for improved model robustness (as in controlled injection for supervision or defense) and an attack surface for adversaries (via structure, embedding, or tokenization manipulation). The phenomenon exposes model weaknesses at the interface of tokenization, semantic processing, and internal representation dynamics. Its study has led to practical advances in defense optimization, structural representation learning, and the quantification and control of alignment drift in LLMs. As the field progresses, the interplay between pseudo-token design, injection mechanism, and model architecture will likely remain central to both trustworthy AI research and the red-teaming of neural systems.