Token Internal Position Awareness (TIPA)

• TIPA is a framework defining how token and subword positions affect model performance, revealing vulnerabilities in adversarial and classification tasks.
• Empirical results show that varying token positions can boost adversarial attack success rates by over 10% and cause a 3–9% drop in F1 scores for token classification.
• Mitigation strategies such as reverse character prediction and random position perturbation enhance model robustness and improve fine-grained linguistic task performance.

Token Internal Position Awareness (TIPA) encompasses a set of principles and methodologies concerned with how neural models—particularly LLMs and token-classification architectures—handle the internal position of tokens or characters within sequences, subwords, or prompts. TIPA addresses critical modeling, generalization, and robustness issues that arise when models are sensitive to the absolute or relative positions of tokens, both within the entire input (sequence-level TIPA) and within individual subword tokens (token-internal TIPA). Empirical results underscore that ignoring token position in model evaluation, adversarial attack design, or tokenization can systematically underestimate failure rates, degrade generalization, and impede fine-grained linguistic tasks.

1. Conceptual Scope and Definitions

TIPA was independently introduced in multiple contexts. In the context of adversarial prompt attacks on LLMs, TIPA refers to explicit consideration of where adversarial tokens are inserted (e.g., prefix, suffix, or internal positions) when crafting and evaluating attacks, rather than defaulting to the suffix. For token classification tasks such as NER and POS tagging, TIPA denotes the model’s ability to assign consistent labels to tokens regardless of their position within the input sequence, discounting spurious position priors learned during fine-tuning. In character-level modeling, TIPA encompasses learning the mapping between subword tokens and their internal character positions, enabling accurate recovery of position-specific information in tasks like spelling correction (Eddoubi et al., 3 Feb 2026, Amor et al., 2023, Xu et al., 2024).

2. Mathematical Formulations

TIPA is instantiated mathematically at multiple levels of granularity.

2.1 Adversarial Prompt Positioning

For gradient-based Greedy Coordinate Gradient (GCG) attacks:

• The classical GCG attack optimizes adversarial tokens as suffixes. TIPA generalizes the objective by introducing a position-weighting function $w(i)$:

$$\mathcal{L}_\text{pos}(x; \theta) = \sum_{i} w(i) \cdot \ell(x_i; \theta)$$

where $\ell(x_i; \theta)$ is the local token-level loss, and $w(i)$ encodes positional weighting (e.g., $w(i)=1$ on the adversarial block, otherwise $0$) (Eddoubi et al., 3 Feb 2026).

2.2 Token Classification with Position Bias

TIPA in token classification is measured by evaluating performance drop as label positions shift:

$$F_1(\alpha) = F_1(D^*_\text{test}(k, \alpha)), \quad \Delta F_1 = F_1(1) - F_1(\alpha)$$

where $D^*_\text{test}(k, \alpha)$ is a synthesized test set with repeated windows. Significant drops in $F_1$ across positions reveal lack of position invariance (Amor et al., 2023).

2.3 Subword-Internal Positioning

For character-level TIPA:

• Decompose every token $t$ into characters $$\mathcal{L}_\text{pos}(x; \theta) = \sum_{i} w(i) \cdot \ell(x_i; \theta)$$0 with reverse mapping $$\mathcal{L}_\text{pos}(x; \theta) = \sum_{i} w(i) \cdot \ell(x_i; \theta)$$1.
• Formulate as sequence prediction: prompt "Reverse characters of token: [t]" and target a linearized mapping "n: $$\mathcal{L}_\text{pos}(x; \theta) = \sum_{i} w(i) \cdot \ell(x_i; \theta)$$2; ...; 1: $$\mathcal{L}_\text{pos}(x; \theta) = \sum_{i} w(i) \cdot \ell(x_i; \theta)$$3".
• The loss is standard cross-entropy over position–character tokens:

$$\mathcal{L}_\text{pos}(x; \theta) = \sum_{i} w(i) \cdot \ell(x_i; \theta)$$4

No base architecture changes are required (Xu et al., 2024).

3. Empirical Evidence and Experimental Results

3.1 Prompt-Positioned Adversarial Attacks

Evaluation on LLMs (deepseek-LLM-7b-chat, Qwen2.5-7B-Instruct, Mistral-7B-Instruct, Llama-2-7B-chat, Vicuna-7B) reveals that attack success rates (ASR) vary substantially with adversarial token positions. Allowing both prefix and suffix placements increases ASR, sometimes by >10 percentage points (e.g., Vicuna-7B: prefix 83% → 99%; suffix 91% → 97%). Cross-model attack transfer is also underestimated if position is fixed; position variation can double or triple transfer success (Eddoubi et al., 3 Feb 2026).

3.2 Position Bias in Token Classification

Position bias leads to an $$\mathcal{L}_\text{pos}(x; \theta) = \sum_{i} w(i) \cdot \ell(x_i; \theta)$$5 drop of 3–9% across datasets as tokens shift to unseen locations. Context perturbation and random position shifting reduce this drop by ~2%, improving robustness (Amor et al., 2023).

3.3 Internal Character Position Awareness

In Chinese Spelling Correction:

• Enhanced TIPA (reverse character prediction training) boosts Position Prediction Accuracy (PPA) from 79.45% (baseline) to 84.72%, and Sentence-Level Accuracy (SA) from 69.58% to 70.70%. Multi-token TIPA (MTIPA) further raises metrics (e.g., PPA 87.52%, SA 72.40%, NESSA 54.67%) (Xu et al., 2024).
• Analogous improvements in character-level F1 are observed across domain benchmarks, confirming the transferability of learned position awareness.

4. Mechanistic Analyses and Attention Dynamics

LLMs allocate attention differently to adversarial blocks depending on their position. Suffix-positioned attacks cause later transformer layers to focus on the adversarial tokens ("attention hijack"), while prefix-positioned tokens are attended to by early layers, yet can be equally effective. This dichotomy reveals that mid-to-late layer attention alone is an incomplete proxy for attack susceptibility (Eddoubi et al., 3 Feb 2026). In token-classification models, over-commitment to position priors in positional encodings (e.g., APE) impedes generalization to new positions, while RoPE and ALiBi improve but do not eliminate the effect (Amor et al., 2023).

5. Architectures and Mitigation Strategies

• Positional Embedding Schemes: Absolute Position Embeddings (APE), Relative Position Embeddings (RPE), and Rotary Position Embeddings (RoPE) vary in their ability to generalize TIPA. RoPE supports better extrapolation but still exhibits non-negligible position bias.
• Training-Time Perturbations: Random Position Perturbation (RPP) and Context Perturbation (CP) are architecture-agnostic, low-overhead methods that enforce position-insensitive learning in token classification—e.g., RPP randomly shifts the tokens, CP permutes batch context (Amor et al., 2023).
• Augmentation for LLM Defenses: Mixing prompt variants with adversarial tokens at multiple positions combats position-dependent vulnerabilities. Defensive protocols such as position-invariant refusal and early-layer monitoring are recommended (Eddoubi et al., 3 Feb 2026).
• Auxiliary Objectives: Reverse character prediction (TIPA) for subwords injects character-level structure without altering model architecture, leading to improved generalization in both position-aware and standard downstream tasks (Xu et al., 2024).

6. Implications, Limitations, and Future Directions

TIPA exposes methodological blind spots in LLM safety, sequence labeling robustness, and tokenization-based generalization. Robust evaluation and defense require systematic testing across all plausible positions of adversarial or labeled tokens. Notably:

• Safety evaluations and jailbreak benchmarks that use only suffix placement markedly underreport LLM vulnerabilities.
• Mitigation includes architecture-agnostic data augmentations and the adoption of evaluation protocols explicitly measuring performance as a function of token position.
• Subword TIPA as demonstrated in Chinese highlights a language-dependent limitation; wider validation in agglutinative and morphologically rich languages is needed.

Planned research avenues include: extending TIPA to arbitrary internal positions and mixed adversarial token placements; developing architectures or training regimes ensuring uniform position sensitivity; integrating explicit position-heads for out-of-vocabulary (OOV) cases; and embedding TIPA into pre-training phases for base checkpoints (Eddoubi et al., 3 Feb 2026, Amor et al., 2023, Xu et al., 2024).

7. Summary Table: Core TIPA Variants

Context	Core Challenge	TIPA Strategy or Metric
Gradient-based LLM jailbreaks	Position sensitivity in attack	ASR@k: measure over both prefix/suffix
Token classification (NER/POS)	Sequence position bias	$$\mathcal{L}_\text{pos}(x; \theta) = \sum_{i} w(i) \cdot \ell(x_i; \theta)$$6 drop, RPP/CP mitigation
Character-level tasks (CSC, OCR)	Subword internal structure	Reverse mapping and seq2seq TIPA loss

The integration of Token Internal Position Awareness across evaluation, adversarial design, and training is essential for exposing and correcting position-dependent vulnerabilities and blind spots in modern LLMs.