REPINA: Representation Projection Invariance

• REPINA is a regularization framework that prevents representation collapse by preserving the geometry of pretrained features through projection-invariance constraints.
• It employs controlled mapping functions to ensure that hidden states remain recoverable, thereby maintaining multitask capacity and robust fine-tuning performance.
• Adapted for low-resource machine translation, REPINA anchors high-resource representations to a stable pretrained manifold, enhancing cross-lingual alignment.

REPINA, short for Representation Projection Invariance, is a regularization framework for fine-tuning pre-trained LLMs that aims to prevent representation collapse or representation degradation by constraining how internal representations change during adaptation (Razdaibiedina et al., 2022). In its original formulation, REPINA preserves the information content of pre-trained representations by requiring that pre-trained features remain reconstructible from fine-tuned features through a controlled function class $\Phi$, rather than forcing exact identity (Razdaibiedina et al., 2022). In later work on low-resource machine translation with Aya-23 8B, the term is adapted to a representation anchoring regularizer that constrains selected hidden states to remain close to a pretrained reference pass, especially to stabilize high-resource-language representations during cross-lingual alignment (Nakai et al., 3 Oct 2025). Across both usages, the central idea is representational regularization in hidden-state space rather than direct regularization of model parameters.

1. Conceptual origin and problem setting

The original REPINA paper studies fine-tuning of pre-trained LLMs such as BERT-large and identifies a failure mode in which sentence-level hidden representations lose expressive power after downstream adaptation (Razdaibiedina et al., 2022). The reported symptoms of this representation collapse include concentration of information into a very low-dimensional subspace, decreased isotropy, weaker performance on auxiliary probing tasks, instability, sub-optimal performance, and weak generalization (Razdaibiedina et al., 2022). The problem is framed not as catastrophic forgetting of the language-model objective, but as a structural degradation of representation geometry.

This formulation differs from several established regularization families. Weight consolidation methods penalize deviations from pretrained parameters; R3F, SMART, and related approaches regularize output distributions or local smoothness; and ReInit alters initialization rather than constraining training dynamics (Razdaibiedina et al., 2022). REPINA instead operates directly on hidden representations. The central claim is that preserving the geometry of the pretrained feature space can retain broad task-supporting capacity while still allowing task-specific adaptation (Razdaibiedina et al., 2022).

A common misconception is that REPINA simply forces a fine-tuned model to mimic its pretrained state. The original formulation explicitly rejects that interpretation. Naively minimizing $\|f_{\text{pre}}(x)-f_{\text{fin}}(x)\|_2^2$ is described as too restrictive because it penalizes harmless transformations such as translations, rotations, or scaling that preserve information content and linear separability (Razdaibiedina et al., 2022). REPINA is designed to preserve recoverability of pretrained features, not exact pointwise identity.

2. Formal definition in the original framework

Let $f_{\text{pre}}: x \mapsto z_{\text{pre}}(x)$ denote the sentence representation produced by the pretrained encoder, let $f_{\text{fin}}: x \mapsto z_{\text{fin}}(x)$ denote the representation produced by the fine-tuned encoder, and let $\mathcal{L}$ denote the downstream task loss (Razdaibiedina et al., 2022). The original paper presents a naive anchoring objective,

$$\hat{\mathcal{L}}=\mathcal{L}+\lambda \sum_{x\in\mathcal{D}} \|f_{\text{pre}}(x)-f_{\text{fin}}(x)\|_2^2,$$

and then replaces it with a projection-invariant regularizer that optimizes over a family of dimension-preserving maps $\Phi$ (Razdaibiedina et al., 2022): $$\mathcal{R}=\min_{\phi\in\Phi}\sum_{x\in\mathcal{D}}\|f_{\text{pre}}(x)-\phi(f_{\text{fin}}(x))\|_2^2,$$

$\hat{\mathcal{L}}=\mathcal{L}+\lambda \mathcal{R}.$

The interpretation of $\Phi$ is a critical design choice. If $\|f_{\text{pre}}(x)-f_{\text{fin}}(x)\|_2^2$0, the constraint is strongest. If $\|f_{\text{pre}}(x)-f_{\text{fin}}(x)\|_2^2$1 contains a shallow MLP, the model is allowed more freedom to apply non-linear deformations so long as pretrained features remain reconstructible (Razdaibiedina et al., 2022). The paper describes this as preserving expressivity, rank, diversity, isotropy, and the ability of the representation space to support many pseudo-tasks with linear heads, while allowing transformations that do not significantly reduce the space of functions representable with a linear head (Razdaibiedina et al., 2022).

The paper also provides a theoretical interpretation in terms of infinitely many pseudo-tasks. For linear $\|f_{\text{pre}}(x)-f_{\text{fin}}(x)\|_2^2$2, the regularizer is shown to be equivalent to minimizing expected error over a family of regression tasks induced by the pretrained representation space (Razdaibiedina et al., 2022). This motivates the view that REPINA preserves the multitask capacity implicit in pretraining rather than only improving optimization for a single downstream objective.

3. Variants, collapse metrics, and empirical behavior

The original paper instantiates REPINA in two concrete forms (Razdaibiedina et al., 2022). REPINAI uses $\|f_{\text{pre}}(x)-f_{\text{fin}}(x)\|_2^2$3 and regularizes intermediate encoder layers $\|f_{\text{pre}}(x)-f_{\text{fin}}(x)\|_2^2$4: $\|f_{\text{pre}}(x)-f_{\text{fin}}(x)\|_2^2$5 REPINAMLP uses a 2-layer MLP $\|f_{\text{pre}}(x)-f_{\text{fin}}(x)\|_2^2$6 with hidden width equal to the representation dimension and regularizes the top-layer pooled sentence representation: $\|f_{\text{pre}}(x)-f_{\text{fin}}(x)\|_2^2$7 The reported hyperparameter range for $\|f_{\text{pre}}(x)-f_{\text{fin}}(x)\|_2^2$8 is $\|f_{\text{pre}}(x)-f_{\text{fin}}(x)\|_2^2$9 (Razdaibiedina et al., 2022).

To quantify collapse, the paper uses both probing-based and geometry-based metrics (Razdaibiedina et al., 2022). The probing protocol freezes an encoder fine-tuned on task $f_{\text{pre}}: x \mapsto z_{\text{pre}}(x)$0, trains new linear heads for other tasks $f_{\text{pre}}: x \mapsto z_{\text{pre}}(x)$1, and averages performance; lower average performance indicates greater collapse. The geometry-based metrics derive from the eigen-spectrum of a Gram matrix $f_{\text{pre}}: x \mapsto z_{\text{pre}}(x)$2 built from sentence representations $f_{\text{pre}}: x \mapsto z_{\text{pre}}(x)$3. The paper defines

$f_{\text{pre}}: x \mapsto z_{\text{pre}}(x)$4

and uses top-$f_{\text{pre}}: x \mapsto z_{\text{pre}}(x)$5 variants such as GM-5, GM-10, GM-20, HM-5, HM-10, and HM-20 because many eigenvalues are near zero in practice (Razdaibiedina et al., 2022). Low GM/HM is interpreted as stronger collapse.

Empirically, REPINA consistently outperforms comparable baselines on most tasks in the paper’s evaluation suite (Razdaibiedina et al., 2022). The abstract states that it outperforms other baselines on 10 out of 13 tasks and improves few-shot behavior and robustness to label perturbation (Razdaibiedina et al., 2022). The probing and geometry-based collapse metrics also favor REPINA variants: the reported average probing score is 48.1 for REPINAI and 47.1 for REPINAMLP, compared with 39.4 for STD++, and the reported mean GM-5 is 575 for REPINAI compared with 481 for STD++ (Razdaibiedina et al., 2022). The paper further reports that REPINAI has the fewest failed runs and that REPINA variants yield small but consistent OOD gains in settings such as IMDB$f_{\text{pre}}: x \mapsto z_{\text{pre}}(x)$6SST2 and Amazon$f_{\text{pre}}: x \mapsto z_{\text{pre}}(x)$7Yelp (Razdaibiedina et al., 2022).

A practical distinction emerges between the two variants. REPINAI is reported as particularly effective in few-shot regimes, whereas REPINAMLP tends to perform better in full-data settings (Razdaibiedina et al., 2022). The paper attributes this to the stronger constraint of identity regularization and the greater difficulty of training extra MLP parameters when data are scarce.

4. Adaptation in low-resource machine translation

In "TRepLiNa: Layer-wise CKA+REPINA Alignment Improves Low-Resource Machine Translation in Aya-23 8B" (Nakai et al., 3 Oct 2025), REPINA is repurposed from projection invariance into a representation anchoring regularizer at a specific transformer layer of a decoder-only multilingual LLM. The setting is low-resource translation from an LRL $f_{\text{pre}}: x \mapsto z_{\text{pre}}(x)$8 to a high-resource pivot language $f_{\text{pre}}: x \mapsto z_{\text{pre}}(x)$9 using Aya-23 8B with QLoRA across MMLoSo language pairs such as Mundari, Santali, and Bhili with Hindi or English pivots (Nakai et al., 3 Oct 2025).

For a parallel pair $f_{\text{fin}}: x \mapsto z_{\text{fin}}(x)$0, the paper defines token-wise hidden states $f_{\text{fin}}: x \mapsto z_{\text{fin}}(x)$1 at layer $f_{\text{fin}}: x \mapsto z_{\text{fin}}(x)$2, together with pretrained reference hidden states $f_{\text{fin}}: x \mapsto z_{\text{fin}}(x)$3 obtained from Aya-23 8B with LoRA disabled (Nakai et al., 3 Oct 2025). REPINA is then applied as

$f_{\text{fin}}: x \mapsto z_{\text{fin}}(x)$4

where $f_{\text{fin}}: x \mapsto z_{\text{fin}}(x)$5 is a stop-gradient identity map (Nakai et al., 3 Oct 2025). Although one equation in the paper’s notation uses superscript $f_{\text{fin}}: x \mapsto z_{\text{fin}}(x)$6, the accompanying text clarifies that the actual use is to anchor HRL states and stabilize $f_{\text{fin}}: x \mapsto z_{\text{fin}}(x)$7 (Nakai et al., 3 Oct 2025).

The intuition in this MT setting is explicit. CKA-only alignment encourages LRL and HRL hidden states to move toward one another, but in low-resource regimes the LRL signal is noisy and sparse, so the HRL representation can drift away from its well-trained state (Nakai et al., 3 Oct 2025). REPINA serves as an anchor that keeps the HRL representation near the pretrained Aya-23 manifold, so cross-lingual transfer is driven mainly by movement of the LRL representation toward a stable HRL reference rather than by mutual drift (Nakai et al., 3 Oct 2025). The paper characterizes this as particularly relevant when gradients come from small LRL datasets and may overfit or induce catastrophic drift.

This adaptation is conceptually similar to L2 regularization toward pretrained weights and to Elastic Weight Consolidation, but the paper stresses that it acts in representation space rather than parameter space (Nakai et al., 3 Oct 2025). It is therefore described as more targeted than uniform weight-space penalties and as a representational analogue of EWC.

5. TRepLiNa: joint CKA and REPINA alignment

The MT paper combines REPINA with Centered Kernel Alignment in a joint method called TRepLiNa—Translation Representation Layer-wise alignment with CKA + REPINA (Nakai et al., 3 Oct 2025). The total objective is

$f_{\text{fin}}: x \mapsto z_{\text{fin}}(x)$8

where $f_{\text{fin}}: x \mapsto z_{\text{fin}}(x)$9 is a label-smoothed causal LM loss, the CKA term aligns LRL and HRL hidden states, and the REPINA term anchors the HRL hidden states to the pretrained reference at the same layer $\mathcal{L}$0 (Nakai et al., 3 Oct 2025).

The translation prompting format is reported as

$\mathcal{L}$1

with prompt tokens masked by $\mathcal{L}$2 in the labels and label smoothing $\mathcal{L}$3 (Nakai et al., 3 Oct 2025). The linear CKA term is computed after truncating both token sequences to the minimum length across the pair, masking pads, flattening tokens across the batch, and mean-centering features: $\mathcal{L}$4 This formulation encourages similarity between the LRL and HRL representations at the selected layer (Nakai et al., 3 Oct 2025).

Layer choice is treated as an empirical design decision. The paper sweeps $\mathcal{L}$5 on Aya-23 8B and reports that CKA-only tends to peak around $\mathcal{L}$6, whereas TRepLiNa tends to peak around $\mathcal{L}$7 for LRL$\mathcal{L}$8HRL directions (Nakai et al., 3 Oct 2025). The final choice for larger experiments is the mid-level layer $\mathcal{L}$9, motivated by prior analysis showing strong neuron overlap across languages at embeddings but weaker overlap in intermediate and higher layers; the paper argues that aligning mid-level layers encourages shared semantic structure without over-constraining early lexical processing or late decoding layers (Nakai et al., 3 Oct 2025).

A useful clarification is that the paper’s adapted REPINA is not the original projection-invariance formulation from (Razdaibiedina et al., 2022). It is instead what the authors call a simpler anchoring variant: a pretrained reference pass with adapters disabled is used as the target, and the stop-gradient identity mapping prevents backpropagation through that reference computation (Nakai et al., 3 Oct 2025).

6. Implementation details, results, and limitations in the MT setting

The TRepLiNa experiments use CohereLabs/aya-23-8B, a decoder-only multilingual LLM, fine-tuned with QLoRA using 4-bit NF4 quantization and LoRA adapters on the projection modules $$\hat{\mathcal{L}}=\mathcal{L}+\lambda \sum_{x\in\mathcal{D}} \|f_{\text{pre}}(x)-f_{\text{fin}}(x)\|_2^2,$$0 with $$\hat{\mathcal{L}}=\mathcal{L}+\lambda \sum_{x\in\mathcal{D}} \|f_{\text{pre}}(x)-f_{\text{fin}}(x)\|_2^2,$$1, $$\hat{\mathcal{L}}=\mathcal{L}+\lambda \sum_{x\in\mathcal{D}} \|f_{\text{pre}}(x)-f_{\text{fin}}(x)\|_2^2,$$2, and dropout $$\hat{\mathcal{L}}=\mathcal{L}+\lambda \sum_{x\in\mathcal{D}} \|f_{\text{pre}}(x)-f_{\text{fin}}(x)\|_2^2,$$3; the base model remains frozen (Nakai et al., 3 Oct 2025). REPINA is applied at representation level rather than weight level, with a reference acquisition step that periodically disables LoRA adapters and runs HRL inputs to obtain $$\hat{\mathcal{L}}=\mathcal{L}+\lambda \sum_{x\in\mathcal{D}} \|f_{\text{pre}}(x)-f_{\text{fin}}(x)\|_2^2,$$4, followed by an anchoring step that re-enables LoRA and computes the regularization term at the same layer (Nakai et al., 3 Oct 2025). This design is stated to avoid backpropagating through the reference pass and to keep memory and compute manageable.

The paper reports several hyperparameter settings. For the 1k-pair, 1-epoch layer sweep, $$\hat{\mathcal{L}}=\mathcal{L}+\lambda \sum_{x\in\mathcal{D}} \|f_{\text{pre}}(x)-f_{\text{fin}}(x)\|_2^2,$$5; for the full 20k-pair, up to 5-epoch runs, $$\hat{\mathcal{L}}=\mathcal{L}+\lambda \sum_{x\in\mathcal{D}} \|f_{\text{pre}}(x)-f_{\text{fin}}(x)\|_2^2,$$6 and $$\hat{\mathcal{L}}=\mathcal{L}+\lambda \sum_{x\in\mathcal{D}} \|f_{\text{pre}}(x)-f_{\text{fin}}(x)\|_2^2,$$7 (Nakai et al., 3 Oct 2025). For Santali with 1k pairs and 1 epoch, $$\hat{\mathcal{L}}=\mathcal{L}+\lambda \sum_{x\in\mathcal{D}} \|f_{\text{pre}}(x)-f_{\text{fin}}(x)\|_2^2,$$8 is reported to over-regularize, whereas reducing $$\hat{\mathcal{L}}=\mathcal{L}+\lambda \sum_{x\in\mathcal{D}} \|f_{\text{pre}}(x)-f_{\text{fin}}(x)\|_2^2,$$9 to $\Phi$0 gives better results (Nakai et al., 3 Oct 2025). Other training settings include PagedAdamW8bit or AdamW, $\Phi$1, weight decay $\Phi$2, learning rate $\Phi$3 with linear warmup $\Phi$4, global batch size $\Phi$5 with gradient accumulation $\Phi$6, and gradient clipping at $\Phi$7 (Nakai et al., 3 Oct 2025).

The main empirical picture is selective rather than universal. On Mundari$\Phi$8Hindi, the reported composite score $\Phi$9 is 34.24 for TRepLiNa, 33.45 for REPINA-only, and 33.36 for NoAlign; on Santali$$\mathcal{R}=\min_{\phi\in\Phi}\sum_{x\in\mathcal{D}}\|f_{\text{pre}}(x)-\phi(f_{\text{fin}}(x))\|_2^2,$$0English, the corresponding numbers are 33.02, 32.28, and 32.14 (Nakai et al., 3 Oct 2025). The BLEU and chrF values show the same ordering for these two pairs. On Bhili$$\mathcal{R}=\min_{\phi\in\Phi}\sum_{x\in\mathcal{D}}\|f_{\text{pre}}(x)-\phi(f_{\text{fin}}(x))\|_2^2,$$1Hindi, however, REPINA-only is marginally best at 48.02, compared with 48.01 for NoAlign and 47.96 for TRepLiNa, with differences described as tiny (Nakai et al., 3 Oct 2025). The paper interprets this as evidence that strong CKA can be slightly harmful for typologically close pairs and that REPINA alone may suffice when the languages are already closely related (Nakai et al., 3 Oct 2025).

The paper also notes several limitations. REPINA and CKA are applied at only a single layer; no curriculum or decay schedule is used for $$\mathcal{R}=\min_{\phi\in\Phi}\sum_{x\in\mathcal{D}}\|f_{\text{pre}}(x)-\phi(f_{\text{fin}}(x))\|_2^2,$$2 or $$\mathcal{R}=\min_{\phi\in\Phi}\sum_{x\in\mathcal{D}}\|f_{\text{pre}}(x)-\phi(f_{\text{fin}}(x))\|_2^2,$$3; and the reported study focuses on LRL$$\mathcal{R}=\min_{\phi\in\Phi}\sum_{x\in\mathcal{D}}\|f_{\text{pre}}(x)-\phi(f_{\text{fin}}(x))\|_2^2,$$4HRL directions rather than HRL$$\mathcal{R}=\min_{\phi\in\Phi}\sum_{x\in\mathcal{D}}\|f_{\text{pre}}(x)-\phi(f_{\text{fin}}(x))\|_2^2,$$5LRL or other domains (Nakai et al., 3 Oct 2025). Hyperparameter sensitivity is explicit: too large $$\mathcal{R}=\min_{\phi\in\Phi}\sum_{x\in\mathcal{D}}\|f_{\text{pre}}(x)-\phi(f_{\text{fin}}(x))\|_2^2,$$6 can over-align and harm language-specific features, and too large $$\mathcal{R}=\min_{\phi\in\Phi}\sum_{x\in\mathcal{D}}\|f_{\text{pre}}(x)-\phi(f_{\text{fin}}(x))\|_2^2,$$7 in very small regimes can “pull back” task learning and cause under-fitting (Nakai et al., 3 Oct 2025). For Santali in Ol Chiki script, the paper also notes that longer maximum sequence length and truncation reduce token-wise overlap for CKA/REPINA, which slows convergence and affects how strong anchoring should be (Nakai et al., 3 Oct 2025).

Taken together, the two papers define a coherent research line. In the original formulation, REPINA preserves pretrained feature geometry by enforcing projection invariance under a restricted transform class (Razdaibiedina et al., 2022). In the MT adaptation, the same name denotes a layer-wise anchoring mechanism that keeps HRL hidden states close to a pretrained reference while CKA aligns LRL and HRL states (Nakai et al., 3 Oct 2025). This suggests a broader interpretation of REPINA as a family of representation-space regularizers whose common goal is to maintain a useful pretrained manifold during downstream adaptation, while the exact operational form depends on the architecture, task, and available supervision.