TALAN: Task-Aligned Latent Adaptation Networks for Targeted Post-Training of Large Language Models

Abstract: Targeted post-training aims to improve reasoning, math, and code without degrading strengths. Low-rank adapters are efficient but task-global; activation interventions are input-aware but often require separate probes, vectors, or inference-time steering. We introduce TALAN (Task-Aligned Latent Adaptation Networks), a sequence-conditioned latent side path inserted into a transformer's residual stream and co-trained with a low-rank adapter in one SFT loop. TALAN compresses the active sequence into latent memory, remixes it into token-level perturbations, and writes them back through a controlled residual update. It is configured along six axes: insertion location, memory size, mixer, writeback rule, trainability scope, and gradient scale. Across four Qwen3-family backbones and four STEM/code benchmarks, TALAN improves matched LoRA and DoRA baselines. With LoRA, it yields a +1.41 point cross-model mean gain, positive on all four backbones and non-negative on all 16 model-benchmark cells. With DoRA, it yields a +1.85 point mean gain, positive on all backbones and on 13 of 16 cells. Paired seed checks support positive average effects but show nontrivial variance, so we treat them as sensitivity checks. Cost is small: <1% trainable parameters relative to the backbone and 1.01-1.02x inference overhead versus matched LoRA. A Llama-3.2-1B transfer probe is also positive under LoRA and rsLoRA across seven paired seeds, supporting a transfer beyond Qwen. Internal-state analyses suggest TALAN is a small complementary activation intervention. The matched adapter update is 80-1,700x larger than the TALAN perturbation, yet their directions have near-zero cosine; per-layer measurements show this small orthogonal perturbation propagates and amplifies through depth. TALAN offers a practical platform for studying steerable activation-level adaptation within standard adapter-based post-training.

• The paper introduces TALAN, a modular, task-aligned latent adaptation network using a summarize–remix–writeback block to enable input-aware, bounded interventions in LLM post-training.
• It employs a six-axis configuration and orthogonal activation perturbations to achieve statistically robust gains over baseline adapters like LoRA and DoRA.
• Empirical evaluations confirm that TALAN delivers amplified improvements in domains such as STEM reasoning and code generation with minimal computational overhead.

Task-Aligned Latent Adaptation Networks (TALAN): A Sequence-Conditioned Intervention for Efficient, Steerable Post-Training

Overview and Motivation

Task-Aligned Latent Adaptation Networks (TALAN) introduce a sequence-conditioned adaptation paradigm for LLM post-training, targeting improved capability in domains such as mathematics, code, and STEM reasoning, while explicitly mitigating regression on pre-existing competencies. In the conventional PEFT landscape, methods such as LoRA and DoRA provide task-global, low-rank modifications but lack granularity for input-aware adaptation. At the other end, activation intervention techniques (e.g., activation steering, CogSteer, ReFT) offer sequence specificity but generally operate outside the end-to-end SFT loop, with extra steps for probe fitting, vector extraction, or inference modifications.

TALAN bridges this gap by embedding a compact, configurable side-path "summarize–remix–writeback" block into the transformer's residual stream Figure 1, jointly co-trained with a low-rank adapter under the same autoregressive loss.

Figure 1: TALAN architecture overview: sequence summarization into a latent memory $Z$, remixing back to token-level perturbations $H$ via one of four mixers, and writeback controlled either by direct merge or a learned gate; configurations vary via six architectural axes.

The method's central claims are twofold: (P1) the activation perturbation induced by TALAN is small and explicitly bounded, and (P2) its direction is architecturally orthogonal to the standard LoRA/DoRA update, offering practical and theoretical advances toward steerable, sequence-conditional post-training.

Architecture: Summarize–Remix–Writeback and Configurability

Summarize–Remix–Writeback Block

The TALAN module is inserted at a configurable residual location $\ell^\star$ and operates in three stages:

• Summarize: Compresses the sequence into $T$ latent slots via attention-like soft assignment between residual tokens and learnable slot queries.
• Remix: Projects the slot memory back to sequence length via one of four possible mixers: multihead local memory attention, cross-attention, gated cross-attention, or per-token MLP.
• Writeback: Reinserts the perturbation using either a direct (scaled) sum or an input-dependent learned gate.

The atomicity of this block ensures that it is modular and that its expressivity is controlled via configuration, not code proliferation.

Six-Axis Configuration Space

TALAN's deployment on a given host LLM is mediated entirely by a six-dimensional axis:

1. Insertion layer ($\ell^\star$): placement in the residual stream.
2. Slot count ($T$): size of the latent memory.
3. Mixer type ($m$): family controlling the remixing mechanism.
4. Writeback rule ($w$): direct merge or learned gating.
5. Trainability scope ($\tau$): which TALAN submodules get gradients.
6. Gradient scale ($\gamma$): scaling on TALAN parameter gradients.

Crucially, the training objective, optimizer, data, and host adapter recipe are held fixed within each experiment; only the 6-axis configuration is manipulated Figure 2.

Figure 2: TALAN configuration axes: selection of insertion point, slot count, mixer, writeback, trainability scope, and gradient scaling per host backbone.

Audit of the 2,590-point configuration space demonstrates that viable, non-regressing configurations are rare (≈7%) but structured, not randomly distributed Figure 3.

Figure 3: Configuration-audit results reveal that successful TALAN configurations are structured in the space of connection depth and slot count, with low slot counts dominating high-value regions.

Empirical Results: Efficacy and Robustness

Datasets and Protocol

Evaluations are conducted across four Qwen3-family LLM backbones (including distillation and MoE variants) and four mathematical/generative code benchmarks (#1GPQA Diamond, GSM8K, MATH-500, MBPP). All experiments leverage a fixed, bench-aligned 5,000-example SFT dataset with matched adapter recipes.

Performance Over Baseline Adapters

TALAN delivers consistent, statistically robust improvement over both LoRA and DoRA across all tested backbones:

• LoRA: Cross-model mean gain of +1.41 percentage points (pp), with non-negative delta across all 16 model-benchmark cells.
• DoRA: Cross-model mean gain of +1.85pp, positive on all four backbones and on 13/16 cells.

Competing approaches (AdapterFusion, DELIFT, CogSteer, ReFT, etc.) reach comparable or lower averages but all show regressions in at least one backbone or cell, lacking TALAN's stringently non-regressive profile.

Cost and Practicality

TALAN incurs <1% parameter overhead relative to the backbone and only 1.01–1.02× training and inference wall-time relative to matched LoRA (Table in Sec. 5.6). Unlike LoRA, the TALAN path is not mergeable post-training and remains a persistent forward component. However, the cost remains modest in practice compared to multi-adapter/fusion ensembles.

Mechanistic Analysis: Perturbation, Orthogonality, Amplification

Perturbation Scale and Directionality

Direct measurement at the insertion layer shows that the TALAN update is 80–1,700× smaller in norm compared to the LoRA update in the same token position, confirming the bounded, non-dominant nature of the intervention (Table, Sec. 6.1). Cosine similarity between TALAN and LoRA/delta directions remains near zero (winner or loser), empirically confirming design orthogonality.

Activation Propagation

Layer-wise L2-difference and cosine-similarity analysis show that, although the TALAN-induced perturbation is small at the injection point (with minimal immediate layer discrepancy), the perturbation is amplified across subsequent transformer layers, resulting in growing divergence through depth (Figures 3, 5–8). This demonstrates that small, input-aware interventions can propagate robustly through deep residual architectures—furnishing downstream behavioral effect without significant initial magnitude.

Figure 4: Per-layer activation divergence (L2-RelDiff) on GPQA; divergence emerges at the configured insertion locus and grows monotonically through later layers.

Figure 5: Qwen3-8B per-layer divergence—amplification ignites at layer 4 (insertion location) for all benchmarks, and impact persists through the network.

Figure 6: DeepSeek-R1-Distill-Qwen-32B per-layer divergence—embedding-level injection shows early and strong amplification notably affecting output.

Cosine Similarity Stability

Cosine analysis (Figures 9–12) shows that the small direction shift at insertion remains stable yet decays, as expected, deeper in the stack—underlining that the sequence-conditioned perturbation occupies a distinct subspace orthogonal to the main LoRA update.

Figure 7: Qwen3-8B per-layer cosine similarity—near 1.0 at insertion, decaying smoothly to lower values at depth.

Comparison to Alternative Architectures

A single-stage TALAN workflow (joint SFT with LoRA/DoRA) is markedly simpler and less costly than AdapterFusion (multi-stage/domain adapters and fusion training), yielding better consistency and generalized non-regression Figure 8.

Figure 8: Workflow comparison—TALAN's end-to-end SFT with latent adaptation contrasts with AdapterFusion's multi-stage, multi-adapter overhead.

Implications, Limitations, and Future Directions

The evidence demonstrates that small, sequence-conditioned, architecturally isolated activation interventions composed with low-rank adaptation can produce targeted gains without macro-level regression. TALAN constitutes a platform for steerable post-training—where location, geometry, and strength of the intervention can be configured (and possibly inferred) from host geometry rather than discovered solely by brute-force search. Importantly, amplifiable, orthogonal modifications allow practitioners to craft domain- or capability-targeted improvements with bounded risk of negative transfer.

The current evaluation is restricted to STEM/code-centric benchmarks and Qwen- and Llama-family hosts. Transfer to other model families (Gemma, Mistral, etc.), dialog or multi-lingual settings, longer SFT regimes, or strategies for automated configuration selection based on model internals remain open extensions. The lack of direct prescription for configuration choice, with observed variance across random seeds and backbones, further underscores the need for future mechanistic studies.

Conclusion

TALAN establishes that a modular, sequence-conditioned latent adaptation side-path—summarizing input, remixing, and perturbing at a selected layer—can robustly enhance PEFT adapters by introducing complementary and amplifiable internal modifications. The method’s bounded scope, low computational burden, rigorous non-regression profile, and composability with existing PEFT methods make it a compelling direction for future research in activation-level steering and modular post-training adaptation.

---

References

See (2606.06902) for full citations and supplementary results.