Supervised In-Context Fine-Tuning (SIFT)

• SIFT is a hybrid framework that integrates supervised fine-tuning, which updates model parameters using labeled data, with in-context learning that uses prompt demonstrations at inference.
• It employs novel loss formulations such as vanilla causal, single-response, and multi-response completions to enhance representational richness and clarify decision boundaries.
• SIFT leverages efficient data selection methods and dynamic adaptation strategies to boost performance in applications like anomaly detection, sequence labeling, and speech-text modeling.

Supervised In-Context Fine-Tuning (SIFT) integrates the parameter updating of supervised fine-tuning (SFT) with the prompt-based adaptation of in-context learning (ICL), producing a joint mechanism for rapidly adapting LLMs to new tasks via a combination of labeled data and contextually provided examples. SIFT generalizes across algorithmic domains, ranging from anomaly detection in computational workflows to sequence labeling and speech-text modeling, and comprises both theoretical and empirical innovations in efficient data selection, interpretability, and hybrid loss formulations.

1. Foundational Principles of SIFT

SIFT is architected to combine the strengths of SFT—direct optimization of model parameters on labeled data—with the flexibility of ICL, which allows models to adapt their output at inference time using carefully crafted prompts or demonstrations. The foundational workflow consists of:

• Supervised Fine-Tuning (SFT):
  • Adapts a pre-trained LLM to a target task via classical supervised training, e.g., sentence classification for anomaly detection (Jin et al., 24 Jul 2024), multinomial logistic regression for token prediction (Deb et al., 20 May 2025).
  • Typically updates only a small subset of parameters (sometimes using low-rank adapters or linear heads), minimizing cross-entropy (e.g., $$L(\theta) = -\sum_{n} [y_n \log p(y_n|x_n;\theta) + (1-y_n) \log(1-p(y_n|x_n;\theta))]$$ for binary anomaly classification).
• In-Context Learning (ICL):
  • Does not update model parameters; instead, uses task description and labeled demonstrations at inference time.
  • Output is conditioned on prompt construction, leveraging the autoregressive nature of decoder-only LLM architectures to elicit task behavior (Jin et al., 24 Jul 2024, Doimo et al., 5 Sep 2024).

SIFT, as synthesized in recent research, casts target tasks (such as anomaly detection (Jin et al., 24 Jul 2024) or generative sequence labeling (Dukić et al., 31 Aug 2025)) as generative response completion problems, capable of handling both closed- and open-ended instruction categories.

2. Hybrid Workflow and Loss Formulation

The hybrid SIFT protocol merges in-context demonstrations (mimicking ICL) into the supervised fine-tuning pipeline, often leveraging distinct loss strategies:

• Vanilla Causal LLMing Loss: Applied over all tokens in the prompt.

$$L_\text{vanilla}(t;\theta) = -\sum_{i=1}^N \log P(t_i | t_{<i}; \theta)$$

• Single-Response Completion (SRC): Loss computed only over the target query response.
• Multi-Response Completion (MRC): Loss computed on all responses (demonstrations and query), to exploit maximal contextual learning:

$$L_\text{MRC}(t; \theta) = -\sum_{i \in (\text{QR} \cup \text{DR})} \log P(t_i | t_{<i}; \theta)$$

where QR refers to query response tokens and DR to demonstration response tokens (Dukić et al., 31 Aug 2025).

This integration allows gradients to flow both from target outputs and in-context exemplars, increasing representational richness and class boundary sharpness in the network’s later layers (Doimo et al., 5 Sep 2024).

3. Internal Representation Geometry and Layer Effects

Recent probability landscape analyses reveal that SIFT induces distinctive hidden state geometries relative to SFT-only or ICL-only strategies (Doimo et al., 5 Sep 2024):

| Layer Half | ICL (few-shot) | SFT (fine-tuning) |\n |------------|------------------|-------------------| | Early | Hierarchical semantic clustering; high ARI with subjects | Mixed, fuzzy clustering; less subject alignment | | Late | Less defined semantic peaks; answer clusters diffuse | Sharper probability modes encoding final answer labels |

A sharp transition occurs in mid-network layers (e.g., layer 16 in typical models), after which the network's representations shift from semantically organized to answer-encoding modes. Strategies such as freezing early layers and concentrating adaptation in later layers (possibly using LoRA) are supported by empirical gains in alignment quality (Harada et al., 17 Jun 2025).

4. Data Selection and Information Gain

Data efficiency in SIFT is amplified by information-theoretic data selection algorithms. FisherSFT (Deb et al., 20 May 2025) and activeft (Hübotter et al., 10 Oct 2024) both exploit uncertainty quantification and log-determinant maximization to select optimal examples:

• FisherSFT: Greedy maximization of $\log \det(V)$, with $$V = I_d + \sum_{i \in S} \sum_j x_{i,j} x_{i,j}^\top$$ over feature embeddings (typically from pre-logit layer), ensures the chosen subset maximizes information gain as measured by the Hessian of the log-likelihood.
• Activeft (SIFT): Employs Gaussian process regression for posterior uncertainty, selecting examples that maximally reduce

$$\sigma^2_X(p) = k(p, p) - k_X(p)^\top [K_X + \lambda' I]^{-1} k_X(p)$$

where $k$ is the kernel on the embedding space (Hübotter et al., 10 Oct 2024).

These methods also provide confidence bounds

$$\Pr\left[\,\forall n\ge1, x\in X:\; d_{TV}\big(s_n(x), s^\star(x)\bigr) \le \beta_n(\delta)\,\sigma_n(x) \right] \ge 1-\delta$$

that link reduction in uncertainty to increased model reliability.

5. Adaptation Strategies and Circuit Dynamics

Attention head activation and internal circuit reconfiguration are foundational in rapid task adaptation under SIFT (Zhao et al., 24 Sep 2024, Yin et al., 7 Oct 2024):

• Fine-tuning selectively enhances activation of task-relevant attention heads, quantified via

$$\mathrm{AL}_{l,h} = \frac{1}{N} \sum_i \left(\Gamma_{l,h}^T \cdot \frac{\partial L(x_i)}{\partial \Gamma_{l,h}} \right)$$

where $\Gamma_{l,h}$ is the attention matrix at layer $l$, head $h$.

• Circuit shift theory posits that ICL enables more dynamic reallocation of circuit components (attention heads and MLP blocks) compared to full parameter fine-tuning, especially for implicit pattern recognition (Yin et al., 7 Oct 2024). The induced reorganization can result in superior performance on tasks with latent structural patterns.

6. Special Topics: Context Awareness and Instruction Conditioning

SIFT workflows for general instruction fine-tuning require vigilance against loss of context awareness (Wang et al., 5 Nov 2024). Standard conversational templates (e.g., "[INST]") can bias models toward internal knowledge and away from input context. Conditional instruction fine-tuning employs:

• A context-dependency indicator ([IND] token), inserted when attention metrics (e.g., $$s_M(Y_m) = \frac{1}{|Y_m|}\sum_{y \in Y_m}\max_{h \in H}(\sum_{x \in X_1\cup\dots\cup X_m}\text{Att}_h(y,x))$$) exceed a threshold.
• Post-hoc attention steering at inference, rescaling attention weights with $(1/Z)\text{Att}(x,y)$ for context tokens $y$ and $(1/Z)\alpha\,\text{Att}(x,y)$ otherwise, to ensure context fidelity is preserved through fine-tuning.

7. Applications and SIFT Extensions

SIFT underpins several active research domains:

• Anomaly Detection: Combining SFT (sentence-classification head, cross-entropy loss) with ICL (few-shot prompt engineering, chain-of-thought reasoning) achieves robust anomaly detection, outperforming classic machine learning baselines and enabling interpretability (Jin et al., 24 Jul 2024).
• Generative Sequence Labeling: In tasks such as NER or aspect-based sentiment analysis, SIFT’s multi-response loss achieves higher micro F1 than both standard SFT and pure ICL, especially when prompt context is compact and extraneous instructions are removed (Dukić et al., 31 Aug 2025).
• Speech-Text Modeling: The SIFT-50M dataset, spanning 14K hours over five languages, and SIFT-LLM, optimized via staged LoRA adaptation, outperform prior benchmarks on instruction-following, closed- and open-ended speech tasks (Pandey et al., 12 Apr 2025).

8. Theoretical Guarantees

Under idealized assumptions (unbounded context, dataset access, Turing completeness), capabilities acquired via SFT are recoverable at inference time through ICL, with approximation errors bounded as

$$TV(\mathcal{P}_\text{base}(y|x,D),\mathcal{P}_\text{fine}(y|x)) \leq \varepsilon$$

for suitably chosen prompt datasets $D$ of size $O((mV)/\varepsilon^2\log(m/\delta))$ in text generation (Sharma, 9 Jun 2025). In practice, retrieval-augmented generation and data-efficient selection mitigate context and access constraints.

9. Empirical Performance and Scaling

Performance metrics in SIFT implementations include accuracy, F1 score, precision, recall, and perplexity. Lower perplexity data in fine-tuning, mid-layer weight updates, and appropriate selection of context-dependent examples all significantly impact post-tuning alignment quality (Harada et al., 17 Jun 2025). SIFT can outperform both SFT and ICL in hybrid regimes, provided context sequence length and data efficiency are appropriately managed.

10. Future Directions

Practical SIFT enhancements include combining multi-stage optimization (e.g., freezing early layers, focusing adaptation on later layers), scaling with informativity-driven selection (FisherSFT, activeft), and ensuring context-awareness preservation during instructional fine-tuning. Ongoing releases of large SIFT-trained model collections and benchmarks are expected to accelerate research across domains, providing fertile ground for theoretical and empirical advances in context-driven model adaptation.

---

In sum, Supervised In-Context Fine-Tuning (SIFT) is a conceptually unified framework for efficiently adapting LLMs to complex downstream tasks, leveraging both parameter and context supervision. Its foundation in probability landscape analysis, circuit dynamics, data selection algorithms, and prompt engineering yields improved task performance, robustness, and interpretability, with strong theoretical guarantees and multidomain practical impact.