Frozen Small Language Models

• Frozen Small Language Models are fixed-weight transformer models augmented with lightweight adapters that allow targeted task-specialization without full retraining.
• They leverage model compression techniques like pruning, quantization, and distillation to optimize performance and reduce inference latency and deployment costs.
• Their modular design facilitates integration into hybrid, multimodal, and safety-critical systems, enabling efficient resource usage and enhanced interpretability.

A Frozen Small LLM (SLM) is a pretrained LLM of relatively small parameter count that is deployed with its main weights fixed—i.e., "frozen"—such that only minimal adaptation (if any) is performed via lightweight modules, architectural plugins, or external adapters. The concept encompasses a spectrum of use cases—from models trained from scratch and then locked, to "frozen" SLMs used either as standalone modules, in hybrid systems that offload specific subtasks, or in ensembles to constrain the behavior of larger autonomous networks. The approach is driven by requirements for computational efficiency, resource-constrained deployment, interpretability, cost minimization, and in certain cases, explicit safety or data governance guarantees.

1. Definition and Architectural Foundations

A Frozen SLM is characterized by a fixed-weight core network—typically implemented as a transformer-based encoder or decoder with a relatively low parameter budget (typically ranging from tens to a few billion parameters). Freezing refers to the operational restriction that, during downstream adaptation or deployment, the main network weights are not updated. Only external lightweight modules (adapters, LoRA modules, task-specific heads), shallow mapping layers, or soft prompts are tunable.

This paradigm is distinct from standard SLMs subjected to full fine-tuning; it is also distinct from frozen LLMs since the core challenge in the SLM regime is to maximize utility and generalizability under tight parameter and compute constraints (Sakib et al., 26 May 2025).

Key architectural components and stabilizing strategies in frozen SLMs include:

• Lightweight backbone architectures (e.g., TinyLLaMA, MobileBERT, DistilBERT, BabyLLaMA).
• Optimized self-attention (FlashAttention, linearized attention, hybrid attention–RNN structures).
• Architectural search for optimal depth and width.
• Well-designed plugin interfaces and adapter insertion points.
• Pruning and quantization applied either pre- or post-freezing to reduce size and memory footprint.

2. Model Optimization: Pruning, Quantization, and Compression

Frozen SLMs benefit from a suite of model compression techniques designed to maximize inference efficiency and minimize deployment costs:

Technique	Description (as used in SLMs)	Impact
Pruning	Removal of weights/neurons based on functional redundancy	Up to ∼60% size reduction
Quantization	Lowering precision/storage to e.g. 8-bit, 4-bit, or FP8	Reduces memory + compute
Distillation	Teacher-student transfer from LLM to SLM while freezing core	Maintains performance

Adapt-Pruner (Pan et al., 5 Feb 2025) introduces adaptive, layer-wise structured pruning for SLMs. The pruning ratio per layer $S^i$ is determined by the functional "distance" from the identity mapping, computed using:

$$I^i = -\text{cosine\_similarity}(\mathcal{L}_{\text{in}}^i, \mathcal{L}_{\text{out}}^i)$$

Sparsity is assigned as:

$S^i = S_{\text{base}} - A \cdot I^i$

where $A$ is a tunable scaling amplitude. Combined with gradient-saliency-based pruning of parameter blocks, this method preserves model capacity in functionally critical layers. Recovery via post-pruning training (Adapt-Accel) helps bring performance close to models pretrained from scratch, with evidence that efficient adaptive pruning can produce SLMs that outperform fixed-sparsity alternatives on multitask linguistic benchmarks.

Quantization (e.g., GPTQ, SmoothQuant) further reduces active memory and energy draw, especially relevant for on-device or edge inference (Sakib et al., 26 May 2025, Jang et al., 22 May 2025).

3. Frozen SLMs in Hybrid, Modular, and Ensemble Architectures

Frozen SLMs are widely adopted in hybrid settings to balance performance and resource utilization:

• Adaptive Inference Systems: AdaptiveLog (Ma et al., 19 Jan 2025) allocates simple logs to a frozen, domain-specialized SLM and escalates only ambiguous samples to a powerful, expensive LLM. Uncertainty estimation (using dropout-based sampling and Bayesian error modeling) governs this dynamic routing, with the SLM acting as cost-efficient first-pass filter.
• Plug-in and Layer Hybridization: The PiFi framework (Kim et al., 9 Jun 2025) inserts a frozen LLM layer (via projection) into an SLM, producing a composite network whose main body is small (and largely fixed), but which leverages LLM-derived linguistic priors. Only the transformation layers and SLM backbone are fine-tuned. Benchmarks demonstrate consistent ∼2–3% accuracy improvements over naive SLM fine-tuning with minimal overhead (<3% increase in FLOPs).
• Ensemble Methods for Purification: Small, trusted frozen SLMs can be ensembled with LLMs at the logit/probability level to sharply reduce copyright, privacy, or data poisoning risks (Li et al., 19 Feb 2024). The ensemble is implemented as:

$p(y|x) = \frac{p_L(y|x) \cdot p_S(y|x)}{Z(x)}$

with $p_L$ the LLM, $p_S$ the SLM, and $Z(x)$ a normalization term. This process yields a $k$-near access-free generative model, bounding risk of emitting harmful content via relative KL-divergence.

4. Few-Shot, Multimodal, and Specialized Applications

Frozen SLMs have demonstrated effectiveness in several non-trivial application domains:

• Multimodal Prompting: "Multimodal Few-Shot Learning with Frozen LLMs" (Tsimpoukelli et al., 2021) shows that a frozen autoregressive LLM, with a trainable visual encoder to produce "visual prefixes," can support in-context multimodal prompting without updating the LLM weights. This supports open-ended captioning, visual question answering, and fast concept binding.
• Speech and Cross-Modal Systems: SLMs can bridge pre-trained speech and LLMs with only a tiny intermediate adapter trained (∼1% of total parameters), enabling instruction-following (contextual ASR, dialog) at high accuracy and low resource cost (Wang et al., 2023).
• Safety and Moderation: Modular frozen SLMs are well-suited to safety-critical front-ends (harmful query detection, content moderation), where they provide fast, interpretable, and easily updatable filters for larger backend generation systems, with performance on par or better than LLMs on task-specific safety benchmarks (Kwon et al., 30 May 2024, Zhan et al., 17 Oct 2024).

5. Sparse Activation and Latency Optimization

Efficient inference with frozen SLMs can be further enhanced through run-time neuron selection and systems-level serving optimizations:

• Sparse Activation: Standard magnitude-based neuron deactivation is ineffective for SLMs due to reduced over-parameterization. Corrected gradient-times-output attribution scores, as introduced by (Song et al., 3 Jun 2024), allow up to 80% neuron deactivation with less than 5% accuracy loss—substantially improving power and latency performance.
• Serving and Batching: SLMs can achieve Pareto-optimal throughput on a single accelerator by batching large numbers of requests until compute saturates memory IO (Recasens et al., 4 Apr 2024). For very small SLMs, concurrent model replication across sub-partitions of accelerator memory improves GPU resource utilization, throughput, and latency.
• Edge Deployment: Performance-cost trade-offs (captured, e.g., by the platform-level "Performance-Cost Ratio," PCR) establish that on-device SLM serving is frequently orders-of-magnitude more efficient (in $PCR = U/{\rm CPR}$, with $U$ a utility combining quality and responsiveness, $CPR$ the cost per response) than cloud alternatives, setting a benchmark for practical deployment (Jang et al., 22 May 2025).

6. Generalizability, Adaptation, and Open Problems

Frozen SLMs, when properly modularized and enhanced, admit several further capabilities:

• Stacked Modular Reasoning: Chains (or stacks) of frozen, specialization-tuned SLMs (each with minimal fine-tuning via adapters or LoRA) can collectively surpass the versatility of a single, non-specialized SLM, approaching LLM-like performance on multitask benchmarks (Liang, 21 Oct 2024). Natural language interfaces between modules enhance interpretability and debugability.
• Data Selection and Distillation: Frozen SLMs can be used as data prospectors to efficiently filter high-quality examples for LLM fine-tuning, trading negligible (<2%) utility drop for ∼58× compute savings (Ni et al., 13 Dec 2024). This "freeze-and-filter" approach supports economically scalable data curation.
• Open Challenges: Issues such as hallucination, bias, and privacy risk are equally relevant for frozen SLMs; freezing the main weights locks in any pre-existing flaws. Adaptation via external modules must be designed carefully to avoid crude overfitting or performance collapse on shifted domains. The compatibility of sparse activation, adapter-tuning, and pruning across diverse inference tasks remains an active area of research (Sakib et al., 26 May 2025).

---

Frozen Small LLMs represent a convergent answer to growing demands for efficient, controllable, and highly adaptive LLM deployment. By strategically restricting weight updates to lightweight components, applying advanced compression/optimization strategies, and modularizing model architectures, frozen SLMs deliver a favorable efficiency–performance trade-off and underpin many contemporary solutions for multimodal, edge, and hybrid systems. Ongoing research continues to improve their robustness, generalizability, and safe operation across an expanding range of domains and deployment environments.