Frozen-Encoder Protocol in ML

• Frozen-encoder protocol is a machine learning method that uses fixed, pretrained backbone models while training only small modular adapters for task-specific improvements.
• It maximizes efficiency by reducing memory and compute requirements, preserves unimodal distributions, and mitigates catastrophic forgetting.
• The protocol enhances modularity, allowing seamless updates and cross-modal adaptations, which is critical for applications in NLP, vision, audio, and beyond.

The frozen-encoder protocol is a paradigm in machine learning that leverages fixed, pretrained backbone networks for one or more input modalities, training only lightweight and typically modular adapter or bridge layers to achieve new tasks or modality alignments. By maintaining the frozen weights of the foundation encoders, these protocols maximize the use of large-scale pretraining, reduce memory and compute requirements, and yield models with strong generalization and modularity across NLP, vision, audio, multimodal, and scientific domains.

1. Core Principles and Motivations

The frozen-encoder protocol rests on several principles:

• Reusability of Foundation Models: Leveraging large, pretrained models (e.g., transformers for vision, language, audio) as fixed feature extractors preserves their representational power and broad coverage (Rowles et al., 24 Oct 2025, Dong et al., 6 Jan 2026, Li et al., 2023).
• Efficiency: Training only small adapters or bridges on top of frozen encoders results in dramatic reductions in parameter count, VRAM consumption, and training time (Rowles et al., 24 Oct 2025, Li et al., 29 Sep 2025, Dong et al., 6 Jan 2026).
• Stable Modality Marginals: Holding foundation encoders fixed ensures that unimodal marginals (e.g., $p_{\rm audio}(\mathbf{a}|\mathrm{text})$ or $p_{\rm video}(\mathbf{v})$) are unchanged. The bridge learns the conditional dependency, e.g., $$p(\mathbf{a}|\mathrm{text},\mathbf{v})\,/\,p(\mathbf{a}|\mathrm{text})$$ (Rowles et al., 24 Oct 2025).
• Modularity: The adapters’ modular structure permits swapping or upgrading foundation encoders with minimal retraining—no end-to-end overhaul is needed when replacing a backbone (Rowles et al., 24 Oct 2025, Li et al., 2023).
• Mitigation of Catastrophic Forgetting: Freezing prevents the catastrophic forgetting observed when naively fine-tuning large, generalist models on specialized tasks (Dong et al., 6 Jan 2026).

2. Protocol Architectures

Frozen-encoder protocols adopt a common architectural motif: foundation encoders map raw input $x$ (text, image, video, audio, etc.) to embeddings $z$, which are then processed by a task- or modality-specific trainable module. Common arrangements include:

2.1. Simple Encoder–Top-Layer Freezing

• NLP: Freeze BERT, train only task-specific CNN/BiLSTM/classifier head (Wang et al., 2021).
• Object-Centric Vision: Freeze a DINOv2 or similar SSL vision encoder; slot attention and decoder reconstruct frozen patch embeddings (Đukić et al., 19 Mar 2025).
• Autonomous Driving: Freeze VLM vision encoder, train transformer adapter and decoder (e.g. GRU for waypoints) (Dong et al., 6 Jan 2026).
• Point Cloud: Freeze CLIP ViT, train patch-tokenizer and head; tokenizer maps 3D patches to tokens in CLIP’s space (Huang et al., 2022).

2.2. Cross-Modal or Multimodal Bridging

• Foley Control (Video-to-Audio): Insert compact video cross-attention blocks into a frozen diffusion-based T2A model; connect via a frozen V-JEPA2 video encoder (Rowles et al., 24 Oct 2025).
• BLIP-2: A two-stage regime where a trainable Q-Former bridges frozen vision and language backbones first for representation learning, then for vision-language generation, with all end-to-end components except for the adapters and small transformers frozen (Li et al., 2023).

2.3. Teacher–Student and Self-Distillation

• SALT: Stage 1: Pretrain a teacher via pixel reconstruction, freeze it. Stage 2: Train a student to predict the teacher’s latent features on masked tokens. Only the student and predictor are trainable in stage 2; the teacher is static (Li et al., 29 Sep 2025).
• Object-Centric EMA: The frozen-encoder baseline fixes the target backbone; OCEBO’s augmentation allows teacher encoder slow-motion bootstrapping via EMA (Đukić et al., 19 Mar 2025).

2.4. Sample-Specific Proxy Control

A proxy network learns to produce sample-specific perturbations of the frozen encoder outputs, steering a fixed decoder towards improved outputs for metrics like COMET or WER—gaining controllability with negligible compute overhead (Fathullah et al., 2024).

2.5. Mixed-Modality with LLM Blocks

Frozen transformer blocks from LLMs serve as additional encoder layers for visual tasks, requiring only dimension-matching projections and a task head, yielding benefit even for visual-only tasks without language (Pang et al., 2023).

3. Training and Optimization Strategies

• Freezing Details: During training, all foundation (encoder) networks’ weights are fixed (no gradients propagated or updated). Only the adapter/projection/top-layer/bridging parameters are updated. Implementationally, this is set by having requires_grad=False on frozen weights (Wang et al., 2021, Rowles et al., 24 Oct 2025, Li et al., 2023).
• Loss Functions: The head or bridge is trained with task-specific objectives (e.g., cross-entropy, v-prediction for diffusion, masked latent regression, CRPS for probabilistic forecasting, InfoNCE for contrastive learning) (Rowles et al., 24 Oct 2025, Li et al., 29 Sep 2025, Filho et al., 14 Nov 2025).
• Regularization: Token-drop or alignment losses (e.g., cross-modal contrastive) improve robustness in multimodal setups (Rowles et al., 24 Oct 2025, Huang et al., 2022).
• Architectural Position: Adapter layers are typically inserted after semantic setting blocks (e.g., after text cross-attention, before feed-forward) to allow frozen priors to set global context and trainable bridges to capture local or cross-modal refinements (Rowles et al., 24 Oct 2025, Pang et al., 2023).
• Compute Allocation: Two-stage regimes often benefit from minimal compute for teacher pretraining and maximal compute for student/adapter learning (Li et al., 29 Sep 2025, Li et al., 2023).
• Early Stopping: Objective signals like validation loss of the adapter or bridge are strongly correlated with downstream performance, enabling simple model selection (Li et al., 29 Sep 2025).

4. Quantitative Footprints and Comparative Outcomes

Protocol	Foundation Type	% Frozen	Trainable Params	Empirical Outcomes
Foley Control (Rowles et al., 24 Oct 2025)	V-JEPA2, Stable Audio Open DiT	~95%	~50M/1B	SOTA temporal/semantic align (MovieGenBench); 1/40× data; 1/20× compute
SALT (Li et al., 29 Sep 2025)	ViT teacher/student	100% (teacher)	varies	Frozen ViT-L: 74.9% SSv2 (vs. 73.7% for V-JEPA2), dominates accuracy-FLOP
BLIP-2 (Li et al., 2023)	CLIP ViT, LLMs	100%	~190M	65.0% zero-shot VQAv2; 8.7% over Flamingo80B with 95× fewer trainable params
FROST-Drive (Dong et al., 6 Jan 2026)	InternVL3 ViT	90–99%	~1–10%	RFS 8.24 (InternVL3-78B, frozen); surpasses all fine-tuned ImageNet ViTs
EPCL (Huang et al., 2022)	CLIP ViT	~91%	~9%	+19 AP50 (ScanNetV2), +4.4 mIoU (S3DIS), +1.2 mIoU (SemanticKITTI)
DINOv3 Rain Nowcasting (Filho et al., 14 Nov 2025)	DINOv3 ViT	~95%	5%	CRPS 3.51 vs. 4.76 for 3D-UNet; 26% efficiency gain
Proxy-Pertain (Fathullah et al., 2024)	Flan-T5, Whisper	100%	<3%	+1.8 COMET (MT), –1.5 WER (ASR) with ≤15% runtime overhead
LLM Visual Plug (Pang et al., 2023)	LLaMA-7B block, ViT	99%	1–3%	+1–3 pts acc across image, video, point cloud, retrieval, forecasting

These results illustrate that the frozen-encoder protocol typically achieves near–state-of-the-art or SOTA on benchmarks while requiring significantly less data, compute, and training time. Empirical evaluations across domains confirm stable learning, sample efficiency, and robust generalization, especially in rare or edge-case scenarios.

5. Theoretical Implications and Limitations

• Preservation of Marginals: By freezing the encoder, $p(\rm output | base\,modal)$ distributions are exactly preserved, ensuring that the model’s predictions for unimodal inputs remain as in pretraining (Rowles et al., 24 Oct 2025). Only the cross-modal or task-specific dependency is updated via the adapter.
• Modularity and Extensibility: Swapping in new, improved encoders only requires re-training the small adapters. This is crucial for evolving foundation models and deployment in constrained environments (Rowles et al., 24 Oct 2025, Li et al., 2023, Huang et al., 2022).
• Performance Ceilings: Frozen targets set an upper bound on probe models (e.g., slot attention for unsupervised object discovery); performance saturates unless the frozen encoder itself is strong or bootstrapping/EMA updates are employed (Đukić et al., 19 Mar 2025).
• Proxy Limits: Sample-specific control via frozen-encoder perturbations assumes reliable proxy learning and sufficient diagonal in the metric manifold; out-of-domain generalization may degrade if proxies are not well aligned (Fathullah et al., 2024).
• Size Requirements: Very small LLM transformer blocks or lightweight backbones do not yield consistent benefit (often causing NaNs or divergence in visual plug-in protocols); capacity thresholds must be respected (Pang et al., 2023).

6. Applications and Domain Expansions

Frozen-encoder protocols have impacted a wide range of domains:

• Audio-Visual Alignment: Foley Control merges video and audio without retraining large backbones (Rowles et al., 24 Oct 2025).
• Video SSL and Masked Prediction: Static teacher–student pipelines outperform EMA-based self-distillation on video representation (Li et al., 29 Sep 2025).
• Vision-Language Pretraining: BLIP-2 demonstrates high efficiency and performance by bridging frozen vision and LLMs (Li et al., 2023).
• Autonomous Driving: Training only adapters above a frozen VLM encoder yields SOTA closed-loop planning metrics (Dong et al., 6 Jan 2026).
• Point Cloud Analysis: CLIP transformers, without any 3D-specific pretraining, function as effective point cloud encoders when accessed via a trained tokenizer (Huang et al., 2022).
• Probabilistic Nowcasting: Frozen satellite vision backbones plus trainable temporal/forecast heads yield superior probabilistic scoring (Filho et al., 14 Nov 2025).
• Controlled Sequence Generation: Sample-specific control in frozen encoder–decoder systems achieves metric-guided output improvements without finetuning (Fathullah et al., 2024).
• Cross-Architecture Plugging: Frozen LLM transformer blocks selectively enhance visual encoding across 2D, 3D, and multimodal tasks (Pang et al., 2023).

7. Best Practices and Operational Guidelines

• Always freeze the backbone weights; train only adapters or bridge layers to prevent catastrophic forgetting and maintain efficiency.
• For cross-modal or multimodal tasks, place the adapter after global semantic layers (e.g., after text CA, before FFN in DiT blocks).
• Aggregate or pool high-dimensional tokens for efficient conditioning, especially for long sequences (e.g., video at 16 FPS) (Rowles et al., 24 Oct 2025).
• Employ cross-modal alignment losses or token-drop for improved robustness as needed (Huang et al., 2022, Rowles et al., 24 Oct 2025).
• Early stopping and validation curves of bridge module loss are reliable proxies for model selection (Li et al., 29 Sep 2025).
• When more capacity or flexibility is required, consider EMA bootstrapping or cross-view filtering to overcome frozen protocol performance ceilings in domains like object-centric representation (Đukić et al., 19 Mar 2025).
• Validate information-filtering and activation amplification, especially when re-purposing large LLM blocks for non-language tasks (Pang et al., 2023).

By following these principles and architectural motifs, the frozen-encoder protocol enables the reuse of powerful foundation models across a growing spectrum of machine learning tasks, maximizing both efficiency and extensibility.