Factor-Aware Mixture-of-Experts with Pretrained Encoder for Combinatorial Generalization

Abstract: The integration of pretrained encoders with diffusion policies has become a dominant paradigm for visual robotic manipulation. However, it still struggles to generalize across complex environments with varying factors such as lighting and surface textures. To address this, we propose FAME, a framework that integrates a factor-aware mixture-of-experts (MoE) with a pretrained encoder to enhance generalization to environmental variations. FAME follows a three-stage training process: (1) policy warmup, where a diffusion policy is trained on standard-environment data with a frozen encoder; (2) factor-specific adapter training, where lightweight adapters inserted between the frozen encoder and the temporarily frozen policy are trained on customized datasets, each targeting a distinct environmental variation; and (3) joint fine-tuning, where a central router and the warmed policy are trained on mixed data to handle multiple factors jointly. FAME is ``factor-aware'' because the central router softly weights frozen factor-specific adapters as a dense MoE, enabling combinatorial generalization across multiple factors. Evaluations on the Meta-World benchmark show that FAME outperforms diffusion policy baselines by 34%. We further validate FAME in a real-world pick-and-place task using a compact model trained on newly collected data, where FAME achieves a 35% improvement in generalization under real-world variations.

• The paper introduces FAME, a framework that uses a factor-aware mixture-of-experts with a frozen pretrained encoder to efficiently address the combinatorial explosion in visual robotic manipulation.
• Its three-stage training process—policy warm-up, factor-specific adapter training, and joint fine-tuning with a dense MoE—delivers a 34% improvement in success rates over strong baselines.
• Experiments on simulation and real-world tasks demonstrate FAME’s data efficiency, robust multi-factor adaptation, and effective zero-shot task transfer under diverse environmental perturbations.

Factor-Aware Mixture-of-Experts with Pretrained Encoder for Robust Combinatorial Generalization in Visual Robotic Manipulation

Introduction

This work introduces FAME (Factor-Aware Mixture-of-Experts with Pretrained Encoder), a new paradigm for enhancing generalization in visual robotic manipulation tasks under environmental variations such as lighting, texture, and camera position (2606.21100). The framework addresses the combinatorial explosion of data requirements encountered when adapting visuomotor policies to multiple factors. FAME leverages a dense Mixture-of-Experts (MoE) architecture integrated with a frozen pretrained visual encoder, providing parameter-efficient adaptation and robust handling of independent, simultaneously varying environmental factors.

Framework and Methodology

Motivation and Design

The key insight in FAME is the factorization of environmental variation. By disentangling distinct factors and developing specialized adaptation modules for each, data complexity grows linearly rather than exponentially with respect to the number of factors. The method circumvents the inefficiency of direct fine-tuning of large encoders, which is computationally expensive, susceptible to overfitting, and suffers from catastrophic forgetting.

Three-Stage Training Architecture

1. Policy Warm-up: In this stage, a diffusion policy head is trained with a frozen DINOv2 visual encoder on standard-environment data, serving as the baseline model.
2. Factor-Specific Adapter Training: Lightweight adapter modules are inserted between the frozen encoder and the frozen policy head. Each adapter is trained on data exhibiting variation in a single environmental factor, specializing the adapters for each unique domain shift without modifying the base encoder or policy.
3. Joint Fine-Tuning with Dense MoE: Factor-specific adapters are aggregated by a gating/central router (the MoE layer), which computes soft weights conditioned on the frozen encoder features. This allows dense fusion of expertise from multiple adapters based on input, enabling combinatorial generalization over all observed factor combinations.

The adapters and visual backbone are frozen during this phase, and only the router and a new policy head are jointly optimized on mixed-environment data.

Experimental Results

Simulation (Meta-World) Benchmarks

FAME achieves a notable average success rate of 54.15% across 9 diverse tasks, an improvement of 34% relative to the best-performing baseline (DP with DINOv2) under challenging test conditions with up to five simultaneous environmental changes. On tasks of increased complexity such as Door Lock, Handle Pull Side, and Peg Insert Side, FAME's gains are pronounced, e.g., 60.33% vs. 6–10% for strong baselines.

Ablation studies confirm:

• Data Efficiency: FAME scales favorably with increased demonstration data, further improving generalization.
• Single- and Multi-Factor Robustness: The architecture maintains high adaptation fidelity for both single-factor and compound variations.
• Zero-Shot Task Transfer: The routing mechanism displays strong cross-task generalization, evidenced by gating activations that transfer between tasks without explicit retraining.

Real-World Robotic Manipulation

A real-world pick-and-place experiment on an X-Arm 6 robot demonstrates the framework's transferability and sim2real performance—FAME attains an overall success rate of 70%, dramatically outperforming ResNet-DP (33%) and DINOv2-DP (35%). Under camera position changes, FAME achieves 87% (ID) and 60% (OOD) success; under compound camera + texture variation, it retains 67% (ID) and 40% (OOD), with baselines failing in these OOD scenarios.

t-SNE visualizations of the learned representations confirm that FAME produces more structured, disentangled feature spaces, supporting robust generalization.

Theoretical and Practical Implications

The FAME framework constitutes a concrete advancement in efficient adaptation for high-dimensional control. It builds a general methodology for decomposing environmental complexity in robotics by reusing a frozen, powerful representation backbone augmented with specialized, parameter-efficient adapters. The dense MoE approach additionally introduces dynamic, context-aware composition of expertise, shifting beyond standard sparse MoE gating and aligning with the growing trend of modular, factorized policy architectures.

Practically, FAME is compelling for scenarios with limited data acquisition budgets and non-stationary domains, as it avoids the cost and instability of full network updates. The explicit modeling of environmental factors supports robust out-of-distribution adaptation, a critical requirement for real-world deployment of autonomous systems.

Future Directions

Open challenges remain regarding the full scalability of FAME to arbitrarily large sets of environmental factors in more complex manipulation and multi-agent systems. The present study's real-world tasks are limited in scope; future work should further validate the architecture on broader task distributions, more intricate factor correlations, and additional hardware platforms. Optimizing adapter architectures and investigating automatic factor discovery (beyond manual annotation) are also important next steps.

Conclusion

FAME presents an effective, theoretically grounded solution for combinatorial generalization in visuomotor policy learning. Through its modular, factor-aware training regimen leveraging frozen encoders and a dynamic dense MoE, the approach achieves robust performance under diverse, compounding environmental perturbations, advancing the state of the art in adaptable robotic manipulation (2606.21100).