Foundation Models & Zero-Shot Control

• Foundation models are large-scale, pre-trained architectures on diverse data that support zero-shot control across multiple domains.
• Zero-shot control enables task execution with novel instructions or constraints without additional retraining or domain-specific supervision.
• Empirical studies show these models achieve robust performance in robotics, geoscience, and vision by integrating multi-modal inputs and similarity-based inference.

Foundation models—large-scale pre-trained architectures trained on diverse data modalities—have enabled zero-shot control across a rapidly growing span of domains including robot manipulation, embodied navigation, geoscientific inference, geometric perception, and bioacoustics. Zero-shot control refers to specifying and achieving tasks at deployment time with novel goals, instructions, or constraints, without any additional task-specific retraining or domain-adapted supervision. This paradigm leverages the generalization capabilities and representational richness of foundation models, allowing diverse forms of user input (images, sketches, text, well logs, reward functions) to directly effect task execution.

1. Foundation Models: Architectures and Pre-training

Foundation models in zero-shot control are characterized by their architecture, pre-training objectives, and scale:

• Vision models (e.g., ResNet50, MoCo, CLIP) are trained on millions of images or image-text pairs using supervised or contrastive objectives. For instance, CLIP employs a symmetric contrastive loss over ~400 million image–text pairs:

$$\mathcal{L}_{\mathrm{CLIP}} = -\frac{1}{2}\sum_{k=1}^N \left[\log \frac{\exp(i_k\cdot t_k/\tau)}{\sum_{n=1}^N \exp(i_k\cdot t_n/\tau)} + \log \frac{\exp(i_k\cdot t_k/\tau)}{\sum_{n=1}^N \exp(i_n\cdot t_k/\tau)}\right]$$

CLIP's multi-modal alignment supports specifying goals in language or visual formats (Cui et al., 2022).

• Multi-modal and promptable models (e.g., ZERO, GEM 3D) combine encoders for text, image, and auxiliary cues, fusing them using transformer backbones to generate domain-appropriate outputs conditioned on free-form prompts, such as instructions or exemplar images (Choi et al., 6 Jul 2025, Dou et al., 1 Jul 2025).
• Behavioral foundation models for control (Meta Motivo, BFM-Zero) learn a joint latent space for motions, goals, and reward embeddings, using Forward-Backward (FB) representations trained from large behavior repositories via unsupervised policy learning (Tirinzoni et al., 15 Apr 2025, Li et al., 6 Nov 2025).
• Geometric foundation models (e.g., Metric3Dv2) are trained on millions of RGB–D images with systematic treatment of camera extrinsics/intrinsics, yielding zero-shot metric generalization across arbitrary scenes, devices, and tasks (Hu et al., 2024).

2. Zero-Shot Control: Goal Specification and Task Prompting

Zero-shot control operates by decoupling model adaptation from explicit downstream supervision, instead relying on prompts or goal specifications to steer inference:

• Robot manipulation: The ZeST pipeline supports goal specification through (i) internet images, (ii) sketches, or (iii) language. These are encoded into a feature space via a foundation model, and similarity between live images and goal encodings drives either retrieval-based policy deployment or offline RL reward assignment (Cui et al., 2022).
• Geoscientific reasoning: GEM 3D supports prompt-driven completion of subsurface tasks—each user prompt (e.g., a well log, binary mask, or horizon sketch) is embedded and fused with latent seismic features, enabling unified reasoning for structural interpretation, segmentation, or property prediction without task-specific retraining (Dou et al., 1 Jul 2025).
• Embodied and behavioral control: Policies such as those in Meta Motivo and BFM-Zero accept latent prompts representing goals, motion trajectories, or reward functions, enabling zero-shot execution of whole-body humanoid tasks. No task- or goal-specific policy weights are required beyond a feed-forward latent computation (Tirinzoni et al., 15 Apr 2025, Li et al., 6 Nov 2025).
• Object detection and visual grounding: Models such as ZERO fuse arbitrary sets of natural-language and visual prompts with images, supporting detection of previously unseen classes—zero-shot—by swapping in new prompts at inference (Choi et al., 6 Jul 2025).
• Navigation and reasoning: In object-goal navigation, foundation models (LLMs, vision-LLMs) are prompted for commonsense judgments about candidate frontiers ("which frontier likely leads to a 'toilet'?"), sidestepping any environment- or class-specific policy adaptation (Yuan et al., 2024).

3. Mathematical Formulation and Inference Mechanisms

A standard formalism frames zero-shot control over Markov decision processes or structured prediction tasks:

• Goal-conditioned policy:

$$\pi(a_t \mid s_t, g)\quad\text{or equivalently}\quad \pi(a_t\mid X_t, g)$$

where $g$ is a prompt or high-level goal. There is no paired (state/observation, goal) data required during training for unseen $g$.

• Similarity-based control: Policies compute a similarity $\phi(X_t, g)$ between current sensory input and prompt encoding, using it as a reward proxy, retrieval criterion, or policy selector. For instance:

$$\phi_{\mathrm{delta}}((X_t,X_0),(g_f,g_0)) = \alpha(\psi(X_t)-\psi(X_0), \psi(g_f)-\psi(g_0))$$

where $\alpha$ is cosine similarity or negative Euclidean distance (Cui et al., 2022).

• Prompt fusion in multimodal models: Given image features $\phi_\mathrm{img}(I)$ and prompt embeddings $$\{\phi_\mathrm{text}(T_i), \phi_\mathrm{vis}(V_i)\}$$, a transformer backbone fuses these for downstream prediction (Choi et al., 6 Jul 2025).
• Latent prompt conditioning in policy learning: Latents $z\in\mathbb{R}^d$ representing reward, goal, or demonstration are computed as:

$$\pi(a_t \mid s_t, g)\quad\text{or equivalently}\quad \pi(a_t\mid X_t, g)$$0

and used as conditional arguments to the policy $$\pi(a_t \mid s_t, g)\quad\text{or equivalently}\quad \pi(a_t\mid X_t, g)$$1 (Tirinzoni et al., 15 Apr 2025, Li et al., 6 Nov 2025).

• Model merging for zero-shot generalization: In domain-specific LLMs, linear interpolation between base and fine-tuned weights can tune domain-expertise vs. instruction adherence in zero-shot settings:

$$\pi(a_t \mid s_t, g)\quad\text{or equivalently}\quad \pi(a_t\mid X_t, g)$$2

with $$\pi(a_t \mid s_t, g)\quad\text{or equivalently}\quad \pi(a_t\mid X_t, g)$$3 chosen empirically for the desired tradeoff (Marincione et al., 7 Nov 2025).

4. Empirical Performance and Cross-Domain Generalization

Zero-shot foundation model control demonstrates strong empirical performance across domains:

• Robot manipulation: ZeST with CLIP achieves $$\pi(a_t \mid s_t, g)\quad\text{or equivalently}\quad \pi(a_t\mid X_t, g)$$460% top-25 success in diverse simulated robot tasks from internet images or sketches, far exceeding random baselines ($$\pi(a_t \mid s_t, g)\quad\text{or equivalently}\quad \pi(a_t\mid X_t, g)$$510%) (Cui et al., 2022).
• Subsurface geoscience: GEM 3D attains IoU $$\pi(a_t \mid s_t, g)\quad\text{or equivalently}\quad \pi(a_t\mid X_t, g)$$60.68 for zero-shot structural interpretation and comparable or superior results to specialized models for new property estimation tasks, including Martian radar data and karst delineation (Dou et al., 1 Jul 2025).
• Behavioral control: Foundation policies in Bio-mimetic humanoid tasks achieve 73% of aggregate task coverage, with single models matching or outperforming task-specific baselines in reward, imitation, and goal-reaching tasks (Tirinzoni et al., 15 Apr 2025, Li et al., 6 Nov 2025).
• Geometric perception: Metric3Dv2 delivers zero-shot metric depth and normal estimation at $$\pi(a_t \mid s_t, g)\quad\text{or equivalently}\quad \pi(a_t\mid X_t, g)$$7 and median normal errors well below 10°, and supports absolute metrology on arbitrary internet images with solely EXIF metadata (Hu et al., 2024).
• Visual detection: ZERO achieves strong zero-shot detection mAP on industrial domains, with prompt diversity and pseudo-labeling contributing measurable improvements (3–5 mAP points) (Choi et al., 6 Jul 2025).
• Navigation: Foundation model-based frontier selection outperforms baselines with up to 4-point higher SPL and 2–3 point higher SR on large-scale navigation datasets using purely zero-shot policies (Yuan et al., 2024).

5. Limitations, Reliability, and Open Challenges

Observed limitations and open challenges include:

• Domain gaps: Pre-trained models may exhibit degraded performance on real-world sensor data or under distributional shift, due to training on web-scale images or simulated data (Cui et al., 2022).
• Viewpoint and occlusion sensitivity: Visual goal specification is susceptible to occlusions and changes in perspective uncommon in training distributions.
• Prompt ambiguity: Sparse, conflicting, or out-of-scope prompts may yield inconsistent or implausible completions, especially in structured domains such as geoscience or robotic assembly (Dou et al., 1 Jul 2025).
• Fine-grained positional control: Tasks requiring precise spatial specification (e.g., "move object 5cm right") are not robustly captured without explicit spatial or object-level representations.
• Scalability constraints: Training foundation models on $$\pi(a_t \mid s_t, g)\quad\text{or equivalently}\quad \pi(a_t\mid X_t, g)$$8 samples or more requires significant computational resources, and deployment in resource-constrained environments needs additional engineering (Hu et al., 2024, Choi et al., 6 Jul 2025).
• Zero-shot–vs.–domain-expert tradeoff: In domain-adapted LLMs, overfitting to specific instructions may reduce generalizability; model merging partially addresses this, but further theoretical understanding is required (Marincione et al., 7 Nov 2025).
• Reliability and commonsense errors: Decision-making pipelines based on foundation model outputs may produce nonsensical or unsafe actions; ensemble approaches (e.g., expert consensus) and explicit voting mechanisms help mitigate these risks in navigation (Yuan et al., 2024).

6. Future Directions

Promising research avenues identified in current literature include:

• Video and dynamic corpus pre-training: For mobile robotics and manipulation, scaling foundation models to video datasets may provide better grounding of temporally extended tasks (Cui et al., 2022).
• Physics-informed and uncertainty-aware extensions: Geoscientific models may benefit from incorporating physics constraints or explicit uncertainty quantification (Dou et al., 1 Jul 2025).
• Object-level and spatial relation integration: Fine-grained control tasks call for embedding architectures that recognize compositional spatial logic (e.g., using DETR-style transformers).
• Few-shot and in-domain adaptation: Lightweight fine-tuning strategies, such as adapter-based residual merging or prompt-based few-shot selection, are being explored to bridge domain gaps without full retraining (Marincione et al., 7 Nov 2025).
• Generalization to unstructured real-world deployments: Further study is needed on robustness under severe environment changes, sensor noise, and real-time constraints, particularly for embodied foundation model deployment (Hu et al., 2024, Li et al., 6 Nov 2025).

7. Cross-Domain Synthesis and Impact

Foundation model–based zero-shot control introduces a unifying paradigm in which large-scale, multi-modal representations mediate the translation of arbitrary human or programmatic goals into executable actions or outputs. The evidence from robotics, geoscience, computer vision, bioacoustics, and navigation indicates:

• Prompts as a formal “control interface” enable compositional and interpretable specification of new tasks, classes, or objectives at inference.
• The underlying representations—learned via highly diverse and large-scale data—provide sufficient “semantic coverage” to allow novel tasks to be solved, often outstripping hand-engineered or narrowly-supervised models.
• Performance in zero-shot tasks is tightly coupled to architectural choices (e.g., contrastive alignment, latent structural frameworks, prompt fusion mechanisms), dataset scale/diversity, and principled prompt conditioning/reward shaping.

This synthesis underscores the ongoing convergence of foundation model research and zero-shot control, offering generalist AI systems with flexible interfaces, scalable learning, and rapid adaptation across modalities and domains (Cui et al., 2022, Dou et al., 1 Jul 2025, Marincione et al., 7 Nov 2025, Tirinzoni et al., 15 Apr 2025, Hu et al., 2024, Choi et al., 6 Jul 2025, Li et al., 6 Nov 2025, Yuan et al., 2024).