---
title: Zero-Shot Control in Modern Systems
url: https://www.emergentmind.com/topics/zero-shot-control
type: topic
---

# Zero-Shot Control in Modern Systems

Searching arXiv for recent papers on zero-shot control to ground the article in current literature.
Zero-shot control denotes a family of methods in which a controller, policy, or inference procedure is expected to handle a new task, system instance, environment, or control specification without task-specific retraining at deployment time. Across the cited literature, the term is used in several technically distinct but structurally related senses: online deployment without a priori plant models or historical data in building control, immediate transfer across parametric optimal control problems by reusing a learned policy basis, direct sim-to-real deployment of policies trained only in simulation, execution of novel computer-interaction tasks without expert traces, and inference-time steering of pretrained generative models without fine-tuning [2206.14191] [2509.18404] [2503.02198] [2310.08740] [2401.10150].

## 1. Scope and operative meanings

The literature does not treat zero-shot control as a single protocol. In building HVAC control, it means no a priori building model, no offline pre-training, and direct online deployment after only a short commissioning period [2206.14191]. In transferable optimal control, it means that the expensive computation is performed once offline, after which a new task is handled by estimating only a low-dimensional coefficient vector or by directly mapping task specifications to coefficients [2509.18404]. In sim-to-real robotics, it means training purely in simulation and deploying on hardware with no additional fine-tuning [2503.02198] [2309.09870]. In computer-use agents, it means control without expert traces or few-shot task traces [2310.08740]. In generative modeling, it commonly means inference-time control with frozen backbone parameters, such as motion, prosody, concept, or style control without retraining [2401.10150] [2408.06827] [2503.06568].

| Domain | Zero-shot meaning | Representative source |
|---|---|---|
| Building control | No simulator or building-specific data before deployment | [2206.14191] |
| Parametric optimal control | Offline basis learning, online coefficient inference | [2509.18404] |
| Sim-to-real robotics | Simulation-only training, direct hardware deployment | [2503.02198] |
| Computer control | No expert traces or task exemplars | [2310.08740] |
| Generative control | Inference-time steering without training/fine-tuning | [2401.10150] |

This variety suggests that zero-shot control is better understood as a deployment constraint than as a single algorithmic family. What unifies the cases is that adaptation is shifted away from conventional retraining. The adaptation variable may instead be a short exploration phase, a coefficient vector in a learned function space, a structured prompt, a reference example, or an inference-time intervention on internal representations.

## 2. Formalizations and recurring algorithmic patterns

One recurrent formalization appears in parametric optimal control with fixed dynamics and task-dependent cost. In that setting, the dynamics are written as
$$
\dot{x}(t)=f(x(t),u(t),t), \qquad x(0)=x_0,
$$
and the optimal feedback family is approximated by a function encoder expansion
$$
u^*(x,t;\theta)\approx \hat u(x,t;\theta)=\sum_{j=1}^p c_j(\theta)\,\phi_j(x,t;\varphi_j).
$$
The offline stage learns reusable neural basis functions, while the online stage estimates only the coefficients \(c_j(\theta)\), either by least squares from measurements or by an operator network \(c(\theta)=\psi(\theta)\) [2509.18404]. A closely related architecture appears in differentiable predictive control with FE-NODE dynamics, where the unknown system instance is represented by a coefficient vector \(\mathbf c\) satisfying
$$
(\mathbf G+\lambda \mathbf I)\mathbf c=\mathbf F,
$$
after which the explicit policy \(\pi_{\mathbf W}(\mathbf x;\boldsymbol\xi,\mathbf c)\) is evaluated online [2511.05757].

A second pattern is commissioning-based system identification followed by model-based planning. In PEARL for building control, the learned one-step dynamics model is probabilistic,
$$
\tilde f_\theta(s_t,a_t)=\mathcal N(\mu_\theta(s_t,a_t),\Sigma_\theta(s_t,a_t)),
$$
with ensemble averaging used to capture epistemic uncertainty. During commissioning, control maximizes predictive variance,
$$
V_\Gamma = Var_P\!\left[\sum_{t=0}^H \mathcal R(s_t,a_t)\right],
$$
and only later switches to reward maximization through MPPI planning [2206.14191]. Here, zero-shot behavior is not training-free; it is enabled by a short active exploration period that substitutes for pre-existing simulators or logs.

A third pattern is inference-time control of frozen pretrained models. Motion-Zero alters initial latent noise, cross-attention maps, and temporal attention in a video diffusion model without any training process [2401.10150]. PRESENT edits duration, pitch, and energy predictions in a FastSpeech2-based TTS model directly at inference time [2408.06827]. Conceptrol constrains where a personalization image can act by masking visual specification with a textual concept mask, again without extra training [2503.06568]. These cases move control from parameter learning to representation steering.

## 3. Control of physical dynamical systems

Zero-shot control is particularly salient when plant-specific modeling is expensive or unavailable. In building control, PEARL was evaluated on three Energym simulations: a Mixed-Use facility in Athens, an Office block in Athens, and a Seminar Centre in Billund. The method uses a short commissioning period, then plans emissions-aware HVAC actions with a probabilistic ensemble model and MPPI. In the Mixed-Use environment, emissions decrease from 68.09 tCO\(_2\) under the RBC to 46.67 tCO\(_2\) under PEARL, a 31.46% reduction, with 0.55% temperature infractions versus 2.47% for the RBC. Across environments, PEARL is reported to have the best mean reward, combining emissions and comfort [2206.14191].

For parametric optimal control, zero-shot transfer is made explicit by reusable policy bases. Numerical experiments cover 2D path planning, a 12D quadcopter, and bicycle navigation with obstacle-dependent running costs. The reported objective errors are below about \(4\%\) on seen and unseen targets in the path-planning setting and about \(0.4\%\) across 27 unseen tasks in the 12D quadcopter problem, indicating near-optimal transfer after coefficient inference rather than re-solving the original OCP [2509.18404].

In plasma shape control, the notion is extended to a goal-conditioned “foundation policy” trained from 1,115 valid HL-3 discharges. The state is \(s\in\mathbb R^7=(I_p,a,\kappa,\delta_l,\delta_u,R,Z)\), the action is a 17-dimensional coil-voltage vector, and zero-shot control is achieved by combining GAIL with Hilbert space representation learning. The learned policy tracks unseen reference trajectories on shots \#5020, \#6338, and \#11638 without task-specific fine-tuning, while latent-space analysis indicates a geometrically structured representation of discharge templates and temporal continuity [2510.17531].

Differentiable predictive control with FE-NODEs generalizes the same theme to system families with unknown or switching dynamics. On Van der Pol, two-tank, glycolytic oscillator, and quadrotor benchmarks, FE-DPC reports MSEs of 0.002683, 0.008452, 0.180299, and 0.022003, with inference times of 0.53 s, 1.13 s, 5.89 s, and 1.93 s, respectively. The corresponding white-box MPC times are 1.21 s, 6.75 s, 136.07 s, and 155.85 s, yielding speedups of about 2.3× to 71.5× while retaining competitive tracking under abrupt parameter switches [2511.05757].

## 4. Sim-to-real and interactive embodied systems

In embodied robotics, zero-shot control often denotes direct transfer from synthetic training to physical execution. FalconGym provides a NeRF-based quadrotor racing environment aligned with real-world track coordinates. The control stack combines a ViT-based Neural Pose Estimator, a Kalman filter, and a self-attention-based multimodal controller trained entirely in simulation with imitation learning. On 30 live flights across three tracks and 120 gates, the deployed policy achieves a 95.8% success rate and an average gate-crossing error of about 10 cm for 38 cm-radius gates, without any real-world fine-tuning [2503.02198].

A more classical control instantiation appears in zero-shot transfer for a scale autonomous vehicle. There, a feed-forward neural controller is trained only in simulation from human-in-the-loop or MPC demonstrations, using RTK-GPS and IMU-based error states. The resulting policies transfer directly to a real parking-lot trajectory in both constant-speed and variable-speed settings. The work emphasizes that the same trained policy is used “as is,” with no real-world training data, and that qualitative differences between human- and MPC-trained controllers persist after transfer: smoother but less aggressive behavior for the former, more precise tracking and sharper speed response for the latter [2309.09870].

Computer control broadens the concept beyond physical actuation. A zero-shot language agent for MiniWoB++ uses compact screen representations, staged planning of executable actions, and structured self-reflection. On 1-screen-1-step tasks it reaches 96.4% completion with one trial and 99.6% with three trials; on 1-screen-\(n\)-step tasks it reaches 94.0% with one trial and 96.2% with three trials; on harder \(n\)-screen-\(n\)-step tasks it improves from 73.5% with one trial to 87.3% with five trials. The mechanism is not task-specific learning from demonstrations but iterative error correction through explicit reflection memory and disabled-action sets [2310.08740].

## 5. Control of pretrained generative models

A substantial branch of the literature extends zero-shot control to pretrained generative systems. Motion-Zero enables bounding-box trajectory control in text-to-video diffusion by combining an Initial Noise Prior Module, gradient-based spatial constraints on cross-attention maps, and a Shift Temporal Attention Mechanism. The method is training-free and backbone-agnostic across ModelScope and ZeroScope, and it reports improvements on ZeroScope from 20.05 to 22.04 in Text Align, from 0.89 to 0.93 in Consistency, and from 18.15 to 20.00 in PickScore [2401.10150].

In speech synthesis, PRESENT treats zero-shot prosody control as an inference-editing problem over explicit duration, pitch, and energy predictors. Applied to an English-only JETS model, it reports character error rates of 12.82% for German, 18.73% for Hungarian, and 5.92% for Spanish, exceeding the previous state of the art by over 2× for all three languages. The same framework also supports subphoneme-level control for Mandarin tone shaping, yielding 25.3% hanzi CER and 13.0% pinyin CER, compared with substantially worse results when tones or duration control are removed [2408.06827].

Conceptrol reframes zero-shot personalized image generation by treating the reference image as a visual specification of a textual concept rather than as a global prompt. The method is training-free, reuses the base model’s own concept localization, and improves the DreamBench++ final score from 0.210 to 0.397 on Stable Diffusion 1.5 + IP-Adapter, from 0.346 to 0.524 on SDXL + IP-Adapter, and from 0.398 to 0.481 on FLUX + OminiControl [2503.06568].

Zero-shot control in audio generation also includes simultaneous voice cloning and style manipulation. ControlSpeech combines FACodec-based disentanglement with a Style Mixture Semantic Density module and reports on unseen speakers a WER of 4.1, MOS-Q of \(3.90\pm0.12\), and MOS-S of \(3.94\pm0.14\), while also supporting textual style control [2406.01205]. TCSinger extends this line to singing voice synthesis with multi-level style control; on zero-shot style transfer it reports MOS-Q \(4.12\pm0.08\), MOS-S \(4.28\pm0.06\), FFE 0.22, MCD 3.16, and Cos 0.92 across 40 unseen singers [2409.15977]. More recent zero-shot TTS work shifts attention to relative and disentangled control: ReStyle-TTS weakens reference-style inheritance with Decoupled Classifier-Free Guidance and then applies style-specific LoRAs; FC-TTS uses two separate references to control timbre and style independently through factorized codec conditioning and a two-stage generator [2601.03632] [2605.24618].

## 6. Limitations, trade-offs, and open questions

A recurrent misconception is that zero-shot control implies zero data. Several sources explicitly contradict this. PEARL requires a short commissioning period and active exploration in the live building [2206.14191]. FE policies can adapt by least squares from task-specific measurements, and their operator-network alternative is fully data-free online only at the cost of more demanding offline training and more difficult generalization when the task descriptor is high-dimensional or implicit [2509.18404]. FE-DPC likewise requires online transitions to estimate the dynamics coefficients \(\mathbf c\) [2511.05757].

Another persistent trade-off is between controllability and fidelity to the reference or the plant. ReStyle-TTS reports that standard CFG preserves timbre well but yields almost no attribute change, whereas DCFG unlocks control and TCO is then required to recover timbre consistency [2601.03632]. FC-TTS explicitly frames its results as sacrificing some naturalness ceiling for stronger control and robustness [2605.24618]. Voice-impression control in zero-shot TTS shows that stronger modulation can reduce naturalness and speaker similarity, especially for extreme expressions, and that impression dimensions are correlated rather than independent [2506.05688].

Sim-to-real systems remain sensitive to simulator fidelity and deployment constraints. FalconGym depends on photorealistic NeRF rendering, a double-integrator assumption in the Kalman filter, and offboard inference with about 39 ms end-to-end latency [2503.02198]. The scale-vehicle study shows residual mismatch from steering slack and parking-lot slope [2309.09870]. PRESENT improves intelligibility in unseen languages but still yields a strong American accent and relatively low MOS in Mandarin [2408.06827].

In generative control, the main technical controversy concerns where control should be exerted: on latents, attention, prompts, codec tokens, or factorized style variables. The existing literature supports all of these routes, but only under different assumptions about what should remain frozen and what should be inferred at test time. This suggests that “zero-shot control” is not converging toward a single universal mechanism. Rather, it is becoming a design principle for separating expensive learning from deployment-time adaptation, whether the controlled object is a building, a quadrotor, a plasma, a browser session, or a pretrained generative model.

Source: https://www.emergentmind.com/topics/zero-shot-control