Zero-Shot Control in Modern Systems

• Zero-Shot Control is a design principle where controllers manage new tasks by using rapid online adaptation or coefficient inference without task-specific retraining.
• It spans diverse applications, including building HVAC systems, sim-to-real robotics, and generative model control, highlighting adaptable methodologies across domains.
• Key approaches leverage offline basis learning, active exploration during commissioning, and inference-time adjustments to efficiently deploy pre-trained models.

Searching arXiv for papers on zero-shot control to ground the article in current literature. Zero-shot control denotes a family of methods in which a controller, policy, or inference procedure is expected to handle a new task, system instance, environment, or control specification without task-specific retraining at deployment time. Across the cited literature, the term is used in several technically distinct but structurally related senses: online deployment without a priori plant models or historical data in building control, immediate transfer across parametric optimal control problems by reusing a learned policy basis, direct sim-to-real deployment of policies trained only in simulation, execution of novel computer-interaction tasks without expert traces, and inference-time steering of pretrained generative models without fine-tuning (Jeen et al., 2022, Li et al., 22 Sep 2025, Miao et al., 4 Mar 2025, Li et al., 2023, Chen et al., 2024).

1. Scope and operative meanings

The literature does not treat zero-shot control as a single protocol. In building HVAC control, it means no a priori building model, no offline pre-training, and direct online deployment after only a short commissioning period (Jeen et al., 2022). In transferable optimal control, it means that the expensive computation is performed once offline, after which a new task is handled by estimating only a low-dimensional coefficient vector or by directly mapping task specifications to coefficients (Li et al., 22 Sep 2025). In sim-to-real robotics, it means training purely in simulation and deploying on hardware with no additional fine-tuning (Miao et al., 4 Mar 2025, Zhang et al., 2023). In computer-use agents, it means control without expert traces or few-shot task traces (Li et al., 2023). In generative modeling, it commonly means inference-time control with frozen backbone parameters, such as motion, prosody, concept, or style control without retraining (Chen et al., 2024, Lam et al., 2024, He et al., 9 Mar 2025).

Domain	Zero-shot meaning	Representative source
Building control	No simulator or building-specific data before deployment	(Jeen et al., 2022)
Parametric optimal control	Offline basis learning, online coefficient inference	(Li et al., 22 Sep 2025)
Sim-to-real robotics	Simulation-only training, direct hardware deployment	(Miao et al., 4 Mar 2025)
Computer control	No expert traces or task exemplars	(Li et al., 2023)
Generative control	Inference-time steering without training/fine-tuning	(Chen et al., 2024)

This variety suggests that zero-shot control is better understood as a deployment constraint than as a single algorithmic family. What unifies the cases is that adaptation is shifted away from conventional retraining. The adaptation variable may instead be a short exploration phase, a coefficient vector in a learned function space, a structured prompt, a reference example, or an inference-time intervention on internal representations.

2. Formalizations and recurring algorithmic patterns

One recurrent formalization appears in parametric optimal control with fixed dynamics and task-dependent cost. In that setting, the dynamics are written as

$\dot{x}(t)=f(x(t),u(t),t), \qquad x(0)=x_0,$

and the optimal feedback family is approximated by a function encoder expansion

$$u^*(x,t;\theta)\approx \hat u(x,t;\theta)=\sum_{j=1}^p c_j(\theta)\,\phi_j(x,t;\varphi_j).$$

The offline stage learns reusable neural basis functions, while the online stage estimates only the coefficients $c_j(\theta)$, either by least squares from measurements or by an operator network $c(\theta)=\psi(\theta)$ (Li et al., 22 Sep 2025). A closely related architecture appears in differentiable predictive control with FE-NODE dynamics, where the unknown system instance is represented by a coefficient vector $\mathbf c$ satisfying

$(\mathbf G+\lambda \mathbf I)\mathbf c=\mathbf F,$

after which the explicit policy $$\pi_{\mathbf W}(\mathbf x;\boldsymbol\xi,\mathbf c)$$ is evaluated online (Iqbal et al., 7 Nov 2025).

A second pattern is commissioning-based system identification followed by model-based planning. In PEARL for building control, the learned one-step dynamics model is probabilistic,

$$\tilde f_\theta(s_t,a_t)=\mathcal N(\mu_\theta(s_t,a_t),\Sigma_\theta(s_t,a_t)),$$

with ensemble averaging used to capture epistemic uncertainty. During commissioning, control maximizes predictive variance,

$$V_\Gamma = Var_P\!\left[\sum_{t=0}^H \mathcal R(s_t,a_t)\right],$$

and only later switches to reward maximization through MPPI planning (Jeen et al., 2022). Here, zero-shot behavior is not training-free; it is enabled by a short active exploration period that substitutes for pre-existing simulators or logs.

A third pattern is inference-time control of frozen pretrained models. Motion-Zero alters initial latent noise, cross-attention maps, and temporal attention in a video diffusion model without any training process (Chen et al., 2024). PRESENT edits duration, pitch, and energy predictions in a FastSpeech2-based TTS model directly at inference time (Lam et al., 2024). Conceptrol constrains where a personalization image can act by masking visual specification with a textual concept mask, again without extra training (He et al., 9 Mar 2025). These cases move control from parameter learning to representation steering.

3. Control of physical dynamical systems

Zero-shot control is particularly salient when plant-specific modeling is expensive or unavailable. In building control, PEARL was evaluated on three Energym simulations: a Mixed-Use facility in Athens, an Office block in Athens, and a Seminar Centre in Billund. The method uses a short commissioning period, then plans emissions-aware HVAC actions with a probabilistic ensemble model and MPPI. In the Mixed-Use environment, emissions decrease from 68.09 tCO$_2$ under the RBC to 46.67 tCO$$u^*(x,t;\theta)\approx \hat u(x,t;\theta)=\sum_{j=1}^p c_j(\theta)\,\phi_j(x,t;\varphi_j).$$0 under PEARL, a 31.46% reduction, with 0.55% temperature infractions versus 2.47% for the RBC. Across environments, PEARL is reported to have the best mean reward, combining emissions and comfort (Jeen et al., 2022).

For parametric optimal control, zero-shot transfer is made explicit by reusable policy bases. Numerical experiments cover 2D path planning, a 12D quadcopter, and bicycle navigation with obstacle-dependent running costs. The reported objective errors are below about $$u^*(x,t;\theta)\approx \hat u(x,t;\theta)=\sum_{j=1}^p c_j(\theta)\,\phi_j(x,t;\varphi_j).$$1 on seen and unseen targets in the path-planning setting and about $$u^*(x,t;\theta)\approx \hat u(x,t;\theta)=\sum_{j=1}^p c_j(\theta)\,\phi_j(x,t;\varphi_j).$$2 across 27 unseen tasks in the 12D quadcopter problem, indicating near-optimal transfer after coefficient inference rather than re-solving the original OCP (Li et al., 22 Sep 2025).

In plasma shape control, the notion is extended to a goal-conditioned “foundation policy” trained from 1,115 valid HL-3 discharges. The state is $$u^*(x,t;\theta)\approx \hat u(x,t;\theta)=\sum_{j=1}^p c_j(\theta)\,\phi_j(x,t;\varphi_j).$$3, the action is a 17-dimensional coil-voltage vector, and zero-shot control is achieved by combining GAIL with Hilbert space representation learning. The learned policy tracks unseen reference trajectories on shots #5020, #6338, and #11638 without task-specific fine-tuning, while latent-space analysis indicates a geometrically structured representation of discharge templates and temporal continuity (Wu et al., 20 Oct 2025).

Differentiable predictive control with FE-NODEs generalizes the same theme to system families with unknown or switching dynamics. On Van der Pol, two-tank, glycolytic oscillator, and quadrotor benchmarks, FE-DPC reports MSEs of 0.002683, 0.008452, 0.180299, and 0.022003, with inference times of 0.53 s, 1.13 s, 5.89 s, and 1.93 s, respectively. The corresponding white-box MPC times are 1.21 s, 6.75 s, 136.07 s, and 155.85 s, yielding speedups of about 2.3× to 71.5× while retaining competitive tracking under abrupt parameter switches (Iqbal et al., 7 Nov 2025).

4. Sim-to-real and interactive embodied systems

In embodied robotics, zero-shot control often denotes direct transfer from synthetic training to physical execution. FalconGym provides a NeRF-based quadrotor racing environment aligned with real-world track coordinates. The control stack combines a ViT-based Neural Pose Estimator, a Kalman filter, and a self-attention-based multimodal controller trained entirely in simulation with imitation learning. On 30 live flights across three tracks and 120 gates, the deployed policy achieves a 95.8% success rate and an average gate-crossing error of about 10 cm for 38 cm-radius gates, without any real-world fine-tuning (Miao et al., 4 Mar 2025).

A more classical control instantiation appears in zero-shot transfer for a scale autonomous vehicle. There, a feed-forward neural controller is trained only in simulation from human-in-the-loop or MPC demonstrations, using RTK-GPS and IMU-based error states. The resulting policies transfer directly to a real parking-lot trajectory in both constant-speed and variable-speed settings. The work emphasizes that the same trained policy is used “as is,” with no real-world training data, and that qualitative differences between human- and MPC-trained controllers persist after transfer: smoother but less aggressive behavior for the former, more precise tracking and sharper speed response for the latter (Zhang et al., 2023).

Computer control broadens the concept beyond physical actuation. A zero-shot language agent for MiniWoB++ uses compact screen representations, staged planning of executable actions, and structured self-reflection. On 1-screen-1-step tasks it reaches 96.4% completion with one trial and 99.6% with three trials; on 1-screen-$$u^*(x,t;\theta)\approx \hat u(x,t;\theta)=\sum_{j=1}^p c_j(\theta)\,\phi_j(x,t;\varphi_j).$$4-step tasks it reaches 94.0% with one trial and 96.2% with three trials; on harder $$u^*(x,t;\theta)\approx \hat u(x,t;\theta)=\sum_{j=1}^p c_j(\theta)\,\phi_j(x,t;\varphi_j).$$5-screen-$$u^*(x,t;\theta)\approx \hat u(x,t;\theta)=\sum_{j=1}^p c_j(\theta)\,\phi_j(x,t;\varphi_j).$$6-step tasks it improves from 73.5% with one trial to 87.3% with five trials. The mechanism is not task-specific learning from demonstrations but iterative error correction through explicit reflection memory and disabled-action sets (Li et al., 2023).

5. Control of pretrained generative models

A substantial branch of the literature extends zero-shot control to pretrained generative systems. Motion-Zero enables bounding-box trajectory control in text-to-video diffusion by combining an Initial Noise Prior Module, gradient-based spatial constraints on cross-attention maps, and a Shift Temporal Attention Mechanism. The method is training-free and backbone-agnostic across ModelScope and ZeroScope, and it reports improvements on ZeroScope from 20.05 to 22.04 in Text Align, from 0.89 to 0.93 in Consistency, and from 18.15 to 20.00 in PickScore (Chen et al., 2024).

In speech synthesis, PRESENT treats zero-shot prosody control as an inference-editing problem over explicit duration, pitch, and energy predictors. Applied to an English-only JETS model, it reports character error rates of 12.82% for German, 18.73% for Hungarian, and 5.92% for Spanish, exceeding the previous state of the art by over 2× for all three languages. The same framework also supports subphoneme-level control for Mandarin tone shaping, yielding 25.3% hanzi CER and 13.0% pinyin CER, compared with substantially worse results when tones or duration control are removed (Lam et al., 2024).

Conceptrol reframes zero-shot personalized image generation by treating the reference image as a visual specification of a textual concept rather than as a global prompt. The method is training-free, reuses the base model’s own concept localization, and improves the DreamBench++ final score from 0.210 to 0.397 on Stable Diffusion 1.5 + IP-Adapter, from 0.346 to 0.524 on SDXL + IP-Adapter, and from 0.398 to 0.481 on FLUX + OminiControl (He et al., 9 Mar 2025).

Zero-shot control in audio generation also includes simultaneous voice cloning and style manipulation. ControlSpeech combines FACodec-based disentanglement with a Style Mixture Semantic Density module and reports on unseen speakers a WER of 4.1, MOS-Q of $$u^*(x,t;\theta)\approx \hat u(x,t;\theta)=\sum_{j=1}^p c_j(\theta)\,\phi_j(x,t;\varphi_j).$$7, and MOS-S of $$u^*(x,t;\theta)\approx \hat u(x,t;\theta)=\sum_{j=1}^p c_j(\theta)\,\phi_j(x,t;\varphi_j).$$8, while also supporting textual style control (Ji et al., 2024). TCSinger extends this line to singing voice synthesis with multi-level style control; on zero-shot style transfer it reports MOS-Q $$u^*(x,t;\theta)\approx \hat u(x,t;\theta)=\sum_{j=1}^p c_j(\theta)\,\phi_j(x,t;\varphi_j).$$9, MOS-S $c_j(\theta)$0, FFE 0.22, MCD 3.16, and Cos 0.92 across 40 unseen singers (Zhang et al., 2024). More recent zero-shot TTS work shifts attention to relative and disentangled control: ReStyle-TTS weakens reference-style inheritance with Decoupled Classifier-Free Guidance and then applies style-specific LoRAs; FC-TTS uses two separate references to control timbre and style independently through factorized codec conditioning and a two-stage generator (Li et al., 7 Jan 2026, Lee et al., 23 May 2026).

6. Limitations, trade-offs, and open questions

A recurrent misconception is that zero-shot control implies zero data. Several sources explicitly contradict this. PEARL requires a short commissioning period and active exploration in the live building (Jeen et al., 2022). FE policies can adapt by least squares from task-specific measurements, and their operator-network alternative is fully data-free online only at the cost of more demanding offline training and more difficult generalization when the task descriptor is high-dimensional or implicit (Li et al., 22 Sep 2025). FE-DPC likewise requires online transitions to estimate the dynamics coefficients $c_j(\theta)$1 (Iqbal et al., 7 Nov 2025).

Another persistent trade-off is between controllability and fidelity to the reference or the plant. ReStyle-TTS reports that standard CFG preserves timbre well but yields almost no attribute change, whereas DCFG unlocks control and TCO is then required to recover timbre consistency (Li et al., 7 Jan 2026). FC-TTS explicitly frames its results as sacrificing some naturalness ceiling for stronger control and robustness (Lee et al., 23 May 2026). Voice-impression control in zero-shot TTS shows that stronger modulation can reduce naturalness and speaker similarity, especially for extreme expressions, and that impression dimensions are correlated rather than independent (Fujita et al., 6 Jun 2025).

Sim-to-real systems remain sensitive to simulator fidelity and deployment constraints. FalconGym depends on photorealistic NeRF rendering, a double-integrator assumption in the Kalman filter, and offboard inference with about 39 ms end-to-end latency (Miao et al., 4 Mar 2025). The scale-vehicle study shows residual mismatch from steering slack and parking-lot slope (Zhang et al., 2023). PRESENT improves intelligibility in unseen languages but still yields a strong American accent and relatively low MOS in Mandarin (Lam et al., 2024).

In generative control, the main technical controversy concerns where control should be exerted: on latents, attention, prompts, codec tokens, or factorized style variables. The existing literature supports all of these routes, but only under different assumptions about what should remain frozen and what should be inferred at test time. This suggests that “zero-shot control” is not converging toward a single universal mechanism. Rather, it is becoming a design principle for separating expensive learning from deployment-time adaptation, whether the controlled object is a building, a quadrotor, a plasma, a browser session, or a pretrained generative model.