---
title: 'BAGEL: Bootstrapping Agents with Guided Language'
url: https://www.emergentmind.com/topics/bagel
type: topic
---

# BAGEL: Bootstrapping Agents with Guided Language

Searching arXiv for the primary BAGEL paper and closely related paradigms mentioned in the provided data.

BAGEL, short for **Bootstrapping Agents by Guiding Exploration with Language**, is a method for training language-model agents to follow natural-language instructions in digital environments without human demonstrations. It addresses settings such as web browsers and tool suites in which observations and actions can be represented as strings, and where even simple instructions may require environment-specific action sequences that are difficult to infer zero-shot. BAGEL converts either randomly explored trajectories or synthetically generated instructions into demonstrations through iterative round-trips between an LM labeler, which maps trajectories to instructions, and a zero-shot LM agent, which maps instructions back to trajectories. The resulting demonstration set is then used for test-time adaptation through retrieval-augmented in-context learning, yielding improvements of over 2–13% absolute on ToolQA and MiniWob++, with up to 13× reduction in execution failures [2403.08140].

## 1. Problem setting and motivation

BAGEL is situated in the problem of instruction following in digital environments such as web browsers and REST-style tool suites, where a language model agent must execute action sequences conditioned on natural-language goals [2403.08140]. In this setting, the agent is a language-conditioned policy $\pi(a_t \mid \tau_{<t}, x)$, where $x$ is the instruction, $\tau$ is the trajectory, and $\tau_{<t}$ is the interaction history up to time $t$ [2403.08140]. Action strings are issued to a low-level controller that maps them to executable operations, such as Selenium commands in MiniWob++ or tool invocations in ToolQA [2403.08140].

The central difficulty is generalization to new environments without environment-specific demonstrations. The motivating examples in BAGEL emphasize that apparently simple goals may require non-obvious, environment-dependent procedures; entering a date, for example, may require navigating a date picker rather than typing directly [2403.08140]. A common remedy is to provide expert demonstrations for conditioning or fine-tuning, but collecting such demonstrations for every environment is laborious and often infeasible as task complexity and horizon length increase [2403.08140].

BAGEL is designed as an alternative to supervised demonstration collection. It starts from either high-entropy exploration trajectories or synthetic instructions, then iteratively refines them into consistent instruction–trajectory pairs accepted by a binary correctness filter [2403.08140]. This suggests a reframing of demonstration acquisition: instead of treating demonstrations as an external prerequisite, BAGEL treats them as latent objects recoverable from the interaction between language priors and environment-grounded execution.

## 2. Core formulation and round-trip mechanism

BAGEL’s formalism centers on two objects: a trajectory and an instruction. A trajectory is

$$
\tau = (o_1, a_1, o_2, a_2, \ldots, o_T, a_T, o_{T+1}),
$$

where $o_t$ are string observations and $a_t$ are action strings, while an instruction is a natural-language string $x$ or $g$ [2403.08140]. The method uses five components: an exploration policy $\pi_{\text{explore}}(a_t \mid \tau_{<t})$, an LM labeler $L$ with distribution $p_{\text{label}}(x \mid \tau)$, a zero-shot LM agent $A$ with induced distribution $p_{\text{agent}}(\tau \mid x)$, a binary demonstration filter $s(x,\tau) \in \{0,1\}$, and, in one variant, an instruction generator $p_{\text{instruct}}(x)$ [2403.08140].

The iterative procedure alternates between labeling and instruction following:

$$
g^t \sim p_{\text{label}}(\cdot \mid \tau^t), \qquad
\tau^{t+1} \sim p_{\text{agent}}(\cdot \mid g^t).
$$

A candidate pair is accepted if the consistency criterion satisfies

$$
C(x,\tau) = s(x,\tau) \in \{0,1\}, \qquad
C(g^t,\tau^t) \ge \delta,
$$

with $\delta = 1$, in which case $(g^t,\tau^t)$ is added to the demonstration set $D$ [2403.08140].

The paper also presents a distributional interpretation. Let $p_k(\tau)$ and $p_k(x)$ denote the marginal trajectory and instruction distributions after $k$ iterations. Then

$$
p_k(\tau) = \sum_{x'} p_{\text{agent}}(\tau \mid x') \, p_{k-1}(x'),
$$

$$
p_k(x) = \sum_{\tau'} p_{\text{label}}(x \mid \tau') \, p_{k-1}(\tau'),
$$

and hence

$$
p_k(\tau) = \sum_{\tau',x'} p_{k-1}(\tau') \, p_{\text{label}}(x' \mid \tau') \, p_{\text{agent}}(\tau \mid x').
$$

The stated effect is a “soft conversion” of the trajectory distribution toward trajectories that are both describable in language and executable by the agent [2403.08140]. This suggests that BAGEL is not merely filtering demonstrations after the fact; it is using language as an iterative projection operator on behavior space.

## 3. Trajectory-first and instruction-first bootstrapping

BAGEL supports two initialization modes. In the **trajectory-first** variant, initialization samples $\tau^0 \sim p_{\text{explore}}(\cdot)$ from an unconditioned exploration policy implemented via a ReAct-style prompt [2403.08140]. In the **instruction-first** variant, initialization samples $x^0 \sim p_{\text{instruct}}(\cdot)$ from an instruction generator based on the initial observation and action space [2403.08140].

In trajectory-first BAGEL, the system begins from semantically noisy behavior. The labeler produces a candidate instruction describing the trajectory, and the agent attempts to follow that instruction, yielding a refined trajectory. Repeated label–follow rounds progressively improve both instruction quality and behavioral coherence until the filter accepts a pair [2403.08140]. In instruction-first BAGEL, the system instead begins from synthetic instructions that may be infeasible or misaligned with the environment; execution and relabeling continue until a consistent pair is found [2403.08140].

The paper’s MiniWob++ choose-date example illustrates this refinement process in concrete terms. Initial exploration may produce clicks on the datepicker and month toggles without a coherent goal. The first labeling step yields “Change month from December to October,” and the first agent round changes the month but does not submit. A second labeling step refines the goal to “Change month to October 7th and submit,” after which the agent navigates the months, selects October 7th, submits, and the filter accepts the pair [2403.08140]. The example is significant because it shows the accepted demonstration emerging from multiple imperfect components rather than from any single accurate model invocation.

The two variants exhibit different empirical profiles. On ToolQA, instruction-first performs best among the BAGEL variants, while on MiniWob++ trajectory-first performs best [2403.08140]. The paper’s practical guidance attributes this to task structure: trajectory-first is often better when plausible instructions are hard to infer from the initial observation alone, as in MiniWob++ landing pages, whereas instruction-first can excel when the LM can propose realistic instructions from the initial state and action space, as in some ToolQA domains [2403.08140].

## 4. Demonstration use at test time

BAGEL does not fine-tune the zero-shot LM agent. Instead, it performs test-time adaptation through retrieval-augmented in-context learning over the mined demonstrations [2403.08140]. Instruction embeddings $f_\theta(x)$ are computed using a T5-XXL model with mean-pooled word embeddings and cosine similarity [2403.08140]. For a test instruction $x_{\text{test}}$, the system retrieves the top-$k$ demonstrations, with $k=3$, by cosine similarity between $f_\theta(x_{\text{test}})$ and the stored demonstration instructions [2403.08140]. These demonstrations are prepended as exemplars in the agent prompt, again in a ReAct-style format, and the agent is rolled out in the environment [2403.08140].

The low-level execution layer is also part of the system design. If an action string causes an exception, such as invalid tool syntax or typing on a non-typeable DOM element, BAGEL re-prompts the agent with the exception message and re-samples up to $m = 5$ times [2403.08140]. The paper reports that this simple recovery scheme reduces execution failures at test time, especially when in-context demonstrations are supplied [2403.08140].

This retrieval stage matters conceptually because it clarifies what BAGEL is optimizing for. The mined demonstrations are not merely a by-product for offline evaluation; they are the mechanism by which a zero-shot LM agent is adapted to environment dynamics at inference time. A plausible implication is that BAGEL’s success depends not only on demonstration correctness, but also on whether the demonstration set captures reusable interaction motifs that can be retrieved by instruction similarity.

## 5. Experimental setting and quantitative results

The paper evaluates BAGEL on **MiniWob++** and **ToolQA** [2403.08140]. MiniWob++ comprises web UI tasks with DOM interactions via Selenium, using a natural-language action space such as Type, Click, Clear, and Move Mouse that a low-level controller maps to driver commands. Evaluation follows Shaw et al., mapping raw reward in $[-1,1]$ to $[0,1]$, averaged over 50 random seeds, and focuses on the hardest 10 tasks with baseline average reward below 0.95 [2403.08140]. ToolQA comprises tool-augmented question answering across 8 domains—Agenda, AirBnB, Coffee, DBLP, Flights, GSM8K, Scirex, and Yelp—with actions expressed in language and parsed into tool calls, and performance measured by F1 of the final answer string against ground truth [2403.08140].

Implementation details are fixed across experiments. The base LM is instruction-tuned PaLM-2; maximum episode length is $T = 15$; the number of round-trip iterations is $T_{\text{iter}} = 5$; and execution-failure re-sampling uses $m = 5$ [2403.08140]. Data collection uses 60 seeds for MiniWob++ and 200 seeds for ToolQA, with either trajectories or synthetic instructions depending on the initialization mode [2403.08140]. Retrieval uses T5-XXL embeddings, cosine similarity, and top-3 demonstrations [2403.08140].

The principal numerical findings are summarized below.

| Setting | Result |
|---|---|
| MiniWob++ | BAGEL improves average reward by over 13 percentage points compared to zero-shot |
| ToolQA | +BAGEL yields over 2 percentage points improvement in average F1 compared to zero-shot across the 8 domains |
| Execution failures | Up to 13× fewer failures on MiniWob++ choose-date, from 1.3 to 0.1 on average |

On MiniWob++, gains are especially strong on choose-date, tic-tac-toe, and use-autocomplete, with about 20% absolute improvement in some cases [2403.08140]. On execution failures, the choose-date task drops from 1.3 to 0.1 average failures, book-flight drops from 3.0 to 0.6, and overall ToolQA drops from 3.0 to 1.9 [2403.08140].

The ablation results isolate the contribution of iteration. On MiniWob++, averages are 46.8 for zero-shot, 52.0 for instruction-first no-itrs, 56.0 for full instruction-first, 53.0 for trajectory-first no-itrs, and 61.0 for full trajectory-first [2403.08140]. On ToolQA, averages are 40.9 for zero-shot, 43.3 for full instruction-first, and 42.2 for full trajectory-first [2403.08140]. These numbers support the claim that multiple round-trips consistently improve performance relative to non-iterative relabeling [2403.08140].

The paper also tests the role of correct instruction–trajectory alignment. On ToolQA, randomly chosen demonstrations and shuffled instruction–trajectory pairs still outperform zero-shot slightly, suggesting some benefit from seeing valid domain trajectories and instruction modes, but aligned demonstrations remain best: 42.2 for BAGEL, 41.4 for Shuffled, and 40.9 for Zero-shot [2403.08140]. This suggests that BAGEL’s gains are partly structural and partly semantic: trajectory priors help, but proper alignment helps more.

## 6. Analysis, limitations, and relation to prior paradigms

The paper identifies several failure modes. **Long-horizon planning** remains difficult: on MiniWob++ book-flight, zero-shot performance is 5%, and BAGEL raises it to 15%, which is an improvement but still leaves substantial room for hierarchical planning and subgoal decomposition [2403.08140]. **Diversity mismatch** can arise when seed exploration clusters around a narrow subset of modes, such as demonstrations that all book December flights, creating a mismatch with a more uniform test distribution [2403.08140]. **Instruction mismatch** appears in domains such as SciRex, where BAGEL tends to generate descriptive-answer questions with generic trajectories, whereas test tasks demand fine-grained retrieval of specific numbers; search-engine tasks requiring precise counting across pages are similarly under-covered [2403.08140]. Finally, **LM component misalignment** persists because the labeler may hallucinate instructions the agent cannot execute, or the agent may produce trajectories the labeler cannot faithfully describe [2403.08140].

Correctness filtering is consequential. The paper reports that manually verifying and removing incorrect demonstrations yields a further 7% absolute improvement on the 10 MiniWob++ tasks [2403.08140]. This indicates that the binary LM-based filter is useful but imperfect, and it motivates future work on stronger verifiers, environment-affordance cross-checking, and confidence calibration [2403.08140].

BAGEL is explicitly positioned relative to several prior paradigms. It uses **ReAct-style prompting** for both exploration and instruction following, but adds an LM labeler and a correctness filter to construct demonstrations without human supervision [2403.08140]. It resembles **Hindsight Experience Replay** in that trajectories are relabeled with goals after the fact, but it operates in natural language, leverages full trajectories, and performs multiple label–follow iterations to reduce noise [2403.08140]. It also intersects with **synthetic data generation** and **self-training**, yet differs in grounding its process in environment interaction and in using the resulting data for in-context adaptation rather than fine-tuning [2403.08140].

A common misconception would be to treat BAGEL as a purely synthetic-data pipeline. The method is better understood as an environment-grounded bootstrapping procedure: the demonstrations are synthetic in origin, but they are accepted only after interaction trajectories and instructions become mutually consistent under a verifier [2403.08140]. Another misconception would be to view it as a replacement for better planning or stronger verification. The reported limitations indicate the opposite: BAGEL reduces demonstration dependence, but does not eliminate the need for improved planning, diversity control, or correctness checking.

## 7. Significance and prospective extensions

BAGEL’s stated novelty is bootstrapping with language-guided exploration through iterative LM-to-LM round-trips, thereby aligning trajectories and instructions without human supervision [2403.08140]. Its practical value is greatest in environments where actions and observations are textual, where zero-shot agents lack exposure to environment dynamics, and where human demonstrations are unavailable or expensive [2403.08140]. MiniWob++ and ToolQA exemplify this regime because both expose sufficiently rich textual cues for relabeling and in-context reuse [2403.08140].

The paper outlines several extensions. These include fine-tuning on BAGEL demonstrations, hierarchical or subgoal policies, diversity-enhanced exploration, confidence-aware filtering, and leveraging limited prior knowledge about the test-time instruction distribution [2403.08140]. These are prospective directions rather than established results, but they make clear that BAGEL is intended as a general bootstrapping principle rather than a fixed pipeline.

Within research on LM agents, BAGEL is significant because it shifts the bottleneck from human demonstration collection to self-generated, filtered, and retrievable exemplars [2403.08140]. The experimental evidence indicates that this shift is operationally meaningful: on the studied benchmarks, a zero-shot agent can be adapted at test time with synthetic yet environment-grounded demonstrations, improving reward and F1 while sharply reducing execution failures [2403.08140]. This suggests that, for a substantial class of digital environments, language can serve not only as a control interface but also as an instrument for reshaping exploration into reusable competence.

Source: https://www.emergentmind.com/topics/bagel