BAGEL: Bootstrapping Agents with Guided Language

• BAGEL is a method that iteratively bootstraps language model agents by converting exploratory trajectories into aligned instruction–trajectory pairs using a round-trip mechanism.
• It eliminates the need for human demonstrations by leveraging a binary correctness filter and retrieval-augmented in-context learning to guide digital task execution.
• Empirical evaluations on MiniWob++ and ToolQA demonstrate significant improvements in performance metrics and reductions in execution failures across diverse tasks.

Searching arXiv for the primary BAGEL paper and closely related paradigms mentioned in the provided data.

BAGEL, short for Bootstrapping Agents by Guiding Exploration with Language, is a method for training language-model agents to follow natural-language instructions in digital environments without human demonstrations. It addresses settings such as web browsers and tool suites in which observations and actions can be represented as strings, and where even simple instructions may require environment-specific action sequences that are difficult to infer zero-shot. BAGEL converts either randomly explored trajectories or synthetically generated instructions into demonstrations through iterative round-trips between an LM labeler, which maps trajectories to instructions, and a zero-shot LM agent, which maps instructions back to trajectories. The resulting demonstration set is then used for test-time adaptation through retrieval-augmented in-context learning, yielding improvements of over 2–13% absolute on ToolQA and MiniWob++, with up to 13× reduction in execution failures (Murty et al., 2024).

1. Problem setting and motivation

BAGEL is situated in the problem of instruction following in digital environments such as web browsers and REST-style tool suites, where a LLM agent must execute action sequences conditioned on natural-language goals (Murty et al., 2024). In this setting, the agent is a language-conditioned policy $\pi(a_t \mid \tau_{<t}, x)$, where $x$ is the instruction, $\tau$ is the trajectory, and $\tau_{<t}$ is the interaction history up to time $t$ (Murty et al., 2024). Action strings are issued to a low-level controller that maps them to executable operations, such as Selenium commands in MiniWob++ or tool invocations in ToolQA (Murty et al., 2024).

The central difficulty is generalization to new environments without environment-specific demonstrations. The motivating examples in BAGEL emphasize that apparently simple goals may require non-obvious, environment-dependent procedures; entering a date, for example, may require navigating a date picker rather than typing directly (Murty et al., 2024). A common remedy is to provide expert demonstrations for conditioning or fine-tuning, but collecting such demonstrations for every environment is laborious and often infeasible as task complexity and horizon length increase (Murty et al., 2024).

BAGEL is designed as an alternative to supervised demonstration collection. It starts from either high-entropy exploration trajectories or synthetic instructions, then iteratively refines them into consistent instruction–trajectory pairs accepted by a binary correctness filter (Murty et al., 2024). This suggests a reframing of demonstration acquisition: instead of treating demonstrations as an external prerequisite, BAGEL treats them as latent objects recoverable from the interaction between language priors and environment-grounded execution.

2. Core formulation and round-trip mechanism

BAGEL’s formalism centers on two objects: a trajectory and an instruction. A trajectory is

$$\tau = (o_1, a_1, o_2, a_2, \ldots, o_T, a_T, o_{T+1}),$$

where $o_t$ are string observations and $a_t$ are action strings, while an instruction is a natural-language string $x$ or $g$ (Murty et al., 2024). The method uses five components: an exploration policy $x$0, an LM labeler $x$1 with distribution $x$2, a zero-shot LM agent $x$3 with induced distribution $x$4, a binary demonstration filter $x$5, and, in one variant, an instruction generator $x$6 (Murty et al., 2024).

The iterative procedure alternates between labeling and instruction following:

$x$7

A candidate pair is accepted if the consistency criterion satisfies

$x$8

with $x$9, in which case $\tau$0 is added to the demonstration set $\tau$1 (Murty et al., 2024).

The paper also presents a distributional interpretation. Let $\tau$2 and $\tau$3 denote the marginal trajectory and instruction distributions after $\tau$4 iterations. Then

$\tau$5

$\tau$6

and hence

$\tau$7

The stated effect is a “soft conversion” of the trajectory distribution toward trajectories that are both describable in language and executable by the agent (Murty et al., 2024). This suggests that BAGEL is not merely filtering demonstrations after the fact; it is using language as an iterative projection operator on behavior space.

3. Trajectory-first and instruction-first bootstrapping

BAGEL supports two initialization modes. In the trajectory-first variant, initialization samples $\tau$8 from an unconditioned exploration policy implemented via a ReAct-style prompt (Murty et al., 2024). In the instruction-first variant, initialization samples $\tau$9 from an instruction generator based on the initial observation and action space (Murty et al., 2024).

In trajectory-first BAGEL, the system begins from semantically noisy behavior. The labeler produces a candidate instruction describing the trajectory, and the agent attempts to follow that instruction, yielding a refined trajectory. Repeated label–follow rounds progressively improve both instruction quality and behavioral coherence until the filter accepts a pair (Murty et al., 2024). In instruction-first BAGEL, the system instead begins from synthetic instructions that may be infeasible or misaligned with the environment; execution and relabeling continue until a consistent pair is found (Murty et al., 2024).

The paper’s MiniWob++ choose-date example illustrates this refinement process in concrete terms. Initial exploration may produce clicks on the datepicker and month toggles without a coherent goal. The first labeling step yields “Change month from December to October,” and the first agent round changes the month but does not submit. A second labeling step refines the goal to “Change month to October 7th and submit,” after which the agent navigates the months, selects October 7th, submits, and the filter accepts the pair (Murty et al., 2024). The example is significant because it shows the accepted demonstration emerging from multiple imperfect components rather than from any single accurate model invocation.

The two variants exhibit different empirical profiles. On ToolQA, instruction-first performs best among the BAGEL variants, while on MiniWob++ trajectory-first performs best (Murty et al., 2024). The paper’s practical guidance attributes this to task structure: trajectory-first is often better when plausible instructions are hard to infer from the initial observation alone, as in MiniWob++ landing pages, whereas instruction-first can excel when the LM can propose realistic instructions from the initial state and action space, as in some ToolQA domains (Murty et al., 2024).

4. Demonstration use at test time

BAGEL does not fine-tune the zero-shot LM agent. Instead, it performs test-time adaptation through retrieval-augmented in-context learning over the mined demonstrations (Murty et al., 2024). Instruction embeddings $\tau_{<t}$0 are computed using a T5-XXL model with mean-pooled word embeddings and cosine similarity (Murty et al., 2024). For a test instruction $\tau_{<t}$1, the system retrieves the top-$\tau_{<t}$2 demonstrations, with $\tau_{<t}$3, by cosine similarity between $\tau_{<t}$4 and the stored demonstration instructions (Murty et al., 2024). These demonstrations are prepended as exemplars in the agent prompt, again in a ReAct-style format, and the agent is rolled out in the environment (Murty et al., 2024).

The low-level execution layer is also part of the system design. If an action string causes an exception, such as invalid tool syntax or typing on a non-typeable DOM element, BAGEL re-prompts the agent with the exception message and re-samples up to $\tau_{<t}$5 times (Murty et al., 2024). The paper reports that this simple recovery scheme reduces execution failures at test time, especially when in-context demonstrations are supplied (Murty et al., 2024).

This retrieval stage matters conceptually because it clarifies what BAGEL is optimizing for. The mined demonstrations are not merely a by-product for offline evaluation; they are the mechanism by which a zero-shot LM agent is adapted to environment dynamics at inference time. A plausible implication is that BAGEL’s success depends not only on demonstration correctness, but also on whether the demonstration set captures reusable interaction motifs that can be retrieved by instruction similarity.

5. Experimental setting and quantitative results

The paper evaluates BAGEL on MiniWob++ and ToolQA (Murty et al., 2024). MiniWob++ comprises web UI tasks with DOM interactions via Selenium, using a natural-language action space such as Type, Click, Clear, and Move Mouse that a low-level controller maps to driver commands. Evaluation follows Shaw et al., mapping raw reward in $\tau_{<t}$6 to $\tau_{<t}$7, averaged over 50 random seeds, and focuses on the hardest 10 tasks with baseline average reward below 0.95 (Murty et al., 2024). ToolQA comprises tool-augmented question answering across 8 domains—Agenda, AirBnB, Coffee, DBLP, Flights, GSM8K, Scirex, and Yelp—with actions expressed in language and parsed into tool calls, and performance measured by F1 of the final answer string against ground truth (Murty et al., 2024).

Implementation details are fixed across experiments. The base LM is instruction-tuned PaLM-2; maximum episode length is $\tau_{<t}$8; the number of round-trip iterations is $\tau_{<t}$9; and execution-failure re-sampling uses $t$0 (Murty et al., 2024). Data collection uses 60 seeds for MiniWob++ and 200 seeds for ToolQA, with either trajectories or synthetic instructions depending on the initialization mode (Murty et al., 2024). Retrieval uses T5-XXL embeddings, cosine similarity, and top-3 demonstrations (Murty et al., 2024).

The principal numerical findings are summarized below.

Setting	Result
MiniWob++	BAGEL improves average reward by over 13 percentage points compared to zero-shot
ToolQA	+BAGEL yields over 2 percentage points improvement in average F1 compared to zero-shot across the 8 domains
Execution failures	Up to 13× fewer failures on MiniWob++ choose-date, from 1.3 to 0.1 on average

On MiniWob++, gains are especially strong on choose-date, tic-tac-toe, and use-autocomplete, with about 20% absolute improvement in some cases (Murty et al., 2024). On execution failures, the choose-date task drops from 1.3 to 0.1 average failures, book-flight drops from 3.0 to 0.6, and overall ToolQA drops from 3.0 to 1.9 (Murty et al., 2024).

The ablation results isolate the contribution of iteration. On MiniWob++, averages are 46.8 for zero-shot, 52.0 for instruction-first no-itrs, 56.0 for full instruction-first, 53.0 for trajectory-first no-itrs, and 61.0 for full trajectory-first (Murty et al., 2024). On ToolQA, averages are 40.9 for zero-shot, 43.3 for full instruction-first, and 42.2 for full trajectory-first (Murty et al., 2024). These numbers support the claim that multiple round-trips consistently improve performance relative to non-iterative relabeling (Murty et al., 2024).

The paper also tests the role of correct instruction–trajectory alignment. On ToolQA, randomly chosen demonstrations and shuffled instruction–trajectory pairs still outperform zero-shot slightly, suggesting some benefit from seeing valid domain trajectories and instruction modes, but aligned demonstrations remain best: 42.2 for BAGEL, 41.4 for Shuffled, and 40.9 for Zero-shot (Murty et al., 2024). This suggests that BAGEL’s gains are partly structural and partly semantic: trajectory priors help, but proper alignment helps more.

6. Analysis, limitations, and relation to prior paradigms

The paper identifies several failure modes. Long-horizon planning remains difficult: on MiniWob++ book-flight, zero-shot performance is 5%, and BAGEL raises it to 15%, which is an improvement but still leaves substantial room for hierarchical planning and subgoal decomposition (Murty et al., 2024). Diversity mismatch can arise when seed exploration clusters around a narrow subset of modes, such as demonstrations that all book December flights, creating a mismatch with a more uniform test distribution (Murty et al., 2024). Instruction mismatch appears in domains such as SciRex, where BAGEL tends to generate descriptive-answer questions with generic trajectories, whereas test tasks demand fine-grained retrieval of specific numbers; search-engine tasks requiring precise counting across pages are similarly under-covered (Murty et al., 2024). Finally, LM component misalignment persists because the labeler may hallucinate instructions the agent cannot execute, or the agent may produce trajectories the labeler cannot faithfully describe (Murty et al., 2024).

Correctness filtering is consequential. The paper reports that manually verifying and removing incorrect demonstrations yields a further 7% absolute improvement on the 10 MiniWob++ tasks (Murty et al., 2024). This indicates that the binary LM-based filter is useful but imperfect, and it motivates future work on stronger verifiers, environment-affordance cross-checking, and confidence calibration (Murty et al., 2024).

BAGEL is explicitly positioned relative to several prior paradigms. It uses ReAct-style prompting for both exploration and instruction following, but adds an LM labeler and a correctness filter to construct demonstrations without human supervision (Murty et al., 2024). It resembles Hindsight Experience Replay in that trajectories are relabeled with goals after the fact, but it operates in natural language, leverages full trajectories, and performs multiple label–follow iterations to reduce noise (Murty et al., 2024). It also intersects with synthetic data generation and self-training, yet differs in grounding its process in environment interaction and in using the resulting data for in-context adaptation rather than fine-tuning (Murty et al., 2024).

A common misconception would be to treat BAGEL as a purely synthetic-data pipeline. The method is better understood as an environment-grounded bootstrapping procedure: the demonstrations are synthetic in origin, but they are accepted only after interaction trajectories and instructions become mutually consistent under a verifier (Murty et al., 2024). Another misconception would be to view it as a replacement for better planning or stronger verification. The reported limitations indicate the opposite: BAGEL reduces demonstration dependence, but does not eliminate the need for improved planning, diversity control, or correctness checking.

7. Significance and prospective extensions

BAGEL’s stated novelty is bootstrapping with language-guided exploration through iterative LM-to-LM round-trips, thereby aligning trajectories and instructions without human supervision (Murty et al., 2024). Its practical value is greatest in environments where actions and observations are textual, where zero-shot agents lack exposure to environment dynamics, and where human demonstrations are unavailable or expensive (Murty et al., 2024). MiniWob++ and ToolQA exemplify this regime because both expose sufficiently rich textual cues for relabeling and in-context reuse (Murty et al., 2024).

The paper outlines several extensions. These include fine-tuning on BAGEL demonstrations, hierarchical or subgoal policies, diversity-enhanced exploration, confidence-aware filtering, and leveraging limited prior knowledge about the test-time instruction distribution (Murty et al., 2024). These are prospective directions rather than established results, but they make clear that BAGEL is intended as a general bootstrapping principle rather than a fixed pipeline.

Within research on LM agents, BAGEL is significant because it shifts the bottleneck from human demonstration collection to self-generated, filtered, and retrievable exemplars (Murty et al., 2024). The experimental evidence indicates that this shift is operationally meaningful: on the studied benchmarks, a zero-shot agent can be adapted at test time with synthetic yet environment-grounded demonstrations, improving reward and F1 while sharply reducing execution failures (Murty et al., 2024). This suggests that, for a substantial class of digital environments, language can serve not only as a control interface but also as an instrument for reshaping exploration into reusable competence.