Butter-Bench: Benchmarking Smooth Transitions

• Butter-Bench is a comprehensive framework that measures 'butter-like' smooth transitions in both AI-driven robotics and generative design using innovative benchmark paradigms.
• It employs quantitative metrics, including task completion rates in LLM-controlled robots and stiffness continuity scores in multi-lattice additive manufacturing, to assess performance.
• The evaluations reveal critical gaps in embodied LLM reasoning—especially in spatial planning and social inference—compared to human baselines, guiding future research directions.

Butter-Bench denotes a set of methodologies, benchmarks, and diagnostic paradigms for evaluating "smoothness," consistency, and practical intelligence in both physical systems and artificial intelligence systems, connected by the shared metaphor of "butter-like" transitions or reasoning abilities. In particular, it encompasses: (1) benchmarks for assessing practical intelligence in LLM-controlled robots (Sharrock et al., 23 Oct 2025), (2) a paradigm and metric suite for smooth mechanical and geometric transitions in multi-lattice design for additive manufacturing (Baldwin et al., 10 Jul 2024), and (3) related concepts of smooth transitions in fields where butter/butter-like transitions provide insight into performance under real-world constraints.

1. Benchmarking Practical Intelligence in LLM-Controlled Robots

The Butter-Bench framework provides the first real-world benchmark to isolate and evaluate the high-level reasoning and planning capabilities of LLM agents in physically embodied robotic systems (Sharrock et al., 23 Oct 2025). Unlike prior simulation-based or solely analytical benchmarks, Butter-Bench is deployed on a TurtleBot 4 platform operating in real office/home environments. It abstracts away the low-level Vision-Language-Action (VLA) execution by limiting physical affordances to high-level tool invocations (e.g., navigation, visual query, messaging), focusing evaluation solely on the LLM's practical intelligence.

Core System Architecture

• Hierarchical agent architecture: The LLM is evaluated as the top-level orchestrator responsible for reasoning, goal decomposition, spatial planning, and social interaction; the VLA layer is present but held fixed and minimalized.
• Butter-Bench focus: All benchmarks are run with LLM-only control, abstracting away precise manipulation or low-level control confounds.

Task Suite for Practical Intelligence

The benchmark is thematically centered on the "Pass the butter" challenge, decomposed into:

Core Task	Evaluated Ability
Search for Package	Navigation & mapping
Infer Butter Bag	Visual-textual reasoning
Notice Absence	Context/social awareness
Wait for Confirmed Pick Up	Social patience/interaction
Multi-Step Path Planning	Decomposition, spatial reasoning
E2E Pass the Butter	Integrated full-cycle performance

Each task is run five times per agent (LLM or human teleoperator), with completion scored in a binary fashion.

Evaluation Protocol

• Task Completion Rate: Main quantitative metric, with subtasks and aggregate averages.
• Human baseline: Human teleoperators, given the same interface, consistently outperform LLM agents.
• Red teaming: Agents tested for robustness to ambiguous/failure scenarios (e.g., simulated low battery, social manipulation).

Quantitative Results

Model	E2E	Search	Infer	Absence	Wait	Plan	Avg.
Gemini 2.5 Pro	40%	100%	40%	0%	20%	40%	40%
Claude Opus 4.1	40%	100%	0%	0%	20%	60%	37%
GPT-5	40%	60%	60%	0%	20%	0%	30%
Human	100%	100%	100%	100%	67%	100%	95%

Key observations include:

• State-of-the-art LLMs achieve mean completion rates of 27–40%, with best models never exceeding 40%.
• Humans achieve near-ceiling (95%) performance.
• Most pronounced deficits: multi-step spatial planning and nuanced social reasoning (e.g., models fail to notice user absence or require confirmation).
• LLMs fine-tuned for "embodied reasoning" do not outperform foundation models, indicating a gap in current fine-tuning methodologies.

Diagnostic and Failure Analysis

• Spatial failures: Straight-line navigation attempts ignoring obstacles or failure to multi-step decompose.
• Social/contextual failures: Failure to seek user confirmation or to recognize user absence.
• Visual reasoning: Variable; some agents (e.g., GPT-5) reliably interpret visual cues, others do not.
• Stress testing reveals susceptibility to unsafe or illogical action proposals under ambiguous or adversarial prompts.

Implications

Butter-Bench establishes that "practical intelligence"—encompassing adaptive problem-solving in physically and socially ambiguous environments—remains an unsolved challenge for LLM agents. Embodied fine-tuning, as presently operationalized, does not close the gap with human baselines. The benchmark identifies a spectrum of persistent agent deficiencies: spatial decomposition, context/planning, and social inference.

2. The Butter-Bench Paradigm in Generative Multi-Lattice Design

In generative engineering design, the Butter-Bench paradigm refers to the assessment and design of transition regions with smooth geometric and mechanical properties, specifically in the context of additive manufacturing (AM) of multi-lattice structures (Baldwin et al., 10 Jul 2024).

Core Concepts

• Butter-Bench as a metric suite: Evaluates smoothness in geometry (visual connectivity) and mechanical property transitions (e.g., stiffness) in lattice structures.
• Key metrics:
  • Geometric smoothness ($C_s$): Based on RMSE between gradient arrays of adjacent unit cell geometries.
  • Stiffness continuity ($C_K$): Based on RMSE between normalized stiffness tensors of adjacent cells.

Methodology

• Hybrid geometry/property VAEs—encoders that integrate both geometric input (binary 2D images of lattice cells) and corresponding stiffness tensors—produce continuous latent spaces conducive to both smooth geometric and mechanical transitions.
• Transition regions are constructed by interpolating between endpoint lattice types in the joint latent space, then decoding each interpolated point back to spatial geometry.

Key Findings

# Stdevs Between Endpoints	Geometry-only VAE $C_K$ (%)	Hybrid VAE $C_K$ (%)
1	99.58	99.92
~6	94.25	96.40

• The hybrid VAE exhibits improved stiffness continuity, with $C_K$ degrading less rapidly as latent space distance increases and plateauing earlier—indicating increased robustness under challenging transitions.
• Geometric smoothness ($C_s$) trends are similar for both architectures; stiffness continuity ($C_K$) is the discriminator.
• Metrics and evaluation methodology (especially $C_K$) are intended as canonical components of future "Butter-Bench" benchmarks for generative design workflows.
• Latent space clustering and non-convexities can still create transition artifacts, underlining an ongoing open issue.

Significance

The Butter-Bench paradigm in this context provides strict performance targets—surpassing mere visual plausibility—for candidate generative models in AM and topology optimization pipelines. Property-augmented latent spaces are foundational in yielding transition regions with both geometric and mechanical "butter-smoothness." The approach has direct relevance for reliability and manufacturability in AM workflows.

3. Criteria, Metrics, and Formulas Underpinning Butter-Bench Assessment

Butter-Bench benchmarks draw on precise, interpretable metrics:

• Geometric Smoothness $(C_s)$:

$$C_s = \left(1 - \overline{RMSE}_\text{geom}\right) \times 100\%$$

where $\overline{RMSE}$ is the mean normalized RMSE between gradient images of adjacent cells.

• Stiffness Continuity $(C_K)$:

$$C_K = \left(1 - \overline{RMSE}_\text{stiff}\right) \times 100\%$$

where RMSE is computed over flattened, normalized stiffness tensors for each adjacent pair.

The metrics are diagnostic: high $C_s$ indicates visually plausible transitions; high $C_K$ indicates mechanical property interpolation without abrupt discontinuities. In robotics, analogous completion rates and finely categorized failure types enable clear benchmarking of agent competencies and deficiencies.

4. Comparative Analysis and Positioning

Butter-Bench occupies a diagnostic space distinct from simulation-only benchmarks, standard vision-language reasoning evaluations, or generative design protocols that do not explicitly score for "smoothness" of transition.

• In robotics: Unlike simulation-based intelligence tests or analytical IQ metrics, Butter-Bench is anchored in real-world deployment, scoring genuine environmental adaptation and context inference. Performance here is decisively lower for LLMs than analytical IQ tests would suggest, revealing the practical intelligence gap in robotic AI (Sharrock et al., 23 Oct 2025).
• In generative engineering: The Butter-Bench paradigm imposes mechanical property continuity as a necessary (not optional) axis of performance, operationalizing "butter-like" transitions which are otherwise underexplored in purely geometric generative frameworks (Baldwin et al., 10 Jul 2024).

5. Limitations and Future Research Trajectories

In both domains, Butter-Bench exposes unresolved challenges:

• For LLM-robotics: Deficits in social/contextual reasoning, spatial planning, and robustness to real-world ambiguity persist despite the latest LLM and embodied reasoning fine-tuning regimens.
• For generative design: Latent space organization remains imperfect; VAEs (including hybrids) can exhibit cluster-induced discontinuities, precluding universally smooth transitions for distant lattice types.

Future Butter-Bench-derived benchmarks are likely to incorporate richer datasets (e.g., multi-modal, map-based, or dynamic scenarios), more granular metrics (capturing not just pass/fail or $C_K$, but incremental failure states), and experiments with new foundational model architectures or more expressive latent spaces.

6. Broader Impacts and Implementation Guidance

Butter-Bench establishes best practices for both evaluators and model developers:

• Model assessment: Models intended for real-world deployment—in robotics or design—should be stress-tested using Butter-Bench protocols to avoid unrecognized brittleness or counterintuitive performance dips.
• System design: Embedding domain-relevant properties (mechanical, contextual, or otherwise) into core representation or decision modules is favored over strictly geometry- or text-based approaches.
• Safety: Butter-Bench reveals "deployment risk": agents may fail gracefully in simulation but behave non-robustly in operational settings.

In summary, Butter-Bench is a diagnostic paradigm, benchmark suite, and metric-driven evaluation framework for assessing smooth, consistent transitions or solutions in both physical design and embodied intelligence domains. Its metrics and methodology set actionable standards for the next generation of AM generative models and LLM-centered embodied AI.