Papers
Topics
Authors
Recent
Search
2000 character limit reached

StructureClaw: Traceable LLM Agents and an Executable Benchmark for Structural Engineering Workflows

Published 16 Jul 2026 in cs.SE, cs.AI, and cs.MA | (2607.14896v1)

Abstract: Addressing a structural-engineering request requires more than a single answer; it requires a chain of interdependent artifacts: interpreted requirements, a computable model, validation records, solver outputs, code-check records, and a final report. Evaluations centered on question answering or script generation rarely verify this complete evidence chain and may therefore reward fluent outputs even when the underlying engineering workflow is incomplete, internally inconsistent, or non-executable. To address this limitation, we present StructureClaw, an artifact-centered workbench in which LLM agents operate through governed engineering skills, typed tools, shared artifact state, and local analysis backends. We also introduce StructureClaw-Bench, an executable benchmark of 150 controlled scenarios spanning standard workflow execution, interactive robustness, and multimodal structural-model reconstruction. A scenario succeeds only when all required artifact- and execution-level assertions pass in a single run. Across ten agent-model configurations, each evaluated on the same 50 standard cases, the average Success Rate rises from 56.8% with the generic-skill baseline to 88.6% with the full automatic workflow. The interactive and multimodal evaluations identify two prominent remaining challenges: safe handling of invalid numerical inputs and fixture-consistent reconstruction of structural models. These findings show that artifact-centered evaluation can expose workflow-level failures that are difficult to identify from final responses alone, providing a more rigorous basis for evaluating and improving structural-engineering agents. The code and benchmark are available at https://github.com/structureclaw/structureclaw.

Summary

  • The paper presents an evidence-chain-centric LLM agent architecture that systematically verifies intermediate engineering artifacts.
  • It implements a ReAct-style execution loop and explicit tool separation, yielding up to 88.6% success rate improvements over generic baselines.
  • The benchmark exposes workflow failures and artifact inconsistencies across multimodal and interactive structural engineering tasks.

StructureClaw: Traceable LLM Agents and an Executable Benchmark for Structural Engineering Workflows

Motivation and Problem Formulation

LLM-driven agentic systems in architecture, engineering, and construction (AEC) have predominantly focused on single-response inference or partial automation, resulting in limited verification of the computational evidence required by engineering tasks. Structural-engineering workflows yield interconnected artifacts: interpreted requirements, computable models, validation/solver outputs, code-check records, and final technical reports. Traditional evaluation has generally prioritized the end-text output, allowing consistency failures, model invalidity, or backend errors to remain undetected. This work introduces StructureClaw, addressing these deficits by organizing workflow execution and evaluation around the engineering evidence chain, making all intermediate steps inspectable and verifiable rather than permitting plausible–but–non-executable outputs.

Figure 1

Figure 1: From a structural-engineering request to reviewable evidence, StructureClaw preserves the skills, tool executions, and artifacts supporting the reported result.

System Design and Architectural Principles

StructureClaw is structured around explicit separation of skills, tools, and artifact state. A domain skill encodes triggers, guidance, and artifact contracts; typed tools perform atomic operations, and providers bind tool contracts to specific analysis backends. The entire workflow is orchestrated through a ReAct-style execution loop, augmenting tool-augmented LLMs with persistent, explicitly type-annotated artifact records at every step. Key design features include governed capability routing, schema-guided artifact construction, backward-compatible validation, backend-aware execution, and trace-linked report generation.

Figure 2

Figure 2: StructureClaw's multimodal context is processed by an agent kernel integrating capability context, tools, and artifact state, with the local sandbox managing artifact-linked model construction, validation, analysis, and reporting.

The artifact protocol operationalizes project metadata, model schema, topology, element properties, supports, and loading semantics—as well as provider invocations and validator outputs—allowing fine-grained post hoc analysis of localized workflow failures. Provider binding ensures that every backend invocation (e.g., OpenSees via OpenSeesPy) is explicit and traceable, with unsupported requests resulting in explicit workflow states rather than silent failures. This approach distinguishes the StructureClaw architecture from prior text-generation or script-centric engineering AI systems.

StructureClaw-Bench: Executable Evidence-Centric Benchmark

Existing benchmarks in the AEC domain (e.g., OSWorld, EngDesign, AECBench) do not require agents to maintain a full evidence chain or to resolve scenarios centered on the fidelity and executability of intermediate engineering artifacts. StructureClaw-Bench, by contrast, covers standard workflows, interactive robustness (handling missing/conflicting/invalid input and edge recovery), and multimodal structure reconstruction from images and DXF files. Each scenario is only successful if all prescribed artifact- and execution-level assertions are satisfied, thus exposing failures that are otherwise masked by fluent outputs.

Figure 3

Figure 3: StructureClaw-Bench's coverage by structure-type, interactive robustness category distribution, and multimodal (image/DXF) and language split.

The benchmark maintains stringent validation: scenario sets are stratified by family, locale, and format, with binary pass/fail metrics computed over the strict conjunction of all required assertions.

Experimental Evaluation and Results

Evaluations were conducted across 10 agent-model configurations for standard and interactive tasks and six for multimodal tasks, yielding 1,800 scenario executions. Metrics include scenario-level Success Rate, assertion-level diagnostic rates, average tool invocations, and wall-clock runtime. All configurations revealed a substantial gain from skill- and protocol-governed execution versus generic structural skill baseline, with mean Success Rate improvements from 56.8% (generic) to 88.6% (automatic).

Figure 4

Figure 4: Success Rate in standard workflow settings: teal (automatic execution) always exceeds purple (generic-only), with labels indicating absolute gain.

Kimi-K2.6 achieved 100% Success Rate in automatic mode; DeepSeek-V4-Flash and GLM-5.2 reached 96%. Notably, mere artifact presence is insufficient: while generic-only cases contain model artifacts in 92% of outcomes, only 70.5% are reference-consistent, compared to 90.8% under governed workflows—a strong claim for protocol-driven traceability in reducing undetected logical errors.

Analysis of the interactive-robustness subset shows 91.0% Success Rate, yet only 70.9% for the invalid-value subcategory, exposing safe state transitions (input validation and abstention) as a clear failure point for current LLM-driven agents.

Figure 5

Figure 5: Interactive robustness Success Rate distribution, with diamonds marking invalid-value subset outcomes.

For multimodal reconstruction, the highest configuration (GLM-5.2/GLM-4.6V) reached 94.0% Success Rate on DXF/image cases, but model matching (consistency with reference schema) trailed recognition and routing, highlighting visual/perceptual reconstruction as a rate-limiting step.

Figure 6

Figure 6: Multimodal model reconstruction Success Rates, split by input format for different configurations.

Diagnostic Analysis and Failure Localization

Descriptive profiling of system behavior Figure 7 confirms that aggregate performance alone can obscure critical family-level or workflow-specific regressions. For example, continuous-beam cases showed a paradoxical 20-point regression in automatic mode for half of evaluated models, underscoring the necessity for scrutiny beyond global averages.

Figure 7

Figure 7: Radial diagnostic display of Success Rates, showing divergent strengths and weaknesses across families.

Assertion grouping isolates two principal classes of agentic failure:

  1. Constraint and Input Validation: Weaknesses in pre-execution constraint checking result in unsafe or premature invocation of analysis routines, especially in invalid-value scenarios.
  2. Artifact Consistency and Reconstruction: Despite high rates of structural-type recognition, successful construction of reference-consistent executable models from multimodal inputs remains below 85% in aggregate diagnostic metrics.

Theoretical and Practical Impact

Artifact-centered protocol design enables auditing and reproducible comparison of agentic decision pathways, providing a formal substrate for analyzing where and why LLM agents fail within engineering workflows. This systematic approach enables not only more reliable benchmarking, but also targeted development of workflow-specific controls (e.g., deterministic value validators, intermediate artifact serialization).

Practically, StructureClaw and StructureClaw-Bench expose the limitations of “text-only” or prompt-centered evaluation protocols for professional engineering workflows. The correlation between artifact-guided execution and evidence-chain consistency has non-trivial implications for safety- and reliability-critical applications of LLM-driven automation within the built environment. The findings call for explicit handling of failure cases, release criteria based on family-level as well as aggregate performance, and controlled ablation studies for workflow attribution in future system releases.

Future Directions

The present results motivate development of:

  • Deterministic safety-check layers prior to model-handling or execution steps.
  • Enhanced trace annotation for causal attribution of errors in artifact construction, perception, schema mapping, and code-check integration.
  • Scaling to larger engineering projects, additional solver backends, and engineer-reviewed equivalence classes for flexible artifact comparison.
  • Routine use of paired scenario-level traces and diagnostics as release gating criteria, replacing or augmenting mere leaderboard-style metrics.

Conclusion

StructureClaw establishes the feasibility and necessity of evidence-chain-centric LLM agent design for structural engineering, formalizing execution, artifact tracking, validation, and report semantics under explicit, formally testable contracts. StructureClaw-Bench delivers an executable, auditable diagnostic testbed targeting the critical limitations of prior evaluation protocols. The system’s strong numerical gains under automatic, governed workflows—88.6% Success Rate vs. 56.8% for the generic baseline—are non-uniform and reveal both structural improvements and regressions, validating the need for fine-grained diagnostics. LLM agent development for engineering domains should prioritize not only output correctness but also workflow transparency, traceability, and explicit failure localization as benchmarks for practical reliability and future research.

Reference: "StructureClaw: Traceable LLM Agents and an Executable Benchmark for Structural Engineering Workflows" (2607.14896)

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Explain it Like I'm 14

What is this paper about?

This paper introduces StructureClaw, a “workbench” that helps AI assistants do structural engineering tasks (like analyzing beams, frames, and trusses) in a safe and trackable way. Instead of just giving a nice-looking answer, the AI must show its work step by step—like a math class where you have to write down every step, not just the final number. The paper also presents StructureClaw-Bench, a set of 150 test cases that checks whether the AI’s whole process—from understanding the request to building a model, running analysis, and writing a report—actually works and can be executed.

What questions were the researchers asking?

They focused on simple, practical questions:

  • Can an AI engineer not only give answers but also show the full “evidence chain” behind those answers?
  • Can we build a test that only counts a task as “successful” if every step (modeling, validation, analysis, reporting) is correct and runnable?
  • Do structured tools and rules help the AI do better than just using a general approach?
  • Where do AI agents still commonly fail—especially when inputs are wrong or when understanding drawings?

How did they approach the problem?

Think of an engineering task like following a recipe. You don’t just serve the cake; you also have the shopping list, measurements, oven settings, and steps written down. The team built two main things to make the AI follow and show the whole “recipe.”

StructureClaw (the system)

  • It’s a controlled environment where an AI uses:
    • Skills (what kinds of structures/tasks it knows)
    • Tools (for reading inputs, building models, checking rules, and analyzing structures)
    • An artifact state (saved “pieces of evidence,” like the model file, solver results, and checks)
  • The AI works in a loop: think → act (use a tool) → store results → think again. This style is called ReAct.
  • It uses a local “engineering sandbox” to actually run structural analysis with real software (like OpenSees via OpenSeesPy).
  • Every piece of the process is saved, so you can see exactly how the final answer was produced.

In everyday terms: it’s like a science fair project with a lab notebook that records every test, every result, and every change, not just the final poster.

StructureClaw-Bench (the test)

  • A set of 150 “scenarios” that the AI must solve.
  • The scenarios cover:
    • Standard tasks (well-described, text-based requests)
    • Interactive robustness (cases with missing info, wrong numbers, confusing units, or conflicts—does the AI ask for clarification or safely refuse to run?)
    • Multimodal reconstruction (turning pictures or DXF drawings into a working analysis model)
  • A scenario only counts as a success if all required checks pass in one go (no retries). That means the model must exist, be consistent with the request, run on a real solver, and the final report must match the actual results.

What did they find?

Here are the key results and why they matter:

  • Using structured workflows clearly helps:
    • In 50 standard cases, the average Success Rate went from 56.8% (generic approach) to 88.6% (full automatic workflow with skills, tools, and checks).
    • One setup achieved 100% on the standard cases.
    • Why it matters: This shows that making the AI follow a clear, enforceable process—rather than just write code or text—greatly improves reliability.
  • The “evidence chain” approach exposes problems that a nice final answer might hide:
    • Because every step is saved, it’s easier to see where things go wrong (like model building vs. solver setup vs. reporting).
    • Why it matters: Engineers and developers can fix the exact weak point instead of guessing.
  • Two big remaining challenges:
    • Handling bad numbers safely (like impossible values in inputs): the AI still sometimes struggles to catch these before running analysis.
    • Rebuilding models from images/drawings consistently (e.g., matching the right geometry and loads): recognition is often okay, but creating a fully correct, runnable model is harder.
    • Why it matters: These are common real-world hurdles—dirty data and messy drawings—so tackling them will make AI more practical on real projects.
  • Interactive robustness is decent but not perfect:
    • When things are unclear or wrong, many agents ask for clarification or safely avoid running analysis, with an overall average around 91% on these tests.
    • But invalid-number cases were notably weaker (about 71% on the observed subset).
    • Why it matters: In real engineering, it’s better to pause than to compute with bad info. The AI needs stricter checks before it touches the solver.
  • Multimodal (images/DXF) results vary more:
    • Some model combinations did very well (up to 94%); others were much lower (around 60%).
    • Recognition and routing (figuring out structure type and what to do) are strong; fully matching the reconstructed model to expected details is harder.
    • Why it matters: Reading drawings and turning them into accurate, executable models is a major step toward helpful AI in engineering.
  • Not all improvements are uniform:
    • For some structure types (like continuous beams), the “automatic workflow” sometimes did worse than the generic approach in a few setups.
    • Why it matters: Even a better average can hide specific weaknesses—these need case-by-case checks before release.

What does this mean for the future?

  • Safer, more transparent AI tools: By forcing the AI to show its work, engineers can trust and review the process, not just the final number.
  • Better training and debugging: The benchmark shows exactly where and why the AI fails, so developers can fix those parts without breaking others.
  • Practical engineering help, not full automation: This is a step toward reliable AI assistants that help engineers, not a replacement for professional judgment or code compliance checks.
  • Clear next goals: Add stricter value checks before analysis, improve how drawings are turned into precise models, test more backends, and use repeated runs and human-reviewed “equivalence” rules for fairer scoring.

In short, the paper shows that making AI follow and record a full engineering workflow—as a chain of evidence—dramatically improves reliability and makes problems easier to find and fix. It’s a practical path toward trustworthy AI helpers in structural engineering.

View Paper Prompt  View All Prompts 

Knowledge Gaps

Knowledge gaps, limitations, and open questions

The paper leaves the following issues unresolved; each item is phrased to guide concrete follow-up research:

  • Component attribution missing: no ablations isolating the effects of skills, validation rules, backend-aware routing, and provider choice on Success Rate; design matched experiments to quantify each component’s contribution.
  • Run-to-run variability unmeasured: single-attempt protocol with no retries obscures stochastic variance; add repeated trials with controlled sampling temperature and seeds to estimate uncertainty and stability.
  • Comparator incompleteness in multimodal reconstruction: model-matching checks only element counts, coordinate spans, and approximate loads, not topology/connectivity/support equivalence; develop graph-based comparators (e.g., structural connectivity isomorphism with tolerance bands).
  • Ground-truth fixture limitations: fixtures may omit valid engineering-equivalent alternatives or accept latent defects; incorporate engineer-reviewed equivalence classes and tolerance-aware assertions for geometry, loads, and boundary conditions.
  • Vision/agent attribution unclear: multimodal results conflate perception and reasoning; perform controlled pairings (fixed LLM, varying vision and vice versa) and cross-over studies to apportion errors.
  • Narrow structural coverage: benchmark centers on frames, beams, trusses, portal/continuous beams, and columns; extend to slabs, shells/plates, shear walls, masonry, timber, composite members, and spatially irregular 3D systems.
  • Analysis scope constrained: limited evidence of nonlinear, second-order (P–Δ), buckling, dynamic/modal/transient, staged construction, or inelastic analyses; add tools and scenarios exercising these regimes with ground-truth comparators.
  • Code-check breadth and validation: “supported code checks” not comprehensively specified or evaluated across standards (e.g., ACI/Eurocode/GB); integrate normative rule sets and assert code-compliance outcomes against reference cases.
  • Backend diversity under-tested: evaluation largely on OpenSeesPy; execute matched scenarios on multiple providers (e.g., PKPM, YJK, commercial solvers) to assess cross-solver consistency and portability.
  • Pre-execution numeric safety: invalid-value handling is a noted weakness; implement deterministic invariant checks (e.g., nonnegative stiffness, feasible dimensions, load magnitudes/ranges) and unit-consistency validators prior to model mutation or solver calls.
  • Units and locale robustness: unit inference and conversion from text, DXF, and images remain error-prone; build a unit-normalization subsystem with explicit unit tags, conversion provenance, and conflict resolution.
  • Report faithfulness verification: current assertions require presence of execution records but do not quantitatively assess report-to-artifact grounding; add span-level factuality checks linking reported numbers to artifacts and solver outputs.
  • Failure localization within traces: marginal diagnostics do not identify the exact failing stage in a single run; add instrumented step-level status codes and causal tagging to attribute failures to routing, extraction, modeling, validation, execution, or reporting.
  • Regression control across structures: automatic workflow regresses on continuous-beam cases; establish per-structure family regression tests and release gates to prevent aggregate improvements masking family-specific degradations.
  • Scalability and performance limits: no analysis of runtime, memory, or solver stability vs. model size/complexity; benchmark large multi-story 3D frames, dense load combinations, and long load histories to chart scaling behavior.
  • Long-horizon, multi-session workflows: persistence/versioning of artifacts across sessions and evolving requirements not evaluated; design memory and artifact lineage mechanisms and test multi-stage projects.
  • Human-in-the-loop effects: optional review paths are not exercised; study how targeted human clarifications/corrections alter success rates, effort, and error profiles.
  • Security and robustness to adversarial inputs: resilience to prompt/tool injection, malicious DXF/image payloads, and solver misuse not tested; introduce red-team scenarios and policy-constrained tool execution.
  • Provenance and tamper-evidence: traces are recorded but not protected against alteration; add immutable logs (e.g., append-only, hashing, signatures) and provenance metadata for auditability.
  • Interoperability with industry schemas: the structural-model protocol’s mapping to IFC/BIM and round-tripping with CAD/BIM tools is untested; implement import/export adapters and fidelity checks.
  • Language coverage beyond zh/en: only Chinese and English prompts included; evaluate additional languages and domain-specific terminology variance to test multilingual robustness.
  • Graded evaluation metrics: Success Rate is all-or-nothing; introduce graded metrics (e.g., graph edit distance, geometric/load deviation) to capture “near-miss” quality and guide incremental improvements.
  • Hyperparameter sensitivity: effects of decoding parameters, tool-selection thresholds, and validation strictness not studied; perform sensitivity analyses and establish robust defaults.
  • Real-world data representativeness: scenarios are controlled and curated; curate and anonymize messy industry projects (mixed files, incomplete specs) to test ecological validity.
  • Solver failure handling: beyond timeouts, strategies for non-convergence, ill-conditioning, and numerical instability are unspecified; add solver health checks, adaptive step controls, and fallback paths.
  • DXF/image parsing edge cases: scale inference, line-weight semantics, symbol disambiguation, and noisy drawings are not systematically evaluated; build perturbation suites to stress the perception-to-model pipeline.
  • Explainability/usability of diagnostics: how validation messages and artifact traces support engineer review is unassessed; conduct user studies to refine diagnostic granularity and presentation.
View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

Below are concrete, deployable use cases that can be built on the paper’s released workbench (StructureClaw) and executable benchmark (StructureClaw-Bench) with current open-source backends (e.g., OpenSeesPy) and the described skill/tool/provider architecture.

  • Structural QA “pre-check” assistant for analysis packages
    • Sector: AEC/software
    • What: Validate model completeness, unit consistency, references, load cases/combos, and backend availability before running solvers; auto-generate trace-grounded reports that cite the exact artifacts and tool calls used.
    • Tools/workflows: StructureClaw agent kernel + validation tools + OpenSeesPy provider; governed skills for frame/beam/truss analysis; export of validation diagnostics and solver traces to a shared artifact state.
    • Assumptions/dependencies: Not a code-compliance certifier; requires professional review; backends (e.g., OpenSees) must be installed and configured; scope limited to supported structure types and checks.
  • Traceable report generation for design reviews and internal audits
    • Sector: AEC/enterprise compliance
    • What: Produce standardized, reviewable reports where every conclusion is linked to model, validation, solver outputs, and code-check artifacts—improving auditability for ISO/EHS/QA processes.
    • Tools/workflows: Report-generation tool bound to the artifact state; governed safe non-execution when inputs are invalid or backends are unavailable.
    • Assumptions/dependencies: Organizational acceptance of artifact schemas; storage and retention policies for artifact traces; alignment with internal QA templates.
  • Interactive requirements clarification and safe non-execution guardrails
    • Sector: AEC/software/customer success
    • What: Deploy an agent that asks for missing data, rejects invalid numerical inputs, and abstains from analysis when the evidence chain is incomplete—instead of fabricating results.
    • Tools/workflows: ReAct-style loop with validation-gated execution; “unsupported termination” pathway when provider or scope constraints are unmet.
    • Assumptions/dependencies: Known weakness on “invalid numerical value” robustness; needs organization-specific guardrail policies and escalation routes to humans.
  • R&D and CI/CD benchmarking of engineering agents
    • Sector: Software/AI tooling
    • What: Use StructureClaw-Bench to regression-test agent updates across routing, modeling, backend execution, safety, and reporting; set release criteria on stage-level diagnostics (not just a single end-to-end score).
    • Tools/workflows: CI runners invoking the 150-scenario benchmark; assertion-level dashboards; Auto–Generic comparisons to detect workflow-level regressions.
    • Assumptions/dependencies: Benchmark scenarios are controlled and finite; fixtures represent operational targets, not all engineering-equivalent solutions; single-run protocol—augment with repeated trials for variance estimates.
  • Multimodal digitization of legacy drawings for preliminary analysis
    • Sector: AEC/digital twin/asset management
    • What: Convert images/DXF of simple frames/beams/trusses into analysis-ready artifacts (counts, spans, approximate loads) and run solvable models where possible.
    • Tools/workflows: Multimodal reconstruction skills + OpenSeesPy provider; artifact-matching comparator (counts/spans/loads) to verify reconstruction.
    • Assumptions/dependencies: Comparator currently checks partial equivalence (not full topology/support equivalence); human verification required; better on standard cases and clean drawings.
  • Education: labs and autograding for structural analysis
    • Sector: Education
    • What: Give students a “glass-box” assistant that shows every modeling and analysis step, enforces units and references, and fails safely when inputs are wrong; auto-grade by executable assertions.
    • Tools/workflows: Course-specific scenarios using StructureClaw-Bench; per-assignment assertions (model validity, solver completion, report grounding); bilingual prompts.
    • Assumptions/dependencies: Instructor-curated tasks and rubrics; compute sandbox/timeouts; academic policies on AI assistance.
  • Vendor evaluation for procurement of AI-assisted engineering tools
    • Sector: AEC/IT procurement
    • What: Compare third-party engineering copilots against the same executable benchmark and stage-level diagnostics to support purchasing decisions and SLAs.
    • Tools/workflows: Side-by-side runs on StructureClaw-Bench; reporting on routing/modeling/backend/report categories and failure localization.
    • Assumptions/dependencies: Harmonized access to proprietary backends (if used); vendors may tune to benchmark—complement with organization-specific cases.
  • Forensic documentation and change tracking
    • Sector: AEC/insurance/legal
    • What: Maintain a traceable evidence chain (inputs→model→validation→solver→report) for design changes or claims, enabling reproducibility and clearer causality.
    • Tools/workflows: Persistent artifact store with versioned records of tool calls and outputs; diffing of model artifacts across revisions.
    • Assumptions/dependencies: Governance for data integrity and chain-of-custody; secure storage and access control.
  • Safe internal chatbots for structural teams
    • Sector: Enterprise software/AEC
    • What: Replace free-form code/execution in chat with typed tools and governed skills that enforce runtime preconditions, reducing “hallucinated calculations.”
    • Tools/workflows: Structure-aware agent policies; typed tool schemas; explicit provider checks and non-execution paths recorded in artifacts.
    • Assumptions/dependencies: Tooling integration in corporate environments; user training; careful scoping to supported tasks.
  • Cross-platform adapter development
    • Sector: Software/CAE vendors
    • What: Build provider adapters to commercial engines (e.g., ETABS, SAP2000, PKPM, YJK) so the same governed skills and artifacts can execute on varied backends.
    • Tools/workflows: Provider interface contracts; conformance tests using StructureClaw-Bench subsets.
    • Assumptions/dependencies: Licensing/SDK access; adapter maintenance with vendor updates; consistent unit/section/material mappings.

Long-Term Applications

These use cases are plausible extensions that need improved robustness (invalid-value handling), richer equivalence criteria, broader code checks, certified accuracy, multi-backend maturity, or scaled integration.

  • Semi-automated code-compliance pre-screening for e-permitting
    • Sector: Government/policy/AEC
    • What: Triage submittals for completeness and traceability, flag missing assumptions/units, verify solver-execution records, and produce review-ready evidence chains.
    • Tools/products: Regulator-facing portal that ingests artifacts; rule packs for local codes; structured deficiency reports.
    • Assumptions/dependencies: Jurisdictional acceptance; standardized artifact schemas; expanded code-check coverage; legal frameworks for AI-assisted reviews.
  • Certifiable design-check pipelines integrated with BIM
    • Sector: AEC/enterprise platforms
    • What: End-to-end pipeline from BIM to governed modeling, solver runs, code checks, and sealed reports with auditable provenance.
    • Tools/products: BIM connectors (IFC/Revit), richer structural-model protocol, multi-backend orchestration, digital signatures on artifacts.
    • Assumptions/dependencies: Stronger equivalence criteria (topology, supports, member-by-member checks), human-in-the-loop sign-off, compliance with professional licensing.
  • Disaster assessment and retrofit triage from field imagery
    • Sector: Public safety/insurance/AEC
    • What: Rapid, on-site reconstruction of approximate structural models from images/drawings to prioritize inspections and propose preliminary retrofit options.
    • Tools/products: Field apps with guided capture; improved multimodal reconstruction; edge or cloud solvers; triage reports tied to evidence artifacts.
    • Assumptions/dependencies: Higher reliability in noisy visual conditions; robust unit inference; integration with emergency workflows; liability management.
  • Continuous compliance monitoring for digital twins
    • Sector: Asset management/energy/infrastructure
    • What: Periodically re-run governed analyses as assets change (load updates, degradation), recording evidence chains and alerting on deviations.
    • Tools/products: Scheduler + artifact diffing; threshold-based alerts; integration with CMMS/SCADA.
    • Assumptions/dependencies: Scalable compute; change-detection hooks from BIM/IoT; model fidelity and uncertainty quantification.
  • Enterprise “engineering copilot” spanning design, analysis, and reporting
    • Sector: AEC/PLM/ERP
    • What: A unified assistant that routes tasks across skills/backends, enforces validation, and standardizes reports across teams and geographies.
    • Tools/products: Skill marketplace; policy packs; centralized artifact registry; governance dashboards.
    • Assumptions/dependencies: Organization-wide adoption; SSO/security; comprehensive adapter coverage; robust multilingual support.
  • Insurance/finance risk analytics backed by executable evidence chains
    • Sector: Insurance/finance
    • What: Scenario testing (e.g., load increases, hazard intensities) with traceable runs to inform underwriting and pricing; audit trails for regulatory scrutiny.
    • Tools/products: Risk simulation libraries bound to governed skills; batch execution and aggregation of solver outputs.
    • Assumptions/dependencies: Certified accuracy and validation against field data; standardized reporting; model risk management policies.
  • Standard-setting for artifact-centered engineering submissions
    • Sector: Standards bodies/regulators
    • What: Formalize schemas for models, validations, solver traces, and reports so evidence chains become a submission norm (beyond PDFs).
    • Tools/products: Open specifications; conformance suites derived from StructureClaw-Bench.
    • Assumptions/dependencies: Multi-stakeholder consensus; alignment with IFC and national code ecosystems; long-term maintenance.
  • Autonomous multi-backend orchestration and vendor lock-in mitigation
    • Sector: Software/CAE vendors/AEC IT
    • What: Agents route tasks to the best-suited engine (accuracy, licensing, performance) while preserving a common artifact protocol.
    • Tools/products: Backend capability registry; cost/performance-aware routing; results reconciliation across solvers.
    • Assumptions/dependencies: Comparable solver semantics; licensing constraints; cross-validation and reconciliation logic.
  • Automated retrofit/design optimization with verified traceability
    • Sector: AEC/energy retrofits
    • What: Propose member sizing/layout changes, run analyses, check constraints, and return traceable design alternatives with sensitivity evidence.
    • Tools/products: Optimization skills (GAs/gradients) wrapped as governed tools; multi-scenario execution; visualization of constraint violations.
    • Assumptions/dependencies: Expanded design-rule knowledge; code-check integration; robust invalid-input handling; human approval gates.
  • Cross-domain transfer of artifact-centered evaluation (MEP, aerospace, robotics)
    • Sector: Engineering software/robotics
    • What: Port the governed-skill + typed-tool + artifact-state pattern to other domains where executable evidence chains matter.
    • Tools/products: Domain-specific skill packs and providers; new benchmarks analogous to StructureClaw-Bench.
    • Assumptions/dependencies: Domain ontologies and solver adapters; safety cases; community-maintained benchmarks.
  • Field-grade, offline analysis tools on tablets/laptops
    • Sector: Construction tech
    • What: Lightweight, governed analysis for simple checks on site with clear non-execution when data are insufficient.
    • Tools/products: Packaged OpenSeesPy + curated skills; local artifact caches; sync when online.
    • Assumptions/dependencies: Compute constraints; simplified scopes; robust UX for data capture; clear disclaimers.
  • Neutral benchmarking platform and public leaderboards for engineering agents
    • Sector: Research/industry consortia
    • What: Community-run, executable evaluation with repeated trials, uncertainty, and richer equivalence criteria to guide the ecosystem.
    • Tools/products: Hosted runners; dataset/versioning; attestation of configs; slice-and-dice diagnostics.
    • Assumptions/dependencies: Funding and governance; legal review for hosting proprietary backends; reproducibility practices.

Notes on feasibility and dependencies across applications:

  • Professional oversight is mandatory; the paper explicitly does not claim construction certification.
  • Current strengths: governed workflows, artifact traceability, OpenSeesPy integration, significant lift over generic agents.
  • Current limitations: invalid numerical input handling is a known weakness; multimodal reconstruction comparator omits full topology/support equivalence; single-run evaluations; benchmark coverage is controlled and non-exhaustive.
  • Operational dependencies: backend availability/licensing, secure artifact storage, alignment with BIM/IFC ecosystems, and organization-specific QA/policy requirements.
View Paper Prompt  View All Prompts 

Glossary

  • AEC (Architecture, Engineering, and Construction): A sector encompassing the design, construction, and operation of built environments. Example: "architecture, engineering, and construction (AEC)"
  • agentic workflows: LLM-driven, multi-step processes where models select and execute tools/actions to achieve a task. Example: "agentic workflows"
  • analysis backends: External engines/software that perform numerical structural analysis when invoked by the agent. Example: "local analysis backends"
  • artifact-centered evaluation: An assessment approach that checks intermediate engineering artifacts and executions, not just final text. Example: "artifact-centered evaluation"
  • artifact-centered workbench: A system that organizes workflows around explicit, inspectable engineering artifacts. Example: "an artifact-centered workbench"
  • artifact state: The maintained, shared record of all intermediate and final artifacts produced during a workflow. Example: "shared artifact state"
  • assertion-level diagnostics: Fine-grained checks and reports on specific benchmark assertions to localize failures. Example: "assertion-level diagnostics"
  • Auto--Generic Lift: The difference in Success Rate between automatic and generic-only modes as a paired system-level comparison. Example: "Auto--Generic Lift"
  • backend-aware execution: Running steps that respect available/compatible analysis engines and explicitly handling unsupported backends. Example: "backend-aware execution"
  • backend providers: Adapters that bind tool contracts to specific analysis engines within the system. Example: "backend providers"
  • BIM (Building Information Modeling): A digital representation of building characteristics enabling coordinated design and analysis. Example: "BIM"
  • capability context: The skill- and domain-specific constraints and knowledge that guide the agent’s planning and tool use. Example: "capability context"
  • code checking: Evaluating designs against building/structural codes with recorded results. Example: "code checking"
  • continuous beams: Structural elements spanning multiple supports without hinges, requiring special analysis. Example: "continuous beams"
  • deterministic solver: A fixed-behavior computational engine yielding repeatable results for given inputs. Example: "policy-based routing to a deterministic solver"
  • DXF (Drawing Exchange Format): A CAD file format used to exchange drawings that can be parsed for structural modeling. Example: "images or DXF files"
  • evidence chain: The linked artifacts, tool calls, validations, and decisions that justify a reported engineering result. Example: "the evidence chain"
  • finite coordinates: A validation requirement ensuring all model geometry uses numerically finite values. Example: "finite coordinates"
  • governed engineering skills: Domain-scoped capabilities with declared inputs/outputs, validations, and dependencies. Example: "governed engineering skills"
  • IFC-based tool interaction: Using the IFC (Industry Foundation Classes) standard to structure model authoring and tool exchanges. Example: "IFC-based tool interaction"
  • invalid numerical inputs: Out-of-range or nonsensical numeric parameters that must be detected and handled safely. Example: "invalid numerical inputs"
  • load combinations: Prescribed combinations of loads (e.g., dead, live, seismic) for structural design/verification. Example: "load combinations"
  • local engineering sandbox: A controlled runtime environment for executing modeling, validation, analysis, and checking. Example: "local engineering sandbox"
  • modal analysis: Structural analysis to extract natural frequencies and mode shapes. Example: "modal analysis"
  • multimodal structural-model reconstruction: Building an analysis-ready model from non-text inputs such as images or DXF files. Example: "multimodal structural-model reconstruction"
  • OpenSees: An open-source platform for structural and geotechnical simulation used as an analysis backend. Example: "OpenSees through its OpenSeesPy interface"
  • OpenSeesPy: The Python interface to OpenSees used to script and run analyses programmatically. Example: "OpenSees through its OpenSeesPy interface"
  • PKPM: A commercial structural analysis/design platform referenced as a possible provider backend. Example: "provider interfaces for PKPM and YJK"
  • policy-constrained tool interaction: Tool-use scenarios governed by explicit policies that the agent must obey. Example: "policy-constrained tool interaction"
  • portal frames: Structural systems with rigid beam–column connections commonly analyzed in building frames. Example: "portal frames"
  • program-aided computation: Having LLMs delegate computations to external programs for correctness and traceability. Example: "program-aided computation"
  • provider availability: The runtime check that confirms whether a required backend is present before execution proceeds. Example: "Provider availability is checked before execution"
  • provider interfaces: Adapter layers that implement tool contracts for specific backends. Example: "provider interfaces"
  • ReAct-style execution loop: An interleaving of reasoning and tool actions to advance a task step-by-step. Example: "a ReAct-style execution loop"
  • reference fixtures: Canonical data or targets used to check whether generated artifacts meet expected conditions. Example: "reference fixtures"
  • reinforced-concrete design: Engineering design of concrete structures with reinforcement, often under code compliance. Example: "code-compliant reinforced-concrete design"
  • retrieval-augmented calculation: Incorporating retrieved knowledge sources to support and verify computations. Example: "retrieval-augmented calculation"
  • routing: Workflow-stage decision of selecting skills/tools and paths based on the task and current artifacts. Example: "attributed to routing"
  • safe non-execution: An explicit, justified choice not to run analysis when inputs are invalid or unsupported. Example: "safe non-execution"
  • schema compliance: Validation that a model meets required structural data schema and references. Example: "schema compliance"
  • shear-wall layouts: Arrangements of shear walls that provide lateral resistance in buildings. Example: "shear-wall layouts"
  • skill manifests: Declarations of skill inputs, outputs, validations, dependencies, and supported scopes. Example: "the skill manifests"
  • structural-model protocol: The shared typed schema describing structural models, loads, materials, and metadata. Example: "a shared structural-model protocol"
  • Success Rate: Primary metric measuring the fraction of scenarios where all required assertions pass in one run. Example: "Success Rate"
  • support representation: How boundary/support conditions are encoded within the structural model. Example: "support representation"
  • structure-type detection: Identifying the structural system category (e.g., frame, truss) from inputs. Example: "structure-type detection"
  • trace-grounded report: A report whose claims are supported by recorded artifacts and execution traces. Example: "a trace-grounded report"
  • trusses: Pin-jointed frameworks of members used to carry loads efficiently. Example: "trusses"
  • typed assertions: Structured, typed checks specifying conditions a scenario’s artifacts/executions must satisfy. Example: "typed assertions"
  • typed tools: Tools with declared input/output schemas and preconditions to enforce correct usage. Example: "typed tools"
  • unit ambiguity: Unclear or conflicting unit specifications that can lead to misinterpretation of values. Example: "unit ambiguity"
  • unit conventions: Explicitly declared measurement units used throughout the model to avoid confusion. Example: "explicit unit conventions"
  • validation diagnostics: Recorded results of model checks that inform repair, clarification, or abstention. Example: "validation diagnostics"
  • validation-guided refinement: Iteratively improving the model/analysis based on validation feedback. Example: "validation-guided refinement"
  • YJK: A commercial structural analysis/design software noted as a potential backend provider. Example: "provider interfaces for PKPM and YJK"
View Paper Prompt  View All Prompts 

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now