Compound LLM Framework: Graph Abstractions & Schedulers

• Compound LLM frameworks are integrated AI systems that embed a base LLM into a graph-based architecture, combining prompts, tools, and scaffold code for enhanced performance.
• They dynamically optimize module selection, routing, and scheduling to balance quality, cost, latency, and reliability across varied applications.
• These frameworks support complex workflows such as generate-test-critique loops, retrieval-augmented generation, and hierarchical agent coordination for robust decision-making.

Searching arXiv for papers on compound LLM frameworks, model selection, scheduling, and scaffolded/agent architectures. A compound LLM framework is a compound AI system in which a base LLM is embedded in a multi-component program rather than used as a single isolated predictor. In the cited literature, such systems appear as compound AI systems that combine multiple LLM calls, scaffolded LLMs with prompts, tools, and scaffold code, uncertainty-aware compound LLM applications, and graph-structured pipelines with stochastic nodes and heterogeneous outputs (Chen et al., 20 Feb 2025, Lin et al., 2024, Zhu et al., 4 Apr 2025, Nilayam et al., 11 May 2026). Typical workflows include self-refine, multi-agent debate, retrieval-augmented generation, generate-test-critique loops, hierarchical agent delegation, and pre-inference or post-inference repair. Taken together, these formulations suggest that the central object of design is no longer a single model invocation but a computation graph whose nodes, interfaces, routing rules, and runtime policies jointly determine quality, cost, latency, and reliability.

1. Scope and conceptual definition

The broadest formalization in the survey literature defines a scaffolded LLM as a compound AI system comprising a base LLM with fixed parameters $\theta$, prompt templates $P$, tool interfaces $T$, and scaffold code $S$, so that the overall system implements

$\hat y = \mathrm{System}(x; \theta, P, T, S).$

Within this view, the non-parametric variables are the prompts, tools, and code, and the system may instantiate architectures such as LLM + Tools, LLM + Code Interpreter, Retrieval-Augmented Generation, or Search Over LLM Calls (Lin et al., 2024).

Other papers specialize this general notion. In model selection for compound AI systems, a static compound AI system is a directed acyclic graph $G=(V,E)$ of LLM modules, where each node is one module such as a generator, critic, or refiner, and a model allocation $f:V\to M$ assigns exactly one model to each module (Chen et al., 20 Feb 2025). In systems work, a compound LLM application is likewise represented as a DAG, but its stages may be regular, LLM, or dynamic, with dynamic stages expanding at runtime (Zhu et al., 4 Apr 2025). In retrieval, a compound retrieval system is defined as a broader class than cascading re-ranking, allowing arbitrary interactions among a first-stage scorer, pointwise relevance predictors, and pairwise LLM relevance predictors (Oosterhuis et al., 16 Apr 2025).

A recurring point across these formulations is that “compound” does not mean only “multi-agent.” It also includes linear pipelines, cascades, retrieval systems, verification loops, and heterogeneous training jobs. This directly contradicts the common simplification that compound design is synonymous with debate-style agents. The surveyed taxonomy explicitly includes RAG, code-interpreter wrappers, and search over LLM calls, while retrieval work shows that even classical ranking systems become compound once multiple predictors are jointly optimized (Lin et al., 2024, Oosterhuis et al., 16 Apr 2025).

2. Architectural primitives and graph formalisms

The graph abstraction is the dominant formal device for describing compound LLM frameworks, but different papers attach different semantics to the nodes and edges. In model-allocation work, nodes are LLM modules and end-to-end performance is

$P(f)=\mathbb E_{z\sim D}[p(f,z)],$

with $p(f,z)\in\{0,1\}$ a binary success indicator (Chen et al., 20 Feb 2025). In systems scheduling, each stage has an uncertain duration, and dynamic stages have uncertain internal membership and edges, so the runtime graph itself is partially stochastic (Zhu et al., 4 Apr 2025). QUIVER extends the graph formalism further to a typed pipeline graph $G=(V,E,T,F)$, where each node computes a stochastic function into a typed output space and node-wise distances are type-dispatched over categorical, set-valued, ordered-list, numeric, or text fields (Nilayam et al., 11 May 2026).

These differences matter because they determine what can be optimized or guaranteed. A static DAG is suitable for model allocation or module-wise search. A DAG with dynamic stages is suitable for scheduling under structural uncertainty. A typed stochastic graph is suitable for perturbation analysis, sensitivity measurement, and bifurcation detection. This suggests that “compound LLM framework” is best understood as a family of graph-based abstractions, not a single canonical architecture.

Several concrete control patterns recur across the literature. LLM-Modulo instantiates a generate-test-critique loop with a base LLM, format critic, constraint critics, and a meta-controller that consolidates failures into a backprompt (Gundawar et al., 2024). TorchOpera uses a linear pre-inference and post-inference pipeline around a core LLM: safety detector, grounding module, core LLM, output safety detector, and repairer (Han et al., 2024). The Concurrent Modular Agent departs from centralized control altogether: modules are independently running Python async functions, a shared vector database stores utterances as embeddings, MQTT carries module-to-module messages, and there is no central clock or controller (Maruyama et al., 26 Aug 2025).

3. Optimization, routing, and model allocation

One core optimization problem is deciding which LLM to use at each module of a compound system. LLMSelector formulates this as the Model Selection Problem, maximizing $P$0 over allocations $P$1, where the search space has size $P$2 (Chen et al., 20 Feb 2025). The framework exploits two empirical insights: end-to-end performance is often monotonic in module performance with other modules fixed, and module-wise performance can be estimated accurately by a sufficiently capable LLM “diagnoser.” Its greedy round-robin procedure estimates

$P$3

for each candidate model at module $P$4, assigns the $P$5, and repeats until convergence or budget exhaustion. Under module-wise intra- and inter-monotonicity and uniqueness of the global optimum, the procedure converges to the globally optimal allocation on the training set within $P$6 rounds, with one full pass costing $P$7 diagnoser calls (Chen et al., 20 Feb 2025). Empirically, on Self-Refine, Multi-Agent Debate, and Locate-Solve, it yields 5%–70% absolute gains over the best homogeneous allocation, including 89% to 95% on LiveCodeBench and 30% to 100% on TableArithmetic (Chen et al., 20 Feb 2025).

AdaptiveLLM addresses a related but narrower routing problem for code generation under cost constraints. It estimates task difficulty from the median Chain-of-Thought length generated by DeepSeek-R1-Distill-Qwen-32B over $P$8 runs, clusters the scalar difficulty proxy into three levels by k-means, fine-tunes CodeBERT with triplet loss to encode difficulty-aware features, and trains an XGBoost classifier to select among eight candidate LLMs (Cheng et al., 12 Jun 2025). At inference, only the embedding, classifier, and selected model are active; CoT generation and difficulty clustering are offline. Reported results show pass@1 of 44.94%, a 7.86% improvement over ComplexityNet while reducing resource consumption by 88.9%, and approximately 15% accuracy improvement over a single model at the same cost consumption (Cheng et al., 12 Jun 2025).

In retrieval, optimization concerns not only which model to invoke but where to invoke it and how to combine its outputs. Compound retrieval systems introduce a learned selection policy over pointwise and pairwise LLM calls and a learned aggregation function, optimized by a trade-off loss

$P$9

On TREC Deep Learning data, supervised compound systems outperform all cascade baselines for $T$0 calls and reach PRP’s nDCG@100 with an order-of-magnitude fewer calls, approximately 50k instead of 1,000,000 (Oosterhuis et al., 16 Apr 2025). A common implication across these works is that homogeneous model assignment and fixed top-$T$1 routing are often suboptimal once modules have heterogeneous strengths and costs.

4. Scheduling and systems support

Compound LLM frameworks are also scheduling problems. LLMSched treats each compound application as a job whose runtime structure is a DAG $T$2 with regular, LLM, and dynamic stages, each stage having uncertain duration and dynamic stages having uncertain membership of inner nodes and edges (Zhu et al., 4 Apr 2025). A Bayesian network profiles discretized stage durations from historical traces; runtime inference updates posterior distributions after each completion event. Uncertainty is quantified by Shannon entropy, and the scheduler computes an uncertainty-reduction score

$T$3

Dispatch then interleaves exploration, selecting the ready stage with maximal $T$4, and exploitation, selecting the ready stage from the job with shortest expected remaining time, through an $T$5-greedy meta-policy (Zhu et al., 4 Apr 2025). On six representative applications, LLMSched reduces average JCT by 14–79% in simulation and 26–66% on a testbed relative to six baselines including FCFS, SJF, Argus, Decima, and Carbyne (Zhu et al., 4 Apr 2025).

Maestro extends compound reasoning to training workloads such as knowledge distillation and multimodal LLM training. It restructures a heterogeneous training job into a section graph $T$6, where each section is annotated by parameter count, execution mode, and effective sequence length, and each section independently chooses configuration

$T$7

A two-stage optimization first identifies the critical section and optimizes its efficiency, then assigns the smallest GPU allocation to auxiliary sections such that they never stall the critical section. To handle data-dependent activation, Maestro introduces a wavefront scheduling algorithm that reorders samples so the critical section has zero idle time (Yuan et al., 11 May 2026). On vision-language training, Maestro achieves 1.4$T$8 and 1.20$T$9 end-to-end throughput over Megatron-LM; on distillation, 1.75$S$0 end-to-end throughput; deployed in production, it reduces GPU consumption by approximately 40% (Yuan et al., 11 May 2026).

These results make clear that compound LLM frameworks are constrained by orchestration at two levels: intra-application routing among modules and cluster-level scheduling among heterogeneous stages. This suggests that improvements in prompting or reasoning can be negated by poor scheduling, especially when execution paths are data-dependent.

5. Reliability, verification, and perturbation analysis

A major motivation for compound design is to impose reliability properties that a single LLM call cannot provide. LLM-Modulo exemplifies this approach in planning and scheduling. The system pairs a generator with a complete set of sound verifiers, rejecting any candidate that fails the format critic or any constraint critic and feeding consolidated feedback back to the generator (Gundawar et al., 2024). The formal guarantee is soundness: if the system returns a plan $S$1, then for all critics $S$2, $S$3. If each critic is sound and each checked constraint is necessary for correctness, then no fallacious plan can ever escape (Gundawar et al., 2024). Across OSU Travel Planner and three Natural Plan scheduling domains, accuracy increases from 8.33% to 23.89% for GPT-4o on OSU, from 3.43% to 40.00% on Trip Planning, and up to 88.80% for Claude-3.5 on Calendar Scheduling (Gundawar et al., 2024).

TorchOpera addresses a different reliability axis: safety, grounding, hallucination detection, and lightweight repair. Its pre-inference phase screens unsafe inputs and grounds the prompt using vector retrieval; its post-inference phase detects hallucinations, generates explanations, and either applies rule-based wrappers or triggers re-grounding and re-inference (Han et al., 2024). The hallucination detector is trained as a joint classification and text-generation model on HaluEval and computes $S$4 from top-10 first-token logits; the repairer can fix structured errors such as unsafe URLs via regex and external APIs without re-querying the core LLM (Han et al., 2024). Reported results include 0.928 hallucination-detection accuracy, input safety detection with Acc 0.877, Rec 0.624, F$S$5 0.656, repair time of approximately 1.06 s on the URL task, and up to $S$6 GPU memory saving through multi-LoRA serving (Han et al., 2024).

QUIVER formalizes reliability as perturbation propagation rather than only end-task success. For each edge $S$7, it defines magnitude sensitivity

$S$8

and occurrence-lift

$S$9

classifying edges as amplifiers, absorbers, or threshold-sensitive (Nilayam et al., 11 May 2026). It further decomposes trajectory divergence into iteration-count divergence $\hat y = \mathrm{System}(x; \theta, P, T, S).$0, structural-shape divergence $\hat y = \mathrm{System}(x; \theta, P, T, S).$1, and value-drift divergence $\hat y = \mathrm{System}(x; \theta, P, T, S).$2, defines bifurcation thresholds for the smallest perturbation that changes structure, and measures distribution faithfulness by $\hat y = \mathrm{System}(x; \theta, P, T, S).$3 (Nilayam et al., 11 May 2026). Validated on two production enterprise pipelines and a public DSPy multihop QA pipeline, with more than 8,200 traces and more than 32,000 pairwise comparisons, QUIVER shows that equal end-to-end divergence rates can arise from mechanically different cascade patterns (Nilayam et al., 11 May 2026). A common misconception is that compound reliability reduces to adding more reasoning; these papers instead show that guarantees arise from critics, detectors, typed instrumentation, and explicit control logic.

6. Hierarchy, multi-agent organization, and asynchronous coordination

Compound LLM frameworks frequently use multiple agents, but the literature is explicit that hierarchy and deliberation are distinct design axes. In an adversarial POMDP study in CybORG CAGE-2, the defender is modeled as

$\hat y = \mathrm{System}(x; \theta, P, T, S).$4

with non-positive reward and a 30-step horizon (Bogdanov et al., 15 May 2026). The study varies context representation, deliberation tools, and hierarchical decomposition. Programmatic state abstraction and compressed history provide the largest returns per token spent, improving mean return by up to 76% over raw observations. Hierarchical decomposition without deliberation achieves the best absolute performance for most models, while distributing self-questioning, self-critique, and self-improvement across sub-agents degrades performance for all five model families, reaching up to 3.4$\hat y = \mathrm{System}(x; \theta, P, T, S).$5 worse mean return while using 1.8–2.7$\hat y = \mathrm{System}(x; \theta, P, T, S).$6 more tokens; this destructive pattern is termed a deliberation cascade (Bogdanov et al., 15 May 2026). The design principle stated in that work is to invest first in deterministic state abstraction and clean task decomposition rather than deeper per-agent reasoning.

The Concurrent Modular Agent proposes a different organizational logic: fully asynchronous modules with a shared persistent vector database and MQTT-based text messaging (Maruyama et al., 26 Aug 2025). Modules wake on timers, incoming messages, or relevant memory retrieval, query context from ChromaDB, call the LLM, and may write memory, publish messages, or invoke hardware. Because no module holds locks on the global state or message channels, failures do not block others, and a Conflict Detector meta-module flags semantically conflicting memory insertions for downstream suppression (Maruyama et al., 26 Aug 2025). In Plantbot, end-to-end conversational latency is 400 $\hat y = \mathrm{System}(x; \theta, P, T, S).$7 50 ms versus 800 $\hat y = \mathrm{System}(x; \theta, P, T, S).$8 200 ms in a monolithic loop, system uptime over 48 h is 99.7%, and memory growth is approximately 5 KB/min before pruning rather than approximately 10 KB/min in the prior centralized implementation. In ALTER3, the system ran for 14 h and autonomously triggered approximately $\hat y = \mathrm{System}(x; \theta, P, T, S).$9 events (Maruyama et al., 26 Aug 2025).

LessonL shows yet another coordination mechanism: a team of LLM agents collaborates over multiple rounds by soliciting, banking, selecting, and reusing “lessons” from one another’s successes and failures (Liu et al., 29 May 2025). Lessons are stored as triples $G=(V,E)$0, where $G=(V,E)$1 is the lesson, $G=(V,E)$2 the speedup at creation time, and $G=(V,E)$3 an effectiveness factor adjusted by later performance. On ParEval and PolyBench, LessonL achieves the best reported geometric mean speedup, including 2.16 in serial mode and 3.46 in OpenMP mode, and on HumanEval it reaches pass@1 of 0.915 (Liu et al., 29 May 2025). Across these studies, more interaction is not inherently beneficial; performance depends on whether the coordination protocol resolves uncertainty, enforces contracts, and controls prompt growth.

7. Domain-specific realizations and broader implications

Domain-specific frameworks show how compound design scales beyond generic chat or coding. In medical multi-image reasoning, M$G=(V,E)$4LLM is built from a five-stage, context-aware instruction generation pipeline over 237,137 compound figures extracted from biomedical literature (Chen et al., 27 Nov 2025). The stages are inline text summarization, medical knowledge complementation, visual perception enhancement, context–question–answer instruction generation, and leakage-prevented context refinement. The final model combines an InternViT vision encoder, a two-layer connector, and a QWen2.5-7B language module with multi-image cross-attention, trained for three epochs with standard next-token cross-entropy plus weight decay (Chen et al., 27 Nov 2025). On PMC-MI-Bench, M$G=(V,E)$5LLM improves BLEU@4 from 9.8 to 15.0 and STS from 74.7 to 78.2 relative to the best baseline in open-ended multi-image VQA, and it also improves disease diagnosis and progression prediction on longitudinal chest X-rays from MIMIC (Chen et al., 27 Nov 2025).

The survey perspective frames these examples as instances of training scaffolded LLMs with language supervision: prompts, tools, and scaffold code are optimized as non-parametric variables through static synthesis, dynamic program analysis, or streaming updates based on textual critiques (Lin et al., 2024). In that formulation, compound frameworks are semi-parametric systems whose behavior can be improved without gradient access to the base model, preserving compatibility with closed-source LLMs and mitigating catastrophic forgetting relative to direct parametric updating (Lin et al., 2024).

Several limits recur across the literature. LLMSched identifies BN structure learning, discretization trade-offs, and multi-tenant resource interference as open problems (Zhu et al., 4 Apr 2025). CMA identifies latency and API-cost overhead, unpredictable emergent behaviors, scalability beyond dozens of modules, and automated conflict resolution as open questions (Maruyama et al., 26 Aug 2025). QUIVER shows that stale evaluation datasets can be localized to specific node-field categories through distribution-faithfulness analysis, implying that aggregate final-output metrics can miss high-sensitivity failure modes (Nilayam et al., 11 May 2026). A plausible synthesis is that compound LLM frameworks should be evaluated simultaneously as algorithms, software architectures, and stochastic dynamical systems: they require optimization of module choice, verification and repair boundaries, scheduling and resource allocation, and production-faithful instrumentation.