JudgeFlow: Agentic Workflow Optimization

• JudgeFlow is a framework for optimizing LLM-based agentic workflows using an Evaluation–Judge–Optimization–Update pipeline and block-level failure analysis.
• It decomposes workflows into reusable logic blocks (sequence, loop, and conditional) to isolate errors and guide precise structural modifications.
• Empirical results on mathematical reasoning and code generation benchmarks demonstrate superior performance and sample efficiency compared to prior methods.

Searching arXiv for the specified JudgeFlow paper and closely related workflow/judge papers. arXiv search query: (Ma et al., 12 Jan 2026) arXiv search query: "JudgeFlow: Agentic Workflow Optimization via Block Judge" JudgeFlow is a framework for optimizing LLM-based agentic workflows through an Evaluation–Judge–Optimization–Update pipeline in which workflows are decomposed into reusable logic blocks, failed executions are analyzed by a dedicated Judge module, and modifications are targeted at the block judged most responsible for failure. In the underlying formulation, this replaces coarse end-to-end supervision with block-level, rank-based responsibility signals, with the stated goals of improving sample efficiency, interpretability, and the scalability of automated workflow optimization. The framework is evaluated on mathematical reasoning and code generation benchmarks, where it reports superior performance and efficiency relative to prior methods such as AFlow, MaAS, and MermaidFlow (Ma et al., 12 Jan 2026).

1. Conceptual setting and motivation

JudgeFlow is defined over agentic workflows, understood as LLM-based systems built by wiring together operators, logic blocks, and complete workflows. Operators are basic actions such as “generate solution”, “test solution”, “self-refine”, “multi-generate and ensemble”, and “programmer”. Logic blocks are higher-level control-flow structures such as sequences, loops, and conditionals that orchestrate one or more operators. A workflow is the end-to-end pipeline composed of multiple logic blocks, taking in a query and emitting a final answer.

The motivating difficulty is that most existing optimization methods rely on coarse end-to-end signals. A workflow is run on a dataset, the final answer is scored, and prompts, structure, or code are then adjusted and re-evaluated. Under this regime, the evaluation function indicates whether the final output is correct, but does not identify which internal step caused failure. In workflows with many LLM calls, conditional branches, or refinement loops, this creates a severe credit-assignment problem: one cannot readily determine whether failure originated in initial reasoning, verification, refinement, or routing.

JudgeFlow is positioned against several prior strands. Prompt optimization and textual-gradient methods such as TextGrad and OPRO typically assume a single node or only a coarse structure. Workflow-as-code methods such as AFlow and MermaidFlow search over structural changes using MCTS or evolutionary programming, but remain mainly guided by end-to-end metrics. According to the paper, this makes search expensive, encourages low-impact modifications, and limits interpretability because improved versions can be identified without a clear account of why they improved (Ma et al., 12 Jan 2026).

2. Logic blocks as the central abstraction

The framework’s defining abstraction is the logic block, which sits between individual operators and the full workflow. A configured operator is written as $O(D)$, where $O$ is a categorical label for the core function and $D$ is its configuration, including backbone model, prompt template, and hyperparameters. A logic block is written as $(B, C)$, where $B \in \mathcal{B}$ is the block type and $C$ is the block configuration.

JudgeFlow uses three block types:

• SequenceLogic (seq): operators executed one after another, each consuming the previous output.
• LoopLogic (for): operators invoked iteratively until a stopping condition is met or a maximum iteration count is reached.
• ConditionalLogic (cond): a condition operator selects either a success or failure branch, and only the selected branch executes.

An agentic workflow is defined as

$$W = \left(\left\{(B_i, C_i)\right\}_{i=1}^M, S\right),$$

where $M$ is the number of logic blocks and $S$ is the ordered top-level sequence of blocks. Execution proceeds block by block, with each block receiving the previous state and producing a new state, and the final output $a'_M$ scored against the ground-truth answer $O$0 by an evaluation function. The overall optimization objective is to maximize expected evaluation performance over a dataset.

This abstraction serves two roles simultaneously. First, it constrains the search space to reusable, configurable structural units that capture fundamental forms of logic. Second, it defines a diagnosis granularity that is coarser than operators but finer than the end-to-end workflow. The paper argues that this is especially useful for conditionals and loops, where operator-level attribution is difficult because only some branches execute in a given run, whereas whole logic blocks remain stable semantic units for both automated diagnosis and human inspection (Ma et al., 12 Jan 2026).

3. The Judge module and rank-based responsibility

JudgeFlow’s distinguishing component is the Judge module, an LLM used as a specialized evaluator of failed execution traces. For a given example $O$1, workflow execution yields a trace containing the problem, the correct answer, the final incorrect answer in failed cases, and the inputs and outputs for each block. If the final score falls below a success threshold $O$2, the run is treated as a failure and passed to the Judge.

The Judge receives the workflow structure, input $O$3, ground truth $O$4, and the intermediate block outputs $O$5. It is prompted as a “workflow failure analyst” and instructed to compare each block’s output with both its input and the correct answer, identify where the first critical deviation is introduced, consider whether later blocks had enough context to fix earlier errors, and use counterfactual reasoning of the form “If this block were correct, would the final answer be correct?”

Its output is a rank vector $O$6 that is a permutation of $O$7. Rank $O$8 denotes the block most responsible for the failure, and rank $O$9 the least responsible. These ranks support two distinct aggregation schemes. The first is the round-wise worst block, the block assigned rank $D$0 for a specific failure; the corresponding failure trace is appended to that block’s log and later used as a few-shot error example during optimization. The second is the OverallWorst block, selected by summing each block’s ranks across failed runs and choosing the block with the lowest cumulative rank, i.e., the block most often judged near rank $D$1 across the dataset.

A central design claim is that JudgeFlow does not rely on a single attribution decision. The paper explicitly notes LLM judge noise and states that responsibility is aggregated across many failed instances because the aggregated failure attribution distribution is more robust than per-instance attribution. The Judge does not explicitly use numeric partial-correctness scores; rather, it reasons from full traces, including wrong code, failed tests, and unreachable branches when these are visible in the execution record (Ma et al., 12 Jan 2026).

4. Optimization, update, and implementation regime

Once the globally weakest block is identified, the LLM-based optimizer receives the current workflow, the selected block, a sample from that block’s failure log, and a set of allowed modification actions. The update step is defined as

$D$2

where the action $D$3 is chosen by the optimizer from a constrained set.

The action space contains exactly three high-level edits:

• Add Block: add a new block near the problematic block.
• Remove Block: remove the selected block and reconnect predecessor and successor.
• Modify Block: change the block’s internal operators or configuration.

The optimizer is given strong constraints. Workflows are limited to at most three blocks, the optimizer must focus on the identified low-performing block, it must not reuse previous optimization solutions, and instruction fields are emphasized as key levers for performance. The modified workflow is then evaluated, inserted into a Top-$D$4 pool of candidate workflows, and future optimization rounds sample starting workflows from that pool using a softmax over scores. This is intended to balance exploitation of high-performing workflows against exploration of non-maximal candidates.

Implementation is deliberately lightweight. The execution LLM, optimizer LLM, and Judge LLM are all gpt-4o-mini-0718 with temperature $D$5, accessed via API. In the AIME extension experiment, gpt-4.1-mini is used as Judge, Optimizer, and Executor. The paper states that there is no additional training or fine-tuning; JudgeFlow is a prompting-based framework built on off-the-shelf LLM APIs. Reported hyperparameters include a maximum of 20 optimization rounds, a success threshold $D$6 for exact correctness, a candidate pool size $D$7, and a maximum of $D$8 blocks per workflow (Ma et al., 12 Jan 2026).

5. Empirical performance, efficiency, and interpretability

JudgeFlow is evaluated on two classes of tasks: mathematical reasoning and code generation. The mathematical benchmarks are GSM8K and MATH, both reported with solve rate (%). The code benchmarks are MBPP and HumanEval, both reported with pass@1. An additional extension uses AIME accuracy. Following prior work such as AFlow, MaAS, and MermaidFlow, each dataset is split 1:4 into training and test, with the training split used during optimization and final performance reported on held-out test data.

On the main four-benchmark comparison with gpt-4o-mini-0718, JudgeFlow reports 93.0 on GSM8K, 58.5 on MATH, 83.8 on MBPP, and 93.4 on HumanEval, for an average of 82.2. MermaidFlow, the strongest prior baseline in the table, reports 92.4, 55.4, 82.3, and 92.9, for an average of 80.8. The paper highlights absolute improvements of +3.1 on MATH, +1.5 on MBPP, +0.6 on GSM8K, +0.5 on HumanEval, and +1.4 average points over MermaidFlow. On AIME 2025, with gpt-4.1-mini as backbone, JudgeFlow reports 44.67% versus 42.00% for AFlow (Ma et al., 12 Jan 2026).

The reported efficiency results are central to the method’s framing. On MBPP learning curves, JudgeFlow’s training and test performance improve rapidly within the first five iterations and converge at a higher test pass@1, approximately 0.8563, whereas AFlow remains flat for many iterations and converges to approximately 0.80645. A cost breakdown for one GSM8K round reports \$D$90.01 for the Judge phase, implying a Judge/Evaluation cost ratio of about 2%.

Interpretability is illustrated through block-level diagnostics. Because each failure is logged against the block deemed most responsible, developers obtain per-block logs $(B, C)$0 of failure traces. The GSM8K case study begins from a two-block workflow: b1 (seq) using multi_generate_ensemble, followed by b2 (seq) using programmer. Different failures lead the Judge to blame different blocks, but aggregated ranks select b1 as OverallWorst. The optimizer then applies Add Block, inserting b3 (seq) with self_refine between b1 and b2, yielding the new order ["b1", "b3", "b2"]. The paper presents this as an example of diagnosis-driven structural change rather than arbitrary mutation (Ma et al., 12 Jan 2026).

Ablation results on MBPP reinforce the importance of both major components. Full JudgeFlow scores 83.8; removing the logic block abstraction yields 81.8; removing both logic blocks and the Judge yields 80.6. Additional experiments replacing the Judge/Optimizer model show 84.5 with GPT-4o and 84.4 with Gemini-2.5-flash, compared with 83.8 for GPT-4o-mini, suggesting that the framework benefits from stronger judging and optimization models while remaining competitive with a smaller one.

6. Limitations, relations, and scope of the framework

The paper identifies several limitations. First, LLM-as-Judge bias and noise remain unresolved: judges can be biased or inconsistent, and single-instance attribution can be wrong, even if aggregation improves robustness. Second, the logic-block search space is deliberately narrow, restricted to seq, for, and cond; more complex control flow or deeply nested structures may be harder to represent exactly. Third, current experiments are limited to workflows with at most three blocks, so scaling to dozens of blocks, tools, or agents may require more sophisticated or hierarchical strategies. Fourth, the Judge itself is not learned or calibrated beyond prompting; no task-specific fine-tuning is performed. The paper also notes that failure cases are not exhaustively cataloged, and identifies likely hard settings such as multimodal tasks, long-horizon interactive environments, or workflows in which many blocks are equally flawed (Ma et al., 12 Jan 2026).

In relation to prior work, JudgeFlow is explicitly situated at the intersection of agentic workflow optimization, self-refinement, and credit assignment. Relative to AFlow, MermaidFlow, GPTSwarm, and MaAS, it replaces predominantly global graph or code search with targeted edits to the block identified as most problematic. Relative to SELF-REFINE, textual gradients, and LLM-AutoDiff, it optimizes workflow structure rather than only prompts or single-call behavior. Relative to semantic backpropagation and related intermediate-feedback approaches, it uses rank-based block attribution rather than scalar gradients.

Two adjacent lines of work are particularly illuminating. LawFlow models complete end-to-end legal workflows as dynamic, modular, and iterative graphs, emphasizing that realistic professional reasoning is better represented as a chain of decisions than as a single input-output mapping; this offers a useful comparison point for JudgeFlow’s commitment to workflow structure, though LawFlow concerns legal practice rather than block-level optimization of LLM systems (Das et al., 26 Apr 2025). “Internal Flow Signatures for Self-Checking and Refinement in LLMs” presents a different kind of flow-based judge: a lightweight GRU validator over internal hidden-state trajectories that predicts hallucination and localizes a culprit depth event, enabling targeted refinement at the activation level rather than at the level of external workflow blocks (Jeong et al., 2 Feb 2026). This suggests a broader research pattern in which “judge” components provide localized diagnostic signals, but JudgeFlow’s specific contribution is to make those signals actionable for structural editing of agentic workflows.

Taken together, JudgeFlow defines a workflow optimization paradigm in which reusable control-flow abstractions, failure-trace inspection, and block-level responsibility ranking are tightly coupled. Its principal novelty lies not in introducing a new execution model for agents, but in changing the supervision signal used to improve them: from coarse outcome scores to structured diagnoses about where a workflow should be changed and why.