Deterministic vs. LLM-Controlled Orchestration for COBOL-to-Python Modernization

Abstract: Modernizing legacy COBOL systems remains difficult due to scarce expertise, large and long-lived codebases, and strict correctness requirements. Recent LLM-based modernization systems increasingly rely on agentic workflows in which the model controls multi-step tool execution. However, it remains unclear whether delegating execution control to the LLM improves correctness, robustness, or efficiency in structured software engineering workflows. We present a controlled empirical study of deterministic and LLM-controlled orchestration for COBOL-to-Python modernization. Using a unified experimental framework, we hold the LLMs, prompts, tools, configurations, and source programs constant while varying only the execution control strategy. This isolates orchestration as the sole experimental variable. We evaluate both approaches using functional correctness, robustness across repeated stochastic runs, and computational efficiency. Across multiple models, deterministic orchestration achieves comparable computational accuracy to LLM-controlled orchestration while improving worst-case robustness and reducing performance variability across runs. Deterministic execution also reduces token consumption by up to 3.5x, leading to substantially lower operational cost. These results suggest that, in structured modernization workflows with explicit validation stages, fixed execution policies provide more stable and cost-efficient behavior than fully agentic orchestration without reducing translation quality.

• The paper demonstrates that deterministic orchestration yields superior worst-case robustness and reduced token consumption compared to LLM-controlled methods.
• Results show LLM orchestration achieves a higher success rate of runnable translations at the expense of increased execution variance and cost.
• Implications suggest employing deterministic pipelines in structured workflows to ensure stability, scalability, and economic viability in legacy modernization.

Deterministic and LLM-Controlled Orchestration for Legacy Code Modernization: An Empirical Assessment

Introduction

Legacy system modernization, especially for COBOL-to-Python migration, has garnered significant interest due to operational, economic, and workforce constraints in critical sectors. Recent advances in LLMs have enabled complex agentic workflows involving multi-step tool integration, yet the effectiveness of delegating execution control to LLMs in structured modernization remains ambiguous. This study ("Deterministic vs. LLM-Controlled Orchestration for COBOL-to-Python Modernization" (2605.09894)) systematically compares deterministic orchestration with agentic LLM-controlled orchestration in a controlled framework, isolating execution control as the experimental variable. The ATLAS system provides the backbone for this empirical analysis.

Figure 1: ATLAS system architecture supporting end-to-end COBOL-to-Python modernization through fixed or adaptive orchestration strategies.

Orchestration Paradigms

Deterministic orchestration refers to a pipeline where tool invocation, execution order, and validation stages are statically defined. Tool ordering and retry logic are invariant across runs, and the LLM is limited to code generation; all orchestration decisions leverage explicit control logic. In contrast, LLM-controlled orchestration delegates tool selection, execution ordering, retries, and termination to the LLM. This agentic paradigm dynamically adapts trajectories based on intermediate output and stochastic sampling, introducing non-determinism into execution traces even with identical base configurations.

Figure 2: Workflow comparison between deterministic and LLM-controlled orchestration, highlighting static vs. adaptive task sequencing and branching.

Experimental Framework

Using the ATLAS platform, experiments are conducted with identical models, prompts, tools, source programs, and configurations, isolating orchestration as the sole variable. The evaluation leverages the NIST COBOL85 Test Suite, encompassing hundreds of COBOL programs with structured test cases. Metrics include:

• Computational Accuracy (CA): Measures semantic equivalence.
• Success Rate (SR): Fraction of runs yielding runnable, correct outputs.
• Worst-case robustness: P5-CA (5th percentile accuracy) and CVaR$_{0.1}$ (severity in failure tails).
• Efficiency: Token consumption per successful translation and associated operational costs.

Results and Analysis

Correctness and Robustness

Both orchestration paradigms attain nearly identical mean translation correctness for finalized outputs. Notably, deterministic orchestration achieves higher CA, P5-CA, and CVaR$_{0.1}$ across all evaluated models, indicating reduced error propagation and higher output stability. LLM-controlled orchestration yields higher SR across repeated stochastic runs, demonstrating more frequent generation of runnable, test-passing translations, but with heavier performance tails and increased execution variance.

Computational Efficiency

Deterministic orchestration achieves substantial reductions in token consumption—up to $3.5\times$ lower compared to LLM-controlled runs, notably in complex COBOL modules (Numeric, Sequential I/O). LLM-controlled orchestration incurs increased token expenditures due to variable-length execution and repeated agentic reasoning steps.

Figure 3: Token usage per successful translation, showing deterministic orchestration minimizes token requirements across COBOL modules.

This resource inefficiency translates directly into operational cost, with deterministic pipelines consistently yielding lower costs per successful translation. The planning-tax associated with agentic orchestration produces significant financial penalties without offsetting quality gains.

Figure 4: Cost comparison by orchestration strategy for COBOL benchmark programs, emphasizing deterministic orchestration's economic advantages.

Test Suite Coverage

Aggregate statistics reveal near parity between approaches in numbers of program executions, errors, and test cases inspected. Minor differences in pass/fail counts align with observed correctness and robustness trends, suggesting deterministic orchestration offers marginally improved reliability for the same set of translated programs.

Limitations

The study's scope is limited to COBOL-to-Python modernization. Potential generalizability issues include distinct language features, workflow dependencies, and integration requirements not captured by standardized benchmarks. Furthermore, correctness is measured relative to test suite coverage, which may omit latent semantic errors. The advantages of deterministic orchestration presuppose explicit validation stages; in less structured workflows, agentic control may provide additional flexibility. Efficiency metrics are framed by token usage and API pricing, not full production deployment costs.

Implications and Future Directions

The empirical results demonstrate that execution control profoundly shapes outcome distribution in LLM-powered modernization workflows. Deterministic orchestration constrains error drift, improves worst-case behavior, and minimizes cost, making it preferable for mature, verifiable workflows. LLM-controlled strategies maximize solution frequency at the expense of efficiency and reproducibility. The findings indicate system architects should decouple generative reasoning from execution control in enterprise modernization tasks, embedding LLMs within deterministic frameworks to attain stable, scalable, and economically viable solutions.

Hybrid orchestration architectures, combining deterministic pipelines with bounded agentic interventions, warrant further exploration. Extending this analysis to heterogeneous environments, mixed-language dependencies, and real-world operational constraints remains an important direction. Long-horizon execution, runtime envelope, and production deployment tradeoffs should be systematically characterized to refine orchestration strategies for broader AI-driven software engineering.

Conclusion

This study rigorously isolates orchestration strategy within LLM-based COBOL-to-Python modernization, revealing that deterministic execution achieves comparable correctness, superior robustness, and lower operational cost relative to agentic LLM-controlled orchestration. The research clarifies the value of fixed execution policies in structured modernization workflows and underscores execution control as a critical systems design parameter. Future work should generalize these results to alternative domains and hybridize orchestration modes to balance adaptivity, reproducibility, and efficiency in large-scale AI-driven software modernization.