Compiler-as-a-Judge Module (CJM) Overview

• Compiler-as-a-Judge Module (CJM) is an automated evaluation mechanism that fuses traditional compiler processes with modern LLM-based judging to assess artifacts.
• It leverages stages like scanning, parsing, and code generation to translate legal texts and validate code correctness with structured, executable feedback.
• CJM bridges formal compiler techniques with modular evaluation pipelines, offering verifiable and scalable judgment across software and legal domains.

Compiler-as-a-Judge Module (CJM) denotes a class of automated evaluation mechanisms that use compiler-style structure, direct compiler feedback, or modular judge architectures to determine validity, correctness, compliance, or desirability of artifacts. Across the literature, the term spans at least three closely related usages: a compiler program for validating and translating Arabic juridical documents into machine-readable XML; LLM-based judging pipelines for compiler validation, verification, and code correctness; and compiler-grounded reward systems, most explicitly in text-to-CAD, where compilation and geometric verification provide a fast and verifiable judgment signal (Bassil, 2019, Sollenberger et al., 2024, Zhou et al., 6 Aug 2025). In parallel, the broader LLM-as-a-Judge literature formalizes evaluation as a mapping from an artifact and its context to labels, scores, explanations, or feedback, which suggests that CJM can be understood as a domain-specialized instantiation of automated judging with unusually strong reliance on executable structure, formal constraints, and pipeline composition (Gu et al., 2024, 2503.02246).

1. Conceptual scope and formalization

CJM is not presented in the literature as a single standardized implementation. Rather, it appears as an explicit module in some works, an extension target in others, and a conceptual analogue in broader judge-system engineering. In software-evaluation terms, LLM-as-a-Judge has been formalized as

$$\mathcal{E} \leftarrow \mathcal{P}_{\mathcal{LLM}}(x \oplus \mathcal{C}),$$

where the evaluation outcome depends on the artifact $x$ and contextual information $\mathcal{C}$, and software-engineering work further describes the judge as

$$E(\mathcal{T}, \mathcal{C}, \mathcal{X}, \mathcal{R}) \rightarrow (\mathcal{Y}, \mathcal{E}, \mathcal{F}),$$

with evaluation type, criteria, items, and optional references producing a result, explanation, and feedback (Gu et al., 2024, 2503.02246).

Within that broader evaluator model, CJM is distinguished by its dependence on compiler-derived structure. In one lineage, the module is literally compiler-based: scanner, parser, and code generator. In another, it is compiler-informed: compilation return codes, STDERR/STDOUT, execution outputs, and reference-tool feedback are injected into an LLM judging pipeline. In a third, compiler behavior becomes a reward oracle, as in CAD-Judge, where the generated sequence is compiled, reconstructed, and compared geometrically against ground truth without rendering or external visual ranking (Bassil, 2019, Sollenberger et al., 2024, Zhou et al., 6 Aug 2025).

This suggests that CJM occupies an intermediate position between classical formal validation and contemporary LLM-based evaluation. It is neither merely a conventional compiler pass nor merely a free-form LLM verdict. Its defining tendency is to ground judgment in structured transformations, executable feedback, or formally checkable intermediate representations.

2. Compiler antecedent in juridical document validation

An early compiler-style antecedent appears in work on legal document translation for digital government, where a compiler program automates the validation and translation of Arabic juridical documents from unstructured plain-text into structured XML (Bassil, 2019). The architecture is explicitly composed of a scanner, a parser, and a code generator. The scanner tokenizes raw text into components such as keywords, numbers, and strings using Deterministic Finite Automata and Regular Expressions; the parser validates syntactic structure using a Context-Free Grammar and outputs a parse tree; the code generator maps the parse tree into target XML using synthesized and inherited attributes in translation rules (Bassil, 2019).

The tokenization layer is tailored to Arabic legal language. Numeric tokens are recognized by a DFA over digits, string tokens by a DFA over letters and relevant symbols, and Arabic labels such as مرسوم, قرار, رقم, and إن are mapped to token types including TYPE, RAQM, and INNA. The parser then checks productions such as

$$document \rightarrow statement\ title\ issuer\ ref\_list\ just\_list\ acknowledge\ article\_list\ loc\_date\ sig\_list$$

and related nonterminals for articles, titles, issuers, and signatures, thereby reconstructing canonical legal structure from loose or inconsistent real-world phrasing (Bassil, 2019).

In encyclopedic CJM terms, the significance of this system lies in its transformation of validation into a compiler problem. The parser flags violations of syntax, including misplaced or missing components, and the output XML mirrors the logical and legal structure of the original text. The work also states that the compiler enforces rules of document structure and can therefore serve as an automated validator, described as a “digital judge” for legal document admissibility and compliance (Bassil, 2019). At the same time, its limitations are explicit: the system can flag syntax errors but not semantic errors or nuanced legal inconsistencies, and simple CFGs may miss implicit elements or misinterpret unusual phrasings common in varied legal practice (Bassil, 2019). A common misconception is therefore to treat compiler-style admissibility checking as equivalent to full legal interpretation; the reported system does not support that conclusion.

3. CJM in compiler validation, verification, and coding tasks

The most direct software-oriented CJM usage appears in LLM4VV, which explores LLM-as-a-Judge for validation and verification test suites targeting compiler implementations of directive-based programming models such as OpenMP and OpenACC (Sollenberger et al., 2024). The framework constructs two groups of test cases through negative probing: one in which valid code is intentionally injected with specific errors, and one containing unchanged valid code. The judge is prompted with explicit criteria, including Syntax, Directive Appropriateness, Clause Correctness, Memory Management, Compliance, and Logic, and is required to emit an exact phrase such as FINAL JUDGEMENT: valid or FINAL JUDGEMENT: invalid so that downstream automation can parse the result deterministically (Sollenberger et al., 2024).

A central architectural contribution is the agent-based prompting approach. Before adjudication, each code file is compiled by a reference compiler, compiler return codes and STDERR/STDOUT are captured, and, if compilation succeeds, the test is executed and the execution outputs are recorded. Only tests that pass compilation and execution advance to the LLMJ stage in a multi-stage pipeline. This structure reduces unnecessary LLM calls, increases throughput with parallelism, and supports synchronous or asynchronous, single- or multi-threaded, GPU- or CPU-based deployment (Sollenberger et al., 2024). The reported results state that agent-based prompting with compilation and execution outputs drastically increases accuracy and reliability, with OpenACC pipeline accuracy at approximately 92% and OpenMP pipeline accuracy at approximately 94–96%, while also reducing positive bias relative to standalone LLM judging (Sollenberger et al., 2024).

A broader benchmark perspective is provided by CodeJudgeBench, which evaluates 26 LLM-as-a-Judge models on 4,260 instances spanning code generation, code repair, and unit test generation (Jiang et al., 14 Jul 2025). The benchmark finds that recent thinking models significantly outperform non-thinking models on code judging tasks, and that even relatively small thinking models, such as Qwen3-8B, can outperform specially trained LLM-as-a-Judge models up to 70B in size (Jiang et al., 14 Jul 2025). Yet all models still exhibit significant randomness, and simply changing the order in which responses are presented can substantially impact accuracy in pairwise judging. The benchmark further reports that pair-wise comparison outperforms scalar point-wise judging, and that retaining comments and reasoning in the full, unprocessed LLM response improves judge performance (Jiang et al., 14 Jul 2025). For CJM design, these findings make clear that code judging is not reliably solved by a single undifferentiated prompt.

4. Modularization, search, and judge-time compute

Recent work increasingly treats CJM not as a monolithic classifier but as a compositional system. One strand builds automated benchmark generation. In work on reliable LLM judgment for code tasks, a directed multigraph $G = (V, E, L)$ represents possible generations among artifact types, with vertices for artifacts and labeled edges for transformations performed by either trusted or tested LLMs (Farchi et al., 2024). Paths and cycles in $G$ are used to formulate self-consistency claims such as translation or summary equivalence, and these claims are then used both to develop candidate judges and to validate them. The paper reports experiments in which LaaJ achieves up to 100% accuracy distinguishing between matching and different summary pairs (Farchi et al., 2024). For CJM, the important contribution is methodological: judgments are anchored in graph-derived expectations rather than only in surface preference labels.

A second strand focuses on judge engineering as architecture search. “Engineering AI Judge Systems” proposes a four-stage framework: Creation of General Constitution, Specialization to Contextualized Constitution, Searching Cognitive Architectures, and Evolving the Judge (Lin et al., 2024). Its Stage III cognitive architecture module is described as akin to a Compiler-as-a-Judge Module in its modular, pluggable approach, searching over jury foundation models, judging heuristics, prompts, metrics, and reference-based or reference-free structures. The case study reports that the AI judge system developed with the framework outperforms the one developed without it by up to 6.2%, and that a significant proportion of principles from the general constitution are reused in contextualized constitutions, with mean reuse around 58% across languages (Lin et al., 2024). The contribution here is to make CJM a reusable engineering artifact rather than an isolated prompt.

Agentic decomposition appears again in Auto-Eval Judge, where evaluation proceeds through a Criteria Generator, Artifact Content Parser, Criteria Check Composer, and Verdict Generator (Bhonsle et al., 7 Aug 2025). The framework decomposes tasks into binary checklist items, retrieves “proof” snippets from reasoning traces and logs, classifies checks as factual, reasoning, or coding, and aggregates evidence-backed decisions into a final verdict. On GAIA and BigCodeBench, the judge agent achieves 4.76% and 10.52% higher alignment accuracy, respectively, than a GPT-4o-based LLM-as-a-Judge baseline (Bhonsle et al., 7 Aug 2025). This shifts CJM toward stepwise, evidence-backed evaluation of process as well as final output.

Judge-time scaling extends this modular trend. Verdict introduces Units, Layers, and Pipelines, with modular reasoning units for verification, debate, and aggregation, and reports state-of-the-art or near-state-of-the-art performance across content moderation, fact-checking, and hallucination detection (Kalra et al., 25 Feb 2025). MCTS-Judge pushes further by explicitly extending CJM with Monte Carlo Tree Search for code correctness evaluation (Wang et al., 18 Feb 2025). It decomposes a task $x=(c,p)$ into subtasks, uses a node-selection strategy that combines LLM-driven self-assessment with the Upper Confidence Bound for Trees,

$$\mathrm{UCT}(s)=\frac{Q(s)}{N(s)}+\alpha \cdot \sqrt{\frac{\ln N_{\mathrm{parent}(s)}}{N(s)}},$$

and applies a unit-test-level reward mechanism based on LLM-driven simulated execution (Wang et al., 18 Feb 2025). On three benchmarks and five LLMs, it improves a base model’s accuracy from 41% to 80% and surpasses the o1-series models with 3x fewer tokens (Wang et al., 18 Feb 2025). A plausible implication is that CJM is evolving from a fixed validation module into a search-based reasoning system whose reliability can be traded against test-time compute.

5. Reward-driven CJM and geometric verification

CAD-Judge provides the clearest explicit formulation of CJM as a reward system outside ordinary source-code compilation (Zhou et al., 6 Aug 2025). In text-to-CAD generation, the module acts as a fast, interpretable, and local reward signal for preference grading and grammatical validation. Given a text prompt $x$, a generated CAD sequence $x$0, and a ground-truth sequence $x$1, both $x$2 and $x$3 are compiled into 3D models through a CAD compiler such as pythonOCC, tessellated to meshes, and sampled into point clouds $x$4 and $x$5. The judge then computes Chamfer Distance,

$x$6

and assigns a binary preference signal: if the sequence compiles and the distance is below a threshold, the sample is desirable; otherwise it is undesirable (Zhou et al., 6 Aug 2025).

This formulation is explicitly contrasted with VLM-based ranking systems that require rendering and may introduce reward hacking. CAD-Judge instead uses rule-based binary preference grounded in compilation and geometry, and couples that signal to prospect-theory-inspired binary utility maximization through KTO (Zhou et al., 6 Aug 2025). The paper reports that CAD-Judge outperformed state-of-the-art models on the DeepCAD dataset, with F1 scores around 0.99 for lines, arcs, circles, and extrusions, lowest median Chamfer Distance of 0.15, and Invalidity Ratio of 1.38% (Zhou et al., 6 Aug 2025). In an ablation table on reward systems, rule-based binary preference identified with CJM achieves the best overall results, with F1(Line) 0.89, F1(Arc) 0.49, median CD 0.62, and IR 1.54% (Zhou et al., 6 Aug 2025).

The theoretical importance of this case is that it decouples “compiler” from ordinary source-language translation while preserving the core CJM property: judgment is produced by compiling structured sequences into a verifiable intermediate or final form and then applying explicit decision rules. This broadens CJM from code assessment to morphological grading and verification.

6. Reliability, bias, and calibration

The CJM literature repeatedly emphasizes that automated judges remain fragile. The LLM-as-a-Judge survey identifies position bias, length or verbosity bias, self-enhancement bias, concreteness or citation bias, overconfidence, and vulnerability to adversarial attacks, and recommends mitigation strategies such as swapping evaluation order, aggregating over multiple orderings, requiring explanations, and using post-processing for stability (Gu et al., 2024). In coding-specific evaluation, CodeJudgeBench confirms that pairwise judgments are sensitive to response order and that model performance varies depending on which LLM generated the candidate code or tests (Jiang et al., 14 Jul 2025). These findings constrain claims that CJM already provides stable human-equivalent judgment.

A more formal statistical critique appears in “Bias and Uncertainty in LLM-as-a-Judge Estimation,” which argues that raw judge outputs are systematically biased unless the judge is perfect (Fiedler, 7 May 2026). For binary outcomes, with judge sensitivity $x$7, specificity $x$8, and Youden’s Index $x$9, the Rogan-Gladen correction is

$\mathcal{C}$0

The paper shows that low $\mathcal{C}$1 makes corrected estimates unstable and that shared calibration across compared models can induce severe bias when cross-model calibration instability $\mathcal{C}$2 is nonzero, including sign reversal in model comparisons with high apparent confidence (Fiedler, 7 May 2026). Its reporting guidance therefore requires explicit disclosure of estimator choice, calibration design, confidence intervals, and per-model diagnostics $\mathcal{C}$3, $\mathcal{C}$4, $\mathcal{C}$5, together with $\mathcal{C}$6 for comparisons (Fiedler, 7 May 2026). For CJM, this means that evaluation modules must be assessed not only by raw accuracy but also by calibration stability.

Adversarial robustness is another unresolved issue. OBJEX(MT) studies objective extraction under multi-turn jailbreaks and finds that no tested judge is robust on hard adversarial data (Kim et al., 23 Aug 2025). Claude-sonnet-4 attains the highest extraction accuracy at 0.515 and best calibration with ECE 0.296 and Brier 0.324, while gpt-4.1 and Qwen3 both reach 0.441 accuracy but show marked overconfidence, with mean confidence around 0.88 and [email protected] around 48–52% (Kim et al., 23 Aug 2025). The paper’s operational guidance is to provide judges with explicit objectives when possible and to use selective prediction or abstention to manage risk (Kim et al., 23 Aug 2025). In CJM settings that involve latent-goal inference rather than direct artifact checking, that recommendation is especially salient.

7. Training strategies and research trajectory

One response to these reliability problems is to train judges as a general ability rather than as a narrowly isolated evaluator skill. “Improve LLM-as-a-Judge Ability as a General Ability” proposes a two-stage pipeline comprising supervised fine-tuning warm-up and direct preference optimization enhancement (Yu et al., 17 Feb 2025). The SFT stage teaches judge style adaptation and step-by-step reasoning over paired responses, while the DPO stage focuses on hard judge scenarios and incorporates a small NLL regularizer. The work also introduces an efficient data synthesis method based on instruction template rewriting, stepwise reasoning, bias control through answer swapping and length balancing, and rule-based filtering (Yu et al., 17 Feb 2025). Using only about 2% to 40% of the data required by other methods, the resulting RISE-Judge-Qwen2.5-32B reaches a RewardBench average score of 92.7, matching the best score in the reported comparison while using far less training data, and its judge signals improve downstream DPO training of internal policy models (Yu et al., 17 Feb 2025). This suggests that future CJMs may be trained not only for task-specific verification but also for generalized judgmental reasoning.

The forward-looking software-engineering roadmap frames LLM-as-a-Judge as a path toward reliable, robust, and scalable human surrogates for evaluating software artifacts by 2030 (2503.02246). It identifies the need for large-scale, diversified, high-quality human-annotated benchmarks; standardization of empirical protocols; stronger domain expertise; hybridization with classical software-engineering tools such as static analyzers and model checkers; and systematic research on adversarial testing and defense (2503.02246). In that framing, CJM becomes a future class of integrated modules for compilers, CI/CD pipelines, and developer environments, capable of multi-faceted assessment over correctness, readability, maintainability, and utility (2503.02246).

Taken together, the literature presents CJM less as a finished artifact than as an organizing principle for automated judgment. Its most mature forms are those with strong external grounding: formal grammars, compilation and execution traces, graph-derived consistency claims, explicit reward rules, or calibrated judge-time search. Its most persistent limitations are likewise consistent across domains: semantic blind spots, order sensitivity, bias, overconfidence, and calibration fragility. The current state of the field therefore supports a precise but limited conclusion: CJM is a viable architecture for structured automated judgment, but only when its verdicts are anchored in verifiable evidence, modularized for analysis, and evaluated with explicit attention to bias and uncertainty (Bassil, 2019, Sollenberger et al., 2024, Zhou et al., 6 Aug 2025).