TeleJudge: Hierarchical LLM Judge for Telecom Patches

• TeleJudge is a hierarchical LLM-based semantic evaluation framework designed to assess commit-level telecom patches with file-level granularity.
• It employs a multi-layered approach integrating diff segmentation, file-level judgments, and holistic meta-judging to complement standard unit tests.
• TeleJudge utilizes a semantic-similarity minus penalty scoring method to capture cross-file inconsistencies and enhance the Ship-Ready Percentage (SRP).

Searching arXiv for the specified paper to ground the article and verify metadata. TeleJudge is a hierarchical LLM-as-a-Judge framework for Diff-level and Commit-level evaluation in TeleSWEBench, a commit-driven benchmark for evaluating LLM-powered software engineering in telecommunications (Gajjar et al., 3 Jun 2026). It operates after an ASE agent produces a multi-file patch and provides a semantic and contextual assessment of the candidate modification at the file level and the commit level. Within TeleSWEBench’s two-stage pipeline, TeleJudge complements standard unit-test evaluation rather than replacing it: on the Scaled Split it alone produces Pass/Fail, while on the Verified Split it runs in parallel to the native unit-test harness, and the joint condition defines the Ship-Ready Percentage (SRP) (Gajjar et al., 3 Jun 2026).

1. Position within TeleSWEBench

TeleJudge is introduced in the setting of TeleSWEBench, which evaluates agent performance on real developer commits mined from the srsRAN 5G repository and organized into structured test cases across three difficulty tiers: Easy, Medium, and Difficult (Gajjar et al., 3 Jun 2026). TeleSWEBench consists of 734 questions accompanied by executable unit tests, but the benchmark also includes a Scaled Split for which no unit tests are available. In that setting, a semantic judge is necessary because test-based evaluation cannot be applied directly (Gajjar et al., 3 Jun 2026).

The framework is therefore positioned as an evaluation mechanism for specialized, mathematically rigorous wireless stacks in which general-purpose coding benchmarks are insufficient. The paper states that contemporary telecom networks now function as immensely intricate and heavily softwareized codebases, and that automated software engineering tools and SWE Agents must be evaluated against stateful logic and strict requirements that are specific to telecommunications (Gajjar et al., 3 Jun 2026). TeleJudge addresses this gap by following an evaluation based on context and semantic similarity in parallel to a standard unit test-based evaluation.

A plausible implication is that TeleJudge is intended to operationalize semantic review for commit-like artifacts in a domain where exact executable checks are either incomplete or unavailable. The paper’s framing emphasizes deployability rather than mere test passing, which explains why TeleJudge is coupled to SRP rather than treated as a standalone score (Gajjar et al., 3 Jun 2026).

2. Multi-layer architecture

TeleJudge has three logical layers that process a candidate patch after generation by an ASE agent (Gajjar et al., 3 Jun 2026).

Layer	Input	Output
Diff Segmentation Layer	question, ground-truth developer diff, agent’s proposed diff	per-file hunks $\{f_1,\ldots,f_m\}$
File-Level Judge (Tier 1)	question, $\Delta_i^{GT}$, $\Delta_i^C$	per-file JSON with verdict, confidence, reasons
Holistic Meta-Judge (Tier 2)	question plus list of file-level verdicts	final commit-level JSON verdict

In the Diff Segmentation Layer, the input is the original “question” understood as the distilled commit intent, together with the ground-truth developer diff and the agent’s proposed diff. The task is to split both the ground truth and the candidate into per-file hunks $\{f_1,\ldots,f_m\}$ (Gajjar et al., 3 Jun 2026).

In the File-Level Judge, a code-aware LLM is invoked with prompt P3. For each file $f_i$, the prompt receives the single-file ground-truth hunk $\Delta_i^{GT}$, the agent’s candidate hunk $\Delta_i^C$, and the question. The per-file output is a verdict and a confidence:

• $verdict_i \in \{\text{“accept”}, \text{“reject”}\}$
• $confidence_i \in [0,100]$

These are encoded as JSON of the form:

$\Delta_i^{GT}$0

The Holistic Meta-Judge then collects all $m$ file-level JSON verdicts and invokes a second LLM with prompt P4, passing the question together with the list of per-file verdicts and reasons. Its output is a final commit-level JSON verdict:

$\Delta_i^{GT}$1

This two-tier structure is explicitly hierarchical. It first localizes judgment at the granularity of individual files and then aggregates these judgments holistically at the commit level. This suggests that TeleJudge is designed to reduce the rigidity of a single exact-match criterion while preserving file-sensitive analysis of multi-file telecom changes (Gajjar et al., 3 Jun 2026).

3. File-level scoring formulation

Although the implementation uses LLM classification for file-level judgments, the paper states that TeleJudge’s internal scoring can be viewed as a continuous “semantic-similarity minus penalty” function (Gajjar et al., 3 Jun 2026). In the formulation given, let the ground-truth diff for file $\Delta_i^{GT}$0 be $\Delta_i^{GT}$1 and the candidate diff be $\Delta_i^{GT}$2.

Code embeddings are computed as

$\Delta_i^{GT}$3

$\Delta_i^{GT}$4

and the cosine similarity is

$\Delta_i^{GT}$5

typically clamped to $\Delta_i^{GT}$6 (Gajjar et al., 3 Jun 2026).

A structural penalty is then defined via normalized edit distance. Let $\Delta_i^{GT}$7 and $\Delta_i^{GT}$8, and let

$\Delta_i^{GT}$9

The penalty is

$\Delta_i^C$0

These components are combined into the file-level score

$\Delta_i^C$1

where $\Delta_i^C$2 balances pure semantic match versus structural faithfulness (Gajjar et al., 3 Jun 2026).

In practice, the LLM’s confidence is mapped onto the file-level score, and the binary verdict is encoded as $\Delta_i^C$3 for “accept” and $\Delta_i^C$4 for “reject.” The unified file score is then

$\Delta_i^C$5

or, if using embeddings, the score follows the semantic-similarity-plus-penalty formulation above (Gajjar et al., 3 Jun 2026).

This dual presentation is important. It indicates that TeleJudge is not described merely as a free-form judge prompt; it is also framed as a structured scoring logic combining semantic similarity and structural faithfulness. A plausible implication is that the framework is intended to remain interpretable even when the actual implementation is LLM-mediated.

4. Aggregation, thresholding, and calibration

Let $\Delta_i^C$6 be the set of files in the ground-truth patch. After computing $\Delta_i^C$7 for each file, TeleJudge forms the commit-level score

$\Delta_i^C$8

The pass/fail decision is obtained by comparing $\Delta_i^C$9 against a threshold $\{f_1,\ldots,f_m\}$0 (Gajjar et al., 3 Jun 2026). The pseudocode given is:

$\Delta_i^{GT}$2

The paper further specifies possible normalization and calibration strategies because $\{f_1,\ldots,f_m\}$1 varies by difficulty tier. One option is tier-specific threshold calibration:

• $\{f_1,\ldots,f_m\}$2
• $\{f_1,\ldots,f_m\}$3
• $\{f_1,\ldots,f_m\}$4

An alternative is Z-score normalization within each tier:

$\{f_1,\ldots,f_m\}$5

with acceptance if $\{f_1,\ldots,f_m\}$6 (Gajjar et al., 3 Jun 2026).

The paper then qualifies this formalization by stating that, in the actual two-tier LLM implementation, the meta-judge LLM P4 effectively performs this weighted aggregation and thresholding in natural language. The explicit formulas therefore describe the underlying logic rather than the literal inference path of the deployed judge (Gajjar et al., 3 Jun 2026).

This distinction matters for interpretation. TeleJudge is simultaneously a practical prompt-based system and a conceptual scoring model. The formal aggregation clarifies what the meta-judge is intended to approximate: commit-level acceptance as an aggregate over file-level semantic adequacy.

5. Parallelism with unit-test evaluation and SRP

On the Verified Split, TeleJudge operates in parallel with Stage 2 functional correctness evaluation on the subset of tasks with exact-match localization (EM) (Gajjar et al., 3 Jun 2026). The paper defines two parallel signals.

The Unit-Test Acceptance Rate (UAR) is

$\{f_1,\ldots,f_m\}$7

where $\{f_1,\ldots,f_m\}$8 is the count of patches that compile and pass all repository-native tests.

The TeleJudge Acceptance Rate (TAR) is

$\{f_1,\ldots,f_m\}$9

where $f_i$0 is the number of patches judged “accept” by TeleJudge (Gajjar et al., 3 Jun 2026).

The Ship-Ready Percentage is then defined by the joint condition

$f_i$1

Accordingly,

$f_i$2

The paper states that this enforces that a patch must both satisfy strict executable semantics and stand up to semantic/contextual scrutiny across files (Gajjar et al., 3 Jun 2026). It further states that TeleJudge catches cases where a candidate passes the unit tests via narrow hacks but breaks higher-level invariants.

This formulation is central to TeleJudge’s role. It is not merely a surrogate metric for missing tests; it is a semantic safety net around standard unit-test evaluation. A common misconception would be to treat a passing test suite as sufficient evidence of deployability. The benchmark explicitly rejects that equivalence by requiring both test success and TeleJudge acceptance for SRP (Gajjar et al., 3 Jun 2026).

6. Empirical behavior, strengths, and limitations

The paper reports several empirical findings for TeleJudge across backbones and difficulty tiers (Gajjar et al., 3 Jun 2026). Across all backbones and difficulty tiers, TeleJudge’s TAR is typically 5–10 points below UAR on Easy tasks, and the gap widens to 15–20 points on Difficult tasks. The stated interpretation is that LLM-based semantic review catches “test-only” hacks.

A concrete example is given for QwenCoder 2.5: it achieves $f_i$3 but $f_i$4 on EM tasks. The paper states that the model often hardcodes to satisfy individual tests but breaks cross-file consistency, which TeleJudge flags (Gajjar et al., 3 Jun 2026).

The strongest model reported is GLM-4.7, which records $f_i$5, $f_i$6, and $f_i$7. The paper attributes this pattern to “extreme caution (high NC in Stage 1),” which trades off recall for high precision in Stage 2 (Gajjar et al., 3 Jun 2026). In the broader benchmark summary, the strongest ASE tools achieve up to 25% of shippable changes, which is consistent with the SRP framing (Gajjar et al., 3 Jun 2026).

The strengths of TeleJudge are explicitly listed as follows:

• it scales to commits without unit tests (Scaled split);
• it captures cross-file semantic consistency;
• it exposes test-passing but semantically broken patches (Gajjar et al., 3 Jun 2026).

The limitations and future work are also explicitly stated:

• context-window exhaustion on very large repos, mitigated by file-by-file prompting;
• occasional over- or under-sensitivity to trivial style changes, addressable by tuning $f_i$8 or adopting more robust diff embeddings;
• manual per-tier threshold calibration, with the suggestion that one can learn $f_i$9 via a small held-out set (Gajjar et al., 3 Jun 2026).

Taken together, these findings characterize TeleJudge as a semantic review mechanism that is stricter than unit-test acceptance in some regimes and differently sensitive in others. This suggests that its primary contribution is evaluative selectivity: it distinguishes patches that merely satisfy available tests from patches that remain coherent under file-level and commit-level semantic scrutiny.