Object Aligner: A Configurable JSON Schema Similarity Score for Graphs, Applied to LLM Prompt Optimization

Abstract: LLMs are often asked to produce JSON conforming to a fixed schema, powering information extraction, tool calling, agentic planning, and knowledge-graph construction. Measuring how closely an output matches a gold reference is essential yet surprisingly hard: exact match is brittle, text similarity ignores structure, and an LLM judge is expensive, opaque, and non-deterministic. We address this with Object Aligner (OA), an open-source Python library that scores two JSON objects deterministically by recursively aligning their trees (the Hungarian algorithm for unordered collections, sequence alignment for ordered ones) and awarding partial credit at the granularity the schema declares. The Object Aligner is configured entirely through a set of JSON Schema extensions, so adapting it to a new task involves annotating a schema rather than writing code. Complex structured data, however, are rarely flat trees: records may form graphs or hypergraphs keyed by arbitrary identifiers, breaking the assumptions of prior similarity metrics. Our central contribution, referential alignment, closes this gap by inferring a bijection between gold and candidate identifiers and scoring every reference through it, so the score is invariant to relabeling. Since recovering this bijection exactly is graph isomorphism, the Object Aligner approximates it with Weisfeiler-Leman color refinement. An order-sensitive sequence regime targets ranking and planning. Since the same alignment localizes every mismatch, the Object Aligner emits ranked repair suggestions at no extra cost. Used as a reward inside the GEPA prompt optimizer, Object Aligner helps or stays neutral across all datasets.

• The paper introduces a schema-driven similarity function that robustly aligns JSON graphs using recursive, referential, and order-sensitive matching.
• Its methodology integrates optimal assignment, sequence alignment, and Weisfeiler–Lehman refinement to overcome challenges in identifier mapping.
• Empirical results show improved prompt optimization with reproducible, actionable feedback and deterministic scoring in structured LLM outputs.

Object Aligner: Schema-Driven Graph Alignment and Evaluation for Structured LLM Outputs

Introduction and Motivation

Structured data extraction and evaluation for LLM-generated outputs present acute challenges due to the complexity, nesting, and referential nature of real-world objects, particularly when encoded as JSON with arbitrary identifier assignment and potentially graph-like cross-references. Common metrics such as exact match or text similarity do not adequately capture structural agreement, and LLM-based judges, although expressive, are non-deterministic and expensive, complicating reproducibility and large-scale prompt optimization.

Object Aligner (OA) (2607.01972) introduces a schema-driven, deterministic similarity function over JSON objects, equipping it to robustly compare structured predictions to references for use as a reward in prompt optimization and model evaluation. The OA framework (i) aligns nested structures, (ii) infers and applies identifier bijections to enable graph and hypergraph matching up to isomorphism, and (iii) provides decomposable, auditable feedback for downstream optimization.

Methods: Recursive Alignment, Referential Invariance, and Schema Configuration

The OA scoring function operates on JSON-schema-typed value trees. Leaves (primitives) are scored with user-supplied comparators (e.g., exact match, Jaro distance, or inverse difference for numerics), while sequences and maps (objects) are recursively matched and scored. Crucially, OA supports per-field configuration: schemas declare whether list order matters (set vs. sequence matching), how to compare keys and values, and which fields represent identifiers or references.

For unordered collections, OA performs optimal assignment (Hungarian algorithm) according to child-similarity scores; for ordered collections, sequence alignment dynamic programming is used.

To handle the pervasive challenge of arbitrary identifier assignment in graphs (where node ids are arbitrary but structure must be preserved), OA introduces referential alignment by:

• Using schema annotations (idScope, ref) to declare identifier scopes and references.
• Masking identifiers in matching, then optimally aligning nodes by content and structure.
• Inferring a gold-to-candidate bijection, using Weisfeiler–Leman (WL-1) color refinement jointly over gold and candidate graphs to disambiguate structurally identical (property-twin) nodes.
• Scoring references through this bijection, making the score invariant to relabeling and faithful to actual structure, not superficial ids.

Figure 2: Joint tie-break 1-WL refinement over the disjoint union $\mathcal G_g\sqcup\mathcal G_c$; the relabeling process disambiguates property-twin nodes for structurally faithful alignment.

Finally, OA decomposes scores fully, enabling the extraction, localization, and prioritization of “repair” suggestions ranked by the score impact of each possible correction.

Intrinsic Validation of Alignment and Robustness

Intrinsic evaluation of OA exhibits two key properties: robust invariance and sensitivity to structural damage. On synthetic graph extraction tasks (Org2Graph), OA reference-alignment achieves a score of 1.0 (variance 0) when comparing gold outputs to identifier-relabeled copies, perfectly resistant to all permutations, even in the adversarial presence of many property-twin nodes.

Conversely, genuine perturbations—record/edge deletions, reference rerouting, or property mutation—result in monotonic degradation of scores, with Spearman correlations between number of edits and score ranging from $-0.66$ to $-0.89$. The ablation in which idScope/ref is omitted (i.e., comparing raw identifiers) suffers substantial, arbitrary score penalties in the relabel-only setting and is blind to pure reference rewirings.

Figure 1: OA’s damage sensitivity: RA is invariant under relabeling but drops monotonically with real edits, unlike value-based baselines that miss structural errors such as reference rerouting.

Comparable validation for the order-sensitive sequence regime shows that OA provides a smooth grading function with respect to Kendall distance from the gold permutation, in contrast to all-or-nothing exact match or order-agnostic matching.

Figure 5: OA sequence alignment: partial credit scales with disorder (Kendall distance), in contrast to exact match or order-agnostic set-based alignment.

Prompt Optimization and Empirical Results

OA’s intended core use is as a reward and feedback generator within automated prompt optimization for LLM-extracted JSON outputs. Using GEPA—a sample-efficient, reflective prompt optimizer—experiments span synthetic and real-world tasks with graph, order, or mixed output structure. Two axes are interrogated:

• Referential alignment (RA): schema reward either includes or omits idScope/ref for referential invariance.
• Order regime: sequence (order-sensitive) vs. set (order-agnostic) alignment, again as controlled by schema.

Three main findings:

1. RA is essential when surface attributes cannot individuate records: In synthetic “coded” Org2Graph, where node identifiers are obfuscated, the RA-enabled reward increases holdout alignment scores by up to 0.21 (absolute), with significance concentrated in the feedback arm. In contrast, on real datasets (SciERC, BioRED), where surface content suffices for node disambiguation, RA offers little benefit.
2. Order sensitivity matters only when output order determines task success: On synthetic Facts2Order (compositionally generated sort tasks), the sequence-vs-set reward difference exceeds 0.4 absolute points, reflecting that order is the information being tested. On NATURAL PLAN (planning), however, order is naturally induced by solution structure; the gain from order-sensitive reward is marginal.
3. Deterministic feedback is broadly useful, especially where structure is hidden: Experiments show that OA’s localized, ranked repair suggestions are crucial for optimizer discovery—without feedback, even property-aware rewards cannot drive optimization on structure-obfuscated tasks.

Figure 3: Contrast effects for referential alignment, order sensitivity, and feedback across datasets and model capacities.

Figure 8: Feedback breadth sweep: adding even one ranked correction item in OA feedback nearly matches the performance of gold-reference reflection across tasks.

Practical and Theoretical Implications

Practically, OA enables deterministic, auditable metric-guided optimization in contexts where existing metrics fall short. It can immediately support any structured output task whose schema can be annotated, including information extraction, function/tool call validation, and structured document or plan synthesis for downstream consumption.

Theoretically, OA demonstrates that for graph- and sequence-structured outputs, robust and actionable measurement can be achieved using standard combinatorial optimization (Hungarian, sequence alignment), generic graph invariants (WL-1), and schema-driven configuration—without LLM-in-the-loop evaluation.

OA’s design also reveals that reward measurement and feedback must be tailored: a reward detecting subtle structure is only actionable if there exists meaningful gradient for the optimizer, which in turn depends on the nature of the task, the optimizer architecture, and the expressiveness of feedback.

Limitations and Future Directions

OA’s identifier mapping is currently limited by the expressiveness of WL-1, and thus fails to distinguish certain isomorphic but structurally ambiguous cases (e.g., regular $k$-cycles). Higher-order coloring (WL-$k$) could strengthen bijection inference but introduces computational challenges. Another limitation is the handling of cycles in mutually referential scopes, which revert to attribute-based alignment.

In terms of applicability, OA’s ordinal nature (scores are not globally calibrated probabilities) could limit certain downstream uses, and differentiation for end-to-end neural training is not directly supported. Potential future directions include:

• Application to text-to-database or database change-diffing, exploiting OA's natural fit with normalized relational structures.
• Development of differentiable approximations (e.g., Sinkhorn assignments) for gradient-based optimization.
• Extension to higher-order graph invariants and fixed-point inference for mutually referential or cyclic scope structures.

Conclusion

Object Aligner represents a robust, schema-declarative, open-source approach to evaluating and guiding LLM-generated structured outputs, fully compatible with graph and hypergraph data, supporting partial credit, and providing localized, actionable feedback for prompt optimization. OA’s referential alignment, order-sensitivity, and decomposable feedback collectively enable deterministic, scalable, and auditable structured-output optimization, making it a viable alternative to LLM-based judges and a general tool for structured data evaluation in advanced LLM systems.