Constraint-Aware Retrieval Module (CARM)

• CARM is a constraint-aware retrieval tool that extracts logical constraint profiles to enhance in-context learning in constraint programming.
• It leverages a fixed constraint ontology and a dedicated type extractor to select high-quality exemplars via Jaccard similarity of constraint sets.
• Empirical results show CARM outperforms dense retrieval methods, yielding up to a 19% gain on benchmarks and improving model fidelity in industrial CP tasks.

The Constraint-Aware Retrieval Module (CARM) is a retrieval mechanism integrated into neuro-symbolic LLM pipelines for constraint programming (CP), specifically designed to enhance formal modeling and solving of industrial-scale constraint optimization problems (COPs). Its defining feature is the analysis of the logical structure of natural language COP descriptions to extract a "constraint profile," which is then leveraged to retrieve in-context exemplars based on semantic constraint similarity rather than surface-level or embedding-based similarity. This approach targets improved in-context learning, code synthesis, and model repair for CP tasks by facilitating more precise and trustable neuro-symbolic AI workflows (Shi et al., 7 Oct 2025).

1. Motivation and Role Within ConstraintLLM

CARM was developed in the context of ConstraintLLM, a neuro-symbolic pipeline intended to automate the generation and solving of COPs at industrial scale. In typical retrieval-augmented generation (RAG) settings, retrieval relies on dense vector similarity, which may overlook distinctions fundamental to symbolic constraint modeling. CARM addresses this by focusing retrieval on the explicit logical structure of constraints (e.g., AllDifferent, Circuit, Cumulative) in the problem statement. Its retrieval mechanism operates at key phases within the ConstraintLLM pipeline:

• Initial modeling: Prior to model generation, CARM identifies $k$ solved cases whose constraint profiles closely resemble the input problem, providing high-quality exemplars.
• Tree-of-Thoughts (ToT): During iterative model construction, CARM supplies relevant modeling patterns, constraint formulations, and variable definitions contextualized to the current partial model.
• Iterative self-correction: In response to solver failures, CARM re-ranks and selects correction exemplars that best match the error context in terms of constraint structure, enabling targeted repair steps.

By infusing domain-level constraint semantics into all major stages of the neuro-symbolic pipeline, CARM is designed to increase final model fidelity and in-context reasoning depth in LLMs (Shi et al., 7 Oct 2025).

2. Architectural Components

CARM consists of three primary components:

1. Constraint Ontology ($\mathcal{O}$):
   • A fixed vocabulary of approximately 50 global and basic constraint types, comprising domain-standard primitives such as AllDifferent, Cumulative, Element, Circuit, NoOverlap, LexDecreasing, and Sum.
2. Constraint Type Extractor ($L_{\mathrm{analyzer}}$):
   • An auxiliary LLM (or fine-tuned variant) configured via a prompt $P$ that maps a natural language problem $Q_{\mathrm{NL}}$ to a constraint profile $C(Q) \subseteq \mathcal{O}$. This module enables semantic parsing requisite for retrieval, implemented as a specialization of the base model used in ConstraintLLM and trained on a constraint extraction dataset.
3. Retrieval Index and Similarity Scorer:
   • A static case library $$\mathcal{D} = \{(D_j,\,C(D_j),\,\mathrm{code}_j)\}_{j=1}^m$$, where $D_{j,\mathrm{NL}}$ is the problem’s natural language description, $C(D_j)$ its precomputed constraint profile, and $\mathrm{code}_j$ the associated CP model. Retrieval uses the Jaccard coefficient to measure overlap between constraint sets:

   $$\mathrm{Sim}\bigl(C(Q),C(D_j)\bigr) = \frac{\bigl|C(Q)\cap C(D_j)\bigr|}{\bigl|C(Q)\cup C(D_j)\bigr|} \in [0,1].$$

• This set-based retrieval is designed to yield exemplars that share maximal logical similarity with the query.

3. Retrieval Algorithm and Implementation

The retrieval algorithm follows a two-phase process: offline index-building and query-time retrieval.

Index-Building (Offline):

• $L_{\mathrm{analyzer}}$ processes each $D_{j,\mathrm{NL}}$ in $\mathcal{D}$ to compute $C(D_j)$. Each case is indexed as $(D_j, C(D_j), \mathrm{code}_j)$.

Query-Time Retrieval:

1. Compute constraint profile of input $Q_{\mathrm{NL}}$: $C(Q) = L_{\mathrm{analyzer}}(Q_{\mathrm{NL}};P)$.

2. For each exemplar, calculate $\mathrm{Sim}_j=\mathrm{Jaccard}(C(Q),C(D_j))$.

3. Rank $\{D_j\}$ by similarity.

4. Return the top-$k$ entries $$\{(D_{r_1},\mathrm{code}_{r_1}),\dots,(D_{r_k},\mathrm{code}_{r_k})\}$$.

Pseudocode:

function BuildIndex(Dataset D): index = [] for each case D_j in D: C_j = L_analyzer(D_j.text, prompt P) index.append((D_j, C_j, D_j.code)) return index function QueryCARM(index, Q_NL, k): Cq = L_analyzer(Q_NL, P) scores = [] for (D_j, Cj, code_j) in index: sim = |Cq ∩ Cj| / |Cq ∪ Cj| scores.append((sim, D_j, code_j)) top_k = select k entries of scores with highest sim return [(D_j, code_j) for (sim, D_j, code_j) in top_k]

Updating: Newly solved or corrected instances are appended to $\mathcal{D}$, storing $(Q,C(Q),\mathrm{code}_Q)$ for future benefit.

Typical implementation choices: $|\mathcal{O}|\approx 50$, $k=4$ for initial/ToT, $k=5$ for self-correction stage, Python set Jaccard for similarity, retrieval latency $\sim 0.17$ ms per query, in-memory tuple-based index. Embedding similarity in two-stage repair utilizes OpenAI’s text-embedding-ada-002.

4. Integration with In-Context Learning and Self-Correction

CARM orchestrates retrieval-driven prompt construction throughout the LLM pipeline:

• Prompt Construction (Initial Modeling): Top-$k$ retrieved exemplars are formatted as few-shot prompt entries, each pairing a reference problem and its CP code, followed by the user’s problem for code generation in PyCSP3.

• Tree-of-Thoughts (ToT): For each decision node within the search tree, the (partial) constraint profile is used to retrieve exemplars relevant to the model fragment being synthesized. This includes:

  • Choosing among global constraints (e.g., AllDifferent vs. Circuit)
  • Selecting variable definitions (arrays, domains)
  • Suggesting auxiliary constructs.

Retrieved patterns are provided as in-context examples, steering ToT exploration.

• Iterative Self-Correction: Upon solver failure, an error context $c_{\mathrm{err}}$ is formed. Self-correction proceeds in two retrieval stages:
  1. Embedding-based trimming: Top-$k$ candidate corrections are selected by text embedding cosine similarity.
  2. Constraint-aware re-ranking: Candidates are sorted using the Jaccard similarity between $c_{\mathrm{err}}$ and exemplar constraint profiles. The top-ranked exemplar is injected into the prompt, guiding repair for up to four iterations.

5. Training and Fine-Tuning

CARM’s core retrieval operation is heuristic, relying on set-based Jaccard ranking without learnable parameters. However, key submodules are improved by supervised fine-tuning:

• Constraint Type Extractor: Trained via cross-entropy loss to maximize $P(C_{\mathrm{gold}}|Q_{\mathrm{NL}})$.
• Base Modeling and Self-Correction Tasks: Fine-tuned on problem-to-code, and (problem, incorrect code, feedback, correct code) datasets. Objective is cross-entropy loss for correct code and repair path generation.

Training regime employs parameter-efficient fine-tuning (QLoRA + AdamW), learning rate $4\times10^{-4}$, 6 epochs, 500 warmup steps, batch size 12, gradient checkpointing, BF16 precision, and 4-bit quantization. The approach is instantiated on an open-source LLM such as Qwen2.5-Coder-32B (Shi et al., 7 Oct 2025).

6. Empirical Evaluation and Generalization

CARM is empirically validated on multiple CP and COP modeling benchmarks. Its performance is benchmarked against a cosine-similarity RAG baseline across four datasets in terms of solving accuracy (SA):

Benchmark	RAG (4-shot)	CARM (4-shot)	Gain
IndusCP	21.8%	40.0%	+18.2%
NL4OPT	88.6%	95.2%	+6.6%
LGPs	82.0%	91.0%	+9.0%
LogicDeduction	92.0%	96.0%	+4.0%

Ablation studies indicate an average relative gain of approximately 19% across all benchmarks for CARM relative to RAG. In cross-domain experiments, where retrieval is limited to IndusCP exemplars for other tasks, solving accuracy remains high (NL4OPT: 92.2%, LogicDeduction: 94.0%), reflecting generalization afforded by the constraint-profile-based retrieval. On the LGPs dataset, in-context learning (ICL) with static 4-shot Chain-of-Thought (CoT) yields solving accuracy of 32%, whereas CARM Top-4 achieves 89%.

These results support the conclusion that constraint-driven retrieval via CARM substantially enhances model synthesis and repair capacities in neuro-symbolic LLM settings, especially for industrial-scale COPs (Shi et al., 7 Oct 2025).

7. Significance and Limitations

CARM exemplifies a domain-aware retrieval paradigm advancing beyond generic vector-based approaches, yielding quantifiable improvements in industrially relevant CP scenarios. Its design and performance suggest broad utility for neuro-symbolic LLM frameworks tasked with structured code generation from natural language. A plausible implication is that constraint-profile-based retrieval architectures may generalize effectively across domains and tasks where symbolic structure is primary.

CARM’s reliance on a manually defined constraint ontology and auxiliary extraction model, however, means it requires domain-specific engineering and annotated data for full deployment. As CARM’s Jaccard-based retrieval is non-parametric, its effectiveness is contingent on both the quality of the indexed library $\mathcal{D}$ and the accuracy of the constraint extraction module. Future work might address dynamic expansion of the ontology and fully automated error-context extraction.

References:

ConstraintLLM: A Neuro-Symbolic Framework for Industrial-Level Constraint Programming (Shi et al., 7 Oct 2025).