Compositional Preference Models

• CPMs are frameworks that explicitly structure, represent, and optimize user or system preferences over compositional objects through modular decomposition.
• They employ formal methods—including dynamic CSPs, qualitative multi-attribute dominance, and soft constraint automata—to aggregate local preferences into global rankings.
• Applications span automated scientific model construction, multi-attribute configuration, and AI alignment, enhancing interpretability and robustness in complex systems.

Compositional Preference Models (CPMs) are a family of frameworks and learning paradigms that explicitly structure, represent, and optimize user or system preferences over compositional objects, such as model components, system behaviors, or candidate outputs in AI systems. These models appear across diverse subfields, including automated scientific model construction, multi-attribute configuration, preference alignment in language and vision models, and preference-aware software or agent systems. CPMs are characterized by the decomposition of global preference assessments into modular, interpretable parts, allowing principled aggregation and optimization under both hard (functional) constraints and soft (preference) requirements (Go et al., 2023, Keppens et al., 2011, Santhanam et al., 2014, Kappé et al., 2016).

1. Formal Structure and Historical Background

The CPM paradigm originated in knowledge-based automated model construction, multi-attribute decision support, and soft constraints, before being extended to large-scale AI alignment and vision-language modeling. Core to all CPMs is the compositional assembly of objects from primitive components, each associated with specific attributes, behaviors, or subprocesses. Preferences are defined at the component or attribute level, often through symbolic or qualitative orders, and then composed or aggregated to yield global rankings or selection criteria (Keppens et al., 2011, Santhanam et al., 2014, Kappé et al., 2016).

Historically, CPMs were introduced to address the limitations of monolithic preference models in compositional systems, where consistency and user guidance must be maintained across combinatorial choices. Recent developments have seen CPMs adapted for aligning generative models and enhancing compositional reasoning in multimodal learning (Go et al., 2023, Mishra et al., 7 Apr 2025).

2. Formalizations and Mathematical Foundations

CPMs admit several formalizations depending on the application domain. The foundational cases are:

1. Dynamic Preference-augmented CSPs: CPMs in scientific model repositories are formalized as activity-based dynamic constraint satisfaction problems (aDCSPs), in which each modeling decision (e.g., inclusion of a subsystem, choice of mathematical formalism) is a variable with an associated domain and optional activation conditions. Soft preferences over alternative choices are integrated via a preference calculus. Preferences are combined using an order-of-magnitude calculus over basic preference quantities (BPQs), allowing symbolic, non-numeric reasoning about trade-offs (Keppens et al., 2011).
2. Qualitative Multi-Attribute Dominance: In multi-attribute composition (e.g., software or team selection), preferences are specified via intra-attribute strict partial orders $\succ_i$ and a relative-importance strict partial order $\rhd$ over attributes. Dominance between composed objects $U, V$ is defined as

$U \succ_d V \iff \exists X_i: \Val(U, X_i) \succ_i^* \Val(V, X_i),\; \forall X_k \text{ with } X_k \rhd X_i \lor X_k \sim X_i: \Val(U, X_k) \succeq_k^* \Val(V, X_k)$

where $\succ_i^*$ is a lifted worst-frontier extension of $\succ_i$ over sets. Key order-theoretic results establish that $\succ_d$ is a strict partial order if $\succ_i$ are SPOs and $\rhd$ is an interval order (Santhanam et al., 2014).

1. Soft Constraint Automata: In preference-aware agents and cyber-physical systems, CPMs are implemented via the compositional algebra of Soft Constraint Automata (SCA). Each component is an automaton whose transitions are labeled by soft constraint satisfaction problems (SCSPs) grounded in c-semirings $(E, \oplus, \otimes, 0, 1)$, enabling explicit modularization and algebraic aggregation of local preferences. Operators such as parallel (product), hiding (projection), renaming, join, and lexicographic composition are used to construct the global preference model, with rigorous algebraic properties (Kappé et al., 2016).
2. Feature-decomposed Aggregation in AI Alignment: Modern CPMs for aligning LMs disaggregate global preference assessments into vectors of interpretable features, each scored by an auxiliary or base model. A simple aggregator (often logistic regression) learns to reproduce human/global preference rankings via convex optimization. This makes the CPM transparent and controllable with respect to feature weights (Go et al., 2023).

3. Construction, Aggregation, and Optimization

The CPM framework typically involves:

• Feature/Component Decomposition: Identification of atomic features (in alignment) or partial models/components (in automated model building) relevant for the composition task.
• Preference Specification: For classical CPMs, qualitative or symbolic preferences are assigned over each choice; for recent alignment-oriented CPMs, feature-scoring prompts are specified to elicit feature-wise scores from an LM or oracle.
• Preference Aggregation: Preferences are aggregated, often via multiset union and order-of-magnitude calculus (model selection), or linear aggregation with learned parameters (AI alignment). In SCA-based CPMs, c-semiring operations are used.
• Optimization: Search or learning algorithms are employed to find globally consistent and maximal-preference solutions. For aDPCSPs this involves best-first search using an admissible heuristic; for multi-attribute dominance, filtering or interleaved construction; for alignment CPMs, convex loss minimization over pairwise comparisons.

The following table summarizes CPM variants:

Domain	Decomposition	Aggregation/Optimization
Scientific modeling	Partial models	Order-of-magnitude calculus + A* search
Multi-attribute config	Attributes	Dominance over attributes + filtering
Agent systems	Automata/ports	c-semiring algebra + automata comp.
Preference alignment	Features	Logistic regression over feature diffs

4. Applications and Benchmarks

CPMs have been instantiated in:

• Automated Model Construction: Ecological model repositories, where domain experts' modeling hypotheses and soft preferences are combined to yield symbolic ODEs consistent with both scientific knowledge and user guidance (Keppens et al., 2011).
• Compositional System Design: Configuration of multi-component systems, web services, or teams, subject to compositional functional and qualitative non-functional preferences (Santhanam et al., 2014).
• Preference-Aware Agents: Workflow synthesis and robot controllers, where system-level objectives (e.g., patrolling, battery management) are modularly encoded, composed, and jointly optimized with robust fallback behavior (Kappé et al., 2016).
• Human Preference Alignment in LMs: Selection of LLM outputs that optimize interpretable desiderata (helpfulness, factuality, etc.), robust against overoptimization and more transparent than conventional black-box preference models (Go et al., 2023).
• Compositional Reasoning in Vision-LLMs: Synthetic-data derived training of multimodal models to improve fine-grained scene composition understanding, as in SCRAMBLe for vision-language compositionality and VideoComp for video-text temporal alignment (Mishra et al., 7 Apr 2025, Kim et al., 4 Apr 2025).

Key benchmarks include Winoground (caption-image compositionality), ActivityNet-Comp (video-text compositional alignment), and EqBen (equivariance tests), with state-of-the-art performance being attained by CPM-based models (Mishra et al., 7 Apr 2025, Kim et al., 4 Apr 2025).

5. Interpretability, Robustness, and Trade-offs

CPMs are motivated by limitations of monolithic or standard preference models, such as poor generalization, lack of transparency, and vulnerability to spurious correlations or reward hacking. The explicit feature or component decomposition underlying CPMs:

• Enables interpretability: learned weights or explicit aggregation functions reveal which features drive global preference (Go et al., 2023).
• Improves robustness: CPMs resist overoptimization on high-variance or limited preference datasets (e.g., less sensitivity to sampling noise, slower divergence under "Goodhart's law").
• Allows modular analysis: component-level design and debugging is possible in multi-agent or system synthesis contexts (Kappé et al., 2016).
• Preserves or improves generalization, often outperforming standard models on out-of-domain preference evaluations (Go et al., 2023, Mishra et al., 7 Apr 2025).

However, CPMs also impose modeling demands, such as the need for domain-specific feature or component vocabularies and the potential for combinatorial explosion in highly compositional systems. In alignment applications, feature selection and prompt sensitivity remain open research challenges (Go et al., 2023). Overfitting to large synthetic preference sets is a risk in multimodal CPMs unless dataset size and adversarial filtering are carefully managed (Mishra et al., 7 Apr 2025).

6. Example Instantiations and Experimental Outcomes

Representative results from recent CPM approaches include:

• LLM Alignment CPMs: In "Compositional preference models for aligning LMs" (Go et al., 2023), CPMs constructed over 13 interpretable features (e.g., helpfulness, detail, factuality) achieved a win-rate of 0.810 (HH-RLHF, GPT-3.5 CPM) vs. 0.588 for standard PMs, and demonstrated near-perfect rank correlation with gold-standard reward models under large-n best-of sampling.
• Vision-Language Compositionality: SCRAMBLe CPMs for MLLMs improved Winoground group accuracy from 49.5% to 54.8% (Molmo-7B), besting prior SOTA for open-weight models, with measured gains of 1–2 pp on general VQA (Mishra et al., 7 Apr 2025).
• Video-Text Alignment: VideoComp CPMs with hierarchical preference loss improved comprehensive binary-classification accuracy on ActivityNet-Comp from 23.4% (contrastive baseline) to 35.0% (with compositionally pre-trained CPM), outperforming prior baselines (Kim et al., 4 Apr 2025).
• Multi-Attribute Configuration: Algorithmic CPMs over software or team compositions offer Pareto-optimal, sound, and in key regimes, efficiently-computed solution sets under strictly partial and interval order relations (Santhanam et al., 2014).

7. Extensions, Limitations, and Open Questions

CPMs continue to evolve along several axes:

• Automatic Feature Discovery: Current CPMs often require hand-chosen features or components. Automated inductive or statistical feature induction remains a key open problem (Go et al., 2023).
• Scalability: While CPMs offer modularity, state-space and feature-space size can become prohibitive. Incremental, local-search, or pruning heuristics are active areas for research (Keppens et al., 2011).
• Beyond Binary Preferences: Extending CPMs to richer forms of feedback, such as ranking CM pairs, continuous rewards, or structured judgments, is underexplored (Go et al., 2023).
• Integration with Soft-constraint Algebra: Algebraic approaches via c-semirings (e.g., SCSPs, SCAs) offer a unifying mathematical language for CPM composition but need further development for complex, real-time or multi-agent domains (Kappé et al., 2016).
• Prompting Robustness and Evaluation: In alignment CPMs, the sensitivity to feature prompt design and lack of full human evaluation pose limitations (Go et al., 2023).

Overall, CPMs bring together advances in preference representation, modular system synthesis, and AI alignment, providing general frameworks for principled, interpretable, and robust optimization under compositional structure.