UltraComposer Model: LLM Instruction & Merging

• UltraComposer is a composite framework that integrates automated prompt composition with derivative-free model merging to enhance LLM instruction synthesis.
• It leverages a standard Transformer foundation alongside sparsity-based denoising and sign-aware scaling to optimize adapter integration under cost constraints.
• Empirical evaluations demonstrate robust generalization, improved cost-performance tradeoffs, and scalability across heterogeneous LLM APIs in limited supervision settings.

UltraComposer encompasses two distinct but convergent paradigms in recent LLM research: as an automated prompt composer facilitating hierarchical instruction synthesis and alignment ("UltraIF: Advancing Instruction Following from the Wild") (An et al., 6 Feb 2025), and as an architectural extension of derivative-free black-box model merging over heterogenous LLM APIs under cost constraints ("Black-box Model Merging for Language-Model-as-a-Service with Massive Model Repositories") (Chen et al., 16 Sep 2025). Both lines exploit structured compositionality, sparse integration, and preference-driven optimization, targeting robust generalization under limited supervision or system-level constraints.

1. Architectural Foundations

UltraComposer, as proposed in (An et al., 6 Feb 2025), is instantiated as a standard Transformer decoder based on LLaMA-3.1-8B-Instruct, featuring pre-layer normalization, rotary embeddings, causal masking, 32 layers, model dimension 4096, MLP dimension 11008, 32 attention heads, and an 8192-token context window. UltraComposer introduces no new architectural modules and leverages the conventional scaled dot-product self-attention mechanism natively.

When formulated as a system-level ensemble method (UltraComposer-as-merger in the editor's parlance), UltraComposer abstracts LLM APIs as "soft" LoRA-style adapters or prompt/prefix-tuning modules, capturing each model’s functional signature in an adapter tuple $(A_i, B_i)$. These adapters are then orchestrated within an API-queryable composite module, using sparsity and scaling parameters optimized for the downstream objective (Chen et al., 16 Sep 2025).

2. Mathematical Framework and Training Objectives

In the context of compositional instruction learning (An et al., 6 Feb 2025), UltraComposer operates on decomposed tuples $(x_i, c_i, q_i)$, mapping simplified query $x_i$ back to the full instruction $X$ and constraint-evaluation question $q_i$:

$$\text{UltraComposer}_\theta(x_i) \longrightarrow (X, q_i)$$

This is cast as a next-token prediction task with cross-entropy loss:

$$\mathcal{L}_{\mathrm{SFT}}(\theta) = -\sum_{(x, y)\in \mathcal{D}_\mathrm{data}} \log\, \pi_\theta(y|x)$$

Successive preference-based learning steps employ Direct Preference Optimization (DPO) and Noise-Contrastive Alignment (NCA):

$$\begin{aligned} \mathcal{L}_{\mathrm{DPO}}(\theta) &= -\mathbb{E}_{(x,y_c,y_r)\sim\mathcal{D}}\,\log\, \sigma\left(\beta\Delta\right) \ \mathcal{L}_{\mathrm{NCA}}(\theta) &= -\mathbb{E}_{(x,y_c,y_r)\sim\mathcal{D}} \left[ \log\, \sigma\left(\beta\Delta_c\right) + \frac{1}{2} \sum_{y \in \{y_c, y_r\}} \log\,\sigma\left(-\beta\Delta_y\right) \right] \end{aligned}$$

For the black-box merging UltraComposer, the optimization decomposes into two derivative-free (CMA-ES) stages (Chen et al., 16 Sep 2025):

• Denoising: Solving

$$\min_{\alpha \in [0, 1]^N} F(\alpha) = \mathcal{L}_{\mathrm{CE}}(\mathcal{D}_{\mathrm{val}}; M_{\mathrm{adapter}}(\alpha)) + \lambda_1 \|\alpha\|_1$$

with $\alpha_i$ controlling the sparsity of adapter $A_i$ via a quantile-based masking operator $S_{\alpha_i}(A_i)$.

• Scaling: Solving

$$\min_{\beta \in \mathbb{R}^N} G(\beta) = \mathcal{L}_{\mathrm{CE}}(\mathcal{D}_{\mathrm{val}}; M_{\mathrm{adapter}}(\beta)) + \lambda_2 \|\beta\|_1$$

with $\beta_i$ permitted to take negative values (enabling constructive/destructive interference).

3. Algorithmic Workflow

UltraIF Data Pipeline (An et al., 6 Feb 2025)

1. Instruction Decomposition: For each real-user instruction $X$, a powerful LLM generates a set of simplified queries $(x_i)$, constraints $(c_i)$, and evaluation questions $(q_i)$.
2. UltraComposer Training: The model is fine-tuned to output $(X, q_i)$ given $x_i$ under cross-entropy, yielding a mapping from simple to complex tasks.
3. Generate-Then-Evaluate Data Curation: Iteratively sample and filter synthetic instruction responses using the model and evaluation questions.
4. Supervised Finetuning (SFT): Minimize $\mathcal{L}_{\mathrm{SFT}}$.
5. Iterative Preference Learning: Apply DPO and NCA losses with increasing task complexity and refined reference policy.

Black-box Merging UltraComposer (Chen et al., 16 Sep 2025)

1. Adapter Abstraction: Represent each API/model as a LoRA or prompt-tuned adapter $(A_i, B_i)$, where only inference access is permitted.
2. Stage 1 (Denoising): Run CMA-ES to prune adapters by optimizing $\alpha$ (adapter sparsity) on validation loss plus $\ell_1$ regularization.
3. Stage 2 (Scaling): Run CMA-ES on $\beta$ (adapter scaling, with sign) to optimize merged performance; allows negative weights to suppress detrimental adapters.
4. Budget-aware Merging: Modify the loss with a cost constraint $\sum_i |\beta_i|c_i \leq$ Budget or add a cost term $\mu \sum_i |\beta_i|c_i$.
5. Practical API Query Minimization: Post-optimization, drop all adapters with $\beta_i \approx 0$ to minimize inference-time API calls; cache and clip queries as needed.

4. Core Mechanisms and Innovations

Sparsity-based Denoising (Chen et al., 16 Sep 2025)

The denoising phase leverages the fact that only a small subset of adapters encode relevant task information, with noise prevalent in adapter parameters $(A_i)$. The quantile-based operator $S_{\alpha_i}(A_i)$ retains only the largest-magnitude entries, with the $\ell_1$ penalty on $\alpha$ driving uninformative adapters towards complete exclusion.

Sign-aware Scaling

In scaling, UltraComposer merges pruned adapters via signed weights $\beta_i$: positive weights amplify synergy; negative weights suppress conflict or hallucination from misaligned models. An $\ell_1$ penalty on $\beta$ restricts the effective number of included adapters and prevents unbounded amplification.

Theoretical Boundedness (Asymmetric Sparsification)

Given update matrices $\Delta W = B\,A$, after sparsification: $$\|\Delta W - \Delta W'(\alpha)\|_F \leq \|B\|_F\,\|A - S_\alpha(A)\|_F$$ Bounding the error in $A$ directly bounds the composition and thus task approximation error under this surrogate.

5. Empirical Performance

UltraComposer demonstrates robust empirical performance for both instruction-following and model-composition tasks:

UltraIF "Composer" Results (An et al., 6 Feb 2025)

• IFEval Pr(S): UltraIF + DPO (scale-up) achieves 71.35 vs. 69.13 for LLaMA-3.1-8B-Instruct.
• InfosBench DRFR: UltraIF + DPO 83.56 exceeds the Instruct baseline (81.33).
• Multi-IF, LiveBench, FollowBench: UltraIF matches or improves over the instruct baseline post scale-up.
• HumanEval Pass@1: UltraIF + DPO (scale) 55.49 vs. 65.24 for the instruct model—a drop, suggesting room for improvement on code generation.
• Ablation studies: Iterative DPO, NCA finishing, and multi-turn SFT all yield multi-point gains. Evaluation-question filtering increases data pass rates from 20% (AutoIF) to 85%.

Black-box UltraComposer-as-Merger Results (Chen et al., 16 Sep 2025)

• Out-of-Domain (OOD): Evo-Merging attains 52.13 F1/53.80 Prec, outperforming LoRaHub by roughly 11 F1/Prec points.
• In-Domain (ID): 38.03 F1 vs. 34.80 for LoRaHub.
• Ablations: Removing denoising drops F1 by ~11, scaling by ~27, and sign-flip-disabling by ~12.
• Robustness: Upon addition of 5 distractors, Evo-Merging's F1 increases by 4 while others drop by 20, highlighting the importance of negative β.
• Scalability: Outperforms all baselines as the number of adapters increases to 100+.
• Sample-efficiency: Achieves 44 F1 on NER_FindVehicle with as few as 50 validation examples.

6. Extension to Arbitrary LLM API Fusion and Real-World Constraints

UltraComposer’s black-box merging methodology generalizes to heterogenous LLM APIs (e.g., GPT-4, Claude, PaLM):

• Adapter Proxying: Each LLM API is mapped to a "soft" adapter using prompt/prefix-tuning or low-rank update approximations.
• Budget-Aware Optimization: The objective incorporates API costs (tokens, latency); multi-objective or constrained CMA-ES is used to trace the Pareto-optimal frontier given performance and budget.
• Pruning for Query Minimization: Post optimization, only active adapters with significant β are queried, reducing API invocations.
• Per-Input Dynamic Weighting: A low-capacity selector network, trained via ES, can substitute for fixed weights, enabling dynamic routing on a per-input basis.

This framework supports real-world deployment scenarios that require fusing LLM specializations, suppressing unreliable contributors, and optimizing cost-performance tradeoffs.

7. Significance, Open Problems, and Limitations

UltraComposer, in both the instruction-alignment and black-box merging regimes, advances scalable and resource-efficient mechanism design for LLM compositionality. It closes much of the gap between open-source and proprietary instruct models under limited supervision, and delivers a robust, theoretically justified system for merging closed weights LLM APIs under inference and cost budgets.

Limitations include:

• Architectural changes are not addressed (e.g., multimodal, retrieval-augmented extensions remain unexplored).
• Dependence on large, high-quality LLM oracles for prompt decomposition and evaluation; possible entrenchment of oracle biases.
• Potential brittleness in settings requiring reasoning over inter-dependent constraints or deep multi-step decomposition.
• Real-world scalability—hardware, latency, and distributed caching—remains to be empirically assessed beyond experimental setups.

A plausible implication is that UltraComposer's compositionality and modularity make it a candidate foundation for more general composite AI systems, especially where intermediate "tools" are entailed. Future research may focus on expanding compositionality to include non-text modalities, richer constraint reasoning, and dynamic expert selection.