CoLLM-NAS: Collaborative LLM-Based NAS

• The paper demonstrates that integrating collaborative LLMs into NAS enhances search efficiency and reduces evaluation cost compared to traditional methods.
• CoLLM-NAS is a two-stage framework that employs a stateful Navigator and a stateless Generator to refine candidate architectures using historical feedback and structured encodings.
• Empirical results on benchmarks like ImageNet and NAS-Bench-201 show improved accuracy and significant reductions in search cost.

Collaborative LLM-based NAS (CoLLM-NAS) denotes a class of neural architecture search systems in which LLMs participate as iterative search operators rather than one-shot generators. In the specific sense established by "CoLLM-NAS: Collaborative LLMs for Efficient Knowledge-Guided Neural Architecture Search," it is a two-stage NAS framework that replaces the conventional second-stage search algorithm with two complementary LLMs and a Coordinator, so that search is driven jointly by pretrained architectural priors and progressive knowledge from iterative feedback and historical trajectory (Li et al., 30 Sep 2025). In a broader research sense, the same label covers closely related designs in which LLMs collaborate with partitioning engines, evolutionary operators, historical feedback memories, zero-cost evaluators, supernets, or MARL-style critics, with the common objective of making search more directed, more data-efficient, and more constraint-aware than generic random search, RL, or unconstrained code generation (Zhu et al., 1 Oct 2025, Gu et al., 12 Mar 2026, Hu et al., 6 Sep 2025).

1. Conceptual foundations and formal setting

The canonical CoLLM-NAS formulation inherits the standard two-stage NAS decomposition. A supernet is trained once over a structured search space, and the search problem is then to identify an architecture $\alpha^*$ maximizing validation performance under a resource constraint $\Lambda$:

$$\begin{aligned} w_{\mathcal{A}^*} &= \underset{w_{\mathcal{A}}}{\arg\min}\, \mathbb{E}_{\alpha \sim \Omega(\mathcal{A})}\left[\mathcal{L}(w_{\mathcal{A}}(\alpha), \mathcal{D}^{\text{train}})\right], \ \alpha^* &= \underset{\alpha \in \mathcal{A}}{\arg\max}\, \mathcal{P}(w_{\mathcal{A}^*}(\alpha), \mathcal{D}^{\text{val}})\quad \text{s.t. } \text{Cost}(\alpha) \leq \Lambda . \end{aligned}$$

CoLLM-NAS changes only the second stage: instead of EA, RL, or random search, it inserts a collaborative LLM loop that reasons over search history, resource constraints, and structured architecture encodings (Li et al., 30 Sep 2025).

This design was introduced against two limitations. Traditional NAS, including one-shot and two-stage variants such as SPOS, OFA, and AutoFormer, still uses generic optimizers in the second stage and therefore requires thousands of candidate evaluations in very large spaces. Existing LLM-based NAS methods such as GENIUS, EvoPrompting, and LLMatic often operate in unconstrained programming token spaces, produce invalid architectures, and frequently train each candidate from scratch rather than using weight sharing (Li et al., 30 Sep 2025).

Within this setting, collaboration is not merely parallel prompting. It is an explicit decomposition of search into interacting functions: history accumulation, strategy induction, candidate synthesis, legality filtering, and evaluator feedback. This decomposition recurs across the literature, even when the outer algorithm differs. PEL-NAS couples an LLM with a partitioning engine, an evolutionary archive, and a zero-cost predictor; LM-Searcher couples an LLM with unified numerical encoding and ranked candidate pools; resource-efficient iterative NAS couples a Code Generator with a Prompt Improver and a structured memory of recent attempts (Zhu et al., 1 Oct 2025, Hu et al., 6 Sep 2025, Gu et al., 12 Mar 2026).

2. Canonical CoLLM-NAS architecture

The narrow, named CoLLM-NAS framework is organized around three modules: a stateful Navigator LLM, a stateless Generator LLM, and a deterministic Coordinator. Its central mapping is

$$\mathcal{H}_t \xrightarrow{\text{Navigator}} \mathcal{S}_t \xrightarrow{\text{Generator}} \mathcal{C}_{t+1},$$

where $\mathcal{H}_t$ is the search trajectory, $\mathcal{S}_t$ is a natural-language search strategy, and $\mathcal{C}_{t+1}$ is the candidate set for the next iteration (Li et al., 30 Sep 2025).

The Navigator LLM is responsible for search strategy. At initialization it receives a target accuracy $P_{\text{target}}$ and a resource constraint $\Lambda$; at later iterations it receives the accumulated history $$\mathcal{H}=\{(\mathcal{S}_k,\mathcal{R}_k)\}_{k=0}^{t-1}$$, where each $\Lambda$0 stores evaluated architectures and their performance and cost. Its output is a strategy in natural language describing which regions of the search space should be emphasized or deemphasized. The Generator LLM receives only the current strategy and a structured description of the search space, then emits candidate architectures in a strict representation format. The Coordinator maintains the visited set $\Lambda$1, the best architecture $\Lambda$2 and best accuracy $\Lambda$3, calls isLegal(α_i), decodes candidates, evaluates them through a supernet or benchmark lookup, appends results to history, and enforces stopping conditions (Li et al., 30 Sep 2025).

A key design decision is the asymmetry between memory-bearing and memoryless components. The Navigator retains full search history because historical trajectory is useful for high-level strategy refinement. The Generator is stateless because retaining generator memory causes “progressive noise accumulation,” increases invalid architectures, and degrades performance on harder datasets such as ImageNet-16-120 and ImageNet. The ablation reported for CoLLM-NAS therefore identifies the best setting as Navigator-memory on and Generator-memory off (Li et al., 30 Sep 2025).

The search spaces used in the canonical study cover both macro and micro NAS. The macro spaces are MobileNet in OFA, ShuffleNet in SPOS, and AutoFormer; the micro space is NAS-Bench-201. In these spaces the Generator does not emit arbitrary code but structured encodings defined in the prompt, such as per-layer kernel sizes and expansion ratios in MobileNet, operator indices in ShuffleNet, layerwise hyperparameters in AutoFormer, or the standard NAS-Bench-201 cell string |op1~0|+|op2~0|op3~1|+|op4~0|op5~1|op6~2| (Li et al., 30 Sep 2025).

3. Knowledge-guided collaboration beyond the canonical framework

The term CoLLM-NAS is also used more broadly for systems in which LLMs collaborate with search infrastructure through explicit memory, role decomposition, or search-space structuring. PEL-NAS is a concrete instantiation of this broader view. It introduces a complexity-driven partitioning engine that divides the search space into disjoint niches, an LLM-powered architecture prompt co-evolution operator, and a zero-cost predictor. Within each niche, the LLM first updates a knowledge base of design heuristics from previous results and then performs guided crossover or mutation under niche-specific constraints. The resulting loop is explicitly described as a “closed-loop collaboration: LLM ↔ evaluation ↔ partitioning ↔ archives,” and the architecture and prompt improve together across generations (Zhu et al., 1 Oct 2025).

A second line of work realizes collaboration through role specialization and bounded memory rather than two-stage supernet search. "Resource-Efficient Iterative LLM-Based NAS with Feedback Memory" uses a Code Generator and a Prompt Improver in a closed loop with one-epoch proxy training on CIFAR-10, CIFAR-100, and ImageNette. Its central memory is a fixed sliding window of $\Lambda$4 recent improvement attempts, with each history entry represented as a diagnostic triple $\Lambda$5. The Markovian truncation rule

$\Lambda$6

keeps context size constant while retaining recent causal structure, and code execution failures are treated as first-class learning signals rather than discarded trajectories (Gu et al., 12 Mar 2026).

A third pattern emphasizes constrained code synthesis. NN-Caption uses DeepSeek-R1-0528-Qwen3-8B to generate image-captioning models under a strict Net API contract, with AST parsing, schema checks, runtime checks, and repair prompting. This is not a multi-LLM system, but it supplies a reusable collaborative protocol for CoLLM-NAS: architecture proposal, static verification, constrained repair, automatic training, and database logging under a shared interface. The paper explicitly recommends splitting these functions into specialized agents such as Architect, Code agent, Debugger, and Analyst in a multi-agent extension (Jesani et al., 7 Dec 2025).

LM-Searcher replaces free-form generation with a ranking-based collaborative interface. It introduces NCode, a universal numerical string representation for architectures, and trains an LLM to choose the best candidate from a pool given a history of architecture-performance pairs. NAS is thereby reformulated from $\Lambda$7 to selection from a candidate set $\Lambda$8, and the prompt format becomes uniform across CNNs, LoRA rank search, efficient transformers, and NAS-Bench-ASR. This suggests a different collaborative topology for CoLLM-NAS: one agent or module generates candidate NCodes, another ranks them, and conventional evaluators close the loop (Hu et al., 6 Sep 2025).

4. Objectives, evaluators, and constraint handling

One defining feature of collaborative LLM-based NAS is that generation is typically separated from scoring. In CoLLM-NAS proper, macro-space candidates inherit weights from a pre-trained supernet and are evaluated on full ImageNet validation, while NAS-Bench-201 candidates are read directly from the tabular benchmark. This allows the collaborative loop to operate within mature two-stage NAS pipelines rather than retraining every candidate from scratch (Li et al., 30 Sep 2025).

PEL-NAS makes this separation explicit in a hardware-aware, multi-objective regime. Accuracy is predicted by an XGBoost ensemble of 13 zero-cost proxies from NAS-Bench-Suite-Zero, with Spearman rank correlation approximately $\Lambda$9 with ground-truth accuracy; latency is obtained from hardware lookup tables on six HW-NAS-Bench devices or from direct profiling on A6000 for ViTs. Search quality is measured by Hypervolume and Inverted Generational Distance. The system reports overall higher HV, lower IGD, and up to 54% lower latency than baselines at similar accuracy, while the search cost drops from days to minutes compared with traditional supernet baselines (Zhu et al., 1 Oct 2025).

FL-NAS introduces a different collaborative objective structure: accuracy, fairness, and hardware deployment efficiency. Its prompt instructs the LLM to improve fairness without decreasing accuracy, to inspect group-wise accuracies, and only then to reduce parameters, latency, and memory. Fairness is quantified by Unfairness, EODD, EOPP1, and EOPP2, while hardware efficiency is measured by parameters, peak GPU memory, and latency on NVIDIA A10 and Raspberry Pi 4. This lexicographic prioritization shows that collaborative LLM-based NAS can be driven by rich metric vectors rather than a single scalar accuracy score (Qin et al., 2024).

The same generate-versus-score separation appears in low-budget open-code systems. The feedback-memory NAS pipeline trains each candidate for one epoch with SGD, momentum $$\begin{aligned} w_{\mathcal{A}^*} &= \underset{w_{\mathcal{A}}}{\arg\min}\, \mathbb{E}_{\alpha \sim \Omega(\mathcal{A})}\left[\mathcal{L}(w_{\mathcal{A}}(\alpha), \mathcal{D}^{\text{train}})\right], \ \alpha^* &= \underset{\alpha \in \mathcal{A}}{\arg\max}\, \mathcal{P}(w_{\mathcal{A}^*}(\alpha), \mathcal{D}^{\text{val}})\quad \text{s.t. } \text{Cost}(\alpha) \leq \Lambda . \end{aligned}$$0, weight decay $$\begin{aligned} w_{\mathcal{A}^*} &= \underset{w_{\mathcal{A}}}{\arg\min}\, \mathbb{E}_{\alpha \sim \Omega(\mathcal{A})}\left[\mathcal{L}(w_{\mathcal{A}}(\alpha), \mathcal{D}^{\text{train}})\right], \ \alpha^* &= \underset{\alpha \in \mathcal{A}}{\arg\max}\, \mathcal{P}(w_{\mathcal{A}^*}(\alpha), \mathcal{D}^{\text{val}})\quad \text{s.t. } \text{Cost}(\alpha) \leq \Lambda . \end{aligned}$$1, learning rate $$\begin{aligned} w_{\mathcal{A}^*} &= \underset{w_{\mathcal{A}}}{\arg\min}\, \mathbb{E}_{\alpha \sim \Omega(\mathcal{A})}\left[\mathcal{L}(w_{\mathcal{A}}(\alpha), \mathcal{D}^{\text{train}})\right], \ \alpha^* &= \underset{\alpha \in \mathcal{A}}{\arg\max}\, \mathcal{P}(w_{\mathcal{A}^*}(\alpha), \mathcal{D}^{\text{val}})\quad \text{s.t. } \text{Cost}(\alpha) \leq \Lambda . \end{aligned}$$2, cosine annealing, and batch size $$\begin{aligned} w_{\mathcal{A}^*} &= \underset{w_{\mathcal{A}}}{\arg\min}\, \mathbb{E}_{\alpha \sim \Omega(\mathcal{A})}\left[\mathcal{L}(w_{\mathcal{A}}(\alpha), \mathcal{D}^{\text{train}})\right], \ \alpha^* &= \underset{\alpha \in \mathcal{A}}{\arg\max}\, \mathcal{P}(w_{\mathcal{A}^*}(\alpha), \mathcal{D}^{\text{val}})\quad \text{s.t. } \text{Cost}(\alpha) \leq \Lambda . \end{aligned}$$3, then feeds back top-1 validation accuracy or an error string to the Prompt Improver. NN-Caption evaluates code-generated captioning models by BLEU-4 after three epochs on MS COCO and stores code, hyperparameters, training curves, and final BLEU-4 in a registry. In both cases, the evaluator is external to the LLM and its outputs are re-injected as structured context (Gu et al., 12 Mar 2026, Jesani et al., 7 Dec 2025).

Constraint-aware backends need not themselves be LLM-based. Puzzle is a hardware-aware NAS framework for very large LLMs that combines blockwise local knowledge distillation with mixed-integer programming under memory, throughput, and latency constraints. The paper explicitly presents this as a useful blueprint for designing collaborative, LLM-in-the-loop NAS systems. This suggests a modular CoLLM-NAS architecture in which LLM agents handle search-space engineering, search planning, or subblock proposal, while BLD and MIP provide scalable local scoring and globally feasible architecture assembly (Bercovich et al., 2024).

5. Reinforcement-learning and Dec-POMDP formulations

A more formal collaborative interpretation treats CoLLM-NAS as a cooperative multi-agent control problem. "Learning Decentralized LLM Collaboration with Multi-Agent Actor Critic" states directly that, for a collaborative LLM-based NAS system, NAS can be framed as a Dec-POMDP in which agents such as an architecture proposer, evaluator, refiner, and search strategist observe text descriptions of current architectures, performance metrics, constraints, and partial search history, and emit macro-actions such as full architecture edits, candidate architectures, or search directives. The objective is to learn a joint policy $$\begin{aligned} w_{\mathcal{A}^*} &= \underset{w_{\mathcal{A}}}{\arg\min}\, \mathbb{E}_{\alpha \sim \Omega(\mathcal{A})}\left[\mathcal{L}(w_{\mathcal{A}}(\alpha), \mathcal{D}^{\text{train}})\right], \ \alpha^* &= \underset{\alpha \in \mathcal{A}}{\arg\max}\, \mathcal{P}(w_{\mathcal{A}^*}(\alpha), \mathcal{D}^{\text{val}})\quad \text{s.t. } \text{Cost}(\alpha) \leq \Lambda . \end{aligned}$$4 maximizing the shared return $$\begin{aligned} w_{\mathcal{A}^*} &= \underset{w_{\mathcal{A}}}{\arg\min}\, \mathbb{E}_{\alpha \sim \Omega(\mathcal{A})}\left[\mathcal{L}(w_{\mathcal{A}}(\alpha), \mathcal{D}^{\text{train}})\right], \ \alpha^* &= \underset{\alpha \in \mathcal{A}}{\arg\max}\, \mathcal{P}(w_{\mathcal{A}^*}(\alpha), \mathcal{D}^{\text{val}})\quad \text{s.t. } \text{Cost}(\alpha) \leq \Lambda . \end{aligned}$$5, while decentralized execution is preserved at search time and CTDE is used during training (Liu et al., 29 Jan 2026).

That paper further proposes two actor-critic variants, CoLLM-CC with a centralized critic and CoLLM-DC with decentralized critics. For NAS-like settings, it explicitly recommends a centralized critic LLM that ingests the joint dialogue or search history and global search state, such as best architectures, Pareto fronts, and budget usage, then outputs TD-based value estimates. It also warns that decentralized critics are likely to be non-stationary and hard to train in long-horizon, high-sparsity regimes, and that pure Monte Carlo methods are suitable only for short-horizon or densely rewarded sub-problems (Liu et al., 29 Jan 2026).

A related line models LLM collaboration as cooperative MARL and introduces Multi-Agent Group Relative Policy Optimization. MAGRPO defines LLM collaboration as a Dec-POMDP with natural-language observations and full-response actions, then updates each agent by PPO-style optimization using a group-relative advantage

$$\begin{aligned} w_{\mathcal{A}^*} &= \underset{w_{\mathcal{A}}}{\arg\min}\, \mathbb{E}_{\alpha \sim \Omega(\mathcal{A})}\left[\mathcal{L}(w_{\mathcal{A}}(\alpha), \mathcal{D}^{\text{train}})\right], \ \alpha^* &= \underset{\alpha \in \mathcal{A}}{\arg\max}\, \mathcal{P}(w_{\mathcal{A}^*}(\alpha), \mathcal{D}^{\text{val}})\quad \text{s.t. } \text{Cost}(\alpha) \leq \Lambda . \end{aligned}$$6

The method avoids per-agent reward engineering by using a single shared reward and no explicit centralized value network. This suggests a plausible CoLLM-NAS training mechanism in which groups of architecture trajectories are sampled, evaluated by shared architecture-level rewards, and used to fine-tune role-specialized LLM agents such as proposer, critic, tuner, or constraint checker (Liu et al., 6 Aug 2025).

These RL-based formulations remain prospective for NAS rather than definitive benchmarks. The explicit transferable principles are the use of macro-actions instead of token-level actions, dense and verifiable intermediate rewards, shared scalar objectives, and centralized training with decentralized execution. In NAS terms, the recommended global information $$\begin{aligned} w_{\mathcal{A}^*} &= \underset{w_{\mathcal{A}}}{\arg\min}\, \mathbb{E}_{\alpha \sim \Omega(\mathcal{A})}\left[\mathcal{L}(w_{\mathcal{A}}(\alpha), \mathcal{D}^{\text{train}})\right], \ \alpha^* &= \underset{\alpha \in \mathcal{A}}{\arg\max}\, \mathcal{P}(w_{\mathcal{A}^*}(\alpha), \mathcal{D}^{\text{val}})\quad \text{s.t. } \text{Cost}(\alpha) \leq \Lambda . \end{aligned}$$7 for critic input includes search iteration, budget remaining, and best-seen performance, so that the critic can recognize where the system is in the search trajectory and provide lower-variance guidance (Liu et al., 29 Jan 2026).

6. Empirical record, limitations, and outlook

The named CoLLM-NAS framework reports state-of-the-art results on both ImageNet and NAS-Bench-201. On MobileNet within OFA, it improves OFA-T from 75.5% Top-1 at 200M FLOPs to 75.7% at 199M FLOPs, OFA-S from 77.2% at 299M FLOPs to 77.6% at 297M FLOPs, OFA-B from 78.2% at 399M FLOPs to 78.4% at 396M FLOPs, and OFA-L from 78.7% at 496M FLOPs to 78.9% at 494M FLOPs. On SPOS, it raises Top-1 from 73.6% to 73.8% while reducing search cost from 0.32 GPU days to 0.09 GPU days and reducing the number of evaluated architectures from 1000 to 250. On AutoFormer, it improves the Tiny, Small, and Base settings while reducing search cost from 1.0 GPU day to 0.1 GPU day at 250 evaluated architectures. On NAS-Bench-201 under a 100-architecture budget, it reaches 94.37% ± 0.01 on CIFAR-10, 73.44% ± 0.15 on CIFAR-100, and 46.79% ± 0.28 on ImageNet-16-120 (Li et al., 30 Sep 2025).

Other collaborative variants reinforce different aspects of the paradigm. PEL-NAS reports approximately 3 minutes per dataset and device with 120 API calls, versus 10 GPU days for FairNAS, 2 GPU days for PRP-NAS, and 17 GPU days for LLMatic, while ablation shows that removing partitioning collapses average HV from 0.978 to 0.516 and increases IGD from 0.0246 to 0.3734 on CIFAR-100. The feedback-memory pipeline completes a 2000-iteration search in approximately 18 GPU hours on a single RTX 4090, with DeepSeek-Coder-6.7B improving from 28.2% to 69.2% on CIFAR-10 and Qwen2.5-7B improving from 50.0% to 71.5%. FL-NAS reports architectures with accuracy up to 75.20%, Unfairness down to 0.0306, parameters as low as 293,954, memory around 158–233 MB, and NVIDIA A10 latency of 0.0011 s per image (Zhu et al., 1 Oct 2025, Gu et al., 12 Mar 2026, Qin et al., 2024).

The main limitations are also consistent across papers. CoLLM-NAS depends on strong reasoning-capable LLMs, careful prompt and system design, structured search-space encodings, and evaluator quality; macro-space search remains bounded by supernet fidelity and validation cost. PEL-NAS identifies manual partitioning design, local mode collapse within niches, dependence on benchmark latency or profiling, and zero-cost predictor generalization as open issues. The feedback-memory and strict-API systems document residual code hallucinations, runtime errors, and context-overload effects. LM-Searcher shows that cross-domain transfer is strong but not uniformly state of the art, and that performance degrades when the architecture-performance mapping in history is shuffled (Li et al., 30 Sep 2025, Zhu et al., 1 Oct 2025, Jesani et al., 7 Dec 2025, Hu et al., 6 Sep 2025).

Taken together, the literature supports a stable definition of CoLLM-NAS as a modular NAS paradigm in which LLMs collaborate with one another or with non-LLM search components through structured histories, constrained representations, and iterative evaluation loops. The most mature instantiations currently favor one of three designs: stateful strategy induction plus stateless candidate synthesis in two-stage NAS, niche-partitioned prompt co-evolution with zero-cost evaluation, or bounded-memory code generation with explicit repair. A plausible next synthesis is a system that combines unified architecture encodings, supernet or surrogate evaluation, and CTDE-style multi-agent learning with centralized critics or group-relative policy updates, while preserving the central CoLLM-NAS principle that search should be guided by both pretrained architectural knowledge and progressively accumulated search experience (Li et al., 30 Sep 2025, Liu et al., 29 Jan 2026).