- The paper introduces an RL-based dynamic framework that enables small language models to selectively request assistance from large language models.
- The paper demonstrates significant improvements in answer quality, interaction efficiency, and privacy preservation across six question-answering datasets.
- The paper shows that learned collaboration policies are transferable across different model scales and domains, supporting robust, privacy-aware deployments.
Dynamic Collaboration Between Small and LLMs: An Expert Review
Motivation and Context
The landscape of LLM deployments is increasingly characterized by a dichotomy: LLMs deliver broad generalization and superior reasoning capacities at considerable cost and privacy risk, while small LLMs (SLMs) facilitate efficient, localized, and privacy-preserving inference but suffer from restricted knowledge and reasoning scope. Conventional strategies for combining SLMs and LLMs rely on static pipelines, where SLMs preprocess queries and trigger LLMs according to fixed heuristics, typically disregarding both privacy and adaptivity. Such approaches fail to leverage the full inferential potential of SLMs, treat LLMs as passive retrieval tools, and often expose sensitive query content unnecessarily. The present paper addresses this gap by formalizing a dynamic, RL-driven collaboration framework that allows SLMs to learn when and how to seek targeted, privacy-aware assistance from LLMs, integrating active LLM feedback rather than mere point responses.
Framework and Methodology
The proposed system is structured as a multi-agent collaborative process with inherent heterogeneity and information asymmetry: the SLM exclusively receives user queries and iteratively decomposes reasoning tasks, autonomously deciding when LLM input is warranted. The LLM, instead of serving as a black-box oracle, supplies adaptive feedback—clarifying underspecified requests, or returning targeted information—and thus participates as a knowledgeable agent in a dialogic sequence. This architecture enables the SLM to optimize a composite objective considering answer quality (Exact Match and BERTScore), interaction efficiency (average interaction turns, INScore), and privacy leakage, as measured both semantically and by instance frequency.
The SLM is trained via end-to-end online RL (using VAPO (Yue et al., 7 Apr 2025)), jointly maximizing response quality and minimizing redundant or privacy-compromising interactions. Penalties for unwarranted requests and privacy leaks are configurable, supporting fine-grained policy optimization. The LLM returns process-level feedback signals, supplementing outcome supervision, which enable the SLM to refine request formulation iteratively.
To robustly evaluate privacy constraints, the authors synthesize a large PrivQA dataset via LLM-driven privacy injection, augmenting public QA corpora with privacy-relevant indirect references and validating through human and LLM-based assessment.
Empirical Results
Across six QA datasets (NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultihopQA, MuSiQuE), dynamic collaboration yields superior results to both static pipeline baselines (e.g., PAPILLON [2025.naacl-long.173]) and direct LLM inference. Specifically, the learned strategies improve response quality by 14.5%–17.4% over SLMs with CoT and 2.8%–9.9% over extant static baselines, while reducing average interaction turns by 0.11–0.15 and privacy leakage rates by 24.3%–32.4%. When SLMs are scaled up, their self-reliance increases, deferring to LLM assistance only for genuinely hard queries, and the interaction patterns become more concise and targeted.
The capacity of both SLMs and LLMs significantly modulates collaboration dynamics: higher-param SLMs request LLM support less frequently, and stronger LLMs provide richer and more relevant signals, reducing the required dialogue rounds. Weak instruction-following SLMs (e.g., Qwen3-0.6B) fail to exploit RL training effectively without supervised bootstrapping, seldom issuing properly formed LLM requests.
Transfer and Generalization
Learned collaboration policies generalize robustly to unseen LLMs. SLMs trained with a moderate LLM can be seamlessly paired at inference with a larger LLM (e.g., Qwen3-Max), inducing additional performance gains (+3.3% EM for Qwen3-4B). The average interaction footprint remains stable across LLM swaps, evidencing policy invariance and transferability.
Efficiency and Privacy
Dynamic collaboration resolves the interaction-efficiency/privacy-accuracy trade-off via explicit penalty tuning. Increasing efficiency penalty weights leads SLMs to favor local inference, decreasing LLM call frequency at a minor cost in answer quality. Privacy penalties enforce strong privacy preservation behaviors; when the penalty exceeds the threshold, SLMs produce near-zero privacy leakage (Privsample ≈ 0.08%), without compromising EM. Ablations confirm active LLM feedback is critical: removing feedback and treating the LLM as a passive tool yields a 2.8% drop in EM.
Static interaction schemes, even with privacy removal modules, remain vulnerable to imperfect intent extraction and frequent privacy leaks, while dynamic collaboration consistently reformulates queries to minimize semantic similarity to labeled private attributes.
Theoretical and Practical Implications
The results substantiate that adaptive collaboration, rather than fixed pipelines, unlocks substantial gains in response quality, efficiency, and privacy. This work reframes cloud-vs-local inference as a dialogic, cost-sensitive policy learning problem that can be addressed with RL, positioning heterogeneous LLM ensembles as flexible, privacy-oriented MASs.
Practically, this architecture is salient for cost-sensitive end-user deployments requiring robust privacy guarantees (healthcare, finance, local device operation). The transferability of learned policies across LLM scales and domains reduces training overhead and enhances long-term maintainability. The privacy injection pipeline and automated leakage evaluation advance evaluation methodology, supporting principled benchmarks for privacy-preserving metamodel strategies.
Theoretically, the work highlights the centrality of information asymmetry, agent capability heterogeneity, and process-level supervision in multi-agent LLM systems. It opens avenues for decentralized RL in MASs, compositional privacy-protective reasoning, and dynamic policy adaptation under opaque or evolving LLM APIs.
Future Directions
- Fine-grained Adaptivity: Incorporation of uncertainty modeling in SLMs to explicitly calibrate request thresholds and balance answer confidence with privacy risk.
- Multi-agent Collaboration: Extension to MASs with multiple heterogeneous SLMs or LLMs interacting concurrently, supporting federated, cross-device collaboration with privacy submodularity.
- Robustness and Adversarial Constraints: Studies of adversarial scenarios, including privacy attacks on query reformulations, and robust policy optimization for long-tail queries.
- Dynamic LLM capabilities modeling: Online update and tracking of LLM capabilities within collaborative agents to support continual learning and cross-model evaluation.
Conclusion
Dynamic RL-mediated collaboration between SLMs and LLMs transforms LLM ensembles from static processing pipelines into adaptive, privacy-aware multi-agent systems. The presented framework attains significant improvements in answer quality, interaction efficiency, and privacy preservation, with robust policy transfer across models and domains. By actively integrating informative feedback and configuring privacy-efficiency trade-offs, this work lays the groundwork for trustworthy, adaptive, and cost-effective cloud-local AI deployments.