Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights (2506.16406v1)

Abstract: Modern Parameter-Efficient Fine-Tuning (PEFT) methods such as low-rank adaptation (LoRA) reduce the cost of customizing LLMs, yet still require a separate optimization run for every downstream dataset. We introduce \textbf{Drag-and-Drop LLMs (\textit{DnD})}, a prompt-conditioned parameter generator that eliminates per-task training by mapping a handful of unlabeled task prompts directly to LoRA weight updates. A lightweight text encoder distills each prompt batch into condition embeddings, which are then transformed by a cascaded hyper-convolutional decoder into the full set of LoRA matrices. Once trained in a diverse collection of prompt-checkpoint pairs, DnD produces task-specific parameters in seconds, yielding i) up to \textbf{12,000$\times$} lower overhead than full fine-tuning, ii) average gains up to \textbf{30\%} in performance over the strongest training LoRAs on unseen common-sense reasoning, math, coding, and multimodal benchmarks, and iii) robust cross-domain generalization despite never seeing the target data or labels. Our results demonstrate that prompt-conditioned parameter generation is a viable alternative to gradient-based adaptation for rapidly specializing LLMs. Our project is available at \href{https://jerryliang24.github.io/DnD}{https://jerryliang24.github.io/DnD}.

• The paper presents a prompt-conditioned zero-shot framework that generates LoRA weights directly from unlabeled prompts, eliminating repeated fine-tuning.
• It leverages a hyper-convolutional decoder and Sentence-BERT embeddings to rapidly synthesize adapter parameters, achieving up to 12,000× efficiency.
• Empirical results demonstrate robust zero-shot generalization and cross-domain transfer, with performance improvements averaging up to 30% on unseen tasks.

Drag-and-Drop LLMs: Prompt-Conditioned Zero-Shot Parameter Generation

This paper introduces Drag-and-Drop LLMs (DnD), a new paradigm for LLM adaptation that leverages prompt-conditioned parameter generators to produce Low-Rank Adapter (LoRA) weights in a zero-shot manner. Instead of running a separate optimization for every downstream task, DnD generates task-specialized parameters directly from a few unlabeled prompts, marking a significant advance in practical efficiency and adaptability of LLMs.

Problem Formulation and Motivation

Conventional Parameter-Efficient Fine-Tuning (PEFT) approaches such as LoRA require explicit optimization—often involving substantial compute resources and time—each time a new task or dataset is encountered. This bottleneck grows acute as the proliferation of LLM-powered applications increases demand for rapid, customized deployment at scale. Recognizing that each LoRA adapter's parameters are inherently a function of its training data, the authors propose learning a direct mapping from representative task prompts (without labels) to the required adapter weights, eliminating the need for repeated fine-tuning.

DnD Architecture and Method

DnD comprises two primary components:

1. Prompt Embedding: Batches of unlabeled task prompts are passed through a pre-trained, frozen text encoder (default: Sentence-BERT), producing compact, information-rich condition embeddings.
2. Hyper-Convolutional Decoder: These embeddings are then input into a specialized hyper-convolutional decoder, which expands them into the full set of LoRA adapter weights for every transformer layer in the base model. The design leverages cascaded convolutional modules, chosen for their efficiency and scalability.

Training:

DnD is trained on pairs of (prompt batch, LoRA weight checkpoint), using an MSE loss between generated weights and the corresponding ground truth LoRA parameters across various tasks and datasets. Strategic pairing and diversity in the training set ensure that the mapping generalizes beyond any single dataset.

Inference:

To adapt an LLM for a new task, only a small batch of representative prompts (typically 32–128 samples, depending on the domain) is needed. The system generates specialized LoRA weights in a single forward pass, which can be integrated immediately with the base LLM for downstream use—no further tuning or labels are required.

Empirical Results

The paper provides extensive experiments across multiple domains, including common sense reasoning, math, code generation, and multimodal tasks. The evaluation focuses both on in-domain (unseen datasets of a known type) and cross-domain (entirely new task types) settings.

Key Outcomes:

• Efficiency: DnD achieves up to 12,000× reduction in adaptation overhead compared to standard fine-tuning, enabling model customization in seconds on a single GPU.
• Zero-Shot Generalization: On unseen datasets, task-specific LoRA weights generated by DnD outperform the strongest training LoRAs, with average improvements up to 30% in several benchmarks.
• Cross-Domain Robustness: DnD handles transfer not only between datasets, but also across task domains (e.g., applying reasoning adapters to science QA) with positive gains.
• Scalability: The approach scales to larger backbones (tested up to 7B parameters), showing consistent performance and compatibility with increasingly complex LLMs.

The authors also perform ablations demonstrating that:

• Prompts alone are the most effective condition; mixing prompts with answers or using answers alone (in tasks with low answer diversity) reduces effectiveness.
• Encoder-based text extractors outperform decoder-only models as the basis for condition embeddings.
• The diversity and volume of training prompt–checkpoint pairs are critical for robust out-of-distribution generalization.

Practical Implications

DnD's design enables rapid, on-the-fly adaptation of LLMs without the traditional barriers of time, compute, or labeled data. Practical deployment scenarios include:

• Enterprise LLM Operations: Instantly customizing a generalist LLM for an organization’s proprietary domains using only raw text samples as prompts.
• LLMaaS Platforms: Commercial providers can offer customers bespoke model adapters without the need to transfer large datasets or perform in-house training.
• On-Device/Edge AI: Generating adapter modules for edge deployments at negligible resource cost, improving privacy and responsiveness.

Beyond LoRA, the method is potentially extensible to other PEFT schemes and weight update formats, subject to further exploration.

Theoretical and Methodological Implications

DnD underscores the feasibility of treating neural network weights as a generative data modality, synthesized directly from semantic task descriptors. This challenges the prevailing doctrine that gradient-based optimization is essential for model specialization, suggesting instead that learned hypernetworks can bridge the data–parameter gap in a single pass.

Limitations and Future Directions

• Scaling: While DnD scales to 7B-parameter models, further algorithmic innovation is needed to address much larger backbones (e.g., 70B or beyond).
• Training Data Requirements: Generalization quality is sensitive to the diversity and representativeness of prompt–checkpoint pairs seen during training.
• Heterogeneity of Adapters: Extensions to structurally different model architectures or hardware constraints may require adapting the hypernetwork’s output format.

The authors suggest investigating:

• Integration with massive, publicly available checkpoint corpora to further generalize the generator.
• Application to other modalities, adapter schemes, or multi-task/multi-modal unified generators.

Concluding Remarks

Drag-and-Drop LLMs enable a paradigm shift in model adaptation from optimization-centric to generation-centric workflows. The practical efficiency, strong zero-shot performance, and minimal deployment barriers position this approach as a promising direction for scalable, rapid, and accessible LLM customization across diverse real-world applications.