The End of Manual Decoding: Towards Truly End-to-End Language Models (2510.26697v1)

Abstract: The "end-to-end" label for LLMs is a misnomer. In practice, they depend on a non-differentiable decoding process that requires laborious, hand-tuning of hyperparameters like temperature and top-p. This paper introduces AutoDeco, a novel architecture that enables truly "end-to-end" generation by learning to control its own decoding strategy. We augment the standard transformer with lightweight heads that, at each step, dynamically predict context-specific temperature and top-p values alongside the next-token logits. This approach transforms decoding into a parametric, token-level process, allowing the model to self-regulate its sampling strategy within a single forward pass. Through extensive experiments on eight benchmarks, we demonstrate that AutoDeco not only significantly outperforms default decoding strategies but also achieves performance comparable to an oracle-tuned baseline derived from "hacking the test set"-a practical upper bound for any static method. Crucially, we uncover an emergent capability for instruction-based decoding control: the model learns to interpret natural language commands (e.g., "generate with low randomness") and adjusts its predicted temperature and top-p on a token-by-token basis, opening a new paradigm for steerable and interactive LLM decoding.

• The paper presents AutoDeco, which dynamically predicts temperature and top-p parameters at each token generation step to replace static, manual tuning.
• It introduces lightweight MLP heads and a differentiable soft top-p mask that enable end-to-end optimization, matching near-oracle performance with minimal computational overhead.
• The emergent natural language-based control allows users to steer decoding style interactively, broadening applications in dialogue, code synthesis, and scientific reasoning.

AutoDeco: End-to-End Adaptive Decoding for LLMs

Introduction

The paper "The End of Manual Decoding: Towards Truly End-to-End LLMs" (2510.26697) introduces AutoDeco, a novel architecture that augments transformer-based LLMs with lightweight prediction heads for dynamic, token-level control of decoding hyperparameters—specifically temperature and top-p. This approach addresses a fundamental limitation in current LLM deployment: the reliance on static, manually-tuned decoding parameters, which are suboptimal and require significant human and computational effort. AutoDeco enables models to self-regulate their sampling strategy in a fully differentiable, end-to-end manner, eliminating the need for manual hyperparameter sweeps and facilitating context-sensitive generation.

Figure 1: AutoDeco architecture augments the transformer with heads that predict temperature and top-p at each generation step, enabling dynamic, context-aware decoding; manual decoding uses static parameters for the entire sequence.

Architecture and Training Methodology

AutoDeco integrates two lightweight MLP heads into the transformer architecture: one for temperature prediction and one for top-p prediction. At each generation step, the model's hidden state is used to infer optimal sampling parameters, which are then applied to the logits before token selection. The top-p head is conditioned on both the hidden state and the predicted temperature, allowing for nuanced interplay between stochasticity and truncation.

A key technical challenge is the non-differentiability of standard top-p sampling, which precludes gradient-based optimization. The authors resolve this by introducing a differentiable "soft" top-p mask, parameterized by a decay steepness $\alpha$, which smoothly attenuates probabilities of tokens outside the nucleus. This enables backpropagation of the cross-entropy loss through the entire decoding pipeline, allowing joint optimization of both heads.

Figure 2: The differentiable soft top-p mask function enables gradient flow for end-to-end training of decoding heads.

Training is performed on reject sampling trajectories from mathematical reasoning datasets, with the base LLM frozen and only the AutoDeco heads updated. To mitigate bias towards conservative temperature predictions (due to "easy" tokens), the loss is masked for a large fraction of such positions. Dynamic fine-tuning further re-weights the loss to focus on tokens with calibrated uncertainty, improving robustness.

Inference and Computational Efficiency

During inference, AutoDeco predicts temperature and top-p for each token in parallel with the standard lm_head. The predicted parameters are used to rescale and filter the logits, producing a dynamically-adjusted probability distribution. The computational overhead is minimal: the additional FLOPs and memory usage are negligible compared to the transformer backbone, and latency increases by only 1–2%.

Empirical Results

AutoDeco is evaluated on eight benchmarks spanning mathematical reasoning and general-domain tasks (QA, code generation, instruction following) using four model families (Llama, Qwen, GPT). The primary metric is Pass@1 accuracy, with extensive oversampling for statistical robustness.

AutoDeco consistently outperforms both Greedy Search and Default Sampling across all models and tasks. Notably, its performance matches or slightly surpasses an oracle-tuned baseline, where static hyperparameters are exhaustively optimized on the test set—a scenario infeasible in practice. The gains are robust across both in-domain and out-of-domain tasks, demonstrating strong generalization.

Figure 3: Fine-grained expert-guided tuning versus AutoDeco on Llama-Nemotron-8B; AutoDeco achieves near-oracle performance without manual sweeps.

Pass@$k$ evaluations ($k=16,32,64$) reveal that AutoDeco's absolute improvements persist at higher $k$ values, with relative error reduction increasing as the baseline error rate decreases. Ablation studies show that each head (temperature or top-p) independently yields substantial gains, but joint optimization provides the best results.

Emergent Instruction-Based Decoding Control

A salient emergent property of AutoDeco is its ability to interpret natural language commands that specify desired generation styles (e.g., "generate with low randomness"). Without explicit supervision, the model spontaneously adjusts its predicted temperature and top-p in response to such meta-instructions. Targeted training with a ranking loss further solidifies this behavior, achieving 95%+ consistency in steering sampling parameters according to user intent.

Figure 4: Token-level $\hat{T}$ and $\hat{P}$ predictions for the same prompt under baseline, high-diversity, and low-diversity commands, demonstrating emergent controllability.

This capability represents a step towards interactive, user-steerable LLMs, where generation style can be modulated via natural language, rather than low-level hyperparameter adjustment.

Training Efficiency

AutoDeco is highly data-efficient, requiring only 400 steps and approximately 6K samples for convergence. The training process is label-free, as supervision is derived directly from the cross-entropy loss on generated tokens, obviating the need for external optimization modules or pre-computed labels.

Figure 5: AutoDeco's training loss curves across models, demonstrating rapid and stable convergence.

Theoretical and Practical Implications

AutoDeco challenges the prevailing paradigm of static, heuristic decoding in LLMs, demonstrating that dynamic, context-sensitive control of sampling parameters can be learned efficiently and deployed with negligible overhead. The approach generalizes across domains and model architectures, suggesting that adaptive decoding is a universal meta-skill rather than a task-specific artifact.

The emergent ability to interpret natural language instructions for decoding control opens new avenues for human-AI interaction, enabling users to specify generation style at a high level. This has implications for applications requiring fine-grained control over creativity, diversity, and factuality, such as dialogue systems, code synthesis, and scientific reasoning.

Future Directions

The current implementation freezes the base LLM and trains only the AutoDeco heads. Joint training of the backbone and heads may enable more precise and granular control, addressing limitations in absolute parameter adjustment and mitigating data biases. Further research is warranted to explore the limits of instruction-based controllability and to extend the approach to other decoding paradigms (e.g., beam search, contrastive decoding).

Conclusion

AutoDeco represents a significant advance in the design of truly end-to-end LLMs, eliminating the need for manual decoding hyperparameter tuning and enabling dynamic, context-aware generation. Its empirical performance matches oracle-tuned baselines, its computational cost is negligible, and its emergent controllability via natural language commands paves the way for more interactive and adaptive LLMs. The paradigm shift from static to dynamic decoding strategies has broad implications for both the theory and practice of neural text generation.