Adaptive RAN Slicing Control via Reward-Free Self-Finetuning Agents

Abstract: The integration of Generative AI models into AI-native network systems offers a transformative path toward achieving autonomous and adaptive control. However, the application of such models to continuous control tasks is impeded by intrinsic architectural limitations, including finite context windows, the lack of explicit reward signals, and the degradation of the long context. This paper posits that the key to unlocking robust continuous control is enabling agents to internalize experience by distilling it into their parameters, rather than relying on prompt-based memory. To this end, we propose a novel self-finetuning framework that enables agentic systems to learn continuously through direct interaction with the environment, bypassing the need for handcrafted rewards. Our framework implements a bi-perspective reflection mechanism that generates autonomous linguistic feedback to construct preference datasets from interaction history. A subsequent preference-based fine-tuning process distills long-horizon experiences into the model's parameters. We evaluate our approach on a dynamic Radio Access Network (RAN) slicing task, a challenging multi-objective control problem that requires the resolution of acute trade-offs between spectrum efficiency, service quality, and reconfiguration stability under volatile network conditions. Experimental results show that our framework outperforms standard Reinforcement Learning (RL) baselines and existing LLM-based agents in sample efficiency, stability, and multi-metric optimization. These findings demonstrate the potential of self-improving generative agents for continuous control tasks, paving the way for future AI-native network infrastructure.

• The paper introduces a Reflective MDP and Actor-Reflector framework that replaces handcrafted scalar rewards with language-based trajectory evaluation and KTO self-finetuning.
• The proposed agent achieves the highest overall utility of 25,702.2, reduces reconfigurations to 21.091, and matches Reflexion on QoS violations after one trajectory and training iteration.
• The results show strong sample efficiency but remain limited by LLM inference latency, Reflector reliability, and validation in larger, more diverse RAN environments.

Overview and motivation

This paper addresses a central obstacle to deploying LLM-based agents in AI-native network control: while LLMs can act without handcrafted reward functions, prompt-based memory mechanisms such as Reflexion and ExpeL are constrained by finite context windows and long-context degradation ("lost in the middle"), confining them to short-horizon, episodic tasks. The authors argue that robust continuous control requires agents to internalize experience into model parameters rather than accumulate it in prompts. They instantiate this argument with a self-finetuning framework evaluated on dynamic RAN slicing, formulated over an AI-RAN architecture with a controller on the RIC performing inter-slice PRB allocation at a 100 ms decision interval.

The work targets the reward engineering bottleneck in network RL: the authors cite evidence that over 90% of RL practitioners rely on manual trial-and-error reward design and nearly 90% acknowledge suboptimal final reward designs. The RAN slicing problem is formulated as a multi-objective optimization problem (MOOP) over three conflicting objectives: maximizing spectrum efficiency (SE), minimizing cumulative Packet QoS (PQoS) violations $V$, and minimizing resource reconfiguration counts $C$.

Methodology

Reflective MDP (R-MDP). The paper formalizes an extension of the MDP, $\langle S, A, \Psi, \Phi, M, P' \rangle$, in which the policy emits a triplet $(\psi_t, a_t, \phi_t)$—a step-level reflection, an action, and a decision analysis—conditioned on the trajectory history $H_{t-1}$. The environment returns a metric vector $M_t$ (latency, throughput, etc.) that is recorded but never collapsed into a scalar reward; the optimization objective substitutes an implicit language-derived reward $r_{lang}$ for the scalar return.

Actor-Reflector (AR) framework. The AR architecture mirrors Actor-Critic but replaces the Critic's scalar value estimate with a Reflector that performs trajectory-level, language-mediated evaluation. The Actor is an LLM (Qwen3-4B) that embeds reflections and analyses in-context as short-term memory; the Reflector (DeepSeek-R1) reviews the full trajectory after each episode, assigning each step a binary quality label $\ell_t$ and proposing an improved action $\hat{a}_t$ for suboptimal steps. This bi-perspective mechanism—step-level reflection within prompts plus trajectory-level retrospective analysis—preserves the advantage-guided update structure of AC while operating entirely in the language domain.

Refine-from-Reflection (RfR). Labeled trajectories are converted into preference datasets and used to fine-tune the Actor with Kahneman-Tversky Optimization (KTO), chosen over DPO because it supports unbalanced positive/negative sets. Each KTO iteration constructs a dataset from two sources: (1) Reflector-labeled examples directly from the trajectory, and (2) "refine-rollout" samples, where the Actor is sampled $m$ times at suboptimal states; outputs matching $C$0 become positives, and rollouts cease for a state once $C$1 exceeds a threshold $C$2. The asymmetric KTO weights are set proportionally to class imbalance. This recursive exploitation of a single trajectory—generating candidate actions offline rather than collecting new environment data—is the claimed source of the framework's sample efficiency.

Experimental setup

Evaluation uses a custom ns-3-based RAN slicing simulator with packet-level fidelity, 3GPP TR 38.901 urban propagation, frequency-selective fading traces, and on/off traffic models for GBR (20 UEs, 10 ms delay bound, 0.5 Mb/s) and non-GBR (4 UEs, 50 ms delay bound, 2 Mb/s) slices. Baselines comprise DQN, SAC, and PPO (Ray RLlib), trained with a weighted scalar reward over SE, reconfiguration penalties, and QoS penalties, and Reflexion adapted with the same backbone models as the proposed method to isolate architectural contributions.

Results

The headline claim is that Self-Finetuning achieves the best overall utility with a single trajectory collection and one training iteration, whereas RL baselines consume 80 rounds × 20 trajectories (1,600 trajectories) with unstable convergence. Key figures:

Algorithm	Avg. SE	Reconf. times	PQoS violations	Utility
Self-Finetuning	5.354	21.091	8.561	25702.2
Reflexion	5.299	29.454	8.630	25314.69
DQN	5.219	46.204	15.911	22519.1
PPO	3.587	51.411	1.997	19277.2
SAC	5.748	44.775	59.967	11704.3

Self-Finetuning reduces reconfigurations by 59% relative to PPO and 28.4% relative to Reflexion, while matching Reflexion on PQoS violations and trailing only PPO, which effectively over-optimizes that single objective at the cost of SE and reconfiguration overhead. SAC attains the highest SE but with severe violation counts and training oscillation, illustrating the reward trade-off pathology the framework aims to avoid. Within a single iteration, six successive KTO rounds on one trajectory reduce reconfiguration frequency by approximately 33% and improve SE slightly, with chosen/rejected KTO rewards converging toward zero—indicating the trajectory's information is fully exploited. The authors attribute Reflexion's weaker stability to its dependence on accumulated prompt history, from which the evaluator cannot distill effective strategies in long-horizon settings.

Limitations and open questions

The authors acknowledge that LLM inference latency precludes real-time deployment at the 100 ms control interval; the paper does not quantify inference overhead, and the reported gains are obtained in simulation with offline fine-tuning between episodes. The Reflector's labels are generated by an LLM rather than validated against ground-truth counterfactuals, so the quality of the preference signal—and its robustness to Reflector errors—is an open question. The evaluation covers a single two-slice traffic configuration; generalization across network scales, slice counts, and traffic regimes is not demonstrated. The proposed distillation of the fine-tuned policy into lightweight models for deployment is left as future work rather than implemented.

Conclusion

The paper contributes a coherent formalism (R-MDP), an Actor-Reflector architecture that recasts the Critic's role as trajectory-level linguistic reflection, and an RfR fine-tuning pipeline that converts reflection-labeled trajectories into KTO preference data. Its strongest empirical claim—competitive multi-objective performance from a single trajectory, where RL baselines require three orders of magnitude more interaction data—is well supported within the evaluated simulator, positioning self-finetuning generative agents as a credible alternative to reward-engineered RL for continuous network control, contingent on resolving inference latency and Reflector reliability.