Style-Conditioned Implicit Q-Learning (SCIQL)

• The paper introduces SCIQL, a unified framework that uses subtrajectory labeling to balance explicit style supervision with high task returns in offline RL.
• It employs a hybrid method combining value-based RL, advantage-weighted regression, and hindsight relabeling to mitigate distribution shift and style-task conflicts.
• Experimental results indicate a 27% increase in style alignment and up to 164% improvement in Pareto-front hypervolume compared to previous methods.

Style-Conditioned Implicit Q-Learning (SCIQL) is an offline reinforcement learning (RL) framework designed to address the challenge of learning policies that are simultaneously style-aligned, according to explicit supervision via subtrajectory labeling, and high-performing with respect to task rewards, under conditions of distribution shift and possible conflict between style and task objectives. SCIQL introduces a unified mathematical definition of behavior style based on subtrajectory windows, integrates this into a goal-conditioned Markov Decision Process (MDP) via explicit style labels, and optimizes policy and value functions through a hybrid of value-based RL, advantage-weighted regression, and hindsight relabeling, ultimately achieving robust performance across diverse style and task requirements (Petitbois et al., 30 Jan 2026).

1. Unified Definition of Behavior Style

SCIQL formalizes style as a function of subtrajectories. Let $\mathcal{T} = \{T_{t:t+h}\}$ denote all overlapping subtrajectories of length $h$ within a trajectory. A style is specified by a labeling function

$$\ell : \mathcal{T} \rightarrow \{0,1,\dots, |\mathcal{Z}|-1\}$$

which assigns each subtrajectory window $T_{t-w+1:t+w}$ of radius $w$ to a discrete style label $z \in \mathcal{Z}$. For an offline dataset $\mathcal{D}$ of trajectories $\tau$, each transition $(s_t, a_t)$ is annotated with

$z_t = \ell(\tau_{t-w+1:t+w})$

leading to an augmented dataset

$h$0

This subtrajectory-based labeling permits multiple styles to coexist within a single episode and enables effective credit assignment when styles operate on different timescales. The formulation generalizes prior definitions based on full-trajectory or state-based styles and provides a unified platform for explicit style supervision (Petitbois et al., 30 Jan 2026).

2. RL Objective and Model Architecture

SCIQL’s objective is to jointly maximize task performance and style compliance within the offline RL paradigm. This is achieved via a Markovian task reward $h$1 given by the environment and a surrogate style reward

$h$2

where $h$3 is a neural label-predictor trained via maximum likelihood on $h$4. Conditioning on a target style $h$5 thus converts the problem into a goal-conditioned MDP.

SCIQL builds upon the Implicit Q-Learning (IQL) framework. It maintains neural approximators $h$6 and $h$7 for the style-conditioned action-value and value functions, respectively. These are optimized using expectile regression: $h$8 where $h$9 is the expectile level. To extract an implicit policy, SCIQL introduces the Gated Advantage-Weighted Regression (GAWR) mechanism, described in §3.

3. Gated Advantage-Weighted Regression and Hindsight Relabeling

SCIQL deploys a unique Gated Advantage-Weighted Regression (GAWR) for policy extraction. The style-conditioned policy $$\ell : \mathcal{T} \rightarrow \{0,1,\dots, |\mathcal{Z}|-1\}$$0 is parameterized via a neural network and is trained using the following procedure:

1. Compute the style advantage,

$$\ell : \mathcal{T} \rightarrow \{0,1,\dots, |\mathcal{Z}|-1\}$$1

and the task advantage,

$$\ell : \mathcal{T} \rightarrow \{0,1,\dots, |\mathcal{Z}|-1\}$$2

where $$\ell : \mathcal{T} \rightarrow \{0,1,\dots, |\mathcal{Z}|-1\}$$3 and $$\ell : \mathcal{T} \rightarrow \{0,1,\dots, |\mathcal{Z}|-1\}$$4 are IQL-learned on $$\ell : \mathcal{T} \rightarrow \{0,1,\dots, |\mathcal{Z}|-1\}$$5.

2. Gate the task advantage,

$$\ell : \mathcal{T} \rightarrow \{0,1,\dots, |\mathcal{Z}|-1\}$$6

where $$\ell : \mathcal{T} \rightarrow \{0,1,\dots, |\mathcal{Z}|-1\}$$7 is the sigmoid function.

3. Train $$\ell : \mathcal{T} \rightarrow \{0,1,\dots, |\mathcal{Z}|-1\}$$8 via regression on the dataset, weighted by $$\ell : \mathcal{T} \rightarrow \{0,1,\dots, |\mathcal{Z}|-1\}$$9:

$T_{t-w+1:t+w}$0

with $T_{t-w+1:t+w}$1 as the inverse-temperature.

This gating mechanism restricts task incentives to cases where style alignment is already strong, thereby mitigating the conflict between task and style and avoiding brittle trade-off tuning. Policy extraction and value learning both incorporate hindsight style relabeling: each $T_{t-w+1:t+w}$2 sample is relabeled with either future style labels observed in its trajectory or uniformly from $T_{t-w+1:t+w}$3, supporting robustness in cases of missing or sparse style annotations (Petitbois et al., 30 Jan 2026).

4. Addressing Distribution Shift and Task–Style Conflicts

SCIQL’s design addresses two principal challenges:

• Distribution Shift: The expectile-based value fit, adopted from IQL, reduces the risk of Q-function overestimation for out-of-distribution actions. Hindsight relabeling further mitigates inference–training style mismatches by densely covering possible style targets, especially in underspecified regions of the dataset.
• Task–Style Conflict: Traditional multi-objective approaches depend on scalarization or explicit constraints, necessitating careful trade-off tuning. SCIQL’s GAWR locally gates task return behind style advantage. If $T_{t-w+1:t+w}$4 is negative (indicating style violation), task advantage is suppressed. If $T_{t-w+1:t+w}$5 is positive, task incentives can augment selection, effectively projecting onto the style-optimal policy set while maintaining strong task returns in a dynamic and robust manner (Petitbois et al., 30 Jan 2026).

5. Experimental Evaluation and Results

SCIQL was empirically validated across three offline RL domains, each augmented with multifaceted style specifications:

Domain	Task	Style Criteria Examples	Dataset Types
Circle2D	Fixed circle	Position, direction, turn type, speed, curvature	--
HalfCheetah	Forward running	Speed, angle, torso/foot heights	Fixed, Stitch, Vary
HumEnv	Humanoid motion	Head height, speed	Diverse, via Metamotivo M1

Metrics include style alignment (fraction of timesteps with correct style label), task return (undiscounted reward sum, normalized), and Pareto-front hypervolume (joint style and task coverage).

Key results:

• Across 14 criteria (all benchmarks, all labels, 5 seeds), SCIQL reached 74.6% style alignment versus 58.9% for the strongest prior (SORL imitation), a +27% relative gain.
• SCIQL dominated prior methods in Pareto-front hypervolume, with improvements of +41% to +164%.
• Ablation studies revealed that (i) while hindsight relabeling provides some benefit, the largest gains arise when combined with value learning and GAWR; (ii) GAWR is essential for raising task returns without sacrificing style; (iii) SCIQL maintains robustness to label noise up to the threshold where noise undermines majority correctness.

SCIQL thus achieves robust offline RL performance, attaining both high style adherence and strong task returns across diverse style definitions, entirely without environment interaction during training (Petitbois et al., 30 Jan 2026).

6. Algorithmic Summary

A high-level description of the key SCIQL algorithmic steps is as follows:

1. Fit the style predictor $T_{t-w+1:t+w}$6 on style-augmented data $T_{t-w+1:t+w}$7.
2. Initialize Q/V networks for both style and task objectives, as well as the policy network.
3. Alternate value learning (updating Q/V with expectile regression, utilizing relabeled style samples) and policy extraction (optimizing GAWR loss with style relabeling).
4. Output the style-conditioned policy $T_{t-w+1:t+w}$8.

All components—hindsight relabeling, Q/V networks, GAWR extraction—are integrated in the training loop, and the framework accommodates arbitrary user-defined subtrajectory style definitions (Petitbois et al., 30 Jan 2026).

7. Significance and Outlook

SCIQL provides the first robust offline RL method to achieve high style alignment and task performance under a scalable, unified definition of style via subtrajectory labeling, without requiring any environment interaction at training time. The method’s gate-based approach to resolving style–task conflicts, and its resilience to distribution shift and label noise, establish a new baseline for style-conditioned offline RL. A plausible implication is that SCIQL’s paradigm of local gating and hindsight relabeling may inform broader areas of multi-criteria or goal-conditioned behavioral learning (Petitbois et al., 30 Jan 2026).