SAFE-GRPO: Socially-Aware Navigation RL

• SAFE-GRPO is a reinforcement learning framework that augments imitation-learned navigation policies with explicit social norm compliance for human-like trajectory generation.
• It leverages flow-based stochastic policies and semantic context from vision-language models to generate trajectories that balance goal achievement and social adherence.
• Empirical results demonstrate dramatic improvements in success rate, rule compliance, and efficient navigation across complex, real-world social environments.

Socially-Aware Flow Exploration GRPO (SAFE-GRPO) is a reinforcement learning framework introduced as a core component of the SocialNav embodied navigation foundation model. SAFE-GRPO augments imitation-learned navigation policies with explicit social norm compliance, operationalizing both robust goal-oriented behavior and human-like trajectory generation via flow-based stochastic policies and reward shaping. By integrating semantic context from high-capacity vision-LLMs with a carefully formulated composite reward, SAFE-GRPO advances the state of the art in socially-compliant autonomous navigation in complex environments (Chen et al., 26 Nov 2025).

1. Motivation and Conceptual Framework

Prevailing imitation learning (IL) approaches for point-goal navigation demonstrate brittleness in unseen or norm-sensitive contexts, where high-level social conventions (e.g., sidewalk adherence, crosswalk use) are more important than shortest-path optimality. SAFE-GRPO is introduced in the final stage of SocialNav’s training pipeline to address this deficiency by infusing norm-aware reward signals, leveraging a flow-based stochastic policy parameterization, and refining the Action Expert network. The method explicitly aligns low-level trajectory generation with semantic priors derived from the system’s Brain module, realized as a vision-LLM (VLM), thereby bridging the gap between expert imitation and robust, context-sensitive reinforcement learning.

2. Mathematical Formulation and Policy Structure

The SAFE-GRPO policy employs a time-indexed velocity field $\bm{v}_\text{flow}$, conditioned on a semantic embedding $\bm{Z}_{\mathrm{VLM}}$ produced by the pretrained Brain module. Trajectory sampling proceeds by integrating a stochastic differential equation (SDE):

$$d\bm{x}_t = \bm{v}_\text{flow}\bigl(\bm{x}_t,t;\,\bm{Z}_{\mathrm{VLM}}\bigr)\,dt + \sigma_t\,d\bm{w}_t$$

where $\sigma_t$ modulates exploration and $d\bm{w}_t$ is Wiener process noise. Candidate trajectory points $\{\bm{x}_{t_k}\}_{k=0}^K$ are generated using Euler–Maruyama discretization. Waypoint-based actions are then decoded from denoised SDE endpoints.

The RL objective is to maximize the expected cumulative return:

$$J(\theta) = \mathbb{E}_{\tau\sim \pi_\theta}[R(\tau)],\quad R(\tau)=\sum_{t=0}^{T-1}\gamma^t r_t$$

with $\theta$ denoting policy parameters. The reward $R(\tau)$ is a convex combination of social compliance, expert similarity, smoothness, and efficiency terms:

$$R(\tau) = R_{\mathrm{social}} + \lambda_{\mathrm{expert}} R_{\mathrm{expert}} + \lambda_{\mathrm{smooth}} R_{\mathrm{smooth}} + \lambda_{\mathrm{eff}} R_{\mathrm{eff}}$$

Policy updates use a REINFORCE-style gradient estimator, averaging over multiple SDE-sampled trajectories and incorporating a learned baseline for variance reduction.

3. Algorithmic Workflow

The SAFE-GRPO algorithm is designed as follows:

1. Semantic Prior Extraction: For each simulated navigation episode, semantic context $\bm{Z}_{\mathrm{VLM}}$ is computed using the frozen Brain module on recent observations and goal.
2. Stochastic Trajectory Generation: For $N$ rollouts per episode, a trajectory is synthesized via SDE integration with prescribed noise.
3. Action Decoding and Reward Evaluation: Waypoints are decoded into actions. The social, expert, smoothness, and efficiency rewards are calculated using the semantic occupancy map, expert demonstrations, and dynamics properties.
4. Policy Gradient Update: The Action Expert parameters $\theta$ are updated using the estimated policy gradient:

$$\nabla_\theta J \approx \mathbb{E}_{\tau\sim\pi_\theta}[ (R(\tau)-b)\,\nabla_\theta\log p_\theta(\tau) ]$$

This process repeats over batches of episodes, using AdamW with a learning rate of $5\times10^{-7}$ and SDE rollout batch size of 128.

4. Reward Engineering for Social Compliance

A distinguishing feature of SAFE-GRPO is the explicit incorporation of a social-compliance reward constructed from semantic occupancy maps $M_{\mathrm{occ}}$. For each trajectory,

• $D(\bm{x})$ gives Euclidean clearance from non-traversable obstacles.
• $\bar d_{\mathrm{pred}}$ is the mean clearance of predicted waypoints, with $\bar d_{\mathrm{gt}}$ the corresponding expert value.

The social reward is:

$$R_{\mathrm{social}} = \beta\,\sigma\left(\frac{\bar d_{\mathrm{pred}}-\bar d_{\mathrm{gt}}}{\alpha}\right)$$

where $\sigma(\cdot)$ is the sigmoid function and hyperparameters ($\alpha=0.5$, $\beta=2.0$) ensure that deviations from expert-like clearance are penalized or encouraged appropriately.

The expert similarity reward combines path and orientation accuracy; smoothness penalizes the standard deviation of step sizes; and efficiency compares path progress to expert demonstrations.

5. Place in the Hierarchical SocialNav Training Pipeline

SAFE-GRPO is deployed in Stage 3 of SocialNav’s hierarchical multi-stage training:

• Stage 1 (Pre-training): Jointly train Brain (Qwen2.5-VL-3B) and Action Expert on video, simulation, and cognitive activation datasets using imitation learning.
• Stage 2 (Fine-tuning): Freeze Brain. Fine-tune Action Expert on high-fidelity real-robot data.
• Stage 3 (SAFE-GRPO RL): Freeze both Brain and waypoint encoder. Apply SAFE-GRPO in large-scale SocCity simulation, optimizing the Action Expert with the stochastic, reward-sensitive pipeline.

The architecture employs the Qwen2.5-VL-3B (36 layers, 16-head attention) as Brain and a 12-layer Diffusion Transformer as the Action Expert.

6. Empirical Performance and Ablative Insights

SAFE-GRPO delivers substantial gains across the SocNav Benchmark, covering nine previously unseen social environments:

Method	SR ↑	RC ↑	SPL ↑	DCR ↑	TCR ↑
CityWalker	47.8	64.7	44.7	36.1	36.6
SocialNav (IL only real)	65.0	78.4	62.3	58.0	56.7
SocialNav (Full)	86.1 (+38.3)	91.2 (+26.5)	77.4 (+32.7)	82.5 (+46.4)	82.9 (+46.3)

Key metrics include Success Rate (SR), Rule Compliance (RC), Success weighted by Path Length (SPL), Distance Compliance Rate (DCR), and Traversability Compliance Rate (TCR). SocialNav with SAFE-GRPO yields dramatic improvements in both goal success and social compliance without significant sacrifices in path efficiency.

Ablation studies reveal that omitting cognitive signals before RL reduces DCR and TCR, underscoring the necessity of pretraining. Removing $R_{\mathrm{social}}$ causes DCR to drop sharply, affirming the central role of the social compliance reward. A minor SPL drop (79.4 → 77.4) is interpreted as prioritizing human-like compliance over strict geometric optimality.

7. Implementation Practices and Research Perspectives

The complete system is trained in BF16 precision with FlashAttention 2 and gradient checkpointing. Inference operates at approximately 5 Hz on NVIDIA A10 hardware, supporting real-time deployment on quadruped robots.

Limitations include the reliance on hand-tuned reward weights and functional reward components; future directions include automating reward discovery via preference inference or LLM-guided shaping, and generalizing the flow-exploration paradigm to multi-agent and physically interactive tasks.

In summary, SAFE-GRPO defines a reproducible, flow-based RL mechanism that injects social norm priors into large-scale navigation models, achieving state-of-the-art trade-offs between robust scene traversal and compliance with human social conventions (Chen et al., 26 Nov 2025).