Rule-VLN: Bridging Perception and Compliance via Semantic Reasoning and Geometric Rectification

Abstract: As embodied AI transitions to real-world deployment, the success of the Vision-and-Language Navigation (VLN) task tends to evolve from mere reachability to social compliance. However, current agents suffer from a "goal-driven trap", prioritizing physical geometry ("can I go?") over semantic rules ("may I go?"), frequently overlooking subtle regulatory constraints. To bridge this gap, we establish Rule-VLN, the first large-scale urban benchmark for rule-compliant navigation. Spanning a massive 29k-node environment, it injects 177 diverse regulatory categories into 8k constrained nodes across four curriculum levels, challenging agents with fine-grained visual and behavioral constraints. We further propose the Semantic Navigation Rectification Module (SNRM), a universal, zero-shot module designed to equip pre-trained agents with safety awareness. SNRM integrates a coarse-to-fine visual perception VLM framework with an epistemic mental map for dynamic detour planning. Experiments demonstrate that while Rule-VLN challenges state-of-the-art models, SNRM significantly restores navigation capabilities, reducing CVR by 19.26% and boosting TC by 5.97%.

• The paper introduces Rule-VLN to bridge the gap between geometric navigation and regulatory compliance by injecting semantic constraints into urban scenarios.
• It proposes the Semantic Navigation Rectification Module (SNRM), a zero-shot component that improves task completion and reduces rule violations without retraining.
• The integration of dual-stage perception and the MPSI pipeline achieves high fidelity and spatial alignment, with robust performance validated by metrics like SSIM and FID.

Rule-VLN: Bridging Perception and Compliance via Semantic Reasoning and Geometric Rectification

Introduction

The paper "Rule-VLN: Bridging Perception and Compliance via Semantic Reasoning and Geometric Rectification" (2604.16993) introduces a new paradigm in Vision-and-Language Navigation (VLN) by directly addressing the compliance deficit inherent in current embodied agents. As agents transition to real-world, urban scenarios, navigation success must extend beyond reaching a target to encompass strict adherence to semantic regulatory rules. This work identifies a persistent "goal-driven trap"—existing agents prioritize geometric traversability, systematically ignoring fine-grained regulatory constraints such as traffic signs. To resolve this, the authors propose two primary contributions: (1) Rule-VLN, a large-scale rule-compliance benchmark with semantic constraints injected into real urban environments, and (2) the Semantic Navigation Rectification Module (SNRM), a universal, zero-shot module for augmenting existing agents with rule comprehension and compliance, achieved without retraining.

Figure 1: The Rule-VLN paradigm: MPSI pipeline semantically augments urban topologies (left); SNRM enables compliant detours, contrary to standard agents (right).

Rule-VLN Benchmark and Dataset Construction

Semantic Constraint Injection and Benchmark Pipeline

Rule-VLN constitutes the first large-scale urban VLN benchmark focusing on compliance with regulatory rules. Leveraging the Touchdown dataset as its geometric backbone, the authors modify the navigation graph by systematically injecting 177 categories of regulatory signs across 8,180 strategic nodes. The semantic augmentation follows a curriculum across four levels, where constraint density and difficulty scale progressively.

The semantic constraints are dynamically mapped to traversable actions—nodes that display, for example, "No Entry" or "No Left Turn" result in corresponding prohibitions over the navigation graph. Mapping from visual semantics to action subspaces is achieved using an LLM-driven pipeline, efficiently translating subtle cues into actionable constraints.

Figure 2: Rule-VLN Construction Pipeline: LLM-action mapping and curriculum-based constraint injection with MPSI.

The CityNav-Rules-73K dataset introduces hybrid representations that interleave raw visual cues with semantic imperatives, reframing navigation from a pattern-matching problem to one of actionable reasoning. This design enforces comprehension of causal relationships between visual cues and urban rule logic.

Figure 3: MPSI Pipeline: Boundary extraction, dual-mask synthesis, and GMM-based semantic filtering.

Mask-Prioritized Semantic Injection (MPSI)

The paper identifies limitations in naive generative inpainting, particularly hallucinations and spatial misalignments when synthesizing regulatory signage. The MPSI pipeline provides geometric and semantic rectification: dual-mask conditioning ensures strict spatial fidelity and semantic consistency. Quality filtering employs CLIP-based alignment, discarding outliers using a GMM, guaranteeing legibility and realism.

Semantic Navigation Rectification Module (SNRM)

The SNRM is a plug-and-play, zero-shot module designed to overlay rule semantics onto any frozen navigation policy. Its architecture is decomposed as follows:

• Dual-Stage Coarse-to-Fine Perception: The pipeline first detects putative rule signals via lightweight detection in panoramic input, then zooms in for fine-grained symbol recognition, using explicit visual prompting to drive the attention of the VLM backbone.
• Knowledge-Driven Rule Grounding: Each detected region is compared against a regulatory knowledge base, allowing deterministic rule inference even in the presence of ambiguous or long-tail visual categories.
• Epistemic Mental Map for Trajectory Correction: Upon detecting a semantic violation, SNRM intercepts the base policy, dynamically replans compliant detours using a penalized greedy heuristic and a visual memory buffer to avoid backtracking and spatial traps.

  Figure 4: SNRM Framework: Dual-stage perception for rule extraction and epistemic correction via mental mapping.

Empirical Evaluation

Baseline Assessment and Compliance Gap

Evaluation utilizes Task Completion (TC), Shortest Path Distance (SPD), and Constraint Violation Rate (CVR), with synthesis fidelity measured by PSNR, SSIM, LPIPS, and FID for image realism. Across all benchmark levels, SOTA models exhibit catastrophic failure when exposed to explicit semantic constraints: TC drops and CVR soars, with frequent illegal action execution on geometrically permissible, but semantically barred, trajectories.

Figure 5: Model performance on the Rule-VLN benchmark; SOTA models collapse sharply in TC and CVR with increasing constraint level.

Figure 6: Trajectory comparison: SNRM executes legal detours, baseline models fail to avoid violations.

SNRM Integration and Ablation

Zero-shot application of SNRM produces significant improvements: it reduces FLAME’s CVR by 19.26% and increases TC by 5.97% at the most challenging curriculum levels—across all tested model classes, SNRM consistently enforces rule-adherent detouring without retraining. Ablation demonstrates that both macro-micro perception and rule-grounding are critical: removing either module increases CVR and/or decreases TC. The mental map further converts static rule recognition into actionable compliant navigation.

Fidelity and Semantic Alignment

Compared to standard inpainting baselines (e.g., FLUX.1-Fill, Google Nano Banana 2), MPSI produces markedly higher pixel-level and perceptual fidelity (SSIM: 0.9280, FID: 7.72 vs. 30.31), with a sharply truncated distribution of low CLIP scores, confirming effective hallucination filtering.

Figure 7: MPSI achieves superior semantic alignment (CLIP score distribution) compared to FLUX.1-Fill.

Figure 8: Qualitative analysis—MPSI produces semantically consistent, spatially aligned signage, outperforming baselines.

Implications and Prospects

The Rule-VLN benchmark is a pivotal resource for embodied-AI, urban navigation, and compliance-critical robotics research. Its curriculum demonstrates that current emphasis on geometric heuristics is substantially inadequate for real-world deployment once regulatory rules are introduced. The SNRM architecture, as a universal, training-free compliance overlay, establishes a robust template for modular behavioral rectification—such modularity is essential for rapid deployment of safety upgrades in heterogeneous agent populations.

The MPSI methodology’s advances in in-context spatial compositing are equally impactful for data generation in generative modeling, simulation, and self-driving pipeline augmentation. Its integration of topological and semantic reasoning could generalize to non-urban regulatory or social-norm navigation as well.

Future work will need to explore:

• Efficient regulatory rule detection from diverse, adversarial, and low-visibility visual signals.
• End-to-end differentiable pipelines where semantic compliance is optimized jointly with navigation.
• Real-time deployment in closed-sensor, multi-agent traffic systems and simulation-to-reality transfer for compliance generalization.

Conclusion

This work formalizes the shift in navigation success metrics—from geometric feasibility to semantic rule compliance—and provides both a high-fidelity benchmark (Rule-VLN) and a lightweight, modular compliance rectification module (SNRM) to catalyze research in this space. These contributions establish new standards for evaluating, enforcing, and measuring compliance in vision-language navigation, advancing embodied-AI towards responsible and lawful operation in real-world settings.