Integrated YOLOP and Lyapunov Control

• Integrated YOLOP Perception and Lyapunov-based Control is a framework that fuses multi-task visual perception with proven Lyapunov-based control for map-free autonomous navigation.
• The approach employs YOLOP for object detection, lane segmentation, and drivable area identification coupled with geometric reconstruction techniques like arc-length resampling and cubic fitting.
• Lyapunov control laws, enhanced by learning-enabled certificates, ensure asymptotic convergence and robustness even under noisy or partially available perception inputs.

Integrated YOLOP Perception and Lyapunov-based Control frameworks combine advanced real-time multi-task visual perception with provably stable tracking controllers to achieve robust, map-free autonomous navigation for nonholonomic differential-drive robots. Core implementations rely on end-to-end pipelines, fusing a single-shot YOLOP (You Only Look Once for Panoptic driving Perception) vision backbone with stable Lyapunov control laws, or learning-enabled hybrid controllers equipped with Control Barrier and Control Lyapunov Functions. These systems deliver real-world performance and formal guarantees of convergence and safety even under partial, noisy, or dynamic perception scenarios (Chen, 1 Dec 2025, Dawson et al., 2022).

1. YOLOP Multi-task Perception Architecture

YOLOP extends a YOLOv5-style CSPDarknet encoder with domain-specific task heads:

• Detection head: Outputs per-anchor, per-scale predictions for objectness, class, and bounding box localization.
• Lane-line segmentation head: Decoder up-samples FPN features to produce a two-channel lane-vs-background mask.
• Drivable-area head: Optionally, in the full model, outputs road vs. non-road regions.

Training utilizes the BDD100K dataset with combined detection, lane mask, and area mask supervision. The compound loss function is

$$\mathcal{L} = \mathcal{L}_\text{det} + \mathcal{L}_\text{da-seg} + \mathcal{L}_\text{ll-seg}$$

featuring:

• $\mathcal{L}_\text{det}$: Binary cross-entropy on objectness, multi-class cross-entropy on categories, CIoU for bounding box regression.
• $\mathcal{L}_\text{ll-seg}$: $\mathcal{L}_\text{CE}$ + $\mathcal{L}_\text{IoU}$ for lane masks, with extra regularization for thin pixels.
• $\mathcal{L}_\text{da-seg}$: Analogous segmentation supervision for drivable area.

At inference, normalized camera images are passed through the shared encoder, yielding detection, lane, and area outputs. Post-processing includes Non-Max Suppression, binary mask thresholding, and extraction of lane pixels for subsequent geometric processing (Chen, 1 Dec 2025).

2. Geometric Lane Reconstruction: 2D-to-3D Projection, Arc-length Resampling, and Cubic Fitting

Semantic masks from YOLOP are mapped to physical coordinates via a calibrated camera geometry under a flat-ground assumption:

• Projection: For each mask pixel $(u,v)$, invert intrinsic $K$ and apply extrinsic vehicle transform $T_{\text{veh}\leftarrow \text{cam}}$ to obtain 3D points in the robot frame.
• Arc-length resampling: Raw point clouds are nonuniform; compute cumulative arc-lengths $S_i$ and interpolate at uniform increments $\mathcal{L}_\text{det}$0 to obtain evenly spaced curve points $\mathcal{L}_\text{det}$1.
• Polynomial Fitting: Fit a cubic $\mathcal{L}_\text{det}$2 to the resampled points via a robust QR least-squares solution; left/right lanes are curve-averaged to define the navigable centerline.

This routine enables the conversion of discrete, possibly noisy mask outputs into smooth, continuous paths required for feedback control (Chen, 1 Dec 2025).

3. Lyapunov-based Control Law Design and Stability Analysis

Tracking control is formulated in the state error space relative to a moving target point $\mathcal{L}_\text{det}$3 represented via polar variables:

• Error dynamics:
  • $\mathcal{L}_\text{det}$4: distance to target
  • $\mathcal{L}_\text{det}$5, $\mathcal{L}_\text{det}$6: orientation errors relative to current and target headings
• Composite Lyapunov function:

$\mathcal{L}_\text{det}$7

• Control Law:
  • Linear velocity:

  $\mathcal{L}_\text{det}$8 - Angular velocity:

  $\mathcal{L}_\text{det}$9

with gains $\mathcal{L}_\text{ll-seg}$0 selected to ensure $\mathcal{L}_\text{ll-seg}$1 except at equilibrium.

Standard Lyapunov analysis ensures asymptotic convergence of both positional and heading errors, with bounded actuator commands and formal closed-loop stability guarantees (Chen, 1 Dec 2025).

4. Integrated Real-time Perception-to-Control Pipeline and Implementation Metrics

The end-to-end integration operates as follows:

1. Acquire stereo camera images (e.g., ZED) at 20–30 Hz.
2. YOLOP inference (≈25 ms) outputs lane masks, processed to extract pixel clusters.
3. 2D-to-3D projection maps pixels to robot-frame coordinates.
4. Arc-length resampling and cubic polynomial fitting (≈5 ms).
5. Centerline generation, with look-ahead points chosen at fixed distances ahead (e.g., 2, 2.5, 3 m).
6. Lyapunov control computes $\mathcal{L}_\text{ll-seg}$2 based on selected targets; commands are sent to differential drive.

All computation executes on embedded hardware (Jetson AGX Orin), supporting closed-loop rates of ≈20–30 Hz with end-to-end latency around 30–40 ms (YOLOP 25 ms, geometric + fitting ≈5 ms, control ≪1 ms). The system was empirically validated on Formula-One–style tracks, achieving mean lateral errors of 0.51 m, orientation MAE of 0.15 rad, and robust stability even under partial or noisy lane perception. Higher speed tests ($\mathcal{L}_\text{ll-seg}$3 m/s) sustained stable, collision-free trajectories with modestly increased errors (Chen, 1 Dec 2025).

Stage	Typical Latency (ms)	Hardware
YOLOP inference	25	Jetson AGX Orin
Projection + resampling + fitting	5	Jetson AGX Orin
Control law evaluation	≪1	Jetson AGX Orin
End-to-end control loop	35	Jetson AGX Orin

5. Extensions: Learning-enabled Perception-to-Control with Certificates

Hybrid control architectures enhance formal guarantees by combining YOLOP front-ends with neural-network–based certificates:

• System architecture: YOLOP multi-head outputs (boxes, lane, area) are encoded into an observation vector $\mathcal{L}_\text{ll-seg}$4, which feeds learned Control Barrier Function (CBF) and Control Lyapunov Function (CLF) networks.
• Hybrid controller: LOCUS (Learning-enabled Observation-feedback Control Using Switching) switches between goal-seeking (solving a QP with CBF and CLF constraints) and exploratory (CBF-level set walk) modes depending on QP feasibility (Dawson et al., 2022).
• Certificate learning: CBF $\mathcal{L}_\text{ll-seg}$5 and CLF $\mathcal{L}_\text{ll-seg}$6 are trained with boundary and dynamics losses, using simulation-generated, domain-randomized datasets. Safety and liveness conditions are imposed directly in observation space, with real-time QP or sample-based control at ≈10 Hz.

This approach provides forward invariance of safety sets, geometric convergence to goals, and explicit bounds on the effect of bounded perception errors. It generalizes to unknown environments and can tolerate partial or corrupted YOLOP outputs if training margins $\mathcal{L}_\text{ll-seg}$7, $\mathcal{L}_\text{ll-seg}$8 are sufficient to buffer against segmentation-classification noise (Dawson et al., 2022).

6. Comparative Perspectives and Implications

Joint YOLOP–Lyapunov control delivers a deterministic, modular approach with closed-form stability guarantees and competitive real-world trajectory smoothness, specifically for track navigation without reliance on prior maps or external localization. Learning-enabled extensions, such as the LOCUS hybrid controller, introduce flexibility and generalization to more diverse or dynamic environments, enabling safety and liveness certificates without a separate state estimator.

The pipeline’s modularity enables adaptation to different perceptual front-ends and embedded platforms, with real-time feasibility confirmed on contemporary embedded GPUs. A plausible implication is broader adoption of such integrated architectures in field robotics tasks demanding formal guarantees, low latency, and robustness to perceptual imperfections.

7. Experimental Outcomes and Significance

Empirical evaluations demonstrate stable, real-time navigation with:

• Lateral and orientation errors lower than conventional baselines (e.g., 0.51 m MAE lateral, 0.15 rad MAE orientation at $\mathcal{L}_\text{ll-seg}$9 m/s).
• Zero-collision, smooth tracking in structured and semi-structured environments.
• Robust closed-loop operation at 20–30 Hz, sustained under dynamic scene changes and partial perception.

Experiments under challenging cases—such as partial lane masking, dynamic obstacles, and increased speeds—verify the resilience and stability of the integrated approach. Both deterministic and learned-certificate controllers leverage YOLOP’s multi-task perception to produce reliable, directly actionable geometric cues without requiring high-definition maps or global localization, setting a precedent for future end-to-end autonomous robotic platforms (Chen, 1 Dec 2025, Dawson et al., 2022).