---
title: Ray-level Panoptic Quality (RayPQ)
url: https://www.emergentmind.com/topics/ray-level-panoptic-quality-raypq
type: topic
---

# Ray-level Panoptic Quality (RayPQ)

Ray-level Panoptic Quality (RayPQ) is a segmentation metric designed for evaluating the joint quality of semantic and instance-level predictions in 3D occupancy settings, where the atomic elements are sensor rays rather than image pixels or 3D voxels. RayPQ retains the structure and spirit of the classical Panoptic Quality (PQ) metric but adapts both the matching and counting logic to operate along sensor-observed rays, which is central to panoptic-occupancy benchmarks in autonomous perception, notably Occ3D-nuScenes. RayPQ is now established as a primary metric for panoptic occupancy, directly aligning with the outputs of multi-view sensor-based methods and addressing the evaluation requirements of modern 3D scene understanding pipelines [2406.10527, 2603.06166].

## 1. Formal Definition and Mathematical Characterization

Let $R = \{r_1, r_2, ..., r_{|R|}\}$ be the set of all sensor (camera or lidar) rays, with one ray per pixel per camera or per voxel column in a regular 3D grid. For each ray $r$, both the prediction and the ground truth specify the first 'visible' occupied voxel encountered, along with its semantic label $\hat{l}(r) \in \{1, ..., N_S\}$ and instance ID $\hat{\imath}(r) \in \mathbb{N}$. For ground truth, denote $l(r)$ and $i(r)$, respectively. For every predicted instance $p$ and ground-truth instance $g$, define:

- Predicted ray segment: $\hat{R}_p := \{ r \in R \mid \hat{\imath}(r) = p \}$
- Ground-truth ray segment: $R_g := \{ r \in R \mid i(r) = g \}$

The RayIoU between predicted instance $p$ and ground truth $g$ is:
$$
\mathrm{RayIoU}(p, g) := \frac{|\hat{R}_p \cap R_g|}{|\hat{R}_p \cup R_g|}
$$

A predicted instance $p$ and ground-truth $g$ are matched as a true positive (TP) if $\mathrm{RayIoU}(p, g) > \tau_{\mathrm{match}}$, usually with $\tau_{\mathrm{match}} = 0.5$. Sets of true positives (TP), false positives (FP, predicted but not matched), and false negatives (FN, ground truth not matched) are:

- TP: all matched $(p, g)$ pairs
- FP: all $p$ not matched to any $g$
- FN: all $g$ not matched to any $p$

RayPQ is then
$$
\mathrm{RayPQ} = \frac{ \sum_{(p, g)\in \mathrm{TP}} \mathrm{RayIoU}(p, g) }{ |\mathrm{TP}| + \frac{1}{2}|\mathrm{FP}| + \frac{1}{2}|\mathrm{FN}| }
$$
The output is a scalar in $[0, 1]$, typically reported as a percentage [2406.10527].

## 2. Computational Workflow in Panoptic Occupancy

RayPQ computation proceeds as follows:

1. Rays are cast from each image pixel (camera-based) or voxel column (lidar-based), according to the sensor model and poses.
2. For each ray, the first encountered occupied voxel (the "visible surface" or "first-hit") is determined in both prediction and ground truth.
3. Record the predicted $(\hat{l}(r), \hat{\imath}(r))$ and ground truth $(l(r), i(r))$ for each ray $r$.
4. Group rays by predicted and ground-truth instances to define ray segments.
5. For all pairs $(p, g)$, compute $\mathrm{RayIoU}(p, g)$.
6. Match $(p, g)$ pairs with $\mathrm{RayIoU} > 0.5$ as TPs; unmatched $p$ are FP, unmatched $g$ are FN.
7. Aggregate results via the RayPQ formula above [2406.10527, 2603.06166].

This produces a global RayPQ score, as well as per-range RayPQ variants, depending on the spatial partitioning of rays.

## 3. Relationship to Related Metrics: PQ and RayIoU

Classical Panoptic Quality (PQ) measures, as established by Kirillov et al. (2019), evaluate panoptic segmentation by matching connected components (segments) across pixels in 2D or voxels in 3D, using IoU as the region overlap criterion. RayIoU, as introduced in Occ3D, is the semantic analog defined over sensor rays, reporting per-class occupancy accuracy via intersection/union counts for sets of rays assigned to each semantic class [2406.10527, 2603.06166].

Key distinctions:
- **PQ (pixel/voxel-level):** Atomic elements are pixels or voxels; segments are connected components; matches are made via pixel-/voxel-IoU.
- **RayIoU (ray-level, semantic):** Ignores instance IDs; compares semantic occupancy labels per ray.
- **RayPQ (ray-level, panoptic):** Atomic elements are rays; segments are ray-segments defined by instance ID and semantic class along the ray's first hit; matches combine instance grouping and per-ray assignment.

Advantages of RayPQ over RayIoU and PQ:
- Simultaneously evaluates segmentation accuracy and instance grouping.
- Penalizes missing (FN) or spurious (FP) instances via denominator.
- Aligned with sensor-ray visibility, handling occlusion natively and avoiding the need to densify to 3D masks [2406.10527].

## 4. Practical Implementation and Interpretative Properties

RayPQ aligns evaluation with the data modality actually sensed at test time, i.e., only visible surfaces as observed via sensor projections. Each ray is evaluated equally, preventing volume bias due to surface thickness or occluded geometry. The focus on first-hit assignment avoids the ambiguity of dense 3D annotation and is computationally scalable, facilitating differentiation in 'soft' evaluations and direct aggregation in post-processing [2406.10527, 2603.06166].

Practical evaluation highlights:
- Reported RayPQ values on Occ3D-nuScenes are: Panoptic-FlashOcc, 16.0 (fully supervised); FreeOcc, 3.1 (training-free), with gains up to 3.9 under weak supervision [2603.06166].
- Instance identification significantly impacts RayPQ (>1 point improvement in ablations), while semantic, geometric, and occupancy refinements also contribute incrementally.
- The metric is sensitive to accurate camera extrinsics: incorrect poses cause substantial RayPQ collapse (2.5 → 1.1 in ablations).
- Use of non-causal/future frames can raise RayPQ by up to 40% (2.5 → 3.5) [2603.06166].

Ablation results demonstrate the cumulative effect of system components, with the largest RayPQ jumps achieved through explicit instance-awareness.

## 5. Comparative Evaluation and Suitability for Panoptic Occupancy

RayPQ is the chosen main metric for panoptic occupancy tasks on Occ3D-nuScenes, precisely because it embodies both panoptic instance grouping and sensor-visibility constraints. Unlike PQ applied directly on voxels—which is not naturally consistent with visibility from sensor viewpoints—or RayIoU, which is purely semantic, RayPQ fully implements the intended evaluation of instance-aware, content-aligned predictions as realized on rays [2406.10527, 2603.06166].

In experimental reporting:
- Panoptic-FlashOcc achieves 16.0 RayPQ at 30.2 FPS, surpassing prior methods on both accuracy and efficiency [2406.10527].
- FreeOcc sets 3.1 RayPQ as the baseline for training-free panoptic occupancy, with substantial performance improvements at longer ranges (RayPQ_4m = 5.8) confirming the metric’s geometric fidelity.

RayPQ correlates with perceptual sharpness and functional task performance, such as driver-centric safety (e.g., obstacle avoidance), establishing its practical relevance for autonomous systems.

## 6. Summary Table: Key Metrics and Their Evaluated Properties

| Metric     | Instance Awareness | Atomic Element | Reporting Base | Occlusion Handling | Penalizes FP/FN |
|------------|-------------------|---------------|---------------|--------------------|-----------------|
| PQ         | Yes               | Pixels/Voxels | 2D/3D grid    | No                 | Yes             |
| RayIoU     | No                | Rays          | Sensor rays   | Yes                | No (semantic)   |
| **RayPQ**  | Yes               | Rays          | Sensor rays   | Yes                | Yes             |

RayPQ generalizes PQ to the 3D occupancy setting, using rays instead of grid elements, and is strictly more informative than RayIoU for panoptic tasks.

## 7. References

- Panoptic-FlashOcc: Y. Zichen et al., "Panoptic-FlashOcc: An Efficient Baseline to Marry Semantic Occupancy with Panoptic via Instance Center," Occ3D-nuScenes Benchmark [2406.10527].
- FreeOcc: "FreeOcc: Training-free Panoptic Occupancy Prediction via Foundation Models" [2603.06166].
- Kirillov, A. et al., "Panoptic segmentation," CVPR 2019 (classical PQ metric).
- Occ3D Benchmark: Tian, X. et al., "Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving," arXiv 2023 (RayIoU context).

Source: https://www.emergentmind.com/topics/ray-level-panoptic-quality-raypq