Point Gaussian Encoder Overview

• Point Gaussian Encoder (PGE) is a method that explicitly models 3D points by converting point features into Gaussian-parameterized representations with defined mean and covariance.
• It supports diverse applications including self-supervised rendering, radar BEV projection, point upsampling, and standard point-cloud compression through tailored architectures.
• PGE leverages factorized Gaussian parameters to directly manage geometry, anisotropy, and feature integration, improving optimization and interpretation across multiple models.

Point Gaussian Encoder (PGE) denotes a class of explicit point-based encoders in which points, point neighborhoods, or point-derived features are mapped to Gaussian-parameterized representations. In the recent literature, this designation covers several closely related constructions: a point cloud encoder whose features are converted into 3D Gaussian primitives for self-supervised rendering in GS$^3$ (Liu et al., 2024); a Gaussian-splatting radar feature encoder that turns each 4D radar point into a continuous, oriented Gaussian primitive for Bird’s-Eye View (BEV) rasterization in RadarGaussianDet3D and its ray-centric refinement in RCGDet3D (Xiong et al., 19 Sep 2025, Xiong et al., 20 May 2026); an anisotropic local Gaussian representation used for explicit point sampling in PU-Gaussian (Khater et al., 24 Sep 2025); a standardized point-cloud-compatible encoding of 3D Gaussian splats in HybridGS (Yang et al., 3 May 2025); and an explicit per-point embedding built on mixtures of learned 3D Gaussians in "3D Gaussian Point Encoders" (James et al., 6 Nov 2025). Across these variants, the common principle is that Gaussian structure is made explicit rather than implicit, so that geometry, orientation, scale, visibility, feature transport, sampling, compression, or filtering can be handled directly.

1. Terminology and conceptual scope

The term PGE is not used uniformly. RadarGaussianDet3D explicitly introduces a "novel Point Gaussian Encoder (PGE)," whereas GS$^3$, PU-Gaussian, and HybridGS are described as realizing the same functionality without always using the name itself (Xiong et al., 19 Sep 2025, Liu et al., 2024, Khater et al., 24 Sep 2025, Yang et al., 3 May 2025). "3D Gaussian Point Encoders" uses the closely related formulation "3D Gaussian Point Encoder" for an explicit per-point embedding built on mixtures of learned 3D Gaussians (James et al., 6 Nov 2025).

This variation in nomenclature reflects a broader conceptual spread. In some works, PGE means a module that predicts Gaussian primitives from point features and then uses rendering or splatting as the supervision or feature-projection operator. In others, it means an interpretable geometric encoding of local shape, or a codec-oriented conversion from Gaussian splats to a standard point-cloud bitstream. A plausible implication is that PGE is best understood as a representational pattern rather than a single fixed architecture.

Paper	PGE interpretation
GS$^3$ (Liu et al., 2024)	Point cloud encoder plus Gaussian parameter heads for self-supervised rendering
RadarGaussianDet3D (Xiong et al., 19 Sep 2025)	Per-radar-point Gaussian primitive encoder for BEV rasterization
RCGDet3D (Xiong et al., 20 May 2026)	Ray-centric PGE (R-PGE) with analytic frame transforms
PU-Gaussian (Khater et al., 24 Sep 2025)	One anisotropic Gaussian per input point for local geometric encoding
HybridGS (Yang et al., 3 May 2025)	Mapping 3D Gaussian splats into a standard point-cloud encoding pipeline
3D Gaussian Point Encoders (James et al., 6 Nov 2025)	Per-point embedding from mixtures of shared Gaussian basis functions

A recurrent misconception is that PGE is synonymous with 3D Gaussian Splatting. The literature is narrower and more varied than that. GS$^3$, RadarGaussianDet3D, and RCGDet3D do use splatting or rasterization, but PU-Gaussian uses reparameterized point sampling, HybridGS uses point-cloud coding, and "3D Gaussian Point Encoders" uses max pooling over Gaussian basis responses rather than rendering (Liu et al., 2024, Khater et al., 24 Sep 2025, Yang et al., 3 May 2025, James et al., 6 Nov 2025).

2. Gaussian parameterization and feature mappings

The primitive-centric form of PGE predicts, for each point or point-derived anchor, a Gaussian mean $\mu \in \mathbb{R}^3$ and a covariance $\Sigma \in \mathbb{R}^{3\times 3}$, commonly decomposed as

$\Sigma = R S S^\top R^\top,$

where $R \in SO(3)$ is a rotation and $S$ is a scaling matrix, commonly diagonal. GS$^3$, PU-Gaussian, RadarGaussianDet3D, and RCGDet3D all use this factored parameterization because it makes anisotropy explicit and guarantees positive semidefiniteness or positive definiteness under the stated constraints (Liu et al., 2024, Khater et al., 24 Sep 2025, Xiong et al., 19 Sep 2025, Xiong et al., 20 May 2026).

In GS$^3$0, the point cloud encoder produces point-wise features $^3$1 with $^3$2, and a feed-forward network $^3$3 maps cross-view encoded point features to $^3$4 Gaussian parameter sets,

$^3$5

with $^3$6. The predicted attributes include center offsets, covariance, opacity, and spherical-harmonics color coefficients (Liu et al., 2024). In PU-Gaussian, one anisotropic Gaussian is regressed per input point rather than a mixture-of-Gaussians per point, and the Gaussian’s mean and covariance are interpreted geometrically: the long axes align to tangent directions of a surface patch, while a short axis aligns to the local normal (Khater et al., 24 Sep 2025).

RadarGaussianDet3D adopts a more constrained design. Each radar point becomes one Gaussian primitive

$^3$7

with $^3$8, fixed opacity $^3$9, learned scale $^3$0, learned unit quaternion $^3$1, and a learned per-point Gaussian feature vector $^3$2; the covariance is then

$^3$3

The design deliberately avoids learned position offsets and learned opacities because ablation results show that learning these additional degrees of freedom degrades accuracy (Xiong et al., 19 Sep 2025). RCGDet3D retains the same Gaussian attributes but predicts them in a ray-aligned coordinate system and reintroduces a learned offset $^3$4 in the ray frame before transforming the mean and covariance to ego coordinates (Xiong et al., 20 May 2026).

The basis-centric form in "3D Gaussian Point Encoders" is structurally different. Rather than assigning one Gaussian primitive to each input point, the encoder maintains $^3$5 shared anisotropic Gaussian basis functions with means $^3$6, precision matrices parameterized by Cholesky factors via

$^3$7

and mixer weights $^3$8. For a point $^3$9, the Gaussian response is

$^3$0

and each embedding channel is

$^3$1

The per-point embedding is then $^3$2, followed by PointNet-style max pooling (James et al., 6 Nov 2025). This suggests that the phrase "Point Gaussian Encoder" can refer either to Gaussian primitives carried by points or to Gaussian basis functions evaluated at points.

3. Encoder architectures and coordinate systems

GS$^3$3 begins from sparse-view RGB-D image pairs, typically $^3$4 views at $^3$5 resolution with camera intrinsics $^3$6 and extrinsics $^3$7. Pixels are back-projected to 3D by the pinhole camera model,

$^3$8

and each 3D point carries RGB color from the source pixel. The back-projected point clouds are then processed by a point cloud encoder $^3$9, implemented either as PointNet++ with 4 Set Abstraction layers and 4 Feature Propagation layers, or as SR-UNet with Sparse Convolution and Sparse Deconvolution blocks. With two input views, GS$\mu \in \mathbb{R}^3$0 can also aggregate across views via a lightweight cost volume module (MVSplat) or an epipolar line transformer (FreeSplat) to produce cross-view enhanced features per view (Liu et al., 2024).

PU-Gaussian uses a Point Transformer backbone. It first projects points via an MLP, then extracts local and global features, and finally applies three prediction heads: $\mu \in \mathbb{R}^3$1 for scales, $\mu \in \mathbb{R}^3$2 quaternion for rotation, and $\mu \in \mathbb{R}^3$3 for the mean offset, with $\mu \in \mathbb{R}^3$4 (Khater et al., 24 Sep 2025). The same source also presents an analytic neighborhood-based view, in which a general-purpose PGE can estimate Gaussian parameters from a $\mu \in \mathbb{R}^3$5-NN neighborhood by sample mean and regularized covariance,

$\mu \in \mathbb{R}^3$6

The eigendecomposition $\mu \in \mathbb{R}^3$7 then yields local tangent directions and a normal estimate from the smallest eigenvector (Khater et al., 24 Sep 2025).

RadarGaussianDet3D organizes feature extraction around Local Feature Aggregation (LFA) and Global Feature Aggregation (GFA). For the $\mu \in \mathbb{R}^3$8-th point, with spherical neighborhood $\mu \in \mathbb{R}^3$9 and default radius $\Sigma \in \mathbb{R}^{3\times 3}$0, LFA computes

$\Sigma \in \mathbb{R}^{3\times 3}$1

while GFA applies point-wise self-attention on raw features in parallel. The implementation emphasizes an Indexing+Scattering strategy based on a neighbor mask and scatter_reduce rather than broadcast-heavy neighborhood processing (Xiong et al., 19 Sep 2025).

RCGDet3D changes the representation space. For each radar point $\Sigma \in \mathbb{R}^{3\times 3}$2 with Cartesian coordinates $\Sigma \in \mathbb{R}^{3\times 3}$3, range $\Sigma \in \mathbb{R}^{3\times 3}$4, azimuth $\Sigma \in \mathbb{R}^{3\times 3}$5, elevation $\Sigma \in \mathbb{R}^{3\times 3}$6, R-PGE defines a local ray-aligned frame with $\Sigma \in \mathbb{R}^{3\times 3}$7 along the radar ray, $\Sigma \in \mathbb{R}^{3\times 3}$8 tangential in the radar $\Sigma \in \mathbb{R}^{3\times 3}$9-$\Sigma = R S S^\top R^\top,$0 plane, and $\Sigma = R S S^\top R^\top,$1 completing a right-handed triad. The rotation from radar to ray frame is

$\Sigma = R S S^\top R^\top,$2

and Gaussian attributes are predicted in this local frame before analytic transformation to radar and ego coordinates (Xiong et al., 20 May 2026). The stated purpose is to decouple coordinate transformation from representation learning and thereby improve geometric consistency.

By contrast, "3D Gaussian Point Encoders" keeps the coordinate system simple but changes the computational role of the encoder. PGE here is a drop-in replacement for PointNet’s per-point MLP, preserving permutation invariance by applying max pooling over the resulting point embeddings rather than projecting point features into an image or BEV grid (James et al., 6 Nov 2025).

4. Rendering, splatting, sampling, and codec interfaces

In GS$\Sigma = R S S^\top R^\top,$3, the predicted Gaussians are rendered by tile-based rasterization. The underlying 3D Gaussian density is

$\Sigma = R S S^\top R^\top,$4

For a given camera, projection to screen space uses a first-order approximation,

$\Sigma = R S S^\top R^\top,$5

followed by a screen-space Gaussian

$\Sigma = R S S^\top R^\top,$6

With opacity $\Sigma = R S S^\top R^\top,$7, front-to-back rendering accumulates transmittance and color as

$\Sigma = R S S^\top R^\top,$8

GS$\Sigma = R S S^\top R^\top,$9 explicitly notes that this matches the 3DGS formulation from Kerbl et al. and avoids per-ray marching by culling Gaussians to overlapping tiles (Liu et al., 2024).

RadarGaussianDet3D adapts the same explicit Gaussian logic to BEV feature formation. A parallel projection with scaling maps 3D means and covariances into the BEV plane,

$R \in SO(3)$0

and the Gaussian contribution to a BEV cell at coordinate $R \in SO(3)$1 is

$R \in SO(3)$2

With $R \in SO(3)$3 and front-to-back transmittance $R \in SO(3)$4, the feature at a pixel is

$R \in SO(3)$5

RCGDet3D keeps the Gaussian-to-BEV idea but describes the feature rasterizer as additive splatting of the 2D marginal over $R \in SO(3)$6,

$R \in SO(3)$7

with truncation to a finite kernel radius for efficiency (Xiong et al., 19 Sep 2025, Xiong et al., 20 May 2026).

PU-Gaussian uses the Gaussian parameters for direct point generation rather than rasterization. Reparameterized sampling is written as

$R \in SO(3)$8

and the method draws $R \in SO(3)$9 samples per Gaussian while discarding samples beyond two standard deviations from the mean. Optional constraints such as tangent-plane projection and truncated Gaussian sampling are presented as mechanisms to preserve edge structure and keep samples close to the underlying surface (Khater et al., 24 Sep 2025).

HybridGS uses a different downstream operator again: standardized compression. Its pipeline produces a compact, explicit, integer-quantized, and unique set of 3D Gaussian primitives and then maps them into a standard point-cloud format so that MPEG G-PCC test model v23 can encode them. Geometry is coded by lossless octree coding and attributes by RAHT in lossless mode; the interface is implemented by splitting 3DGS into multiple xyza-style attribute streams and then decoding and merging them back into Gaussian primitives (Yang et al., 3 May 2025). This suggests that PGE is not merely a feature extractor but also an interface layer between Gaussianized point representations and downstream operators such as renderers, BEV accumulators, samplers, and codecs.

5. Supervision, optimization, and stability

GS$S$0 is trained by RGB-only self-supervision. Its total loss is

$S$1

with

$S$2

During pre-training, GS$S$3 also applies Masked Point Modeling (MPM): $S$4 of back-projected points are masked, Gaussians are predicted from only the visible points, and the render is compared against full RGB targets. The reported pre-training setup uses ScanNet v2, 2 sparse views per scene with frame interval 5, batch size 4, 100 epochs, AdamW with initial learning rate $S$5, weight decay $S$6, cosine annealing down to $S$7, and consistent random rotations applied to both point clouds and camera poses (Liu et al., 2024).

PU-Gaussian uses two-stage upsampling supervision. The coarse and refined point sets are optimized with Chamfer Distance,

$S$8

together with a Gaussian regularization term

$S$9

supervised by nearest mean-to-ground-truth-point assignment. A lightweight refinement network based on Point Transformer predicts residual displacements, and at inference an extra refinement pass is used for stability (Khater et al., 24 Sep 2025).

RadarGaussianDet3D integrates PGE into a standard CenterPoint-style detector. The classification loss is unchanged, whereas regression combines the original box regression loss with Box Gaussian Loss (BGL),

$^3$0

A 3D box $^3$1 is converted to a Gaussian with

$^3$2

and BGL is defined as the KL divergence between predicted and ground-truth Gaussians (Xiong et al., 19 Sep 2025). RCGDet3D follows RadarGaussianDet3D’s formulation and training recipe, adding Semantic Injection (SI), in which deformable attention samples image features near each projected radar point and concatenates them with radar point features before Gaussian attribute prediction (Xiong et al., 20 May 2026).

"3D Gaussian Point Encoders" addresses a different optimization problem: learning Gaussian basis parameters directly is described as difficult with standard optimizers, so the paper uses natural gradients and distillation from pretrained PointNets. The distillation loss is

$^3$3

and the mean update under the Mahalanobis metric is

$^3$4

A fuller Fisher Information preconditioner is also used for joint updates of means and Cholesky parameters (James et al., 6 Nov 2025). HybridGS, by contrast, is optimization-aware at the compression stage: it uses Uniform Quantization with Straight-Through Estimator or a Robust Quantizer with perturbation-aware affine transforms and DAT-R reconstruction, while explicitly controlling bits per primitive through

$^3$5

This explicit relation between quantization settings and rate is central to its interpretable rate-control scheme (Yang et al., 3 May 2025).

6. Applications, empirical behavior, and limitations

In GS$^3$6, the primary application is point cloud unsupervised pre-training. The reported transfer results cover 3D object detection, 3D semantic segmentation, 3D instance segmentation, and 3D scene reconstruction. On SUN RGB-D, GS$^3$7 + VoteNet improves from $^3$8 and $^3$9 to $^3$00 and $^3$01; on ScanNet v2 with H3DNet it reaches $^3$02 and $^3$03; with MinkUNet (2cm) it improves ScanNet v2 semantic segmentation from $^3$04 to $^3$05; with PointGroup (2cm) it improves ScanNet v2 instance segmentation from average AP $^3$06 to $^3$07; and for ConvONet-based reconstruction it improves IoU from $^3$08 to $^3$09. At the systems level, GS$^3$10 reports approximately $^3$11 pre-training speedup and less than $^3$12 memory cost compared to Ponder (Liu et al., 2024).

In radar detection, PGE is primarily an efficient densification mechanism for sparse 4D radar points. RadarGaussianDet3D reports on TJ4DRadSet a test-set $^3$13 mAP of $^3$14, BEV mAP of $^3$15, and $^3$16 FPS on V100; on View-of-Delft it reports EAA mAP $^3$17, ROI mAP $^3$18, and $^3$19 FPS on V100. Its ablations show a baseline pillar encoder at $^3$20 $^3$21 mAP, $^3$22 with Gaussian splatting, $^3$23 with the complete PGE, and $^3$24 with PGE+BGL (Xiong et al., 19 Sep 2025). RCGDet3D further reports VoD EAA mAP $^3$25, ROI mAP $^3$26, and $^3$27 FPS on V100, while on TJ4DRadSet it reports $^3$28 mAP $^3$29, BEV mAP $^3$30, and $^3$31 FPS. Its ray-centric ablations on VoD give $^3$32 EAA mAP for switching ego-centric to ray-centric Gaussian coordinates, $^3$33 for adding $^3$34 offsets, and a further gain to $^3$35 with SI using deformable attention (Xiong et al., 20 May 2026).

In point upsampling, PU-Gaussian evaluates on PU-GAN and PU1K using CD, HD, and P2F. Reported values include PU-GAN $^3$36: CD $^3$37, HD $^3$38, P2F $^3$39; PU-GAN $^3$40: CD $^3$41, HD $^3$42, P2F $^3$43; and PU1K $^3$44: CD $^3$45, HD $^3$46, P2F $^3$47. The paper also reports robustness to noise and sparsity, for example under added Gaussian noise $^3$48 and $^3$49, and notes that PU-Gaussian ranks second behind APU-LDI while remaining single-pass and efficient (Khater et al., 24 Sep 2025).

In recognition, "3D Gaussian Point Encoders" reports that a distilled and filtered PGE is $^3$50 times faster than a comparable-accuracy PointNet, with $^3$51 less memory and $^3$52 fewer FLOPs on ScanObjectNN using 2048 points and an RTX 4070 mobile GPU. On CPU, it reports $^3$53 throughput on RK3588, and when used inside Mamba3D it reports $^3$54 times faster runtime with memory and FLOP reductions of $^3$55 and $^3$56 respectively (James et al., 6 Nov 2025). In compression, HybridGS reports typical encoding and decoding speed of $^3$57-$^3$58 seconds for many scenes, GPCC-compatible standard bitstreams, and large size reductions relative to vanilla 3DGS, such as $^3$59 dB at $^3$60 MB on playroom versus vanilla 3DGS at $^3$61 dB and $^3$62 MB, and $^3$63 dB at $^3$64 MB on dance versus vanilla 3DGS at $^3$65 dB and $^3$66 MB (Yang et al., 3 May 2025).

The limitations are domain-specific but structurally related. GS$^3$67 assumes reasonably accurate RGB-D inputs; strong noise or missing depth can degrade back-projection and Gaussian placement, and very sparse views under-constrain occluded regions (Liu et al., 2024). RadarGaussianDet3D notes that 4D radar point clouds remain sparse and noisy, and that very large or very small learned scales can blur boundaries or fail under sparsity; it also reports that learning offsets and opacities degrades performance in its setting (Xiong et al., 19 Sep 2025). RCGDet3D states that radar alone lacks strong semantics, that SI depends on image quality and calibration, and that the current experiments assume a front-facing single-radar setup (Xiong et al., 20 May 2026). PU-Gaussian observes that high noise, extreme sparsity, sharp corners, and non-manifold or intersecting structures can make a single Gaussian insufficient for local modeling (Khater et al., 24 Sep 2025). "3D Gaussian Point Encoders" emphasizes brittle optimization under naive end-to-end Euclidean gradients and the memory cost of voxel filtering in higher dimensions (James et al., 6 Nov 2025). HybridGS states that it does not include modules aimed at improving 3DGS quality during generation, that bit allocation is uniform across attributes in its second rate-control method, and that there is no end-to-end rate-distortion training (Yang et al., 3 May 2025).

Taken together, these results indicate that PGE is a family of explicit Gaussian encoders whose main advantage is not tied to a single task. The shared benefit is structural: once mean, anisotropy, orientation, feature payload, and sometimes opacity are explicit, one can rasterize to BEV, render RGB, sample new points, compress to standard bitstreams, or filter basis evaluations using computational geometry. This suggests that the enduring significance of PGE lies less in any single implementation than in the move from implicit point processing to explicit Gaussianized point representations.