N4MC: Neural 4D Mesh Compression

Abstract: We present N4MC, the first 4D neural compression framework to efficiently compress time-varying mesh sequences by exploiting their temporal redundancy. Unlike prior neural mesh compression methods that treat each mesh frame independently, N4MC takes inspiration from inter-frame compression in 2D video codecs, and learns motion compensation in long mesh sequences. Specifically, N4MC converts consecutive irregular mesh frames into regular 4D tensors to provide a uniform and compact representation. These tensors are then condensed using an auto-decoder, which captures both spatial and temporal correlations for redundancy removal. To enhance temporal coherence, we introduce a transformer-based interpolation model that predicts intermediate mesh frames conditioned on latent embeddings derived from tracked volume centers, eliminating motion ambiguities. Extensive evaluations show that N4MC outperforms state-of-the-art in rate-distortion performance, while enabling real-time decoding of 4D mesh sequences. The implementation of our method is available at: https://github.com/frozzzen3/N4MC.

• The paper introduces a neural 4D mesh codec that converts topology-varying meshes into TSDF-Def volumes, compresses key frames, and synthesizes intermediate frames with tracked-motion priors and a transformer.
• N4MC achieves leading quality at roughly 4 Mbps on MPEG V-DMC sequences, including D2-PSNR values up to 73.129 and SSIM up to 0.9791, while reaching comparable mixed-scene quality at 2.469 Mbps versus 4.931 Mbps for NeCGS.
• The method enables topology-agnostic and on-device decoding but remains limited by sequence-specific model overfitting, voxel-resolution trade-offs, long-range interpolation errors, and mobile latency of 144–346 ms per frame.

N4MC introduces a neural framework for compressing time-varying (4D) mesh sequences, positioned by its authors as the first neural method designed explicitly for this task. The central observation motivating the work is that existing neural mesh compressors—such as NeCGS (2602.20312), DMGC, and Squeeze3D—process each frame independently and therefore leave temporal redundancy unexploited, while classical dynamic mesh coders like Apple's VSMC (the basis of MPEG V-DMC) require topology-consistent re-meshing that is unreliable on real depth-sensor data. N4MC instead borrows the inter-frame prediction paradigm of 2D video codecs: it stores only key frames in a compact latent space and synthesizes intermediate frames with a learned interpolation model.

Representation: TSDF-Def tensors

The pipeline begins by converting each irregular mesh frame into a regular volumetric tensor. Building on NeCGS's TSDF-Def representation, each mesh becomes a 4D tensor of shape $k \times k \times k \times 4$, where each voxel stores one truncated signed distance value plus a three-dimensional deformation vector. This conversion removes dependence on vertex count and connectivity, which is what allows the framework to handle topology-varying sequences uniformly; it also means quality is fundamentally bounded by the voxel resolution $k$, a trade-off made explicit in the ablations.

Latent compression via an autoencoder–decoder pair

A 3D convolutional autoencoder with a ConvNeXt3D backbone encodes TSDF-Def tensors into embedded features at reduced spatial dimension ($k' < k$). A quantization-aware auto-decoder reconstructs the full-resolution tensor using 3D PixelShuffle upsampling, trained with a joint loss combining L1, a near-surface masked L1 (restricted to voxels where $|\text{TSDF}| < \alpha$), and SSIM. Both the decoder and all linear layers of the downstream transformer are quantized to 8 bits so that the model itself can be streamed as part of the bitstream—a design choice that produces lightweight, sequence-specific models rather than a general codec.

Motion priors from volume tracking

Because latent features alone carry no explicit motion structure, interpolation across long gaps is ambiguous. N4MC addresses this by running As-Rigid-As-Possible volume tracking over the sequence, extracting $p = 2000$ tracked volume centers per frame that trace non-rigid motion trajectories. A PointNet-style encoder maps start/target/end center sets into descriptors; a temporal delta feature and a sinusoidal time embedding are concatenated and passed through an MLP to yield a 32-dimensional latent deformation code per intermediate frame. An ablation disabling this latent mapping module shows complete interpolation failure, indicating the motion prior is not merely an accelerant but a necessary conditioning signal for the transformer to converge to meaningful intermediates.

Transformer-based latent interpolation

Every $n$-th frame is a key frame; the remaining $n-1$ frames are synthesized by a cross-attention transformer. Key-frame embeddings form memory tokens; FiLM-conditioned queries built from the latent code and time embedding attend to these memories through two encoder and two decoder layers (four attention heads, $4\times$ feedforward expansion). Only key-frame embeddings, latent codes, and the quantized models are entropy-coded into the final bitstream; decoding proceeds by interpolation, auto-decoder reconstruction, and Deformable Marching Cubes mesh extraction.

Evaluation

Experiments use four 300-frame MPEG V-DMC sequences ("Dancer", "Basketball player", "Mitch", "Thomas"), a synchronized four-sequence "Mixed" dataset, custom RGB-D captures, and Thingi10K synthetic meshes. Baselines include Draco, KLT-based TSDF compression, TVMC, and NeCGS. At approximately 4 Mbps, N4MC achieves the best scores on every quality metric across all four MPEG sequences:

Sequence	D2-PSNR	SSIM	PSNR
Dancer	67.276	0.9712	33.567
Basketball player	66.107	0.9671	33.188
Mitch	73.129	0.9779	37.274
Thomas	72.759	0.9791	37.811

TVMC, the strongest classical competitor, degrades sharply on "Basketball player" (SSIM 0.8795 versus N4MC's 0.9671) because large non-rigid motions break deformation-based inter-frame prediction—an outcome that supports the paper's argument that explicit motion priors are more robust than reference-mesh warping. On desktop hardware (RTX 4090), per-frame decoding takes about 38 ms at resolution 256, exceeding real-time (over 24 FPS), though notably slower than Draco's roughly 2 ms.

The multi-object results are arguably the most striking claim in the evaluation: on the "Mixed" dataset, N4MC reaches comparable quality (D2-PSNR around 61 dB, SSIM about 0.94) at 2.469 Mbps while NeCGS needs 4.931 Mbps, KLT exceeds 10 Mbps, and TVMC and Draco require roughly $4\times$ or more of N4MC's bitrate. Because all content is rasterized into identically shaped TSDF tensors, bitrate is largely decoupled from geometric complexity—an advantage over vertex-count-dependent codecs that grows with scene complexity. Ablations show monotone quality gains with resolution $k$ (D2-PSNR rising from 54.99 at $k$0 to 66.11 at $k$1, at the cost of decoding time growing from 4.1 ms to 38.5 ms), and gradual degradation as either group size $k$2 or total sequence length increases, consistent with error accumulation in long-range interpolation.

On mobile deployment, the authors provide a Unity plugin and report decoding entirely on-device: 346 ms per frame on Meta Quest 3, 144 ms on Pixel 9, and 156 ms on Xiaomi 12S Pro, split between model inference and Marching Cubes extraction. These figures fall short of real time on current mobile hardware but demonstrate, to the authors' knowledge, the first fully on-device neural 4D mesh decoder including a standalone VR headset. On static synthetic data with interpolation disabled, N4MC attains a compression ratio of 89.56 on Thingi10K—about $k$3 Neural Progressive Meshes—at slightly worse geometric accuracy ($k$4 increasing from 4.12 to 10.42 $k$5).

Limitations and open questions

The paper concedes several constraints. First, the framework is sequence-specific: models are overfit per mesh sequence and shipped inside the bitstream, so generalization to unseen content is limited—the supplementary material explicitly identifies this overfitting and proposes larger-scale training as future work. Second, quality is bounded by voxelization, and the uniform TSDF grid is memory-intensive during training; a pruned near-surface representation is suggested but not evaluated. Third, decoding speed depends on voxel construction and isosurfacing, leaving mobile latency well above interactive rates. Finally, the volume-center prior assumes trackable coherent volumes; behavior under severe occlusion, topological splitting/merging, or very fast non-rigid motion beyond the tested sequences remains an open question.

Conclusion

N4MC combines a topology-agnostic implicit volumetric representation, tracked-volume motion priors, and a quantized latent-space interpolation transformer to exploit temporal redundancy in mesh sequences exceeding 100 frames—far beyond the 5–10 frame groups typical of prior inter-frame methods. Its rate-distortion results across real, synthetic, single-, and multi-object benchmarks are consistently superior at matched bitrates, and its decoupling of bitrate from geometric complexity is a practically meaningful property for dense multi-object scenes. The main unresolved issues are per-sequence model specialization, voxel-bounded fidelity, and mobile inference cost.