Video-RDMesh: Dynamic Mesh Reconstruction

• Video-RDMesh is a family of video-driven mesh formulations characterized by explicit meshes that dynamically encode geometry, appearance, and motion.
• It integrates techniques such as differentiable rendering, Gaussian splatting, and hybrid skinning to achieve robust view-dependent feature alignment and deformation.
• The approach ensures temporal consistency and compression efficiency through topology-preserving registration and adaptive inter-frame coding strategies.

Video-RDMesh denotes a family of video-driven mesh formulations in which a dynamic mesh remains the primary geometric representation while appearance, motion, and sometimes compression are organized around that mesh. In the recent literature, the expression appears both as a conceptual pipeline—“video-driven reconstruction via differentiable rendering (DR) of a mesh (Mesh)” and, more generally, a system that “ingests a monocular video,” “produces a deformable mesh whose surface carries radiance (view-dependent appearance),” and “deforms that mesh over time conditioned on the video”—and as the name of a large-scale dataset of dynamic mesh sequences curated for pose-misaligned 4D animation (Jena et al., 2023, Li et al., 2024, Wu et al., 13 May 2026). Across these uses, the mesh is treated as the first-class carrier of geometry, while view-dependent appearance may be represented by textures or surface-bound Gaussians, motion may be parameterized by skinning, sparse deformation graphs, or latent trajectory decompositions, and temporal consistency is enforced through shared topology, rectification, or inter-frame correspondence.

1. Terminology and scope

The term has two closely related but distinct usages in current work.

Usage	Description	Representative source
Pipeline notion	“video-driven reconstruction via differentiable rendering (DR) of a mesh (Mesh)”	(Jena et al., 2023)
Radiance-mesh pipeline	“Ingests a monocular video,” “Produces a deformable mesh whose surface carries radiance (view-dependent appearance),” and “Deforms that mesh over time conditioned on the video”	(Li et al., 2024)
Dataset name	513,690 high-quality, 64-frame vertex-trajectory clips from 252,823 unique dynamic assets	(Wu et al., 13 May 2026)

A recurring dispute in this area concerns whether explicit meshes are too restrictive for video-based reconstruction and animation. “Mesh Strikes Back” explicitly presents “a counter viewpoint to this fundamental premise” that mesh-based representations cannot capture complex clothing and textures from monocular RGB, while DreamMesh4D argues that binding radiance Gaussians to a mesh surface and deforming that mesh via a sparse graph and hybrid skinning “constrains the solution space and yields robust spatiotemporal consistency” (Jena et al., 2023, Li et al., 2024). This suggests that Video-RDMesh is best understood not as a single standardized architecture, but as an explicit-mesh research program spanning reconstruction, animation, avatar modeling, and coding.

2. Mesh-centered representations

A central design decision in Video-RDMesh systems is how appearance is attached to the mesh. DreamMesh4D uses a triangle mesh with per-face 3D Gaussians that are differentiably anchored to the surface. For a face $f=(v_a,v_b,v_c)$, each Gaussian center satisfies the barycentric constraint

$$\mu_g=\pi_a v_a+\pi_b v_b+\pi_c v_c,\qquad \pi_a+\pi_b+\pi_c=1,\ \pi_i\ge 0,$$

and the covariance is parameterized by a scale $s_g\in\mathbb{R}^3$ and a rotation $q_g$ through

$\Sigma_g=R(q_g)\operatorname{diag}(s_g^2)R(q_g)^T.$

The implementation attaches $x=6$ flat Gaussians per face, with view-dependent color realized by low-order spherical harmonics, so radiance is explicitly surface-aligned rather than represented in free space (Li et al., 2024).

RMAvatar adopts a related but more human-specific formulation, described as “rectified mesh-embedded Gaussians.” Gaussians are embedded into triangular faces and inherit low-frequency motion from the mesh, while a pose-related Gaussian rectification module predicts fine non-rigid corrections,

$$(\Delta\mu,\Delta r,\Delta s)=F_\theta(\gamma(\mu^*),\theta),$$

followed by

$$\mu'=\mu^*+\Delta\mu,\qquad r'=r^*\cdot \Delta r,\qquad s'=s^*+\Delta s.$$

This division lets the explicit mesh represent motion and coarse deformation, while Gaussian splatting carries photorealistic appearance and pose-dependent residual detail (Peng et al., 13 Jan 2025).

Other Video-RDMesh formulations remain texture-centric rather than Gaussian-centric. “Mesh Strikes Back” reconstructs a clothed human with an SMPL+D mesh and a learned multi-resolution texture representation based on per-vertex 3D texture coordinates and a multi-resolution hash encoding; the mesh topology is fixed to SMPL, while geometry is enriched by per-vertex displacement $D\in\mathbb{R}^{N\times 3}$ (Jena et al., 2023). TexMesh similarly separates detailed per-frame geometry from sequence-level appearance by predicting spatiotemporally consistent detailed per-frame meshes together with “a single high-resolution full-body albedo texture $A(u,v)$ shared across the sequence” (Zhi et al., 2020).

Taken together, these systems indicate two dominant representational strategies. One binds radiance primitives directly to a deformable surface; the other keeps appearance in UV or neural texture space while refining mesh geometry. Both preserve an explicit surface suitable for export, retargeting, and downstream graphics pipelines.

3. Motion, skinning, and rectification

Video-RDMesh methods differ most sharply in how they parameterize motion. DreamMesh4D builds a sparse deformation graph over the refined static mesh by uniformly sampling $$\mu_g=\pi_a v_a+\pi_b v_b+\pi_c v_c,\qquad \pi_a+\pi_b+\pi_c=1,\ \pi_i\ge 0,$$0 control points on the surface and connecting each mesh vertex to $$\mu_g=\pi_a v_a+\pi_b v_b+\pi_c v_c,\qquad \pi_a+\pi_b+\pi_c=1,\ \pi_i\ge 0,$$1 control nodes by shortest geodesic distance. A deformation MLP $$\mu_g=\pi_a v_a+\pi_b v_b+\pi_c v_c,\qquad \pi_a+\pi_b+\pi_c=1,\ \pi_i\ge 0,$$2 predicts, for each control node $$\mu_g=\pi_a v_a+\pi_b v_b+\pi_c v_c,\qquad \pi_a+\pi_b+\pi_c=1,\ \pi_i\ge 0,$$3 and frame $$\mu_g=\pi_a v_a+\pi_b v_b+\pi_c v_c,\qquad \pi_a+\pi_b+\pi_c=1,\ \pi_i\ge 0,$$4,

$$\mu_g=\pi_a v_a+\pi_b v_b+\pi_c v_c,\qquad \pi_a+\pi_b+\pi_c=1,\ \pi_i\ge 0,$$5

where $$\mu_g=\pi_a v_a+\pi_b v_b+\pi_c v_c,\qquad \pi_a+\pi_b+\pi_c=1,\ \pi_i\ge 0,$$6, $$\mu_g=\pi_a v_a+\pi_b v_b+\pi_c v_c,\qquad \pi_a+\pi_b+\pi_c=1,\ \pi_i\ge 0,$$7, $$\mu_g=\pi_a v_a+\pi_b v_b+\pi_c v_c,\qquad \pi_a+\pi_b+\pi_c=1,\ \pi_i\ge 0,$$8, and $$\mu_g=\pi_a v_a+\pi_b v_b+\pi_c v_c,\qquad \pi_a+\pi_b+\pi_c=1,\ \pi_i\ge 0,$$9. The method then fuses Linear Blend Skinning and Dual Quaternion Skinning through a per-vertex rigidity $s_g\in\mathbb{R}^3$0, producing

$s_g\in\mathbb{R}^3$1

This “adaptive hybrid skinning” is intended to preserve DQS rotation quality while retaining LBS’s ability to incorporate nonrigid strain (Li et al., 2024).

RMAvatar addresses a different failure mode: even when mesh motion is available, classical LBS cannot adequately model clothing wrinkles, hair motion, and other high-frequency non-surface deformations. Its rectification module learns pose-conditioned corrections in Gaussian position, rotation, and scale on top of mesh-guided motion, which the ablations identify as important: removing rectification lowers PSNR and yields “burr textures on loose clothing” (Peng et al., 13 Jan 2025).

R-DMesh frames the problem at the sequence level as one of pose misalignment. Given a static conditional mesh $s_g\in\mathbb{R}^3$2 and a target dynamic sequence $s_g\in\mathbb{R}^3$3, it decomposes motion into a “rectification jump” and relative trajectories,

$s_g\in\mathbb{R}^3$4

The model then processes geometry, jump offset, and trajectory jointly with Triflow Attention,

$s_g\in\mathbb{R}^3$5

followed by shared aggregation over $s_g\in\mathbb{R}^3$6. Generation is handled by a Rectified Flow-based Diffusion Transformer trained with

$s_g\in\mathbb{R}^3$7

The explicit decomposition is designed to separate initial alignment from subsequent motion (Wu et al., 13 May 2026).

A related alignment strategy appears in V2M4, which treats per-frame meshes from a native 3D mesh generator as misposed and recovers object motion by searching for a camera pose $s_g\in\mathbb{R}^3$8 that matches the input frame, then applying the inverse of that camera to the mesh. This “camera search and mesh reposing” is followed by condition embedding optimization for appearance refinement and later topology consolidation (Chen et al., 11 Mar 2025).

4. Reconstruction and optimization workflows

The reconstruction lineage of Video-RDMesh predates the recent radiance-mesh literature. “Photometric Mesh Optimization for Video-Aligned 3D Object Reconstruction” formulates object mesh recovery from RGB videos as multi-view photometric consistency constrained by a shape prior, poses the problem as a “piecewise image alignment problem for each mesh face projection,” and introduces rasterization from a virtual viewpoint to avoid “a degeneracy of zero photometric gradients” (Lin et al., 2019). This line established the idea that explicit meshes can be optimized directly against video evidence without depth or mask supervision.

TexMesh extends that philosophy to RGB-D human capture with known illumination. Its inputs are RGB-D video, an HDR environment map, and a coarse per-frame human mesh from RGB-D tracking; its outputs are “spatiotemporally consistent and detailed per-frame meshes” and “a single high-resolution full-body albedo texture.” The pipeline is partitioned into AlbeNorm for per-frame albedo and normal prediction under environment lighting, TexGen for UV texture refinement, and MeshRef for UV-space 3D displacement prediction and self-supervised adaptation on real sequences (Zhi et al., 2020).

NeuralMeshing targets complete object meshing from casual capture rather than dynamic animation. It operates on “two or more videos,” requires “the specification of one known point in at least one frame of each video,” estimates intrinsics and extrinsics through Structure-from-Motion, applies SAM2 for segmentation, trains per-video neural fields with instant-ngp and a NeuS2 surface loss, and aligns partial meshes across videos via coarse correspondences and ICP before extracting the final mesh with marching cubes. Its stated objective is to generate “a complete object mesh” by “using multiple videos and merging results,” thereby avoiding “hole filling” (Erich et al., 22 Aug 2025).

V2M4 translates single monocular video into a “usable 4D mesh animation asset” through a structured workflow: per-frame coarse mesh generation with TRELLIS, camera search and mesh reposing, condition embedding optimization, pairwise mesh registration for shared topology, global texture map optimization, and export as a GLTF animation with a static base mesh, a shared texture map, and a deformation tensor $s_g\in\mathbb{R}^3$9 (Chen et al., 11 Mar 2025). In contrast to methods that rely on multi-view image or video generation priors, V2M4 is based on “native 3D mesh generation models.”

These pipelines show that Video-RDMesh is not tied to one supervision regime. It can be driven by RGB-only photometric consistency, RGB-D plus lighting, monocular diffusion-guided optimization, or multi-video neural field fusion, provided the end product remains an explicit, temporally organized mesh.

5. Topology consistency, temporal correspondence, and compression

A persistent technical obstacle is that video sequences often violate constant-topology assumptions. The current V-DMC inter-frame coding pipeline is limited to mesh frames with constant topology; when topology varies, inter-frame tools are disabled and frames fall back to intra-like coding (Huang et al., 2024). To address this, “Inter-Frame Coding for Dynamic Meshes via Coarse-to-Fine Anchor Mesh Generation” introduces an inter-frame anchor mesh for a target frame $q_g$0 that preserves the exact connectivity of a reference base mesh $q_g$1 while adapting vertex positions to the target surface.

The anchor is generated in two stages. First, an octree is built over the target mesh vertices, and each base vertex $q_g$2 is matched to a coarse anchor by motion-compensated nearest-neighbor search. Motion prediction is graph-aware:

$q_g$3

and the coarse match defines the motion

$q_g$4

Second, a local Quadric Error Metrics refinement computes a finer anchor position by minimizing accumulated plane quadrics. For a homogeneous vertex $q_g$5,

$q_g$6

and for an incident edge collapse the optimal position is obtained from the combined quadric $q_g$7 (Huang et al., 2024).

The key architectural point is that connectivity is borrowed from the reference mesh while geometry is adapted to the target frame. This permits subdivision, displacement coding, and attribute mapping to proceed “without re-meshing at the decoder.” The reported effect is a geometry BD-rate gain of 7.2% to 10.3% compared with V-DMC TM v7.0, with encoding time approximately 108% and decoding time approximately 102% of the baseline (Huang et al., 2024).

A different route to temporal consistency is used by V2M4. Instead of handling topology-varying meshes at the codec stage, it registers each frame back to the topology of the first refined mesh, optimizes a single texture map on the shared UV layout, and exports the animation as a base mesh plus per-frame vertex offsets (Chen et al., 11 Mar 2025). This suggests two complementary interpretations of Video-RDMesh: one in which temporal correspondence is created for compression, and one in which it is imposed during reconstruction so that the asset is already animation- and codec-ready.

6. Dataset and empirical landscape

As a dataset, Video-RDMesh was introduced to support R-DMesh’s pose-rectified 4D generation setting. It contains 252,823 unique dynamic assets and 513,690 high-quality, 64-frame vertex-trajectory clips paired with rendered reference videos. Training and evaluation use topology and resolution filters of vertex count $q_g$8 and face-to-vertex ratio $q_g$9, with padding to 8,192 vertices and 20,480 faces. Each example includes a dynamic mesh sequence $\Sigma_g=R(q_g)\operatorname{diag}(s_g^2)R(q_g)^T.$0, a reference video rendered in Blender, a conditional static mesh $\Sigma_g=R(q_g)\operatorname{diag}(s_g^2)R(q_g)^T.$1, derived motion components $\Sigma_g=R(q_g)\operatorname{diag}(s_g^2)R(q_g)^T.$2 and $\Sigma_g=R(q_g)\operatorname{diag}(s_g^2)R(q_g)^T.$3, and video features $\Sigma_g=R(q_g)\operatorname{diag}(s_g^2)R(q_g)^T.$4 extracted from Wan2.2-TI2V-5B (Wu et al., 13 May 2026).

On the Video-RDMesh benchmarks, R-DMesh reports PSNR 25.8, Subject Consistency 0.949, Motion Smoothness 0.995, EucD 0.012, and inference time about 10 s, compared with PSNR 23.4 and time about 40 m for SC4D, PSNR 22.3 and time about 30 s for L4GM, PSNR 13.5 and time about 8 s for AnimateAnyMesh, and EucD 0.035 with time about 25 m for Puppeteer (Wu et al., 13 May 2026).

Outside that benchmark, DreamMesh4D reports representative averages on the Consistent4D test split of PSNR(ref)=37.04, SSIM(ref)=0.980, LPIPS=0.126, FVD=474.96, FID-VID=29.14, and CLIP=0.938, outperforming the baselines listed in the paper on all metrics (Li et al., 2024). RMAvatar reports strong novel-view results on PeopleSnapshot and ZJU-MoCap, with examples including PSNR 34.12, SSIM 0.985, and LPIPS 0.013 on male-3-casual, and PSNR 34.30, SSIM 0.979, and LPIPS 0.024 on ZJU sequence 392 (Peng et al., 13 Jan 2025). NeuralMeshing, in a multi-video object setting, reports mean MAE 0.11, RMSE 0.16, and PSNR 16.80 over five objects, compared with NeRF at MAE 0.22, RMSE 0.27, PSNR 11.56 and NeuS2 at MAE 0.20, RMSE 0.25, PSNR 12.13 (Erich et al., 22 Aug 2025).

These results are not directly interchangeable because the tasks differ—video-guided animation, monocular 4D generation, avatar reconstruction, dynamic mesh coding, and complete object meshing—but they collectively indicate that explicit-mesh pipelines can compete across appearance quality, temporal coherence, runtime, and compression efficiency.

7. Limitations and open problems

Several limitations recur across the literature. DreamMesh4D depends on Zero123 SDS during training, assumes a fixed camera around a static object, and notes that a deformation graph with fixed connectivity may struggle with “extreme topology changes or severe self-occlusions”; it also models view dependence with spherical harmonics rather than explicit physically based materials (Li et al., 2024). RMAvatar inherits the limitations of LBS-guided human meshes and can fail when garments or hair deviate strongly from the underlying human topology (Peng et al., 13 Jan 2025).

R-DMesh identifies “mesh interpenetration” and degraded synthesis quality on rare object categories as current failure modes, attributing some of the difficulty to noisy ground-truth data and data sparsity (Wu et al., 13 May 2026). V2M4 assumes a fixed-camera monocular video, minimal topology change, and sufficiently good initial geometry from TRELLIS; poor generator outputs propagate into later registration and texture optimization stages (Chen et al., 11 Mar 2025). NeuralMeshing remains sensitive to reflective or transparent surfaces, autofocus blur, low texture, insufficient baselines, and fiducial errors, all of which can destabilize SfM or neural field learning (Erich et al., 22 Aug 2025). In compression, the coarse-to-fine anchor strategy retains its gains mainly where inter-frame tools remain effective; at high bitrates, denser subdivisions and overlapping patches reduce the benefit, and over-aggressive QEM can smooth sharp features unless boundary- or feature-aware constraints are used (Huang et al., 2024).

A final misconception is that Video-RDMesh implies a single representation or objective. The evidence points instead to a broader convergence: explicit meshes are being combined with differentiable rendering, radiance Gaussians, learned deformation fields, topology-preserving registration, and rate–distortion tools. This suggests that the long-term significance of Video-RDMesh lies less in one canonical model than in a shared commitment to explicit, temporally coherent, graphics-compatible 4D assets grounded directly in video (Jena et al., 2023, Li et al., 2024).