VIRTUS-FPP: Virtual Fringe Projection Profilometry

• VIRTUS-FPP is a virtual sensor modeling framework for fringe projection profilometry that replicates the structured-light pipeline with full mathematical fidelity.
• It integrates virtual calibration, phase decoding, and triangulation-based reconstruction within NVIDIA Isaac Sim, enabling rapid prototyping and sensor design iteration.
• The framework achieves sub-pixel calibration accuracy and digital-twin correspondence by validating 3D reconstructions against precise ground truth geometry.

VIRTUS-FPP is a virtual sensor modeling framework for fringe projection profilometry (FPP) implemented in NVIDIA Isaac Sim. It is presented as the first comprehensive physics-based virtual sensor modeling framework for FPP in Isaac Sim, with the explicit aim of reproducing the complete structured-light pipeline—from calibration through phase decoding to triangulation-based reconstruction—while maintaining full mathematical fidelity to the standard camera–projector formulation used in physical FPP systems (Haroon et al., 18 Sep 2025). In this setting, simulation is not treated as a visual approximation alone: the framework models a pinhole camera, a projector realized as a textured rectangular light, phase-shift fringe projection, virtual calibration, wrapped and unwrapped phase recovery, and 3D reconstruction against exact scene geometry, and it further extends this pipeline to a digital-twin configuration intended to replicate a real FPP setup.

1. Definition, scope, and motivation

VIRTUS-FPP addresses a set of practical constraints that are intrinsic to physical fringe projection profilometry. The paper identifies complex calibration requirements, a bulky system footprint, and sensitivity to environmental conditions as central limitations of real FPP systems. It also emphasizes that physical development is hindered by the difficulty of controlled experiments under varying ambient light, reflectance, specularity, translucency, and scattering, and by the lack of exact geometric ground truth for validating reconstruction (Haroon et al., 18 Sep 2025).

Within this framing, VIRTUS-FPP is defined as an end-to-end virtual FPP system in which the same mathematical objects that appear in real structured-light metrology—camera and projector intrinsics and extrinsics, phase-shift image sequences, wrapped and unwrapped phase, and stereo-style triangulation—are instantiated inside Isaac Sim. The framework is intended for rapid virtual prototyping, sensor design iteration, controlled environmental testing, synthetic data generation, and digital-twin replication of real camera–projector systems. A central implication of this design is that calibration, acquisition, and validation can all be performed in a purely virtual environment before any physical implementation is built (Haroon et al., 18 Sep 2025).

A common misconception is to treat VIRTUS-FPP as a generic rendering pipeline for projected stripe patterns. The paper instead presents it as a virtual counterpart to a real FPP system whose significance lies precisely in preserving the structured-light reconstruction pipeline rather than only producing visually plausible projected fringes.

2. Simulation architecture in Isaac Sim

The framework is implemented in NVIDIA Isaac Sim and uses the simulator’s combination of USD scene representation, Python APIs and extension system, PhysX, RTX/OptiX ray tracing, MDL materials, and Replicator. The paper’s architectural description organizes the system into scene and object modeling, virtual camera–projector sensing, fringe generation and projection, synthetic image capture, calibration, phase extraction, triangulation-based reconstruction, and validation against ground truth (Haroon et al., 18 Sep 2025).

The virtual camera is a pre-defined pinhole camera asset with an attached RGB render product. The projector is modeled using a UsdLux rectangular light source configured with isProjector = True and a texture image containing the fringe pattern. This is a deliberate approximation to a physical projector within Isaac Sim’s native sensor primitives. Because Isaac Sim does not directly expose a full projector-intrinsics interface, the framework supplements the simulator’s geometric light definition with an explicit inverse camera model for projector geometry; this is one of the paper’s principal technical claims.

The paper describes the software implementation as a custom Python extension rather than a pure OmniGraph workflow. Three components are emphasized. setup_scene constructs the camera–projector system, calibration targets, scanned objects, lighting, and materials. setup_post_load manages runtime operations such as board reorientation, texture loading, and image acquisition. setup_post_reset restores repeatable initial conditions. Fringe projection is driven by an update_texture_callback that updates the projector texture, triggers rendering, waits briefly for completion, and captures the camera frame as a grayscale RGB image. This structure makes the simulation operate as a synchronized structured-light sensor rather than as a static rendered scene (Haroon et al., 18 Sep 2025).

The paper also reports concrete virtual sensor parameters. The camera uses a focal length of 50 cm, horizontal aperture 20.9995 cm, vertical aperture 15.2908 cm, near clip 0.5 m, far clip 1000 m, and resolution 960 × 960. The projector is configured with intensity 40 nits, height 0.625 m, width 0.5 m, white color $(1,1,1)$, exposure 0.0, and the fringe pattern as texture. For triangulation experiments, the projector is placed 0.1 m below and 0.125 m to the left of the camera (Haroon et al., 18 Sep 2025).

3. Mathematical formulation of the FPP pipeline

A defining feature of VIRTUS-FPP is that it reproduces the classical mathematical structure of fringe projection profilometry inside simulation. The paper uses the standard $N$-step phase-shifting intensity model

$$I_n(x,y) = I'(x,y) + I''(x,y)\cos\left(\phi(x,y) + \frac{2\pi n}{N}\right),$$

with $N=18$ phase shifts in the reported experiments (Haroon et al., 18 Sep 2025).

The wrapped phase is recovered by

$$\phi(x,y) = -\tan^{-1}\left(\frac{\sum_{n=1}^{N} I_n(x,y) \sin \delta_n}{\sum_{n=1}^{N} I_n(x,y) \cos \delta_n}\right), \qquad \delta_n = \frac{2\pi n}{N},$$

and the unwrapped phase is written as

$\Phi(x,y) = \phi(x,y) + 2\pi k(x,y),$

where $k(x,y)$ is the fringe order obtained through temporal unwrapping with Gray coding (Haroon et al., 18 Sep 2025).

The camera and projector are represented as calibrated pinhole devices with projection matrices

$$M^c = A^c[R^c\,|\,t^c], \qquad M^p = A^p[R^p\,|\,t^p].$$

For a 3D point $(x,y,z)$, the camera and projector image coordinates satisfy

$$s^c \begin{bmatrix} u^c\ v^c\ 1 \end{bmatrix} = M^c \begin{bmatrix} x\ y\ z\ 1 \end{bmatrix}, \qquad s^p \begin{bmatrix} u^p\ v^p\ 1 \end{bmatrix} = M^p \begin{bmatrix} x\ y\ z\ 1 \end{bmatrix}.$$

Triangulation is then posed as

$N$0

where $N$1 and $N$2 are assembled from the entries of $N$3, $N$4, and the recovered image correspondences $N$5, $N$6 (Haroon et al., 18 Sep 2025).

This explicit preservation of the FPP equations is the main reason the paper describes the framework as maintaining “full mathematical fidelity” rather than merely simulating projected texture appearance.

4. Virtual calibration and reconstruction workflow

The calibration target is an asymmetric circular board generated procedurally and applied as a diffuse texture to a plane mesh. Its dimensions are computed from the intended pattern specification using

$N$7

$N$8

with $N$9, and then scaled to the supporting plane by

$$I_n(x,y) = I'(x,y) + I''(x,y)\cos\left(\phi(x,y) + \frac{2\pi n}{N}\right),$$0

The resulting simulated circle diameter and center spacing are

$$I_n(x,y) = I'(x,y) + I''(x,y)\cos\left(\phi(x,y) + \frac{2\pi n}{N}\right),$$1

The paper reports that the generated board dimensions matched the expected values within 7% error, corresponding to about 1–2 mm difference (Haroon et al., 18 Sep 2025).

For calibration, the board is initially placed 0.5 m in front of the camera–projector system and then moved through 18 poses. These poses include vertical and lateral translations of $$I_n(x,y) = I'(x,y) + I''(x,y)\cos\left(\phi(x,y) + \frac{2\pi n}{N}\right),$$2 m and tilts of 5 to 15 degrees about the $$I_n(x,y) = I'(x,y) + I''(x,y)\cos\left(\phi(x,y) + \frac{2\pi n}{N}\right),$$3 and $$I_n(x,y) = I'(x,y) + I''(x,y)\cos\left(\phi(x,y) + \frac{2\pi n}{N}\right),$$4 axes. For each pose, the full set of 18 phase-shifted fringes is projected and captured, producing 936 fringe images in total. The paper states that this takes just over 5 minutes, equivalent to about 10,530 captures per hour, and describes the acquisition rate as about 3 FPS (Haroon et al., 18 Sep 2025).

After feature extraction and phase processing, the framework performs camera calibration, stereo calibration between camera and projector, and reprojection-error evaluation. The reported calibration errors are stereo reprojection error = 0.055506 and projector error = 0.048609, which the paper describes as sub-pixel (Haroon et al., 18 Sep 2025).

For reconstruction accuracy, the paper uses a sphere mesh of radius 50 mm as ground truth. The reconstructed point cloud is aligned to the exact sphere using ICP and a sphere is fit using MATLAB pcfitsphere, based on MSAC. The reported fitted radius is 50.512 mm for a true radius of 50.0 mm, giving an absolute radial error

$$I_n(x,y) = I'(x,y) + I''(x,y)\cos\left(\phi(x,y) + \frac{2\pi n}{N}\right),$$5

a relative error of 1.02%, and 99.7% inliers $$I_n(x,y) = I'(x,y) + I''(x,y)\cos\left(\phi(x,y) + \frac{2\pi n}{N}\right),$$6 (Haroon et al., 18 Sep 2025).

5. Physics-based rendering and environmental modeling

The rendering model is central to the framework’s claims. VIRTUS-FPP uses Isaac Sim’s RTX/OptiX-based renderer to account for optical effects relevant to structured-light imaging, including reflection, refraction, scattering, multi-bounce transport, global illumination, shadowing, and ambient occlusion. Through MDL materials and path tracing, the simulator can alter fringe appearance in response to surface roughness, metallicity, specularity, and ambient lighting (Haroon et al., 18 Sep 2025).

The paper evaluates this capability using a 50 mm radius sphere under four lighting conditions: baseline overhead sphere light, No_Ambient, One_Ambient, and Two_Ambient. It also varies materials across three cases: a baseline matte-like material with roughness = 0.95, specular = 0.15, and ambient occlusion to diffuse ratio = 0.95; a variant with AO-to-diffuse ratio = 0; and a metallic material with metallic = 1.0 and roughness = 0.2 (Haroon et al., 18 Sep 2025).

The reported conclusion is qualitative but specific: fringe visibility degrades as ambient illumination increases and as surfaces become more specular or metallic. This suggests that the simulator can reproduce categories of failure that are well known in physical FPP systems. A plausible implication is that the framework can be used not only for nominal reconstruction studies but also for controlled robustness evaluation and synthetic-data generation in adverse optical regimes. The paper also links this capability to future machine-learning use cases such as phase unwrapping, denoising, and reconstruction training (Haroon et al., 18 Sep 2025).

At the same time, the paper is explicit that some physical effects remain simplified. It identifies simplified handling of lens distortion, sensor noise, and projector gamma response, and notes that path-tracing noise can itself corrupt phase maps. It further reports that sampled direct lighting mode had to be disabled to avoid pixel-level artifacts, indicating that optical realism in the simulator still requires parameter choices that are tailored to FPP processing (Haroon et al., 18 Sep 2025).

6. Digital-twin formulation and real-system correspondence

A major extension of the framework is its digital twin capability. Here the goal is not only to run a virtual FPP system, but to replicate a real camera–projector arrangement in Isaac Sim and compare predicted fringe geometry and reconstruction performance against physical measurements (Haroon et al., 18 Sep 2025).

The camera intrinsics are transferred into simulation through Isaac Sim’s documented relations

$$I_n(x,y) = I'(x,y) + I''(x,y)\cos\left(\phi(x,y) + \frac{2\pi n}{N}\right),$$7

where $$I_n(x,y) = I'(x,y) + I''(x,y)\cos\left(\phi(x,y) + \frac{2\pi n}{N}\right),$$8 are calibrated focal lengths, $$I_n(x,y) = I'(x,y) + I''(x,y)\cos\left(\phi(x,y) + \frac{2\pi n}{N}\right),$$9 are image dimensions, and $N=18$0 is the physical pixel size. The projector is treated through an inverse pinhole model: $N=18$1 with

$N=18$2

Because $N=18$3 is non-square, the paper uses a Moore–Penrose pseudo-inverse via SVD. To remove depth ambiguity, it parameterizes the solution by $N=18$4, giving

$N=18$5

This inverse-camera model is then used to recover the physical dimensions of the projected fringe field from calibration data (Haroon et al., 18 Sep 2025).

The method is first validated internally on the virtual system. The paper reports projector error = 0.058357 and stereo reprojection error = 0.065518 in this setting. For projected image dimensions at $N=18$6 m, it obtains

$N=18$7

and over distances from 400 mm to 1000 mm it reports width MAE = 1.7014 mm and height MAE = 1.4170 mm between theoretical and measured projected dimensions (Haroon et al., 18 Sep 2025).

The same procedure is then applied to a real physical FPP system. The recovered metric projected fringe dimensions are

$N=18$8

while the real measured values at 0.4 m are Width = 204.20 mm and Height = 326.56 mm. The agreement is close, though the paper notes a simulator-specific artifact: correct correspondence in Isaac Sim required using projector dimensions computed at $N=18$9 m even when the desired physical screen distance was 0.4 m. The authors attribute this to the implementation of RectLight, which appears to use 1 m as a reference scaling distance (Haroon et al., 18 Sep 2025).

For geometric digital-twin validation, the paper uses a 3D-printed astronaut figurine scanned both physically and virtually from the same STL geometry. After ICP alignment to the source mesh, both real and virtual reconstructions yield Cloud-to-Mesh distance histograms that are left-skewed and peak in the 0–1 mm range. The paper interprets this as evidence of sub-millimeter reconstruction fidelity and close real–virtual agreement (Haroon et al., 18 Sep 2025).

7. Significance, limitations, and research uses

The framework’s reported significance lies in combining virtual calibration, physics-based image formation, structured-light mathematical fidelity, and digital-twin replication in a single Isaac Sim system. The paper claims this enables rapid virtual prototyping, system configuration studies, sensor design exploration, environmental stress testing, and synthetic dataset generation without dependence on a pre-calibrated real projector–camera pair (Haroon et al., 18 Sep 2025).

Several practical uses follow directly from the reported experiments. Because scene geometry is exact, reconstruction algorithms can be validated against ground truth rather than against another measurement device. Because lighting and materials are programmable, failure regimes can be explored systematically. Because calibration and acquisition are automated, the framework can generate structured-light datasets at high throughput. The conclusion explicitly points toward machine-learning applications, multi-view FPP, robotic manipulation and automated scanning, and hybrid physical–virtual workflows (Haroon et al., 18 Sep 2025).

The paper is equally clear about remaining limitations. Real-time operation requires strong RTX-capable hardware; the discussion gives NVIDIA GeForce RTX 3070 with at least 8 GB VRAM as a practical minimum. There is still a sim-to-real gap caused by simplified projector gamma, lens distortion, and sensor-noise modeling. Path-tracing noise can interfere with phase extraction. The projector model depends on the behavior of Isaac Sim’s RectLight, and an empirical scale factor of 1.37 was needed for accurate real-world correspondence in virtual reconstruction. These details indicate that the framework is physics-based but not a complete optical-electronic model of a real FPP instrument (Haroon et al., 18 Sep 2025).

A broader misconception would be to equate “digital twin” here with exact numerical identity between real and simulated measurements. The paper does not claim that. Instead, it presents a calibrated correspondence in projected dimensions and reconstruction accuracy that is sufficiently close to support design and validation workflows. This suggests a digital twin in the metrological sense of operational equivalence for system development, not exact duplication of every physical effect.

Quantity	Reported value	Context
Stereo reprojection error	0.055506	Virtual calibration
Projector error	0.048609	Virtual calibration
Sphere true radius	50.0 mm	Reconstruction target
Sphere estimated radius	50.512 mm	Reconstructed fit
Absolute radius error	0.512 mm	Sphere validation
Relative error	1.02%	Sphere validation
Inlier ratio	99.7%	Sphere fitting
Calibration poses	18	Virtual calibration
Phase shifts	18-step	Phase-shifting acquisition
Captured fringe images	936	Calibration run
Capture rate	about 3 FPS	Reported throughput

In sum, VIRTUS-FPP denotes a simulation-native, mathematically explicit FPP framework in Isaac Sim that reproduces the standard camera–projector structured-light pipeline, validates it against exact mesh geometry, and extends it to digital-twin comparison with physical systems. Its importance lies less in any single rendering feature than in the integration of optical simulation, virtual calibration, reconstruction mathematics, and real–virtual correspondence into a single metrology-oriented platform (Haroon et al., 18 Sep 2025).