---
title: 'VIRTUS-FPP: Virtual Fringe Projection Profilometry'
url: https://www.emergentmind.com/topics/virtus-fpp
type: topic
---

# VIRTUS-FPP: Virtual Fringe Projection Profilometry

VIRTUS-FPP is a virtual sensor modeling framework for **fringe projection profilometry (FPP)** implemented in **NVIDIA Isaac Sim**. It is presented as the **first comprehensive physics-based virtual sensor modeling framework for FPP in Isaac Sim**, with the explicit aim of reproducing the complete structured-light pipeline—from calibration through phase decoding to triangulation-based reconstruction—while maintaining **full mathematical fidelity** to the standard camera–projector formulation used in physical FPP systems [2509.22685]. In this setting, simulation is not treated as a visual approximation alone: the framework models a pinhole camera, a projector realized as a textured rectangular light, phase-shift fringe projection, virtual calibration, wrapped and unwrapped phase recovery, and 3D reconstruction against exact scene geometry, and it further extends this pipeline to a digital-twin configuration intended to replicate a real FPP setup.

## 1. Definition, scope, and motivation

VIRTUS-FPP addresses a set of practical constraints that are intrinsic to physical fringe projection profilometry. The paper identifies **complex calibration requirements**, a **bulky system footprint**, and **sensitivity to environmental conditions** as central limitations of real FPP systems. It also emphasizes that physical development is hindered by the difficulty of controlled experiments under varying ambient light, reflectance, specularity, translucency, and scattering, and by the lack of exact geometric ground truth for validating reconstruction [2509.22685].

Within this framing, VIRTUS-FPP is defined as an **end-to-end virtual FPP system** in which the same mathematical objects that appear in real structured-light metrology—camera and projector intrinsics and extrinsics, phase-shift image sequences, wrapped and unwrapped phase, and stereo-style triangulation—are instantiated inside Isaac Sim. The framework is intended for **rapid virtual prototyping**, **sensor design iteration**, **controlled environmental testing**, **synthetic data generation**, and **digital-twin replication** of real camera–projector systems. A central implication of this design is that calibration, acquisition, and validation can all be performed in a purely virtual environment before any physical implementation is built [2509.22685].

A common misconception is to treat VIRTUS-FPP as a generic rendering pipeline for projected stripe patterns. The paper instead presents it as a **virtual counterpart to a real FPP system** whose significance lies precisely in preserving the structured-light reconstruction pipeline rather than only producing visually plausible projected fringes.

## 2. Simulation architecture in Isaac Sim

The framework is implemented in **NVIDIA Isaac Sim** and uses the simulator’s combination of **USD scene representation**, **Python APIs and extension system**, **PhysX**, **RTX/OptiX ray tracing**, **MDL materials**, and **Replicator**. The paper’s architectural description organizes the system into scene and object modeling, virtual camera–projector sensing, fringe generation and projection, synthetic image capture, calibration, phase extraction, triangulation-based reconstruction, and validation against ground truth [2509.22685].

The virtual camera is a **pre-defined pinhole camera asset** with an attached RGB render product. The projector is modeled using a **UsdLux rectangular light source** configured with `isProjector = True` and a texture image containing the fringe pattern. This is a deliberate approximation to a physical projector within Isaac Sim’s native sensor primitives. Because Isaac Sim does not directly expose a full projector-intrinsics interface, the framework supplements the simulator’s geometric light definition with an explicit **inverse camera model** for projector geometry; this is one of the paper’s principal technical claims.

The paper describes the software implementation as a **custom Python extension** rather than a pure OmniGraph workflow. Three components are emphasized. `setup_scene` constructs the camera–projector system, calibration targets, scanned objects, lighting, and materials. `setup_post_load` manages runtime operations such as board reorientation, texture loading, and image acquisition. `setup_post_reset` restores repeatable initial conditions. Fringe projection is driven by an `update_texture_callback` that updates the projector texture, triggers rendering, waits briefly for completion, and captures the camera frame as a grayscale RGB image. This structure makes the simulation operate as a synchronized structured-light sensor rather than as a static rendered scene [2509.22685].

The paper also reports concrete virtual sensor parameters. The camera uses a focal length of **50 cm**, horizontal aperture **20.9995 cm**, vertical aperture **15.2908 cm**, near clip **0.5 m**, far clip **1000 m**, and resolution **960 × 960**. The projector is configured with intensity **40 nits**, height **0.625 m**, width **0.5 m**, white color \((1,1,1)\), exposure **0.0**, and the fringe pattern as texture. For triangulation experiments, the projector is placed **0.1 m below** and **0.125 m to the left** of the camera [2509.22685].

## 3. Mathematical formulation of the FPP pipeline

A defining feature of VIRTUS-FPP is that it reproduces the classical mathematical structure of fringe projection profilometry inside simulation. The paper uses the standard \(N\)-step phase-shifting intensity model
\[
I_n(x,y) = I'(x,y) + I''(x,y)\cos\left(\phi(x,y) + \frac{2\pi n}{N}\right),
\]
with **\(N=18\)** phase shifts in the reported experiments [2509.22685].

The wrapped phase is recovered by
\[
\phi(x,y) = -\tan^{-1}\left(\frac{\sum_{n=1}^{N} I_n(x,y) \sin \delta_n}{\sum_{n=1}^{N} I_n(x,y) \cos \delta_n}\right),
\qquad
\delta_n = \frac{2\pi n}{N},
\]
and the unwrapped phase is written as
\[
\Phi(x,y) = \phi(x,y) + 2\pi k(x,y),
\]
where \(k(x,y)\) is the fringe order obtained through temporal unwrapping with Gray coding [2509.22685].

The camera and projector are represented as calibrated pinhole devices with projection matrices
\[
M^c = A^c[R^c\,|\,t^c],
\qquad
M^p = A^p[R^p\,|\,t^p].
\]
For a 3D point \((x,y,z)\), the camera and projector image coordinates satisfy
\[
s^c
\begin{bmatrix}
u^c\\
v^c\\
1
\end{bmatrix}
=
M^c
\begin{bmatrix}
x\\
y\\
z\\
1
\end{bmatrix},
\qquad
s^p
\begin{bmatrix}
u^p\\
v^p\\
1
\end{bmatrix}
=
M^p
\begin{bmatrix}
x\\
y\\
z\\
1
\end{bmatrix}.
\]
Triangulation is then posed as
\[
\begin{bmatrix}
x\\
y\\
z
\end{bmatrix}
=
\mathbf{A}^{-1}\mathbf{b},
\]
where \(\mathbf{A}\) and \(\mathbf{b}\) are assembled from the entries of \(M^c\), \(M^p\), and the recovered image correspondences \((u^c,v^c)\), \((u^p,v^p)\) [2509.22685].

This explicit preservation of the FPP equations is the main reason the paper describes the framework as maintaining “full mathematical fidelity” rather than merely simulating projected texture appearance.

## 4. Virtual calibration and reconstruction workflow

The calibration target is an **asymmetric circular board** generated procedurally and applied as a diffuse texture to a plane mesh. Its dimensions are computed from the intended pattern specification using
\[
W_{\mathrm{pattern}}=
\left(2L_{\mathrm{border}}+\frac{(C-1)D_{\mathrm{centers}}}{2}+D_{\mathrm{circle}}\right)\cdot k_{\mathrm{mm2m}},
\]
\[
H_{\mathrm{pattern}}=
\left(2L_{\mathrm{border}}+(R-1)D_{\mathrm{centers}}+D_{\mathrm{circle}}\right)\cdot k_{\mathrm{mm2m}},
\]
with \(k_{\mathrm{mm2m}}=0.001\), and then scaled to the supporting plane by
\[
S_{\mathrm{pattern}}=
\min\left(
\frac{W_{\mathrm{plane}}}{W_{\mathrm{pattern}}},
\frac{H_{\mathrm{plane}}}{H_{\mathrm{pattern}}}
\right).
\]
The resulting simulated circle diameter and center spacing are
\[
D_{\mathrm{circle}}^{\mathrm{sim}} = D_{\mathrm{circle}}\cdot k_{\mathrm{mm2m}}\cdot S_{\mathrm{pattern}},
\qquad
D_{\mathrm{centers}}^{\mathrm{sim}} = D_{\mathrm{centers}}\cdot k_{\mathrm{mm2m}}\cdot S_{\mathrm{pattern}}.
\]
The paper reports that the generated board dimensions matched the expected values within **7% error**, corresponding to about **1–2 mm difference** [2509.22685].

For calibration, the board is initially placed **0.5 m** in front of the camera–projector system and then moved through **18 poses**. These poses include vertical and lateral translations of \(\pm 0.015\) m and tilts of **5 to 15 degrees** about the \(x\) and \(z\) axes. For each pose, the full set of **18 phase-shifted fringes** is projected and captured, producing **936 fringe images** in total. The paper states that this takes **just over 5 minutes**, equivalent to about **10,530 captures per hour**, and describes the acquisition rate as about **3 FPS** [2509.22685].

After feature extraction and phase processing, the framework performs camera calibration, stereo calibration between camera and projector, and reprojection-error evaluation. The reported calibration errors are **stereo reprojection error = 0.055506** and **projector error = 0.048609**, which the paper describes as **sub-pixel** [2509.22685].

For reconstruction accuracy, the paper uses a **sphere mesh of radius 50 mm** as ground truth. The reconstructed point cloud is aligned to the exact sphere using ICP and a sphere is fit using MATLAB `pcfitsphere`, based on **MSAC**. The reported fitted radius is **50.512 mm** for a true radius of **50.0 mm**, giving an absolute radial error
\[
R_{\mathrm{abs}} = |R_{\mathrm{est}}-R_{\mathrm{act}}| = |50.512-50.0| = 0.512\ \mathrm{mm},
\]
a **relative error of 1.02%**, and **99.7% inliers** \((133{,}802 / 134{,}512)\) [2509.22685].

## 5. Physics-based rendering and environmental modeling

The rendering model is central to the framework’s claims. VIRTUS-FPP uses Isaac Sim’s RTX/OptiX-based renderer to account for optical effects relevant to structured-light imaging, including **reflection**, **refraction**, **scattering**, **multi-bounce transport**, **global illumination**, **shadowing**, and **ambient occlusion**. Through MDL materials and path tracing, the simulator can alter fringe appearance in response to surface roughness, metallicity, specularity, and ambient lighting [2509.22685].

The paper evaluates this capability using a **50 mm radius sphere** under four lighting conditions: baseline overhead sphere light, **No_Ambient**, **One_Ambient**, and **Two_Ambient**. It also varies materials across three cases: a baseline matte-like material with **roughness = 0.95**, **specular = 0.15**, and **ambient occlusion to diffuse ratio = 0.95**; a variant with **AO-to-diffuse ratio = 0**; and a **metallic** material with **metallic = 1.0** and **roughness = 0.2** [2509.22685].

The reported conclusion is qualitative but specific: fringe visibility degrades as ambient illumination increases and as surfaces become more specular or metallic. This suggests that the simulator can reproduce categories of failure that are well known in physical FPP systems. A plausible implication is that the framework can be used not only for nominal reconstruction studies but also for controlled robustness evaluation and synthetic-data generation in adverse optical regimes. The paper also links this capability to future machine-learning use cases such as phase unwrapping, denoising, and reconstruction training [2509.22685].

At the same time, the paper is explicit that some physical effects remain simplified. It identifies simplified handling of **lens distortion**, **sensor noise**, and **projector gamma response**, and notes that path-tracing noise can itself corrupt phase maps. It further reports that **sampled direct lighting mode** had to be disabled to avoid pixel-level artifacts, indicating that optical realism in the simulator still requires parameter choices that are tailored to FPP processing [2509.22685].

## 6. Digital-twin formulation and real-system correspondence

A major extension of the framework is its **digital twin** capability. Here the goal is not only to run a virtual FPP system, but to replicate a real camera–projector arrangement in Isaac Sim and compare predicted fringe geometry and reconstruction performance against physical measurements [2509.22685].

The camera intrinsics are transferred into simulation through Isaac Sim’s documented relations
\[
f = \frac{f_x + f_y}{2}\cdot s_{\mathrm{mm}},
\qquad
A_h = W\cdot s_{\mathrm{mm}},
\qquad
A_v = H\cdot s_{\mathrm{mm}},
\]
where \(f_x,f_y\) are calibrated focal lengths, \(W,H\) are image dimensions, and \(s_{\mathrm{mm}}\) is the physical pixel size. The projector is treated through an **inverse pinhole model**:
\[
\begin{bmatrix}
X\\
Y\\
Z
\end{bmatrix}
=
\left(M_{\mathrm{ext}}\right)^{-1}
\left(M_{\mathrm{int}}\right)^{-1}
\begin{bmatrix}
u\\
v\\
1
\end{bmatrix},
\]
with
\[
M_{\mathrm{int}}=
\begin{bmatrix}
f_x & 0 & o_x\\
0 & f_y & o_y\\
0 & 0 & 1
\end{bmatrix},
\qquad
M_{\mathrm{ext}}=[R_p \mid T_p].
\]
Because \(M_{\mathrm{ext}}\) is non-square, the paper uses a **Moore–Penrose pseudo-inverse via SVD**. To remove depth ambiguity, it parameterizes the solution by \(Z\), giving
\[
\begin{bmatrix}
x_w\\
y_w\\
1
\end{bmatrix}
=
Z
\left(
\left(M_{\mathrm{ext}}\right)^{-1}
\left(M_{\mathrm{int}}\right)^{-1}
\begin{bmatrix}
u\\
v\\
1
\end{bmatrix}
\right).
\]
This inverse-camera model is then used to recover the physical dimensions of the projected fringe field from calibration data [2509.22685].

The method is first validated internally on the virtual system. The paper reports **projector error = 0.058357** and **stereo reprojection error = 0.065518** in this setting. For projected image dimensions at \(1\) m, it obtains
\[
\begin{bmatrix}
x_w\\
y_w
\end{bmatrix}
=
\begin{bmatrix}
626.3\\
501.1
\end{bmatrix},
\]
and over distances from **400 mm to 1000 mm** it reports **width MAE = 1.7014 mm** and **height MAE = 1.4170 mm** between theoretical and measured projected dimensions [2509.22685].

The same procedure is then applied to a real physical FPP system. The recovered metric projected fringe dimensions are
\[
\begin{bmatrix}
x_w\\
y_w
\end{bmatrix}
=
\begin{bmatrix}
202.7\\
323.3
\end{bmatrix},
\]
while the real measured values at **0.4 m** are **Width = 204.20 mm** and **Height = 326.56 mm**. The agreement is close, though the paper notes a simulator-specific artifact: correct correspondence in Isaac Sim required using projector dimensions computed at **\(Z=1\) m** even when the desired physical screen distance was **0.4 m**. The authors attribute this to the implementation of `RectLight`, which appears to use **1 m** as a reference scaling distance [2509.22685].

For geometric digital-twin validation, the paper uses a **3D-printed astronaut figurine** scanned both physically and virtually from the same STL geometry. After ICP alignment to the source mesh, both real and virtual reconstructions yield **Cloud-to-Mesh distance histograms** that are **left-skewed** and peak in the **0–1 mm** range. The paper interprets this as evidence of **sub-millimeter reconstruction fidelity** and close real–virtual agreement [2509.22685].

## 7. Significance, limitations, and research uses

The framework’s reported significance lies in combining **virtual calibration**, **physics-based image formation**, **structured-light mathematical fidelity**, and **digital-twin replication** in a single Isaac Sim system. The paper claims this enables **rapid virtual prototyping**, **system configuration studies**, **sensor design exploration**, **environmental stress testing**, and **synthetic dataset generation** without dependence on a pre-calibrated real projector–camera pair [2509.22685].

Several practical uses follow directly from the reported experiments. Because scene geometry is exact, reconstruction algorithms can be validated against ground truth rather than against another measurement device. Because lighting and materials are programmable, failure regimes can be explored systematically. Because calibration and acquisition are automated, the framework can generate structured-light datasets at high throughput. The conclusion explicitly points toward **machine-learning applications**, **multi-view FPP**, **robotic manipulation and automated scanning**, and **hybrid physical–virtual workflows** [2509.22685].

The paper is equally clear about remaining limitations. Real-time operation requires strong RTX-capable hardware; the discussion gives **NVIDIA GeForce RTX 3070** with at least **8 GB VRAM** as a practical minimum. There is still a **sim-to-real gap** caused by simplified projector gamma, lens distortion, and sensor-noise modeling. Path-tracing noise can interfere with phase extraction. The projector model depends on the behavior of Isaac Sim’s `RectLight`, and an empirical scale factor of **1.37** was needed for accurate real-world correspondence in virtual reconstruction. These details indicate that the framework is physics-based but not a complete optical-electronic model of a real FPP instrument [2509.22685].

A broader misconception would be to equate “digital twin” here with exact numerical identity between real and simulated measurements. The paper does not claim that. Instead, it presents a calibrated correspondence in projected dimensions and reconstruction accuracy that is sufficiently close to support design and validation workflows. This suggests a digital twin in the metrological sense of **operational equivalence for system development**, not exact duplication of every physical effect.

| Quantity | Reported value | Context |
|---|---:|---|
| Stereo reprojection error | 0.055506 | Virtual calibration |
| Projector error | 0.048609 | Virtual calibration |
| Sphere true radius | 50.0 mm | Reconstruction target |
| Sphere estimated radius | 50.512 mm | Reconstructed fit |
| Absolute radius error | 0.512 mm | Sphere validation |
| Relative error | 1.02% | Sphere validation |
| Inlier ratio | 99.7% | Sphere fitting |
| Calibration poses | 18 | Virtual calibration |
| Phase shifts | 18-step | Phase-shifting acquisition |
| Captured fringe images | 936 | Calibration run |
| Capture rate | about 3 FPS | Reported throughput |

In sum, VIRTUS-FPP denotes a simulation-native, mathematically explicit FPP framework in Isaac Sim that reproduces the standard camera–projector structured-light pipeline, validates it against exact mesh geometry, and extends it to digital-twin comparison with physical systems. Its importance lies less in any single rendering feature than in the integration of optical simulation, virtual calibration, reconstruction mathematics, and real–virtual correspondence into a single metrology-oriented platform [2509.22685].

Source: https://www.emergentmind.com/topics/virtus-fpp