Part-Level Force Visualization

• Part-level force visualization is the process of localizing physical forces to specific regions of an object using meshes, voxel fields, tactile surfaces, and finite-element methods.
• It employs diverse mathematical representations and inference strategies—including inverse mechanics and calibrated sensor data—to accurately capture and decompose force vectors.
• Visualization techniques such as vector glyphs, heatmaps, and stress trajectory lines provide actionable insights into force direction, magnitude, and impact on motion and stability.

Searching arXiv for relevant papers on part-level force visualization and closely related force/stress/contact visualization methods. Part-level force visualization is the problem of localizing physical loading to specific regions of an object, contact surface, or solid, rather than reporting only a global force or torque. In current work, the localized entity may be a set of contact points on a known rigid-object mesh, a per-voxel scalar field over a 3D scene, a dense traction field on a visuotactile membrane, a per-Gaussian force decomposition on a deformable object, a family of principal stress trajectories inside a finite-element part, or an end-effector displacement glyph that makes controller-generated forces visible in augmented reality (Ehsani et al., 2020, Hanai et al., 2023, Zhang et al., 2019, Peng et al., 2023, Gao et al., 30 Nov 2025, Wang et al., 2021, Brandt et al., 26 Mar 2026). Across these formulations, the central objective is consistent: to show where forces act, in which directions they act, and how they relate to observed motion, deformation, stability, or control.

1. Spatial meaning of “part” in force visualization

The term “part” is not tied to a single representation. In video-based human–object interaction, contact points are predicted in the object-coordinate frame as 3D points on the object mesh surface, and these points can be mapped to mesh parts such as a handle, spout, or lid if a segmentation is available; otherwise, proximity to semantic keypoints or mesh regions is used (Ehsani et al., 2020). In static clutter scenes, the representation may not contain explicit part labels at all: a per-voxel scalar field can still localize force intensity to bottom faces, edges contacting walls, and inter-object interfaces, so that object parts emerge from the spatial distribution of contact intensity rather than from an explicit segmentation (Hanai et al., 2023).

On tactile sensing surfaces, “part” usually means a contact patch or a subregion of a membrane. The FingerVision formulation works on the 2D sensor surface and visualizes normal force heatmaps, tangential traction arrows, and torsional maps over the contact patch, while the soft-bubble formulation estimates one traction vector per mesh vertex and interpolates over triangles for dense surface maps (Zhang et al., 2019, Peng et al., 2023). In differentiable inverse graphics for deformable objects, the native support is per-Gaussian: semantic masks, grounded segmentation, clustering, or keypoint-based soft memberships can define parts, and per-part net force and torque are then obtained by aggregating per-Gaussian forces (Gao et al., 30 Nov 2025).

Finite-element stress visualization uses a different notion of part. In 3D-TSV, the object is a single solid part, and the visualization emphasizes how tensile and compressive action travels through the interior by tracing principal stress lines of the Cauchy stress tensor field (Wang et al., 2021). By contrast, the augmented-reality teleoperation formulation begins at end-effector scale: the target pose glyph and displacement line visualize the virtual spring deflection of an impedance controller, and the paper treats true part-level force overlays as an extension based on per-contact vectors or surface stress maps (Brandt et al., 26 Mar 2026). This suggests that part-level force visualization is better understood as spatial attribution of load than as a fixed data structure.

2. Core representations and mathematical forms

The principal representational forms used in current work can be summarized as follows.

Support	Localized quantity	Representative form
Known rigid object mesh (Ehsani et al., 2020)	Per-finger contact force vectors	$c_i(t)$, $F_i(t)$, $\tau_i(t) = r_i(t) \times F_i(t)$
3D voxel grid (Hanai et al., 2023)	Scalar contact-force magnitude field	$F(x) \in \mathbb{R}_{\ge 0}$
2D tactile surface (Zhang et al., 2019)	Normal, shear, torsional components	$f_n(x,y)$, $\mathbf{f}_t(x,y)$, $\tau_z(x,y)$
Bubble membrane mesh (Peng et al., 2023)	Per-vertex traction, normal and shear	$t_i$, $t_{n,i}$, $t_{t,i}$
Per-Gaussian deformable object (Gao et al., 30 Nov 2025)	External and internal force decomposition	$F_i(t)$0, $F_i(t)$1, $F_i(t)$2
Finite-element solid (Wang et al., 2021)	Principal stress directions and magnitudes	$F_i(t)$3, $F_i(t)$4, PSLs
End-effector/controller frame (Brandt et al., 26 Mar 2026)	Impedance-generated wrench proxy	target disk, displacement line

In mesh-grounded contact prediction, the local quantity is a 3D force vector applied at a surface point, and rotational effect is made explicit through torque:

$F_i(t)$5

The visualization can further separate normal and tangential force components using a mesh normal $F_i(t)$6 and, for plausibility checks, the inferred Coulomb condition $F_i(t)$7 (Ehsani et al., 2020).

In force-map prediction, the localized entity is not a vector but a scalar “force density” on a regular voxel grid:

$F_i(t)$8

with $F_i(t)$9 the 3D Gaussian kernel and $\tau_i(t) = r_i(t) \times F_i(t)$0 given by a temporal moving average after stabilization. The map represents only magnitudes, not directional decomposition (Hanai et al., 2023).

Tactile and membrane approaches localize force through deformation inversion. On the FingerVision surface, the measured displacement field is decomposed as

$\tau_i(t) = r_i(t) \times F_i(t)$1

so that $\tau_i(t) = r_i(t) \times F_i(t)$2 captures compression-like patterns and $\tau_i(t) = r_i(t) \times F_i(t)$3 captures shear-like or torsional patterns (Zhang et al., 2019). On the bubble sensor, the global FE equilibrium yields

$\tau_i(t) = r_i(t) \times F_i(t)$4

followed by a per-vertex traction decomposition $\tau_i(t) = r_i(t) \times F_i(t)$5, $\tau_i(t) = r_i(t) \times F_i(t)$6, and $\tau_i(t) = r_i(t) \times F_i(t)$7 (Peng et al., 2023).

In deformable inverse graphics, the support is per Gaussian, and a part $\tau_i(t) = r_i(t) \times F_i(t)$8 is a set of Gaussian indices. The net part force and torque are

$\tau_i(t) = r_i(t) \times F_i(t)$9

This directly supports type-wise aggregation into wind, gravity, elastic, damping, and contact contributions (Gao et al., 30 Nov 2025). In stress visualization, the localized structure is an integral curve of a principal direction:

$F(x) \in \mathbb{R}_{\ge 0}$0

with $F(x) \in \mathbb{R}_{\ge 0}$1 encoding tensile or compressive intensity along the trajectory (Wang et al., 2021).

3. Inference, supervision, and inverse mechanics

A major divide in part-level force visualization is whether the localized quantity is directly measured, indirectly supervised, or inferred only through physical consistency. In “Use the Force, Luke!,” there is no ground-truth force supervision. Predicted contact points and forces are applied in PyBullet, and the model is trained so that the simulated object motion matches the observed motion through a 2D keypoint reprojection loss,

$F(x) \in \mathbb{R}_{\ge 0}$2

with gradients through PyBullet obtained by finite differences and a joint objective that couples contact localization to effect consistency (Ehsani et al., 2020). The result is a force estimate whose meaning derives from the effect it reproduces.

Force-map learning uses the opposite strategy: privileged simulation contact data become direct supervision. PyBullet provides sparse contact positions and force vectors, and these are converted into smooth labels with 3D weighted KDE and temporal averaging. A ResNet-50 encoder with a Residual U-Net style decoder then maps a single RGB image to multi-channel $F(x) \in \mathbb{R}_{\ge 0}$3-slices of the 3D force map (Hanai et al., 2023). The scalar target emphasizes rough contact-force patterns rather than precise vectors.

Tactile methods are calibration-based inverse mappings. The FingerVision pipeline tracks markers, interpolates a dense displacement field, solves the Poisson problems for the Helmholtz–Hodge decomposition under natural boundary conditions, and then maps low-dimensional summaries of the curl-free, raw, and divergence-free components to net normal force, tangential force, and torsional torque using RANSAC linear regression or a small MLP (Zhang et al., 2019). The bubble formulation is more explicitly mechanical: RGB-D and pressure sensing provide noisy nodal displacements and $F(x) \in \mathbb{R}_{\ge 0}$4, and the unknown correction $F(x) \in \mathbb{R}_{\ge 0}$5 is found through a convex group-lasso inverse problem that promotes sparse, patch-like external tractions while penalizing large displacement corrections (Peng et al., 2023).

Differentiable inverse graphics extends this logic from contact surfaces to deformable objects in video. “Seeing the Wind from a Falling Leaf” represents a time-varying force field with a causal tri-plane and recurrent time encoder, couples it to MPM-style dynamics, and backpropagates a sparse 4D tracking loss through the simulator to recover the force field that best explains observed motion (Gao et al., 30 Nov 2025). Stress visualization in 3D-TSV is not itself an inverse problem; it assumes the stress tensor field is already available from FEA and focuses on trajectory extraction, spacing, and rendering (Wang et al., 2021). In augmented-reality teleoperation, the inferred quantity is not contact force distribution but controller-generated wrench, computed from pose and velocity errors by the impedance law

$F(x) \in \mathbb{R}_{\ge 0}$6

and visualized through target pose and displacement (Brandt et al., 26 Mar 2026).

4. Visualization operators and perceptual encodings

The most direct part-level encoding is the localized vector glyph. For mesh-grounded human–object interaction, arrows are rendered at the predicted contact locations $F(x) \in \mathbb{R}_{\ge 0}$7, oriented along $F(x) \in \mathbb{R}_{\ge 0}$8, scaled by $F(x) \in \mathbb{R}_{\ge 0}$9, and color-coded by finger identity: thumb orange, index red, middle blue, ring green, pinky purple. Because the contacts lie on the mesh, the arrows immediately distinguish object regions such as pitcher handle versus rim or drill grip versus body. The same framework supports color-coded magnitude heatmaps, directional histograms, cumulative impulse over time, torque arc glyphs, and temporal trails showing how contacts migrate across parts during manipulation (Ehsani et al., 2020).

Scalar-field methods trade vector detail for spatial coverage. In force-map prediction, the localized quantity is a per-voxel scalar field that peaks at contact patches such as basket bottoms, walls, and inter-object interfaces. The field is sliced along the $f_n(x,y)$0-axis and treated as a multi-channel image; its gradients are then used to approximate outward contact normals and to select lifting directions that avoid pushing into neighboring contact gradients (Hanai et al., 2023). This yields a part-level cue without explicit vector decomposition.

Tactile visualization typically separates normal, tangential, and torsional structure. The FingerVision module renders a normal-force heatmap $f_n(x,y)$1, tangential traction arrows $f_n(x,y)$2, and a torsional map $f_n(x,y)$3, often together with friction-cone plots and phase labels such as stable, incipient slip, slipping, and recovery (Zhang et al., 2019). The bubble sensor likewise renders a 3D surface colormap of normal traction, a tangential magnitude map, and a vector-field overlay for the full traction vector, with the thresholded contact patch shown explicitly (Peng et al., 2023).

Finite-element and inverse-graphics methods add internal or volumetric views. 3D-TSV extracts evenly spaced, mutually orthogonal principal stress lines, converts them to renderable line sets, and optionally combines two principal directions into oriented ribbons; halos, desaturation with distance, ambient occlusion, outlines, and a translucent outline hull improve depth perception and reduce clutter (Wang et al., 2021). The deformable inverse-graphics formulation produces per-Gaussian forces, which can be visualized as color maps on Gaussians, arrow plots, normal/tangential decompositions, streamlines of $f_n(x,y)$4, and time series of $f_n(x,y)$5 and $f_n(x,y)$6 (Gao et al., 30 Nov 2025). In augmented reality, the perceptual encoding is deliberately simpler: a translucent blue “target disk” at the target pose and a translucent blue displacement line from the current end-effector position to the target position convey translational force direction, translational force magnitude, and rotational intent in real time (Brandt et al., 26 Mar 2026).

5. Empirical evidence and reported performance

The current literature supports part-level force visualization with several distinct forms of evidence.

Setting	Quantitative evidence	Reported effect
Joint contact–force prediction (Ehsani et al., 2020)	CP error 6.71e−2 vs. 6.83e−2; KP 99.38 px vs. 104.57 px; rotation 0.183 vs. 0.218; translation 0.212 m vs. 0.250 m	Joint training improves both contact localization and motion imitation
Force-map planning (Hanai et al., 2023)	52 lifts: linear 5.63 ± 5.55 cm vs. 4.15 ± 4.62 cm; angular 0.710 ± 0.984 rad vs. 0.430 ± 0.769 rad	26% reduction in translation displacement and 39% reduction in angular displacement
HHD tactile estimation (Zhang et al., 2019)	Tangential force RMSE: 0.241 ± 0.033 N with RANSAC linear; torsional torque RMSE: 5.862 ± 1.547 N·mm	Decomposition-based regressors outperform or match a large MLP on raw vectors
Bubble FEM inversion (Peng et al., 2023)	Avg. net force error 1.24 N vs. 1.95 N; contact mIOU 0.513 vs. 0.560	36% lower force error with comparable contact patch detection
AR impedance visualization (Brandt et al., 26 Mar 2026)	Lifting completion time reduced by ~24%; $f_n(x,y)$7, $f_n(x,y)$8, $f_n(x,y)$9	Improvement appears when force regulation is critical

The video-based human–object work also reports that using ground-truth contacts but training only with $\mathbf{f}_t(x,y)$0 yields KP 102.40 px across objects, whereas joint training with predicted contacts achieves 99.38 px, and qualitative examples show opposing arrows on airplane wings during twisting and large early force magnitudes during skillet lifting and rotation (Ehsani et al., 2020). Force-map prediction reports 5,400 synthetic scenes with domain randomization, qualitative transfer to real scenes, and a real-robot example in which the force-map-based lifting direction causes less motion of a surrounding brown object than a straight-up lift (Hanai et al., 2023).

The tactile literature emphasizes calibration stability and interpretability. FingerVision uses 6 objects and 300 contact trials, and the decomposition resembles simulated patterns for pure normal, tangential, and torsional loads; the normal-force case shows higher variance because the monocular camera sees mostly in-plane motion (Zhang et al., 2019). The bubble method evaluates 170 test trajectories across five indenter geometries and reports near real-time operation at 1–2 Hz for $\mathbf{f}_t(x,y)$1, with most runtime spent in convex optimization (Peng et al., 2023).

For solid mechanics, 3D-TSV reports case studies on a Cantilever, Rod, Femur, Bracket, Bearing, and Parts1, with extraction times from 0.4 s to 33.4 s and interactive rendering below 10 ms per frame on an RTX 2070 SUPER. The main qualitative claim is that simultaneous, evenly spaced seeding across principal directions reduces clutter and preserves symmetries better than separate seeding strategies (Wang et al., 2021). In differentiable inverse graphics, synthetic scenarios report PSNR, SSIM, LPIPS, force magnitude error, and direction error, and the causal tri-plane ablation improves force accuracy relative to K-Planes and point forces; the paper further reports real-world re-simulation that closely matches observed motion and deformation (Gao et al., 30 Nov 2025).

6. Limitations, misconceptions, and research directions

Limitations are strongly modality-dependent. Video-only inverse methods face monocular ambiguity, occlusion, initial-pose sensitivity, and force-scale ambiguity; static-image force maps are vulnerable to shadows, occlusion, unseen regions, and uniform-friction assumptions; tactile decomposition has higher variance for normal-force estimation because the monocular camera observes mostly in-plane marker motion; bubble inversion degrades for very light touches and large curvature changes under the linear plane-stress membrane approximation (Ehsani et al., 2020, Hanai et al., 2023, Zhang et al., 2019, Peng et al., 2023). For internal stress visualization, near-degenerate regions where principal stresses approach equality make direction assignment ill-conditioned and can produce ribbon flips; for differentiable inverse graphics, monocular observability, errors in $\mathbf{f}_t(x,y)$2, $\mathbf{f}_t(x,y)$3, and $\mathbf{f}_t(x,y)$4, and long-horizon gradient instability remain central difficulties (Wang et al., 2021, Gao et al., 30 Nov 2025).

Two misconceptions recur. First, part-level force visualization does not necessarily require explicit semantic part segmentation. Mesh segments, semantic keypoint regions, contact patches, Gaussian subsets, and even interior principal stress trajectories can all function as “parts,” and some methods deliberately operate without explicit segmentation so that part-level cues emerge from force concentration itself. Second, visualization does not necessarily imply direct force sensing. Some systems estimate forces only through effect consistency or privileged simulation labels, some recover them from calibrated visuotactile deformation, and the AR teleoperation formulation visualizes controller-generated forces through target displacement rather than true distributed contact tractions (Ehsani et al., 2020, Hanai et al., 2023, Brandt et al., 26 Mar 2026).

A plausible implication is that future work will increasingly combine these strands: explicit part segmentation with voxel or Gaussian force fields, multi-view constraints for video-based force recovery, uncertainty visualization for inverse estimates, and tighter coupling between controller-level causes and contact-level effects. The existing literature already points in that direction through few-shot transfer to novel objects, optional aggregation of scalar force maps over instances or parts, per-part force and torque aggregation on deformable objects, and extensions from end-effector glyphs to surface stress heatmaps (Ehsani et al., 2020, Hanai et al., 2023, Gao et al., 30 Nov 2025, Brandt et al., 26 Mar 2026).