Neural-Pull: Differentiable Geometry & Interpretability

• Neural-Pull is a method that employs differentiable pulling operations to reconstruct implicit surfaces by learning signed distance functions from raw point clouds.
• It also extracts interpretable linear decision boundaries from ReLU networks via excitation pullbacks, enhancing the clarity and stability of feature representations.
• Experimental results show improved geometric fidelity, reduced artifacts, and robust network interpretability compared to traditional reconstruction and visualization methods.

Neural-Pull encompasses a set of advanced methodologies in neural representations, distinguished by their use of "pulling" operations either to reconstruct geometric surfaces from point clouds or to extract interpretable features and decision boundaries from deep neural networks. Two principal instantiations are currently established: (a) surface reconstruction via learning Signed Distance Functions (SDFs) from raw point clouds without ground-truth distances, and (b) uncovering aligned representations in ReLU networks through excitation pullbacks, which reveal the underlying linear structure and perceptually aligned boundaries of high-performing classifiers. These methods leverage differentiable pulling or gradient operations for improved fidelity, interpretability, and training efficacy (Ma et al., 2020, Satkiewicz, 30 Jul 2025).

1. Foundational Principle: Pulling as a Supervised Signal

The central concept of Neural-Pull in geometric learning involves reconstructing implicit surfaces using a continuous SDF $f_\theta\colon \mathbb{R}^3\to\mathbb{R}$ such that $f_\theta(x)\approx 0$ for surface points, $f_\theta(x)>0$ outside, and $f_\theta(x)<0$ inside the surface (Ma et al., 2020). Training is performed only with an unoriented point cloud $P=\{p_j\}$, absent explicit SDF labels. The pivotal innovation is a differentiable pulling operation that moves query points $q_i$ toward the surface, guided by both the predicted signed distance and gradient—both outputs of the neural network. The error signal penalizes discrepancies between the pulled query and its true nearest neighbor in $P$, thereby indirectly learning the SDF and its gradient field by supervision solely from spatial proximity.

In contrast, Neural-Pull applied to ReLU networks formalizes and extracts the implicit linear model embedded via input-dependent gating patterns. Here, the "pullback" refers to reconstructing the local decision boundary in input space by propagating the path-space weights through the gating structure, with modifications that enhance stability and interpretability (excitation pullbacks) (Satkiewicz, 30 Jul 2025).

2. Mathematical Formulation of the Pulling Operation

For surface reconstruction, given $q_i\in\mathbb{R}^3$, the neural network predicts: $$s_i = f_\theta(q_i),\qquad g_i = \nabla_x f_\theta(q_i),\qquad \hat g_i = \frac{g_i}{\|g_i\|_2}$$ and the pulled query is: $$\tilde q_i = q_i - s_i \hat g_i = q_i - f_\theta(q_i) \frac{\nabla_x f_\theta(q_i)}{\|\nabla_x f_\theta(q_i)\|_2}$$ For $q_i$ outside ($s_i>0$) or inside ($s_i<0$), this adjusts the point toward the surface. The loss penalizes pulling error: $\mathcal L = \sum_{i=1}^I \|\tilde q_i - t_i\|_2^2$ where $t_i = \arg\min_{p\in P} \|q_i - p\|_2$ (Ma et al., 2020).

In the context of ReLU networks, the logit $f(x)$ is expressed as a linear model in exponentially large path space: $f(x) = \langle \omega,\, \phi_G(x)\rangle$ where $\omega$ are path weights and $\phi_G(x)$ feature activations under the gating pattern induced by $x$ (Satkiewicz, 30 Jul 2025). The excitation pullback modifies the backward pass by introducing a smooth excitation function, $\Gamma_\ell(x)[i] = \sigma(z_\ell(x)[i]/T)$, replacing the hard gate with a temperature-parameterized sigmoid, stabilizing the extracted boundary.

3. Network Architecture and Training Paradigm

The SDF learning model adopts an occupancy/SDF MLP backbone, consisting of eight fully connected layers, 512 hidden units per layer, and ReLU activations. Conditional tasks use separate encoders (e.g., ResNet or SoftRas’s CNN) to obtain latent codes concatenated at each layer input. Gradients are automatically computed (Ma et al., 2020).

Training employs point sampling strategies that focus queries near the surface ($2\times 10^4$ points per shape; 25 Gaussian samples per centroid), mini-batch sizes of $5\times 10^3$, Adam optimization (learning rate $10^{-4}$, 2,500 epochs), and geometric initialization toward a unit sphere. Regularization is not required beyond the pulling loss.

For excitation pullbacks, standard pretrained architectures (ResNet-50, VGG11_BN, DenseNet-121) are used. The backward pass for each ReLU is replaced by a surrogate function, which multiplies the incoming gradient by the excitation value (Satkiewicz, 30 Jul 2025):

class ExciteReLU(torch.autograd.Function): @staticmethod def forward(ctx, z, temp=0.3): ctx.save_for_backward(z) ctx.temp = temp return torch.relu(z) @staticmethod def backward(ctx, grad_out): (z,) = ctx.saved_tensors gate = torch.sigmoid(z / ctx.temp) return grad_out * gate, None

4. Differentiability and Optimization Implications

The pulling map $(q_i,\theta)\mapsto \tilde q_i$ is differentiable almost everywhere due to the $C^1$ MLP implementation. Loss gradients propagate through both $f_\theta$ and $\nabla f_\theta$, updating all parameters toward the SDF optimum (Ma et al., 2020). Similarly, excitation pullbacks in ReLU nets ensure the extracted boundary is continuous with respect to the pre-activations, suppressing erratic flips and yielding stable, interpretable gradients in input space (Satkiewicz, 30 Jul 2025).

This suggests that differentiable pulling not only improves geometry learning but also facilitates principled exploration of latent structures in neural feature spaces, supporting knowledge discovery and improved interpretability in deep learning systems.

5. Experimental Results and Benchmarking

Neural-Pull SDF learning is evaluated on datasets including ABC, FAMOUS, and ShapeNet subsets. Metrics comprise L2-Chamfer distance (L2-CD), L1-CD, Normal Consistency, F-score, and Earth Mover’s Distance (EMD). Mesh extraction utilizes Marching Cubes on the zero-level set. Notable improvements are observed:

• L2-CD on FAMOUS: from $1.0\text{–}1.6\times10^{-2}$ to $0.35\times10^{-2}$
• L2-CD on ABC: $0.48\times10^{-2}$ (vs. much higher baselines)
• ShapeNet: average L2-CD drops from $0.07$ (next best) to $0.038$
• Single-image reconstruction: L1-CD from $0.020$ to $0.011$; EMD from $\sim 1.7\times10^{-2}$ to $1.72\times10^{-2}$

Neural-Pull produces reconstructions with increased sharpness, reduced artifacts, and robustness to noise, outperforming PSR, BPA, ATLAS, DeepSDF, MeshP, NUD, SALD (Ma et al., 2020).

Excitation pullbacks reveal high-resolution, target-specific regions in ImageNet classifiers. Compared to vanilla gradients (which are noisy and often highlight backgrounds), excitation pullbacks produce crisp, class-specific contours (e.g., eyes, wheels, handles). Projected gradient ascent demonstrates that excitation-guided perturbations manifest semantically aligned structures after few steps, with superior perceptual alignment (quantified via edge correlation and user studies) (Satkiewicz, 30 Jul 2025).

6. Relationship to Prior Methods and Advantages

Neural-Pull surface reconstruction does not require ground-truth signed distances, in contrast to DeepSDF [Park et al., 2019]. Prior unsigned distance/occupancy methods (IGR, SAL, NUD) require regularizers or heuristic constraints. Neural-Pull directly learns signed distances from raw points without additional loss terms or sign disambiguation heuristics, resulting in conceptual simplicity and accuracy gains (Ma et al., 2020).

Excitation pullbacks expose the implicit path-space linear model of ReLU networks, distinct from prior gradient-based visualization approaches. By replacing hard gates with excitation functions, they recover more interpretable, stable boundaries and confirm the existence of semantically meaningful, aligned representations (Satkiewicz, 30 Jul 2025).

A plausible implication is that both instantiations of Neural-Pull advance the state of the art not only in their respective domains (geometry processing, neural interpretability), but also as exemplars of leveraging differentiable pulling for supervised and unsupervised signal extraction.

7. Broader Significance and Applications

In surface reconstruction, Neural-Pull enables flexible, accurate learning of continuous implicit surfaces from minimal unlabeled data, benefiting 3D computer vision, graphics, and robotics.

In neural interpretation, excitation pullbacks suggest that dependable AI systems can be constructed by exposing the true local linearization of classifier boundaries, improving trust, out-of-distribution monitoring, and potentially regularizing decision surfaces via excitation gating.

Together, these paradigms indicate a wider utility for differentiable, gradient-driven pulling operations in interpretable, robust, and high-fidelity neural learning (Ma et al., 2020, Satkiewicz, 30 Jul 2025).