The Geometry of Last-Layer Model Stealing

Abstract: This paper uses geometry to explain how a machine learning model can be stolen using an already existing well-known method. The author has shown the exact conditions required to perfectly copy the final layer of a transformer network. When looking deeper into the hidden layers the author has explained clear limits. The author has also demonstrated that a hidden network cannot be fully reverse engineered just by looking at the final results. The research clearly maps out what can and cannot be stolen from a model.

• The paper establishes that last-layer model stealing is recast as the recovery of the output manifold’s polar space under linear and quadratic constraints.
• It employs exterior differential systems to characterize spectral rank gaps and normalization-induced quadrics, leading to closed-form attack success when regularity conditions hold.
• The analysis further delineates identifiability limits below the last layer, showing that sublayer parameters remain non-identifiable despite full hidden-state recovery.

Geometric Characterization and Identifiability Limits in Last-Layer Model Stealing

Geometric Analysis of Last-Layer Model Stealing

This work provides a rigorous geometric formalization of the last-layer model-stealing attacks, specifically those elucidated by Carlini et al. (Carlini et al., 2024), using the formalism of exterior differential systems (EDS) and their extensions to Lie algebroids. The main technical result asserts that the output logits generated by a transformer under typical architectural constraints constitute the integral variety of an ideal in the space of logit vectors, governed by a combination of linear and quadratic constraints. These constraints emerge as a linear subspace, determined by the width $h$ of the hidden state, and a quadratic form imposed by the normalization layer projecting the hidden states onto a sphere and, post-projection, onto an ellipsoid in logit space.

Crucially, the last-layer extraction attack is reinterpreted as the recovery of the polar space of the output manifold, revealing that exact attack success hinges on two regularity conditions: (1) a robust rank gap in the logit singular spectrum, and (2) a nondegeneracy criterion for the normalization-induced quadric. When these conditions hold, recovery is information-theoretically optimal, up to an inherent orthogonal gauge corresponding to rotations within the hidden space.

Figure 1: The logit singular spectrum in a toy model, with a steep fourteen-order magnitude drop at index $h=64$, indicating precise recovery of the hidden dimension.

The EDS language is instrumental as an organizational tool; the underlying mathematical content relies primarily on established linear-algebraic and manifold-analytic machinery. The framework unifies the phenomena of rank detection, ellipsoid alignment, and the identification of irreducible orthogonal ambiguities. The system is Frobenius-integrable due to the ideal being generated by closed 1-forms, providing a conceptual explanation for why the attack is closed-form rather than iterative.

Regularity Analysis and Robustness

The analysis delineates the precise conditions under which the attack is robust or fragile. Spectral regularity and quadric nondegeneracy are both necessary and sufficient for stable recovery. Spectral rank estimation is extremely robust even under significant logit noise—rank $\hat{h}$ is reliably recovered for unit-scale noise—while projection recovery (alignment of $W$ beyond the linear subspace) is highly sensitive, with root-mean-square error growing linearly with noise amplitude.

The formalization here provides a concrete characterization of failure modes: if activations span a subspace of dimension $k<h$, the attack recovers only the effective, not nominal, hidden width. This is empirically demonstrated in anomalies such as those occurring in GPT-2-Small models.

Figure 2: Left: Spectral rank gap degrades as $1/\sigma$ under growing logit noise but remains above one, ensuring robust rank recovery; projection error grows linearly. Right: The linear span overstates sublayer content, but intrinsic manifold dimension exposes the true low-rank nonlinear bottleneck.

Identifiability Below the Last Layer

The study extends beyond the last linear layer, rigorously analyzing the identifiability of the penultimate network block. After reconstructing the hidden-state manifold up to a global rotation, the intrinsic dimension is introduced as an observable that is invariant under the symmetries unaddressed by SVD. Notably, for a nonlinear sublayer (e.g., an MLP over a $k$-dimensional content space), the actual information content is much lower than indicated by the linear span: the intrinsic dimension estimator recovers $k$, even if the ambient span is significantly larger.

Beneath the last layer, parameter non-identifiability becomes dominant. The sublayer is only determined up to its input-output behavior (within the accessible input support) and subject to a large fiber of irrecoverable parameters, including:

• Full freedom in $W_1$ on the orthogonal complement of input support
• Arbitrary expansion of MLP width without altering outputs
• Freedom over input reparametrizations and support

These are not theoretical artifacts but are substantiated with floating-point exactness on toy models, where up to 87.5% of weights can be hidden within non-identifiable components with no observable effect on output.

Implications and Future Research Directions

This geometric framework does not confer new attack capabilities but provides clarifying insight into the recovery mechanism and its boundaries. It offers a sharp delineation: the attack's limit is an identifiability wall, not algorithmic insufficiency. Sub-layer parameters are, by construction, not extractable from API access to logits alone. Accordingly, any proposed attack aiming to recover parameters below the last layer must directly contend with the high-dimensional non-identifiable fiber described here.

In practice, this establishes a rigorous boundary for model extraction attacks and identifies intrinsic dimension measurements as potential probes for detecting hidden network bottlenecks invisible to classical linear-algebraic analysis. Future research should aim at formal impossibility theorems quantifying recovery limits in multi-layer, nonlinear, high-rank architectures and establishing exact conditions for partial identifiability.

Conclusion

The paper delivers an authoritative, unified geometric characterization of last-layer model stealing. The attack succeeds up to an orthogonal gauge precisely under spectral and quadric regularity, and this is not further improvable using only logits. Below the last linear layer, parameter recovery is obstructed by intrinsic non-identifiability, with the observable reduced to the intrinsic manifold geometry of hidden states rather than network parameters. This geometric perspective organizes previous observations and sharpens identifiability questions for future theoretical and empirical research.