A Deeper Dive into the Irreversibility of PolyProtect: Making Protected Face Templates Harder to Invert

Abstract: This work presents a deeper analysis of the "irreversibility" property of PolyProtect, a biometric template protection method initially proposed for securing face embeddings. PolyProtect transforms embeddings into protected templates via multivariate polynomials, whose coefficients and exponents are distinct for each subject enrolled in the face recognition system. A polynomial is applied to consecutive sets of elements from a given embedding, where the amount of overlap between the sets is a tunable parameter. We begin our irreversibility analysis by demonstrating that PolyProtected templates are easier to invert using a numerical solver based on cosine distance, as opposed to Euclidean distance (used in the earlier PolyProtect work). To make this inversion more difficult, we then propose a "key selection algorithm", which tries to choose "keys" (coefficients and exponents of the PolyProtect polynomial) that enhance the irreversibility of PolyProtected templates, compared to when the keys are purely random. Our experiments show that this algorithm is effective at generating PolyProtected templates that are significantly more difficult to invert, and that it approximately equalises the irreversibility of PolyProtected templates generated using different "overlap" parameters. This allows for better control of the irreversibility versus accuracy trade-off, known to exist across different overlaps. We also show that accuracy in the PolyProtected domain can be affected by the range in which the embedding elements lie, but that this can be improved by normalizing the embeddings prior to applying PolyProtect. This work is reproducible using our open-source code.

• The paper identifies embedding-range sensitivity and shows that L2 normalization restores near-baseline verification accuracy, achieving 0.1–0.3% FNMR at 0.1% FMR on key datasets.
• The paper demonstrates that a cosine-distance BFGS attacker is stronger than the original Euclidean solver, exposing higher inversion risk as overlap increases.
• The paper introduces rejection-based key selection that reduces inversion success from 85.6–99.6% to below 1%, though it increases FNMR and remains untested against broader attack classes.

Context and motivation

Face embeddings produced by modern deep face recognition models are invertible: reconstruction networks can recover approximate face images from them, and soft-biometric attributes (sex, age, race) can be extracted directly from the embedding space. This motivates biometric template protection (BTP), and in particular feature-transformation methods that map embeddings into a protected domain. The paper under review revisits PolyProtect, a handcrafted BTP method that transforms an $n$-dimensional embedding $V$ into a lower-dimensional protected template $P$ by applying subject-specific multivariate polynomials — parameterised by coefficients $C$ and exponents $E$ ("keys") — to overlapping windows of consecutive embedding elements. The work is situated in the PRiMEAiD project on privacy-preserving face identification for humanitarian aid distribution, which motivated the choice of a lightweight, modular, open-source, handcrafted BTP method over learned approaches, biometric cryptosystems, and homomorphic encryption.

The paper makes three contributions: (1) it identifies and fixes a previously unreported accuracy pathology related to the numerical range of embeddings; (2) it strengthens the worst-case attacker model by showing that a cosine-distance-based numerical solver inverts PolyProtected templates more effectively than the Euclidean-distance solver used in the original evaluation; and (3) it proposes a key selection algorithm that substantially improves irreversibility against this stronger attacker and approximately equalises irreversibility across overlap settings.

Experimental set-up

Five state-of-the-art models generating 512-dimensional embeddings were benchmarked on three datasets spanning different acquisition conditions: Multi-PIE (controlled, 337 subjects), SOTERIA (mobile phone selfies, 70 subjects), and iCarB-Face (near-infrared in-car capture, 198 subjects). iResNet100 and EdgeFace emerged as the two most accurate extractors and were carried forward. PolyProtect was applied with $m = 5$ elements per polynomial term — motivated by the Abel–Ruffini theorem, since degree-5 polynomials lack closed-form solutions — with exponents drawn uniformly from $[1, 5]$ and coefficients from non-zero integers in $[-50, 50]$, evaluated at overlaps 0 through 4.

Accuracy: range sensitivity and the normalization fix

A notable finding is that although iResNet100 and EdgeFace embeddings achieve comparable unprotected verification accuracy, iResNet100 templates degrade severely after PolyProtect. The authors attribute this to element range: EdgeFace embeddings lie roughly in $[-0.7, 0.7]$, whereas iResNet100 embeddings span about $[-4, 4]$. After transformation, the protected iResNet100 template range explodes to approximately $V$0 — ten times the protected EdgeFace range — inflating intra-class variance, as confirmed by t-SNE visualisations. The consequence is stark: at 0.1% FMR on Multi-PIE, unnormalized iResNet100 FNMR ranges from 10.7% to 53.3% across overlaps, versus 0.1% unprotected.

The proposed remedy is simple: L2-normalize embeddings before applying PolyProtect. This restores near-baseline accuracy regardless of model — e.g., normalized iResNet100 achieves 0.1–0.3% FNMR at 0.1% FMR on Multi-PIE and SOTERIA across all overlaps, and 0.8–1.8% on iCarB-Face. The implication is practical: PolyProtect is applicable to any embedding model provided normalization precedes the transform, removing what would otherwise be a hard dependency on the feature extractor's output distribution.

Irreversibility: a stronger attacker

Theoretically, the inverse transform remains an underdetermined system (e.g., 103 equations in 512 unknowns at overlap 0, leaving 409 degrees of freedom), so exact inversion is impossible. The empirical question is whether approximate inversion succeeds well enough to impersonate the identity. Under a full-disclosure threat model per ISO/IEC 30136 — and with initial guesses drawn from distributions estimated on the very embeddings under attack, making this attacker even more informed than in prior work — the paper compares the original Levenberg–Marquardt solver minimising Euclidean distance against a BFGS solver minimising cosine distance between the re-protected candidate and the target template.

The result challenges the original attacker definition: because inversion success is judged by cosine distance, recovering only the embedding's direction suffices, and the cosine-based solver consistently produces inverted vectors closer to the true embeddings than the Euclidean solver. The Euclidean solver arguably paints a "truer" picture of irreversibility since it recovers both magnitude and direction, but it is not the right tool given the cosine-based success criterion. The paper's recommendation — that the worst-case attacker should be defined via the cosine solver — is a substantive revision to how PolyProtect's security should be assessed.

As expected, higher overlaps yield easier inversion, confirming the known trade-off: larger overlap improves accuracy but leaks more information about the original embedding.

Key selection algorithm

The core methodological contribution is a rejection-sampling scheme for keys. For each subject's reference template, random $V$1 are drawn; the resulting protected template is attacked with the cosine solver; if inversion succeeds (at a deliberately loose threshold corresponding to 20% FMR, so that failure implies failure at all stricter thresholds), new keys are drawn until inversion fails.

The quantitative effect is dramatic. With random keys, inversion success rates (ISR) reach 85.6–99.6% at 0.1% FMR and up to 98.7% at 0.01% FMR. With key selection, ISR drops below 1% in every configuration and to essentially 0% at 0.01% FMR — improvements of 68.1 to 98.8 percentage points depending on dataset and overlap. Equally important, key selection approximately equalises irreversibility across overlaps 0–3, decoupling the security level from the overlap parameter.

This comes at a cost: recognition accuracy degrades, with FNMR increases of 1.3–14.2 percentage points at 0.1% FMR (e.g., from 0.3% to 8.3% on Multi-PIE at overlap 0). However, the authors argue — convincingly, given the magnitude asymmetry — that the security gain far outweighs the accuracy loss. Because irreversibility is now approximately uniform across overlaps while accuracy still improves monotonically with overlap, the trade-off collapses into a simple prescription: choose the largest sensible overlap (overlap 3 here). Overlap 4 is explicitly discouraged: the algorithm failed to find suitable keys for all templates on Multi-PIE and iCarB-Face, indicating that highly-overlapped templates may be intrinsically difficult to protect with this scheme.

Limitations and open questions

Several caveats deserve emphasis. First, the key selection algorithm is validated only against the specific cosine-based numerical solver used during selection; resistance to other attack classes — learned inversion networks, gradient-free optimisers, or attacks exploiting multiple enrolled templates per subject — is not established. Second, the algorithm's guarantee is per-template and threshold-relative: keys are selected so that one particular solver fails from ten random initialisations, and no formal bound on the suitable-key space is given; the authors themselves flag characterising this space as future work, noting its connection to renewability and unlinkability. Third, the accuracy degradation induced by key selection, while modest relative to the security gain, is non-trivial at strict thresholds (e.g., FNMR rising to 23.2% on iCarB-Face at overlap 0 and 0.01% FMR), and the proposed remedy — adding an accuracy check to the selection loop — remains unrealised. Fourth, results are reported in detail only for iResNet100, with EdgeFace findings asserted to be comparable but relegated to reproducible code rather than shown. Finally, the evaluation covers three relatively small datasets (70–337 subjects); behaviour at operational scale is unverified.

Conclusion

This work materially sharpens the security analysis of PolyProtect. It demonstrates that the original Euclidean-solver-based attacker understates inversion risk, replaces it with a demonstrably stronger cosine-based attacker, and then neutralises that attacker through a key selection algorithm that reduces inversion success rates from near-certain to near-zero while equalising irreversibility across overlap configurations. Combined with the embedding-normalization fix for cross-model accuracy, the paper leaves PolyProtect in a considerably stronger position for deployment. The principal open questions are the size and structure of the suitable-key space, and whether the irreversibility achieved against the simulated numerical solver generalises to broader classes of inversion attacks.