Explain asymmetric attribute behavior during LFA latent-direction traversals

Determine the underlying mechanism causing the observed asymmetric and sometimes non-monotonic attribute changes when traversing ArcFace face recognition embeddings along latent directions discovered by Latent Feature Alignment and decoding with Arc2Face (e.g., the Young attribute increasing under both positive and negative traversals and the Female attribute remaining flat under negative traversal). Ascertain whether these effects arise from non-linear structure of the embedding manifold, underfitting of the face recognition embeddings, or entanglement among demographic and appearance attributes.

Background

The paper introduces Latent Feature Alignment (LFA), an attribute-label-free algorithm that discovers semantically coherent groups in face recognition embeddings by aligning samples along latent directions. The authors evaluate interpretability by traversing embeddings along discovered directions and decoding images using Arc2Face to visualize semantic changes.

Quantitative analysis with an attribute classifier shows that relevant attributes often change monotonically with traversal strength. However, the authors report exceptions in which attributes behave asymmetrically or non-monotonically, such as Young increasing in both traversal directions and Female not decreasing under negative traversal. They explicitly state that they do not yet have a definitive explanation for these effects and hypothesize potential causes including non-linear manifold structure, underfitting, or attribute entanglement.