Camera Pose Auto-Encoders (PAEs)

• Camera Pose Auto-Encoders are models that encode camera pose into structured latent variables to improve tasks such as absolute pose regression and view synthesis.
• Teacher–student and conditional variational approaches enable these models to refine pose estimates with measurable accuracy improvements and efficient latent sampling.
• Self-supervised and radiance-field variations leverage latent encoding for effective image reconstruction, multi-modal pose hypotheses, and object-centric localization.

Camera Pose Auto-Encoders (PAEs) are auto-encoding formulations in which camera pose, or a latent variable that is structurally tied to camera pose, is encoded into a compact representation and then decoded either back to pose parameters or into pose-conditioned image evidence. In the explicit terminology of “Camera Pose Auto-Encoders for Improving Pose Regression,” PAEs are multilayer perceptrons trained via a Teacher-Student approach to encode camera poses using Absolute Pose Regression (APR) networks as teachers (Shavit et al., 2022). In a broader conceptual sense, the literature also contains conditional variational, view-synthesis, video, and radiance-field formulations that operate as PAEs because they encode pose or pose ambiguity into latent variables and decode them through pose regression, image reconstruction, or pose-conditioned rendering (Zangeneh et al., 2024).

1. Conceptual scope and defining characteristics

The core invariant across PAE formulations is the use of an auto-encoding factorization in which camera pose is not merely an output target but an explicit latent object of learning. Depending on the formulation, the encoded variable may be a pose vector $p = (x,q)$ with position $x \in \mathbb{R}^3$ and orientation $q \in S^3$, a rigid transform $y = [R \mid t] \in SE(3)$ with $R \in SO(3)$ and $t \in \mathbb{R}^3$, or a trajectory $\{P_t\}$ of relative poses. The decoder side may be an APR head, a conditional generator of poses, a renderer of images, or a volumetric/radiance-field decoder. What makes these models “pose auto-encoders” is that the latent code is constrained to carry pose-relevant structure rather than unconstrained appearance statistics alone (Shavit et al., 2022).

A useful distinction in the literature is between deterministic and probabilistic PAEs. Deterministic variants encode a pose into a latent that reproduces teacher APR latents or supports view synthesis. Probabilistic variants instead model a distribution over possible poses conditioned on an image, so that the latent variable organizes camera-pose ambiguities and the decoder generates samples from a multi-modal posterior. In the conditional variational formulation for relocalization, the latent $z$ is explicitly described as summarizing camera-pose ambiguities while the decoder, conditioned on the observed image $x$, generates poses $y \in SE(3)$ (Zangeneh et al., 2024).

The term also spans different supervision regimes. Some PAEs are fully supervised by posed images and APR teachers; others are self-supervised from video by assuming that scene structure is static within a short clip and that only camera pose changes. Still others are object-centric rather than scene-centric, inferring object pose relative to the camera and thereby becoming relevant to camera pose when a stable world-referenced object is available (Lai et al., 2021).

Formulation	Encoded variable	Decoder target
Teacher–student APR PAE	Pose-to-teacher latent	APR pose branches or train-image reconstruction
Conditional variational PAE	Pose ambiguity latent $x \in \mathbb{R}^3$0	Pose samples in $x \in \mathbb{R}^3$1 conditioned on image
View-synthesis/video PAE	Pose latent or trajectory	Image/video reconstruction by rendering
Pose-conditioned radiance-field PAE	Camera pose with content latents	NeRF-based image rendering

2. Teacher–student PAEs for absolute pose regression

The explicit PAE construction introduced for APR improvement uses a student network that maps a camera pose to latent vectors whose dimensionality matches the latent output of an APR teacher, with separate latent vectors for position and orientation. The student is a 4-layer MLP, and Fourier Features are applied to the input with $x \in \mathbb{R}^3$2 levels. The supplementary ablation on KingsCollege with a PoseNet+MobileNet teacher reports 1.26 m and 3.54 deg for a 4-layer MLP without Fourier Features, 1.36 m and 3.27 deg for $x \in \mathbb{R}^3$3, and 1.15 m and 3.58 deg for $x \in \mathbb{R}^3$4, with the $x \in \mathbb{R}^3$5 configuration selected due to the lowest position error (Shavit et al., 2022).

The operational purpose of these PAEs is not only latent imitation. They are used for a light-weight test-time optimization in which nearby training poses are fetched in PAE encoding space and combined affinely to refine camera position. The refinement is of the form

$x \in \mathbb{R}^3$6

where the supplement states that neighbors are fetched based on PAE encoding, but does not specify the exact distance metric, the value of $x \in \mathbb{R}^3$7, or the distance-to-weight mapping. The same affine-combination idea was evaluated for orientation, but it degrades accuracy, so the effective refinement is position-only (Shavit et al., 2022).

The empirical effect is substantial on the reported benchmarks. For single-scene APRs on KingsCollege, PoseNet+MobileNet improves from 1.24 m to 0.91 m, PoseNet+ResNet50 from 1.56 m to 1.31 m, and PoseNet+EfficientNet from 0.88 m to 0.81 m. For MS-Transformer, the average median position error improves from 1.28 m to 0.96 m on Cambridge Landmarks and from 0.18 m to 0.15 m on 7Scenes, while the orientation medians remain 2.73° and 7.28°, respectively. The supplement characterizes this as a new state-of-the-art position accuracy for APR on those benchmarks (Shavit et al., 2022).

A second contribution is image reconstruction from pose encoding. The authors train a decoder to reconstruct training images from the learned pose encodings using 64×64 crops, with similar position accuracy and image quality observed at 256 px resolution. The supplement does not specify the decoder architecture or the exact reconstruction loss, but the reported result is used to argue that train-set visual information can be carried through compact pose codes rather than full-resolution images or large retrieval descriptors. The same source gives an order-of-magnitude comparison in which retrieval/relative-pose approaches require gigabytes, whereas single-scene APR, multi-scene APR, and multi-scene APR augmented with PAEs require megabytes (Shavit et al., 2022).

3. Conditional variational PAEs and probabilistic relocalization

A probabilistic PAE formulation appears in the conditional variational relocalization model “Conditional Variational Autoencoders for Probabilistic Pose Regression,” which can be viewed as a Camera Pose Auto-Encoder even though the paper does not use the term explicitly. Here the image is $x \in \mathbb{R}^3$8, the camera pose is $x \in \mathbb{R}^3$9 with $q \in S^3$0, and the latent is $q \in S^3$1. The decoder is a conditional generator $q \in S^3$2 implemented by a neural network $q \in S^3$3, while the encoder is instantiated as $q \in S^3$4. The prior is chosen as an unconditional standard normal, so inference uses

$q \in S^3$5

The resulting ELBO is

$q \in S^3$6

and training minimizes a negative ELBO with a $q \in S^3$7-weighted KL term and a pose reconstruction loss $q \in S^3$8 (Zangeneh et al., 2024).

This formulation is designed for ambiguous relocalization, particularly in environments with repetitive structures such as corridors, stair landings, and ceiling panels. The latent space is meant to organize pose modes so that sampling $q \in S^3$9 at test time yields multiple plausible hypotheses for a single image. The method does not fit an explicit parametric mixture such as a Gaussian–Bingham mixture; instead, uncertainty is represented nonparametrically through samples. The recommended procedure is to draw $y = [R \mid t] \in SE(3)$0 latent samples, typically $y = [R \mid t] \in SE(3)$1, decode $y = [R \mid t] \in SE(3)$2, and treat the sample set as an approximation to $y = [R \mid t] \in SE(3)$3 (Zangeneh et al., 2024).

Pose outputs are parameterized continuously. Translation is a 3-vector, and rotation is produced in the 6D continuous representation of Zhou et al. and then projected to $y = [R \mid t] \in SE(3)$4 by orthonormalization. The pose-aligned reconstruction loss is

$y = [R \mid t] \in SE(3)$5

which corresponds to a Gaussian-like likelihood in those metrics. Architecturally, the decoder uses a ResNet-18 backbone with the last layer set to identity to produce a 512-D image feature, fuses image and latent features after projecting both to 64-D, and predicts a 9-D pose output with a five-layer MLP of width 128. The encoder is a 5-layer MLP of width 128, the latent dimension is $y = [R \mid t] \in SE(3)$6, batch size is 16, optimizer is Adam with learning rate $y = [R \mid t] \in SE(3)$7 and decoupled weight decay $y = [R \mid t] \in SE(3)$8, and $y = [R \mid t] \in SE(3)$9 is annealed with warm-up for 4000 iterations starting after 1000 (Zangeneh et al., 2024).

The reported behavior is qualitatively different from deterministic regressors and from probabilistic regressors that require preset mixture cardinality or winners-take-all fractions. In scenes with repetitive patterns, the model yields multiple concentrated pose clusters that include the ground-truth mode and visually indistinguishable alternatives. The data block further notes that NeRF augmentation strengthens generalization and posterior stability in short ambiguous sequences, while the main limitations are the usual APR gap to structure-based methods, possible failure on out-of-distribution images, and sampling cost that grows proportionally to $R \in SO(3)$0, even though per-sample inference is fast and parallelizable (Zangeneh et al., 2024).

4. View-synthesis PAEs and self-supervised structure–motion disentanglement

A distinct PAE lineage treats camera pose as the latent control variable of image formation rather than as the direct target of a pose decoder. In “Learning Neural Representation of Camera Pose with Matrix Representation of Pose Shift via View Synthesis,” each degree of freedom $R \in SO(3)$1 is encoded as a $R \in SO(3)$2-dimensional unit-norm sub-vector $R \in SO(3)$3, and local camera movement is represented by a skew-symmetric generator $R \in SO(3)$4 acting on that sub-vector. Finite motion is modeled by the exponential map

$R \in SO(3)$5

with the full pose vector formed by concatenating the six sub-vectors. A scene latent $R \in SO(3)$6 and the pose latent $R \in SO(3)$7 are decoded through a deconvolutional renderer $R \in SO(3)$8. Training combines an image reconstruction term with rotation-consistency losses enforcing the Lie-algebraic movement model. On 7 Scenes with a PoseNet backbone, the learned representation averages 0.21 m and 7.92° versus 0.23 m and 8.12° for PoseNet-17 with quaternions, while on GQN rooms, Gibson rooms, and ShapeNet car it is reported to be more robust to pose noise and more effective in camera pose regression than several hand-designed alternatives (Zhu et al., 2021).

This Lie-structured perspective is important because it makes motion composition consistent by construction in latent space. The paper does not learn $R \in SO(3)$9 and $t \in \mathbb{R}^3$0 directly; instead, it learns a neural representation in which physical movement acts through orthogonal matrices in the latent space. The stated motivation is that classical parameterizations can be non-injective, discontinuous, over-parameterized, or difficult to constrain during neural optimization. A plausible implication is that PAEs can be understood not only as compression devices for pose, but also as learned coordinate systems in which group structure is easier to regularize than in raw Euclidean or quaternion coordinates (Zhu et al., 2021).

Self-supervised video autoencoding extends this view from single images to trajectories. “Video Autoencoder: self-supervised disentanglement of static 3D structure and motion” encodes a short clip into a temporally-consistent deep voxel feature volume $t \in \mathbb{R}^3$1 and a per-frame camera pose trajectory $t \in \mathbb{R}^3$2, each $t \in \mathbb{R}^3$3 being relative to the first frame. The first frame alone produces $t \in \mathbb{R}^3$4, while a trajectory encoder predicts $t \in \mathbb{R}^3$5 for each frame pair. The decoder rigidly warps the voxel volume by the predicted pose, refines it with 3D convolutions, reshapes it to 2D features, and renders $t \in \mathbb{R}^3$6. Training uses a per-frame reconstruction term with $t \in \mathbb{R}^3$7 and $t \in \mathbb{R}^3$8, a WGAN-GP adversarial term with $t \in \mathbb{R}^3$9, and a 3D structure-consistency loss with weight $\{P_t\}$0 (Lai et al., 2021).

The model is self-supervised because it requires no ground-truth 3D or camera pose annotations and no camera intrinsics. On RealEstate10K, after Umeyama alignment, it reports Absolute Trajectory Error over 1000 test clips with Mean 0.017, RMSE 0.019, Max err. 0.041, Density 100%, outperforming the listed self-supervised baselines and also COLMAP under that setup. For novel view synthesis it reports PSNR 23.21, SSIM 0.73, and LPIPS 1.54 on RealEstate10K, along with competitive or robust results on Matterport3D and Replica. The main limitations are large viewpoint changes, the static-scene assumption, scale ambiguity up to a similarity transform, dynamic-scene failure modes, and the memory/computation cost of 3D volumes (Lai et al., 2021).

5. Pose-conditioned radiance fields and object-centric extensions

PAEs also appear in the radiance-field regime, where the decoder is an implicit volumetric renderer rather than an APR head or convolutional image decoder. AE-NeRF formulates an auto-encoder in which an encoder infers appearance code $\{P_t\}$1, shape code $\{P_t\}$2, and camera pose $\{P_t\}$3 from a single image,

$\{P_t\}$4

and a pose-conditioned NeRF predicts density and color,

$\{P_t\}$5

for rendering. The camera pose head predicts rotation in the 6D continuous representation, converted to $\{P_t\}$6, together with scale $\{P_t\}$7 and translation $\{P_t\}$8 following GRAF’s camera model. The training scheme is explicitly stage-wise: decoder pre-training with adversarial patch rendering, encoder pre-training on pseudo pairs from the fixed decoder, and end-to-end fine-tuning with reconstruction, perceptual, GAN, Global-Local Attribute Consistency (GLAC), and Swapped-Attribute Classification (SwAC) losses. On CARLA, reconstruction FID is 42.92 for AE-NeRF versus 58.31 for SA, 128.82 for CodeNeRF, and 76.63 for EditNeRF; on camera-swapped manipulation AE-NeRF reports FID 42.29 (Kim et al., 2022).

Although AE-NeRF is object-centric rather than scene-relocalization-oriented, it provides a direct example of a PAE whose latent space explicitly contains camera pose alongside other disentangled 3D attributes. Its significance for the PAE literature lies in showing that pose encoding can be coupled to high-quality volumetric rendering without test-time optimization, while stage-wise training and attribute-swapping losses materially improve disentanglement (Kim et al., 2022).

CVAM-Pose pushes the conditional variational paradigm toward multi-object monocular pose estimation. It estimates an object pose $\{P_t\}$9 relative to the camera rather than camera pose directly, but the data block makes the relation explicit: if the world frame coincides with an object frame, or if an object pose in the world is known, then the camera pose can be inferred by inverting and composing the estimated object pose. The method uses a label-embedded CVAE with an adapted ResNet-18 encoder, SiLU activations, a latent dimensionality ablated over $z$0 and best at $z$1, and three label-embedded MLP heads that regress a 6D continuous rotation representation, a projective center $z$2, and a projective distance $z$3. Translation is recovered by the pinhole equations

$z$4

On Linemod-Occluded under the BOP protocol, CVAM-Pose reports $z$5, $z$6, $z$7, and $z$8, compared with 0.090/0.095/0.254/0.146 for AAE and 0.150/0.153/0.346/0.217 for Multi-Path. The same source states that it is “25% and 20% better than AAE and Multi-Path” on $z$9 and that the single latent space works well up to about a dozen objects with $x$0 before accuracy deteriorates beyond roughly 15 objects if capacity is not increased (Zhao et al., 2024).

6. Relative pose regression, deployment trade-offs, and limitations

The teacher–student PAE framework was extended to Relative Pose Regression (RPR) in “Relative Pose Regression with Pose Auto-Encoders: Enhancing Accuracy and Data Efficiency for Retail Applications.” In this formulation, a camera pose is represented by 3D position $x$1 and orientation $x$2, and a PAE $x$3 encodes $x$4 into high-dimensional latent representations $x$5. An APR network $x$6 acts as both teacher and decoder: it supplies latent targets from images and its regressor layers decode PAE outputs back to pose. The paper states that PAEs are implemented with sine functions and MLPs, and that their outputs are trained to capture sufficient geometric and visual information to allow accurate decoding by the APR teacher (Shavit et al., 12 Aug 2025).

The proposed PAE-based RPR replaces the reference image in a Siamese RPR pipeline with a PAE encoding of the reference pose. The baseline image-based and PAE-based RPRs both use ResNet34, ResNet50, or EfficientNet-B0 backbones; in the compared architectures, two FC layers produce 256-dimensional latent representations for translation and rotation, and each translation/rotation head is a three-layer MLP with ReLU. On 7Scenes, the reported median errors are very close: with ResNet50, image-based RPR gives 0.22 m / 9.75° and PAE-based RPR gives 0.23 m / 9.45°; with EfficientNet-B0, image-based RPR gives 0.21 m / 9.36° and PAE-based RPR gives 0.22 m / 8.77°. The paper summarizes this as similar localization accuracy, slightly weaker translation, and favorably lower orientation error for the PAE-based variant (Shavit et al., 12 Aug 2025).

The same work uses a Transformer PAE-based RPR to refine APR estimates without storing reference images or pose databases at inference. The refinement pipeline is: first, a pre-trained APR predicts an initial pose $x$7 from the query image; second, the PAE-based RPR estimates a refinement motion $x$8 from the query image and the current pose estimate; third, the pose is updated; the process can be iterated. The paper does not provide an explicit $x$9 composition formula for this update. With MS-Transformer on 7Scenes, the mean of medians improves from 0.18 m / 7.28° to 0.17 m / 6.69°, corresponding to relative improvements of 5.5% in position and 8.0% in orientation. Under subset training, refinement improves the same APR teacher trained on 70%, 50%, and 30% of the data, with the 30% case moving from 0.20 m / 8.16° to 0.19 m / 7.62°. Each refinement iteration takes approximately 37 ms and the PAE-based RPR model size is 86 MB (Shavit et al., 12 Aug 2025).

Across the literature, several limitations recur. In teacher-distilled PAEs, generalization to unseen scenes is inherited from the APR teacher, and the RPR extension notes this explicitly. In probabilistic CVAE PAEs, absolute pose regression remains less accurate than structure-based methods in general, out-of-distribution images can cause failure, and generating large posterior sets increases compute proportionally to the number of latent samples. In self-supervised video PAEs, the static-scene assumption, similarity-transform ambiguity, dynamic objects, and large viewpoint changes remain important failure modes. A plausible synthesis is that PAEs consistently improve the accuracy–memory–runtime trade-off of learned localization, but they do not remove the dependence of monocular pose inference on scene coverage, viewpoint distribution, and decoder inductive bias (Shavit et al., 12 Aug 2025).

The literature therefore positions PAEs less as a single architecture than as a family of pose-centric latent-variable models. Their explicit forms range from APR-compatible MLPs that distill teacher pose latents, to CVAEs that sample multi-modal posteriors in ambiguous environments, to view-synthesis and video autoencoders that discover pose by enforcing rendering consistency, to NeRF-based encoders that disentangle pose from shape and appearance. The common contribution is to make pose a structured latent quantity that can be compressed, regularized, sampled, or rendered from, thereby turning camera localization from direct regression into a broader auto-encoding problem over geometry, ambiguity, and scene priors.