AdjustAR: AI-Driven AR Alignment

• AdjustAR is an AI-driven, in-situ adjustment mechanism that preserves the original authored AR scene by realigning virtual content when physical environments change.
• It employs multimodal large language models to infer 2D corrections that are back-projected using geometric principles to update AR anchor positions.
• The system automatically handles misalignment scenarios with fallback strategies for missing referents while addressing latency and precision challenges.

AdjustAR is an AI-driven, in-situ adjustment mechanism for site-specific outdoor augmented reality in which authors continue to design experiences against static, georeferenced 3D models, while runtime corrections realign virtual elements when the physical environment changes or localization drifts. The system treats the originally authored scene as the canonical representation of author intent, compares that canonical view to the current live view from the same perspective, and uses multimodal LLMs to infer revised placements that are then backprojected into 3D space and applied to the running AR scene (Numan et al., 9 Aug 2025).

1. Problem setting and authored intent

AdjustAR addresses a failure mode of outdoor site-specific AR in which content is authored relative to a static georeferenced 3D model and later deployed through a Visual Positioning System. This pipeline is effective only while the physical site still matches the model and runtime localization remains accurate. The paper identifies three sources of breakdown: physical environments change over time, geospatial models become out of date, and localization drifts. Under those conditions, labels, arrows, characters, and other AR elements become misaligned with their intended real-world referents, degrading interpretability and experience quality (Numan et al., 9 Aug 2025).

A central design decision is to preserve authored intent rather than to re-author the experience or to retarget it to an unrelated layout. The authored scene is therefore treated as canonical: the “correct” spatial relation is whatever relation held in the original authored view. This makes AdjustAR a maintenance layer rather than a replacement for model-based authoring. It operates after deployment, at runtime, and attempts to restore the relation between virtual content and the physical referent that the author originally specified (Numan et al., 9 Aug 2025).

The system is positioned against several prior approaches. Static model-based pipelines, including VPS-centered authoring workflows, assume a stable site model and do not automatically compensate for later environmental change. Semantic and optimization-based layout adaptation systems typically focus on mixed-reality UI layout and usability rather than on preserving a particular site-specific referential relation. Geometry-only snapping methods can align content to nearby planes or edges, but they do not reason about what an element is intended to denote in the authored experience. AdjustAR’s contribution is to insert visual-semantic reasoning directly into the runtime correction loop while keeping the original authored model as the normative reference (Numan et al., 9 Aug 2025).

2. Runtime architecture

The runtime pipeline extends a conventional outdoor AR stack built with the Niantic SDK for Unity and VPS localization. During authoring, AR objects are placed relative to a site-specific georeferenced 3D model and stored as world-space anchors. During execution, the user localizes on site via VPS and sees the scene rendered from those original anchors. AdjustAR adds an “evaluation and adjustment” pass that can be triggered manually or periodically (Numan et al., 9 Aug 2025).

At trigger time, the system captures two views from the same camera pose. The first is the live AR view, consisting of the real camera image plus the currently rendered virtual overlays. The second is a synthetic authored view rendered from that same pose using the original 3D model and the authored anchors. The authored rendering uses unique flat colors to identify individual AR elements in a content-agnostic way. These two images are assembled into a side-by-side composite that becomes the input to the multimodal reasoning stage (Numan et al., 9 Aug 2025).

The MLLM stage is two-tiered. Gemini 2.5 Flash performs a fast pass that checks whether each outlined element is correctly aligned relative to its physical referent and whether the referent is visible in the live image. Gemini 2.5 Pro is then invoked only for elements flagged as misaligned and still visible. For those elements, it returns corrected normalized 2D anchor coordinates in the live image and an optional vertical offset estimate. The geometric stage then uses cached camera intrinsics, extrinsics, and depth at trigger time to backproject the new 2D anchor into a 3D world coordinate and update the anchor in Unity (Numan et al., 9 Aug 2025).

The pipeline includes an explicit fallback for irrecoverable cases. If the model judges that the referent no longer exists or is fully occluded, AdjustAR does not synthesize a new anchor. Instead, it presents a snapshot of the original authored view as a proxy for the intended experience. This preserves access to the content while acknowledging that the original referential relation cannot currently be restored (Numan et al., 9 Aug 2025).

3. Misalignment reasoning and geometric correction

The reasoning stage is organized around the paper’s notion of a “physical referent,” defined in the prompts as the real object or surface to which an AR element was originally attached or which it was intended to indicate. To identify elements without exposing the application’s internal scene graph, the system outlines each one with a unique color selected from green, blue, magenta, red, orange, yellow, or cyan. The models are repeatedly reminded that the left image is the live AR view and the right image is the authored intended view, and they are instructed to return JSON in a fixed schema (Numan et al., 9 Aug 2025).

Gemini 2.5 Flash performs alignment checking by comparing the outlined object’s relation to its referent in the authored image and the corresponding relation in the live image. The output includes the name of the physical referent, the referent’s position relative to the outlined object in the authored image, whether the placement in the live image is correct, and whether the referent is visible in the live image. Gemini 2.5 Pro then classifies each candidate as Correct, Missing, or Misaligned, and, for the last category, predicts normalized image coordinates in the range $[0,1000] \times [0,1000]$ together with a vertical offset in centimeters (Numan et al., 9 Aug 2025).

The geometric stage uses a standard pinhole camera model. For a world point $(X,Y,Z,1)^\top$ projected to image coordinates $(u,v)$ with scale factor $s$,

$$s \begin{bmatrix} u \ v \ 1 \end{bmatrix} = K \begin{bmatrix} R & t \end{bmatrix} \begin{bmatrix} X \ Y \ Z \ 1 \end{bmatrix}.$$

Normalized MLLM outputs are first converted back to pixels via

$$u = \frac{X_{\text{norm}}}{1000}\cdot W,\qquad v = \frac{Y_{\text{norm}}}{1000}\cdot H.$$

Depth is then sampled at that pixel, $Z_c = D(u,v)$, and inverse projection is performed by forming $\tilde{p} = [u,v,1]^\top$, computing $\tilde{r} = K^{-1}\tilde{p}$, and obtaining the camera-space point $[X_c,Y_c,Z_c]^\top = Z_c \cdot \tilde{r}$. World coordinates are recovered through

$(X,Y,Z,1)^\top$0

A vertical offset $(X,Y,Z,1)^\top$1 from the MLLM is converted from centimeters to meters and applied along the world up vector. The resulting point becomes the new anchor, interpreted as the bottom center of the AR element’s bounding box (Numan et al., 9 Aug 2025).

This design couples semantic inference to strict geometric update. The MLLM is not asked to hallucinate 3D directly; it is asked for a 2D correction relative to the observed referent, and the system then enforces a geometric reconstruction from cached pose and depth. A plausible implication is that the method uses the MLLM where semantic ambiguity dominates and conventional projective geometry where metric consistency dominates.

4. Evaluation status, latency, and failure modes

AdjustAR was introduced as a UIST Adjunct system description and design exploration rather than as a benchmark-driven evaluation. The paper explicitly reports no formal quantitative evaluation, no user study numbers, and no statistical tests. Its evidence is qualitative, centered on scenarios in which some elements remain correctly aligned, some are corrected after misalignment, and others are classified as missing and handled through the canonical-view fallback (Numan et al., 9 Aug 2025).

The implementation uses the Niantic SDK for Unity for camera pose, intrinsics, VPS localization, and access to georeferenced 3D models, and relies on Gemini 2.5 Flash and Gemini 2.5 Pro through Google APIs. The current prototype exhibits a correction latency of about 10–20 seconds per correction cycle, including screenshot capture, server upload, MLLM inference, backprojection, and Unity update. The paper identifies reducing this latency and improving accuracy as a central future direction (Numan et al., 9 Aug 2025).

Several limitations are explicit. The current correction assumes static image pairs in which both the referent and the AR content are visible in the same frame. Occluded referents and out-of-frame referents are therefore difficult cases. The system is also exposed to standard MLLM failure modes, including referent misidentification, over-confident but wrong anchor choices, and sensitivity to prompt phrasing. Dense urban scenes with many similar doors, windows, trees, poles, or signs can create ambiguity when the canonical and live views diverge strongly. On the geometric side, the method currently uses a bottom-center anchoring heuristic plus an optional vertical offset, which is not sufficient for all edge-, corner-, or surface-attached content. Backprojection quality is further limited by the depth map: noisy, missing, or aliased depth can corrupt the 3D anchor even when the 2D correction is semantically correct (Numan et al., 9 Aug 2025).

The future work agenda follows directly from these constraints. The paper proposes lower-latency inference, explicit use of multi-view sequences, depth and mesh as richer inputs, spatiotemporal reasoning for occluded and out-of-frame referents, richer anchoring strategies beyond bottom-center plus vertical offset, author-defined semantic constraints, and empirical study of what alignment tolerance is acceptable across application domains (Numan et al., 9 Aug 2025).

5. Relation to other content-adjustment frameworks

AdjustAR belongs to a broader class of AR systems that modify placement or alignment at runtime, but its operating assumptions differ from several established approaches. “Retargetable AR” constructs a 3D scene graph from RGB-D semantic mapping and oriented bounding boxes, then performs subgraph matching so that content authored against an abstract context such as “chair in front of TV” can be semantically registered in varied indoor scenes (Tahara et al., 2020). That framework is relation-centric and environment-agnostic: it seeks a semantically equivalent configuration in the current room. AdjustAR, by contrast, is referent-preserving and site-specific: it tries to restore the originally authored relation to the same physical referent in a changing outdoor site.

“ARfy” represents both the authored virtual scene and the scanned physical environment as point clouds and solves a constrained point-cloud registration problem with a single global transformation $(X,Y,Z,1)^\top$2, where rotation is constrained to yaw and scale is uniform (Caetano et al., 2024). Its strength is annotation-free geometric adaptation of an entire scene to arbitrary physical spaces. AdjustAR does not perform global scene embedding; it performs per-element maintenance against a canonical authored view and uses visual-semantic reasoning to decide which referent each element should still denote.

A third line concerns manual or semi-manual layout control. The HoloLens 2 miniature-room technique based on World-in-Miniature creates and updates a room mesh from depth sensing, generates miniature counterparts of virtual objects, and lets users move and scale those objects by manipulating the miniature rather than the full-scale content. In evaluation, this miniature method significantly reduced workload in physical and temporal demand, although there was no significant difference in the total manipulation time (Ihara et al., 7 Jan 2026). That work addresses ergonomic adjustment by the user. AdjustAR addresses automated runtime correction without requiring the user to manually re-layout the scene.

Taken together, these comparisons clarify that “adjustment” in AR spans at least three distinct regimes: semantic retargeting to a new environment, geometry-driven embedding of a scene into arbitrary space, and runtime repair of authored site-specific referential alignment. AdjustAR occupies the third regime.

6. Adjustment as a multilayer AR design problem

The significance of AdjustAR becomes clearer when placed alongside other layers at which AR systems perform adjustment. One layer is perceptual quality optimization. “TransformAR,” “TransformAR-KD,” and “TransformAR-KD+” model AR image quality from two references—the AR foreground and the real background—using ViT encoders, patch-wise shift representations, cross-attention decoders, and, in the KD variants, feature-level knowledge distillation. The method is designed for monocular visual confusion and classical distortions in superimposed AR imagery, and the paper explicitly frames it as a potential feedback signal for modifying opacity, contrast, compression, resolution, and even placement or style choices in an AR pipeline (Sekhri et al., 2024). In that sense, AdjustAR’s referent-maintenance objective complements rather than replaces quality-aware adaptation.

A second layer is visual transformation of the user’s viewport. “ShadAR” lets a Meta Quest 3 user describe a desired perceptual effect in natural language, transcribes the request with Whisper, uses OpenAI o3-mini to generate HLSL shader code, compiles the shader through Unity in headless batch mode, and applies it to the passthrough feed. The demonstrated targets include accessibility simulation, creative stylization, and search-oriented filtering, with a latency of about 30–45 seconds from voice command to deployment (Mei et al., 19 Feb 2026). This is adjustment of appearance rather than alignment: it changes how the world is seen, not where site-specific content should be anchored.

A third layer is hardware-level personalization. “Prescription AR” integrates a free-form combiner inside the prescription lens itself and customizes the optical design to the user’s prescription, interpupillary distance, eye relief, and frame geometry. The paper reports a 169$(X,Y,Z,1)^\top$3 dynamic prototype with a $(X,Y,Z,1)^\top$4 virtual image, 23 cpd resolution at center field, a $(X,Y,Z,1)^\top$5 eye box, and varifocal capability from 0.5–3$(X,Y,Z,1)^\top$6 (Wu et al., 2019). This is adjustment at the level of optics and visual comfort, again distinct from AdjustAR’s runtime maintenance of semantic anchoring.

These systems suggest that AR adjustment is not a single technique but a stack. AdjustAR operates at the semantic-spatial layer, preserving authored intent in the face of environmental drift. Other work adjusts visibility, perceptual transformation, or hardware fit. A plausible implication is that robust long-lived AR experiences will require coordination across these layers: site-specific anchors that remain semantically correct, rendered content whose contrast and blending remain perceptually legible, and display hardware whose optics remain individualized to the wearer.