Online Geometry Control Methods

• Online Geometry Control is a set of methods that integrate real-time geometric priors, states, or constraints directly into decision processes.
• These approaches employ dual-stream feature fusion, semantic distance fields, and 3D bounding box prompts to improve spatial understanding and control precision.
• Key trade-offs include balancing geometric fidelity, computational latency, and complexity to support applications from streaming 3D vision to aggressive UAV maneuvers.

Searching arXiv for papers on online geometry control and closely related geometry-aware online control frameworks. Online geometry control can be understood as a family of methods in which geometric state, priors, or constraints are injected directly into an online inference or control loop, rather than being deferred to offline reconstruction or downstream post-processing. In recent work, this paradigm appears in streaming 3D vision-language modeling, semantic-aware safe navigation from monocular video, 3D-aware image editing with online reinforcement learning, and singularity-free aggressive UAV control on $SO(3)$ (Yu et al., 5 Jun 2026, Zhang et al., 1 Jun 2026, Wang et al., 22 Jun 2026, Yao, 2023). A related but boundary-case direction is local geometry-controllable CAD generation, where geometric instructions govern the regeneration of a masked local part, although the setting is not streaming in the same sense (Zhang et al., 12 Jun 2025).

1. Scope and problem setting

The common feature across these systems is that geometry is not treated as a static annotation. Instead, it enters the decision process at runtime: a LLM decides whether to continue ingesting video or begin responding; a safety filter queries a semantically inflated distance field before solving a QP; a diffusion model conditions on projected 3D box prompts and is further adapted through online RL; and a quadcopter controller evaluates a near-optimal state-feedback law while representing attitude directly on $SO(3)$ (Yu et al., 5 Jun 2026, Zhang et al., 1 Jun 2026, Wang et al., 22 Jun 2026, Yao, 2023).

The term “online” has domain-specific meanings in these papers. In Stream3D-VLM it denotes real-time spatial understanding from streaming video, with decision tokens that gate whether more frames are consumed or text generation starts (Yu et al., 5 Jun 2026). In the monocular safe-control framework it denotes a perception-to-control loop running at $10$--$20$ Hz from RGB-only input (Zhang et al., 1 Jun 2026). In BoxCtrl it refers to an online RL stage on unpaired real-world data, where each edit is treated as an RL rollout (Wang et al., 22 Jun 2026). In the quadcopter work it denotes per-step evaluation of a closed-form near-optimal control law with online computational efficiency relative to finite-time SDRE (Yao, 2023). GeoCAD, by contrast, performs local loop editing from a prompt and masked part; this is geometry-controllable inference, but not streaming control in the same sense (Zhang et al., 12 Jun 2025).

Work	Online setting	Geometric mechanism
Stream3D-VLM	Streaming video	VSFI, GAVC, decision tokens
Semantic-aware safe control	Monocular closed loop	semantic-aware ESDF and CBF QP
BoxCtrl	Online RL adaptation	RGB 3D bounding-box prompts
Quadcopter flip control	Per-step feedback control	$SO(3)$ error and $\theta$–D law
GeoCAD	Interactive local editing	geometry instruction for masked loop

A plausible implication is that online geometry control is less a single algorithmic family than a systems pattern: geometry is represented explicitly, queried repeatedly, and made causally effective in runtime decisions.

2. Geometric representations and how they are injected

A central design choice is the representation by which geometry enters the online loop. Stream3D-VLM uses a dual-stream representation: a 2D vision encoder produces visual patch tokens $H^{2D}_t \in \mathbb{R}^{N \times d^{vis}}$, while a streaming 3D reconstructor produces latent geometry tokens $G_t \in \mathbb{R}^{K \times d^{geo}}$ and a camera token $c_t$. The Visual–Spatial Feature Integration (VSFI) module first projects geometry,

$$H^{3D}_t = \mathrm{MLP}([c_t; G_t]) \in \mathbb{R}^{(K+1)\times d^{vis}},$$

then fuses it into the visual stream by cross-attention, treating $SO(3)$0 as queries and $SO(3)$1 as keys and values, yielding geometry-enhanced visual tokens $SO(3)$2. The Geometry-Adaptive Voxel Compression (GAVC) module then back-projects each patch into 3D, adds positional encoding, clusters by spatial K-Means, and aggregates tokens with a dual-attention weighting that mixes feature similarity and positional proximity (Yu et al., 5 Jun 2026).

The monocular safe-control framework represents geometry as a dense point map from MASt³R-SLAM-style reconstruction, then integrates it into a TSDF and finally an ESDF. Semantic labels are fused into the same map by a positive/negative vote rule. Class-dependent inflation is then applied before ESDF computation:

$SO(3)$3

The resulting $SO(3)$4 and $SO(3)$5 retain efficient ESDF queries while making high-risk objects exert a larger spatial influence in the safety field (Zhang et al., 1 Jun 2026).

BoxCtrl uses a different representational strategy. A 3D bounding box $SO(3)$6 is defined by eight corners $SO(3)$7 together with $SO(3)$8 and $SO(3)$9, and its three orthogonal faces are painted red, green, and blue to encode orientation. Projection is performed through

$10$0

The rendered source and target boxes are stacked with the source image as visual prompts. Translation is encoded by the pixel-space shift of the box center, scale by projected box size, and rotation by the visible RGB face pattern (Wang et al., 22 Jun 2026).

In the quadcopter setting, geometry enters through the state manifold itself. Rather than defining attitude error in Euler angles, the controller uses a coordinate-free error on $10$1:

$10$2

with corresponding vector error

$10$3

Because $10$4 is used directly, the formulation avoids the singularity issue during a full $10$5 flip (Yao, 2023).

GeoCAD represents geometry textually. A CAD model in FlexCAD is written as a sequence of loops $10$6; a local part $10$7 is masked, the remaining loops $10$8 are concatenated with a geometric instruction $10$9, and the model predicts the masked loop autoregressively. Here the controlled geometric object is a local closed 2D path rather than a streaming 3D scene (Zhang et al., 12 Jun 2025).

3. Objectives, constraints, and control laws

Online geometry control methods differ sharply in objective form, but all make geometry operative inside the optimization.

Stream3D-VLM formulates streaming control as part of next-token prediction. Two disjoint token sets are defined: $20$0 for decision tokens $20$1 and $20$2 for all other tokens. The joint loss is

$20$3

with

$20$4

During inference, $20$5 means “keep listening” and $20$6 means “trigger response” (Yu et al., 5 Jun 2026).

The semantic-aware safe-control framework uses a Control Barrier Function defined from the semantically inflated ESDF:

$20$7

with safe set $20$8. Let $20$9 be the semantic class of the nearest obstacle. The class-dependent CBF condition is

$SO(3)$0

and the runtime safety filter solves

$SO(3)$1

This directly couples geometric clearance, semantic risk, and control admissibility (Zhang et al., 1 Jun 2026).

BoxCtrl uses a two-stage objective. In supervised fine-tuning, it optimizes a Flow Matching loss on the diffusion velocity field:

$SO(3)$2

The online RL stage then maximizes a joint reward with translation/scaling, rotation, and visual-similarity components,

$SO(3)$3

$SO(3)$4

$SO(3)$5

combined as

$SO(3)$6

The RL objective is a weighted flow-matching loss using “push” and “pull” policies (Wang et al., 22 Jun 2026).

The quadcopter paper poses a finite-horizon nonlinear optimal control problem with cost

$SO(3)$7

then derives the near-optimal $SO(3)$8–D feedback law

$SO(3)$9

The distinction from finite-time SDRE is that the online stage requires matrix multiplies and vectorizations, but no real-time Riccati solve (Yao, 2023).

GeoCAD trains an autoregressive LLM with LoRA using a masked-loop generation objective,

$\theta$0

and stage-specific losses that recover augmented loops from captions and masked loops from the remaining model plus instruction. The model is optimized with AdamW on the LoRA parameters (Zhang et al., 12 Jun 2025).

4. Training data, supervision, and benchmark construction

The recent literature shows that online geometry control often depends on nontrivial dataset construction, because streaming or geometry-labeled supervision is typically scarce.

Stream3D-VLM addresses this by generating Stream3D-1M and Stream3D-Bench. RGB-D video datasets ScanNet, ScanNet++, and ARKitScenes are combined with instance segmentations. A Core Annotation Engine computes per-frame metadata including object visibility via depth z-buffer occlusion, camera kinematics, inter-object distances, object sizes, and room area. Twenty-nine task templates are then populated across Backward Tracing, Realtime Perception, and Forward Response, using rule-based generation for geometric queries and QA transfer from ScanQA with GPT-5 verification. The result is over $\theta$1M online spatio-temporal 3D QA pairs and a $\theta$2K-sample held-out benchmark balanced over tasks, answer types, and datasets (Yu et al., 5 Jun 2026).

BoxCtrl likewise treats data generation as fundamental. For supervised fine-tuning, it constructs a synthetic paired dataset using Kubric and GSO, with $\theta$3 object meshes in fixed camera scenes with random HDRI backgrounds and random transformation parameters $\theta$4. Paired natural-language prompts are generated automatically. The second stage then uses unpaired real images from PIE-Bench and Subjects200k for online RL, with source 3D boxes from DetAny3D and zero-reward fallback for detection failures under the specified confidence thresholds (Wang et al., 22 Jun 2026).

GeoCAD builds supervision through a complementary captioning strategy over $\theta$5K distinct local parts. Simple parts are captioned by vertex-based rules using coordinates, edge lengths, and angles; complex parts are rendered to 2D and captioned by a VLLM, followed by human verification and correction for low-confidence captions. The training data come from DeepCAD, containing $\theta$6K sequences split $\theta$7 train/val/test (Zhang et al., 12 Jun 2025).

A plausible implication is that online geometry control tends to require synthetic generation, temporal grounding, or automated geometric annotation, because the supervisory signal must specify not merely what is present, but how geometry should evolve, be queried, or constrain action over time.

5. Runtime pipelines and empirical trade-offs

Stream3D-VLM is organized as a single end-to-end pipeline from raw RGB-D frames to an LLM’s next-token output. At timestamp $\theta$8, the system ingests $\theta$9 and optionally $H^{2D}_t \in \mathbb{R}^{N \times d^{vis}}$0, $H^{2D}_t \in \mathbb{R}^{N \times d^{vis}}$1, and $H^{2D}_t \in \mathbb{R}^{N \times d^{vis}}$2, fuses 2D and 3D features in VSFI, lifts the result into spatial voxels, compresses them by GAVC, and prepends or appends decision tokens before sending the sequence to the LLM. On Stream3D-Bench at $H^{2D}_t \in \mathbb{R}^{N \times d^{vis}}$3 fps input, the $H^{2D}_t \in \mathbb{R}^{N \times d^{vis}}$4B-parameter model reports answer accuracy $H^{2D}_t \in \mathbb{R}^{N \times d^{vis}}$5, Answer-Timing Accuracy $H^{2D}_t \in \mathbb{R}^{N \times d^{vis}}$6, Time-To-First-Token $H^{2D}_t \in \mathbb{R}^{N \times d^{vis}}$7 ms, end-to-end latency $H^{2D}_t \in \mathbb{R}^{N \times d^{vis}}$8 s per frame, and memory usage $H^{2D}_t \in \mathbb{R}^{N \times d^{vis}}$9 GB. Without VSFI, average accuracy drops by $G_t \in \mathbb{R}^{K \times d^{geo}}$0 points; without GAVC, inference latency doubles; at $G_t \in \mathbb{R}^{K \times d^{geo}}$1 retention, the model retains $G_t \in \mathbb{R}^{K \times d^{geo}}$2 of its top-line performance while halving latency (Yu et al., 5 Jun 2026).

The monocular safe-control framework runs the full perception-to-control loop at $G_t \in \mathbb{R}^{K \times d^{geo}}$3--$G_t \in \mathbb{R}^{K \times d^{geo}}$4 Hz on a single GPU+CPU. The pipeline consists of SLAM, semantic segmentation, TSDF update, occupancy extraction with class-dependent inflation, ESDF update, and a small QP with one inequality. Reported timing is $G_t \in \mathbb{R}^{K \times d^{geo}}$5--$G_t \in \mathbb{R}^{K \times d^{geo}}$6 ms for the full loop, with SLAM plus segmentation taking $G_t \in \mathbb{R}^{K \times d^{geo}}$7 ms, ESDF update $G_t \in \mathbb{R}^{K \times d^{geo}}$8 ms, and the QP $G_t \in \mathbb{R}^{K \times d^{geo}}$9 ms (Zhang et al., 1 Jun 2026).

BoxCtrl reports both synthetic and real-world results. On unseen Kubric it gives PSNR/SSIM/LPIPS of $c_t$0, IoU $c_t$1, RotAcc $c_t$2, and CLIP-I/DINO-I $c_t$3. On the real-world benchmark it reports Translation IoU $c_t$4 vs next best $c_t$5, Scaling IoU $c_t$6 vs $c_t$7, Rotation RotAcc $c_t$8 vs $c_t$9, Composite IoU $$H^{3D}_t = \mathrm{MLP}([c_t; G_t]) \in \mathbb{R}^{(K+1)\times d^{vis}},$$0 vs $$H^{3D}_t = \mathrm{MLP}([c_t; G_t]) \in \mathbb{R}^{(K+1)\times d^{vis}},$$1, and overall CLIP-I/DINO-I approximately $$H^{3D}_t = \mathrm{MLP}([c_t; G_t]) \in \mathbb{R}^{(K+1)\times d^{vis}},$$2. Ablations further show that real IoU rises from $$H^{3D}_t = \mathrm{MLP}([c_t; G_t]) \in \mathbb{R}^{(K+1)\times d^{vis}},$$3 and RotAcc from $$H^{3D}_t = \mathrm{MLP}([c_t; G_t]) \in \mathbb{R}^{(K+1)\times d^{vis}},$$4 when moving from SFT to SFT+RL (Wang et al., 22 Jun 2026).

The quadcopter controller separates offline and online computation. Offline, $$H^{3D}_t = \mathrm{MLP}([c_t; G_t]) \in \mathbb{R}^{(K+1)\times d^{vis}},$$5 is integrated backward and a vectorized Lyapunov operator is precomputed; online, $$H^{3D}_t = \mathrm{MLP}([c_t; G_t]) \in \mathbb{R}^{(K+1)\times d^{vis}},$$6 and $$H^{3D}_t = \mathrm{MLP}([c_t; G_t]) \in \mathbb{R}^{(K+1)\times d^{vis}},$$7 are obtained by matrix multiplies and vectorizations, then assembled into the feedback gain. Simulations report average online CPU time of $$H^{3D}_t = \mathrm{MLP}([c_t; G_t]) \in \mathbb{R}^{(K+1)\times d^{vis}},$$8 ms/step for $$H^{3D}_t = \mathrm{MLP}([c_t; G_t]) \in \mathbb{R}^{(K+1)\times d^{vis}},$$9–D against $SO(3)$00 ms for SDRE, with control energy $SO(3)$01 versus $SO(3)$02, and no attitude singularity during the $SO(3)$03-rad flip (Yao, 2023).

GeoCAD reports generation quality, validity, and text-to-CAD consistency. In the main test-set comparison, GeoCAD achieves COV $SO(3)$04, MMD $SO(3)$05, JSD $SO(3)$06, PV $SO(3)$07, Ver-score $SO(3)$08, VLLM-score $SO(3)$09, and Realism $SO(3)$10; with $SO(3)$11-shot prompting it further improves to COV $SO(3)$12, MMD $SO(3)$13, PV $SO(3)$14, Ver-score $SO(3)$15, VLLM-score $SO(3)$16, and Realism $SO(3)$17 (Zhang et al., 12 Jun 2025).

These results indicate a recurring trade-off between geometric fidelity, latency, and token or state complexity. Stream3D-VLM makes this trade-off explicit through GAVC retention ratio, while BoxCtrl and the semantic-ESDF framework expose a different compromise between geometric precision and visual fidelity or control conservatism.

6. Conceptual boundaries, recurring misconceptions, and research direction

Several of these works explicitly reject simplified views of geometry in online systems. The semantic-ESDF framework argues against assigning the same distance-based safety margin to all mapped obstacles or using semantics only as a downstream controller adjustment; instead, semantic risk is embedded into the distance field before control optimization (Zhang et al., 1 Jun 2026). BoxCtrl argues against text-only or coarse 2D-guided editing for precise translation, scaling, and rotation, and attributes its effectiveness to RGB 3D bounding boxes that decouple geometric control from appearance control (Wang et al., 22 Jun 2026). Stream3D-VLM is presented as an alternative to offline 3D Large Multimodal Models that require complete scene observations or predefined video clips (Yu et al., 5 Jun 2026). The quadcopter work avoids Euler-angle singularities by moving the attitude subproblem to $SO(3)$18 (Yao, 2023).

This suggests a common misconception: that geometry can be appended as a late-stage constraint without changing the internal online state. In the surveyed systems, the opposite design is preferred. Geometry is fused into tokens before language decoding, embedded into occupancy before ESDF computation, encoded in prompts before diffusion denoising, or built into the state manifold before deriving control error.

A second boundary concerns what counts as “online.” GeoCAD demonstrates robust local geometry-controllable CAD generation through masked-loop prediction, complementary captioning, and two-stage fine-tuning, but its primary interaction pattern is user-directed local editing rather than continuous closed-loop perception or streaming response (Zhang et al., 12 Jun 2025). It is therefore better viewed as adjacent to online geometry control than identical with it.

A plausible research implication is that the field is converging on a common architectural template: explicit geometric representation, lightweight fusion or compression, a runtime decision mechanism that queries that representation directly, and data-generation pipelines tailored to temporal or local geometric supervision. Across the cited works, the central issue is not merely whether geometry is available, but whether it remains computationally accessible and decision-relevant at the moment an action, answer, or edit must be produced.