InFlux++: Real and Synthetic Data for Estimating Dynamic Camera Intrinsics
Abstract: Camera intrinsics are vital for recovering 3D structure from 2D video. However, most 3D algorithms assume fixed intrinsics throughout a video, an assumption that often fails for real-world in-the-wild videos. Consequently, estimating per-frame intrinsics from RGB images is critical for making 3D methods robust to videos with dynamic intrinsics. InFlux previously advanced this research direction by establishing the first real-world benchmark with per-frame ground truth intrinsics for dynamic intrinsics videos. Nevertheless, existing methods remain inaccurate due to two obstacles: (i) training data is scarce and lacks intrinsics diversity; and (ii) benchmarks, including InFlux, have limited scene and camera motion diversity, making it difficult to properly evaluate methods. To address both gaps, we present InFlux++, consisting of two components. InFlux++ Synth is a large-scale procedurally generated synthetic video dataset with 441K+ annotated frames from 1841 high-resolution videos, providing accurate per-frame ground truth intrinsics for training dynamic intrinsics prediction models; a subset also includes per-frame pose, depth, and normals. The videos feature rich intrinsics diversity through changes in camera zoom and focus, as well as dynamic objects and realistic rendering effects such as lens distortion and defocus blur. InFlux++ Real is a large-scale real-world benchmark that extends InFlux with 514K+ newly captured frames across 334 high-resolution videos, spanning a wider range of scenes and camera motions. Finetuning existing intrinsics prediction methods on InFlux++ Synth consistently improves focal length estimation across both InFlux++ Real and InFlux, suggesting that synthetic supervision is promising for RGB-based intrinsics prediction. For the dataset, benchmark, code, videos, submission instructions, and live leaderboard, please visit https://influx.cs.princeton.edu/ .
Summary
- The paper introduces InFlux++, a comprehensive data suite enabling per-frame dynamic camera intrinsics prediction using both synthetic and real-world data.
- It employs innovative synthetic data generation and board-based calibration protocols to enhance focal length, distortion, and principal point estimations.
- Empirical evaluations reveal current estimation limits, guiding future research toward robust self-calibration in applications like AR/VR and 3D vision.
InFlux++: A Comprehensive Data Suite for Dynamic Camera Intrinsics Estimation
Motivation and Problem Scope
Estimating dynamic camera intrinsics from RGB video is essential for precise 3D reconstruction and geometric reasoning, yet most contemporary 3D vision algorithms assume static intrinsics throughout a sequence. This assumption fails in real-world scenarios, especially with variable optical zoom, focus adjustments, and lens distortion inherent to in-the-wild footage. The present work addresses this by introducing InFlux++, a large-scale benchmark and training suite specifically for per-frame camera intrinsics prediction, comprising both synthetic and real-world data sources.
Previous efforts, notably InFlux [influx], provided the initial real-world benchmark enabling systematic evaluation of frame-wise intrinsics prediction, but were hampered by limited diversity in camera motion, scene variety, and training data, leading to suboptimal performance in SOTA models. InFlux++ bridges these gaps by delivering increased diversity for both training and benchmarking, detailed annotations, and robust calibration protocols.
Dataset Design and Characteristics
InFlux++ consists of two complementary components:
- InFlux++ Synth: A procedurally generated synthetic dataset utilizing the Infinigen engine [infinigen, infinigen2024indoors], yielding 441K+ annotated frames from 1841 high-resolution videos. It enables supervised learning for dynamic intrinsics estimation through:
- Physically accurate lens modeling via thin lens simulations.
- Dynamic camera motion and intrinsics (zoom, focus, lens distortion, defocus blur).
- Diverse indoor and nature scenes, with dynamic objects and non-Lambertian surfaces.
- Per-frame annotations: intrinsics, pose, depth, surface normals (for a subset).
- InFlux++ Real: An expanded real-world benchmark with 514K+ frames from 334 high-resolution videos. Innovations include:
- Board-based calibration experiments at fixed locations, replacing the prior drone-based approach for large field-of-view spatial footprint (FSF) calibration.
- Richer diversity in urban, suburban, and domestic environments; active subject motion; increased camera translation; and longer video durations.
- Comprehensive privacy visual preprocessing (multi-scale face and license plate blurring).
Figure 1: A gallery of InFlux++, showing the diversity present in both synthetic (top rows) and real-world (bottom rows) video sequences across scenes, lens effects, and activities.
Figure 2: Gallery of InFlux++ Synth’s procedurally varied indoor and nature scenes with realistic materials, geometry, lighting, and dynamic visual elements.
Figure 3: Gallery from InFlux++ Real, illustrating the expandend coverage of environments, activities, and camera perspectives.
The synthetic component leverages random walks in LFL and LTO, producing frame-wise continuous variation in focal length and focus. Radial distortion is sampled using Brown–Conrady parameterization at control points, tuned for realism, and augmentations are applied to simulate photometric and geometric variability. The real-world component utilizes lens metadata and look-up tables constructed via calibration board sweeps at multiple vantage points.
Figure 4: Histograms of per-frame intrinsics generated in InFlux++ Synth, evidencing broad and continuous parameter coverage in focal length, distortion coefficients, and focus distances.
Methodological Innovations
This paper advances both data collection and evaluation procedures:
- Synthetic Data Generation: Camera motion is constructed with RRT-based planners favoring trajectories with strong geometric cues, rejecting views dominated by planar surfaces to enhance training signal. Lens breathing and other optical phenomena are rigorously simulated.
- Calibration Protocols: Large-FSF board-based calibration simplifies ground truth acquisition and enhances reliability, with diverse board patterns projected onto multi-story screens and camera sweeps used to excite rotational axes.
- Evaluation Metrics: EPE recall is computed using a refined 3D point visibility filter, cutting off points outside the monotonic radial mapping support to remove non-physical projections. LUT-reliable EPE recall is calculated by filtering frames based on leave-one-out validation at specific pixel thresholds for interpolation reliability.
Empirical Evaluation
A suite of baseline methods was evaluated on InFlux++ Real and InFlux, including AnyCalib [anycalib], GeoCalib [geocalib], Perspective Fields [perspectivefields], UniDepthV2 [unidepthv2], WildCamera [wildcamera], COLMAP [colmap], and DroidCalib [droidcalib], among others.
Figure 5: LUT-reliable EPE recall plots for baseline methods, with AnyCalib outperforming competitors, though achieving modest recall for low pixel thresholds.
The results demonstrate that dynamic intrinsics prediction is challenging and remains far below optimal. Best-performing models (AnyCalib and its finetuned variant) achieve only 25–34% recall at 10% pixel error thresholds for focal length, and less than 26% EPE recall at 50 px. Performance is higher on the real-world component with more indoor scenes, but principal point estimation and distortion parameters remain weak, underscoring difficulty in generalizing across datasets.
AnyCalib finetuned with InFlux++ Synth demonstrates improved focal length prediction for both benchmarks but yields slightly worse distortion and principal point recall, attributed to its training loss function (spherical ray distance) emphasizing central pixels at the expense of boundary regions where distortion is pronounced.
Benchmarking and Data Accessibility
The benchmark design ensures high experimental reproducibility, with splits provided for validation and test sets, intrinsics annotation protocols, and leaderboard infrastructure for result submission and tracking. Data, code, and loaders are freely released under open licenses, and privacy is preserved via rigorous face/license plate blurring.
Implications and Future Directions
The introduction of InFlux++ is foundational for robust per-frame intrinsics estimation in unconstrained video. Its synthetic-optical diversity and real-world coverage enable multiple avenues:
- Enhanced Training Regimes: Synthetic supervision improves focal length estimation—a promising approach for future architectures incorporating domain adaptation and simulation-to-real transfer.
- Algorithmic Advances: Improved evaluation metrics and reliable calibration boards support algorithmic innovation, particularly for models capable of handling wider scene and camera motion distributions.
- Downstream Tasks: Accurate, dynamic intrinsics unlock reliable SLAM, visual odometry, AR/VR overlays, 3D Gaussian splatting [GS], and neural radiance fields [NeRF] in realistic conditions.
Given current limitations in principal point and distortion estimation, future work should address loss function alignment with boundary-aware metrics, domain adaptation to reconcile synthetic-real gap, and integration of additional multi-modal signals (e.g., depth, surface normals).
Figure 6: Visualization of ground-truth modalities, exemplifying the annotation granularity available for RGB, depth, surface normals, and camera trajectories in synthetic videos.
Figure 7: Lens breathing effect demonstrated in synthetic renders, validating simulation fidelity of lens-induced FOV variations with thin lens modeling.
Figure 8: Temporal plots of LFL, LTO, and CFL confirming natural, smooth changes of intrinsics across frames.
Conclusion
InFlux++ provides a rigorously curated data suite and benchmark for dynamic camera intrinsics estimation, filling gaps in diversity, annotation accuracy, and evaluation rigor. Synthetic supervision is established as a promising direction, yet significant challenges persist. The resources, protocols, and empirical findings herein catalyze further research toward robust video camera self-calibration, with implications for improved downstream 3D vision systems and real-world deployment across robotics, AR/VR, and spatial computing.
(2607.05389)
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
What is this paper about?
This paper is about teaching computers to understand a camera’s “inside settings” as they change over time in real videos. These inside settings, called camera intrinsics, include things like how zoomed-in the camera is (focal length), where the image center is, and how the lens bends light (lens distortion). Most 3D computer vision methods assume these settings stay the same throughout a video. But in real life, people zoom and refocus while filming, which breaks that assumption. The authors build two big datasets—one synthetic (computer-generated) and one real-world—to help AI models learn to estimate these changing settings for every single frame of a video.
What questions were the researchers trying to answer?
They focused on two simple questions:
- How can we train computers to guess a camera’s changing intrinsics (like zoom and focus) from just the video frames?
- What kind of data do we need to train and fairly test those computer methods so they work on real videos, not just idealized ones?
How did they do it?
To answer these questions, the team created a two-part data suite called InFlux++.
Two new datasets: Synthetic and Real
- InFlux++ Synth (synthetic):
- 441,000+ frames from 1,841 high‑resolution, computer-generated videos.
- Every frame has perfect ground-truth camera intrinsics because the scenes are rendered in software.
- Some frames also include extra information like camera pose, depth, and surface normals.
- Videos include realistic changes in zoom and focus over time, moving objects, lens distortion, and blur from out-of-focus areas.
- InFlux++ Real (real-world):
- 514,000+ newly captured frames across 334 high‑resolution videos.
- Covers a wide variety of scenes (indoors, city streets, sports, nighttime, moving vehicles) and more realistic camera motion (not just turning in place, but also moving through space).
- Each frame has ground-truth intrinsics measured using a careful calibration process.
Why both? Synthetic data is great for training because it’s big and perfectly labeled. Real data is great for testing because it reflects actual filming conditions.
How they made the synthetic videos realistic
- Procedural scenes: They used a system called Infinigen to automatically build diverse indoor and outdoor worlds with realistic lighting and materials—like forests, living rooms, kitchens, even reflective surfaces and water.
- Natural camera motion: They planned smooth camera paths that move and turn through the scene (think of a camera gliding around rather than teleporting).
- Zoom and focus changes over time: Instead of just setting “zoom” directly, they used a simple physics-based lens model (like what real cameras use) so changing focus can slightly change the field of view (“lens breathing”), just like in real life.
- Lens distortion and blur: They added common lens effects—curved straight lines near edges (distortion) and soft blur when parts are out of focus—to match real cameras.
- Controlled variety: They let zoom and focus “wander” smoothly over time (a bounded random walk) so the intrinsics don’t jump around but still change enough to be interesting.
Think of it like a well-made animated movie that also keeps a detailed log of the camera’s exact settings every frame.
How they collected and calibrated the real videos
- Smart lenses: They used zoom lenses that record per-frame lens settings, like zoom level and focus distance.
- Building a lookup table (LUT): They ran careful calibration experiments across many zoom and focus settings. This created a “map” from lens settings to the exact intrinsics for that lens.
- A new large-scene calibration trick: For very wide shots (where a huge area is in focus), it’s hard to move a giant calibration board around. Instead, they kept a large board still and moved the camera around it to collect strong calibration views—simpler and more reliable.
- Privacy: They automatically blurred faces and license plates in the real videos.
How they tested existing methods
They evaluated several methods that try to estimate intrinsics from video frames, including the current leading approach, AnyCalib. They measured:
- How close the predicted zoom (focal length) is to the true value.
- How close the image center is.
- How far the projected 3D points land from the correct pixel positions (a “pixel error” score called EPE).
They also tried improving AnyCalib by fine-tuning it on the new InFlux++ Synth dataset to see if synthetic training helps on real videos.
What did they find?
- Synthetic training helps with zoom: Fine-tuning the top method (AnyCalib) on InFlux++ Synth improved its ability to estimate focal length (how zoomed-in the camera is) on both the new real-world benchmark and the earlier InFlux benchmark.
- The task is still hard: Even the best methods still make significant mistakes, especially on details like the exact image center or lens distortion, which are crucial for accurate 3D reconstruction.
- Better data matters: The new datasets cover many more scenes, activities, and camera movements, providing a stronger training ground and a fairer, tougher test for future methods.
Why is this important? Many popular 3D methods—like those used for AR, VR, robotics, and 3D scene capture—break when a camera’s intrinsics change mid-video. If we can estimate these intrinsics for each frame, those same methods can work on everyday videos where people zoom and refocus.
Why does this matter?
- For everyday videos: Phones and cameras zoom and refocus all the time. Better per-frame intrinsics estimation means more reliable 3D effects, AR overlays, and measurements from ordinary videos.
- For robotics and drones: Robots need accurate geometry from cameras. Handling changing intrinsics makes navigation and mapping more robust.
- For research and industry: InFlux++ provides large, diverse, well-labeled data to train and benchmark future methods, speeding up progress across computer vision applications.
In short, this paper delivers the training fuel (a realistic synthetic dataset), the real-world test track (a bigger, tougher benchmark), and shows that training on synthetic videos can make today’s best methods better—pushing us closer to 3D systems that work reliably on real, in-the-wild videos. You can find the datasets, code, and leaderboard at https://influx.cs.princeton.edu/.
Knowledge Gaps
Unresolved gaps, limitations, and open questions
Below is a single consolidated list of what remains missing, uncertain, or unexplored in the paper, stated concretely so future work can act on it:
- Synthetic-to-real gap: Finetuning on InFlux++ Synth improves focal length but degrades principal point and EPE; the root causes (e.g., distortion modeling, photometric pipeline, data distributions) are not diagnosed with controlled ablations.
- Distortion realism: Distortion is applied post-render with Brown–Conrady; real lenses exhibit non-polynomial, asymmetric, and lens-specific behaviors (e.g., rational models, fisheye, anamorphic). The impact of in-render vs post-process distortion and richer models is untested.
- Principal point variability in Synth: The paper does not specify whether c_x/c_y are varied in synthetic data; if principal points remain near the image center, Synth may under-supervise off-center principal points prevalent in the Real benchmark.
- Skew and pixel aspect ratio: Zero skew and square pixels appear assumed; the effects of modeling skew and non-unity pixel aspect on training and evaluation are unexplored.
- Aperture and bokeh modeling: Defocus blur is rendered via a thin-lens model, but aperture size/shape variation (f-stop, blade geometry, cat-eye effect) is not described or evaluated as a cue for intrinsics.
- Rolling shutter and motion blur: Synth does not model rolling shutter or motion blur, which are common in handheld and vehicle videos; the effect on intrinsics prediction and transfer is unknown.
- Photometric pipeline realism: Augmentations omit blur, resolution changes, vignetting, chromatic aberration, camera response function (CRF), compression, and auto-exposure/white-balance dynamics; their absence may limit transfer but is not studied.
- Temporal intrinsics beyond LFL/LTO: Only zoom (LFL) and focus (LTO) vary; other time-varying intrinsics (e.g., principal point drift, skew changes, digital zoom, stabilization-induced variations, temperature drift) are not modeled.
- Realistic lens breathing: Thin-lens–based “breathing” may not match lens-specific, non-linear breathing curves or focus shift; no validation against measured breathing profiles is provided.
- Distortion–defocus interaction: Applying distortion after rendering ignores physical interaction between optical distortion and depth-of-field blur; the realism and its effect on learning are unquantified.
- Synthetic scene coverage: Urban scenes with glass facades, dense crowds, LED screens, fine text patterns, and extreme specularities are not explicitly represented; coverage of such challenging cues remains unclear.
- Motion statistics: RRT-based camera paths lack a quantitative comparison to real handheld or vehicle egomotion (e.g., acceleration spectra, jerk, path curvature); Synth may not reflect realistic motion patterns.
- Missing sensor/IMU signals: The benchmarks do not include IMU/gyro metadata; the potential of fusing inertial cues with visual intrinsics estimation is unexamined.
- Real GT accuracy: The new large-FSF board-based calibration replaces drone-based capture, but absolute ground-truth accuracy, systematic biases, and comparison to drone-based results are not quantified.
- LUT coverage and density: The grid spacing and coverage at extreme LFL/FD are not reported; how often test frames lie far from calibration nodes (and induced GT error) remains unknown.
- Evaluation filtering: The LUT-reliable EPE excludes frames in poorly supported LUT regions; potential evaluation bias and how many frames are filtered (per scene/motion/lens) are not disclosed.
- Distortion metrics: Evaluation focuses on EPE and focal/principal point percent errors; there are no direct error metrics for distortion parameters (e.g., k1/k2/tangential), limiting diagnostic insight.
- Uncertainty estimation: No calibrated uncertainty/confidence is reported for predicted intrinsics, hindering downstream use where robust fusion requires uncertainty.
- Downstream utility: The paper does not test whether supplying predicted per-frame intrinsics improves SLAM/VO/NeRF/GS pipelines versus fixed-intrinsics baselines on real videos.
- Methodological breadth: Only AnyCalib is finetuned; training from scratch on Synth, finetuning other baselines, or evaluating temporal/sequence models that exploit video coherence is left unexplored.
- Loss-function alignment: The suspected mismatch between FOV-field losses and EPE (especially near boundaries with distortion) is not empirically addressed with alternative objectives or weighting schemes.
- Parameter range limits: Synth varies LFL from 8–100 mm; ultra-wide fisheye and long-telephoto regimes common in action cams and wildlife/sports are not covered.
- Resolution/FPS gap: Synth is 1280×720 at 24 FPS; Real is 3424×2202 at 23.976 FPS; with resolution augmentations disabled, the impact of resolution/FPS mismatch is not investigated.
- Pose/depth/normals subset: The size and coverage of Synth frames with additional supervision (pose/depth/normals) are unspecified; joint training benefits and protocols are not evaluated.
- Privacy blurring effects: Automatic face/license-plate blurring in Real could remove calibration cues (e.g., edges, corners); its effect on model performance is not analyzed.
- Split design and leakage: The paper does not detail how lenses, scenes, and motion types are partitioned across train/val/test to prevent leakage or overfitting to specific lens LUTs.
- Lens diversity: Real captures use the same lens types as InFlux; smartphone cameras, action cams, and specialty optics (fisheye/anamorphic/zoom-by-wire, variable aperture) are absent, limiting generalizability.
- Reproducibility specifics: Key settings (distortion sampling ranges, RRT tuning, calibration setup, augmentation hyperparameters) are deferred to the supplement; a minimal main-paper recipe is lacking.
Practical Applications
Immediate Applications
Below is a focused set of concrete use cases that can be acted on now, drawing directly on the paper’s dataset, methods, and findings.
- Industry (software/vision), Academia — Training and evaluation of dynamic intrinsics models
- What: Use InFlux++ Synth for supervised training and InFlux++ Real for benchmarking and leaderboard submission to improve per-frame focal length and principal point estimation on in-the-wild videos.
- Tools/workflows: Model training pipelines (e.g., finetuning AnyCalib), CI-integrated benchmark evaluation against LUT-reliable EPE metrics.
- Dependencies/assumptions: Synthetic-to-real gap; current models still have low EPE recall at strict thresholds; need compute and data access.
- Robotics, Drones, AR/VR — Plug-in module to supply per-frame intrinsics to SLAM/VO pipelines
- What: Insert a dynamic intrinsics estimator (finetuned on InFlux++ Synth) before geometric back-ends (e.g., COLMAP, DROID-SLAM, ORB-SLAM2) so downstream computations use frame-accurate calibration when zoom/focus changes occur.
- Tools/workflows: A preprocessor that outputs per-frame intrinsics, passed into existing SLAM configs via API hooks.
- Dependencies/assumptions: Improvements are strongest for focal length; principal point/distortion remain challenging; real-time performance may require model distillation.
- Film/VFX, Broadcast — Match-moving and camera solve stabilization for shots with zoom/focus changes
- What: Use per-frame intrinsics (from InFlux++-trained models or lens LUTs) to reduce geometric drift in camera solves and improve 2D-3D alignment for compositing.
- Tools/workflows: A “dynamic intrinsics assist” plug-in in match-moving software (e.g., PFTrack/Nuke pipelines).
- Dependencies/assumptions: Remaining distortion errors near image borders may still require manual adjustments.
- Mobile/AR apps — Offline AR overlay correction for user videos with zoom/focus
- What: Pre-process recorded videos to estimate per-frame intrinsics and adjust virtual overlays or measurements to mitigate calibration errors caused by autofocus and lens breathing.
- Tools/workflows: Batch post-processing within app; export corrected overlays for sharing.
- Dependencies/assumptions: Offline (not on-device real-time) inference; acceptable latency and battery use.
- Camera/Lens OEMs, Production Houses — Large-FSF board-based calibration to build lens-specific LUTs
- What: Adopt the paper’s improved board-based procedure (moving camera around a static large board) to construct accurate LUTs mapping (LFL, FD) → intrinsics for zoom lenses.
- Tools/workflows: AprilTag board setup, capture at multiple vantage points, LUT generation, validation via LOO reliability checks.
- Dependencies/assumptions: Lens metadata access (per-frame LFL and FD), space for large-board setup, controlled capture conditions.
- Vision R&D, Dataset curators — Privacy-preserving video pipelines
- What: Integrate face/license plate blurring (adapted RetinaFace/EgoBlur) to safely release dynamic-intrinsics videos for research and product QA.
- Tools/workflows: Automated blurring stage in data curation with audit logs.
- Dependencies/assumptions: Detector recall/precision in varied lighting; policy-compliant retention of unblurred originals.
- CV/Depth research — Synthetic supervision beyond intrinsics (pose/depth/normals subset)
- What: Use InFlux++ Synth’s subset with pose/depth/normals for multi-task training (e.g., depth-from-video with dynamic intrinsics).
- Tools/workflows: Joint training regimens, curriculum learning with thin-lens defocus signals.
- Dependencies/assumptions: Distribution differences vs real world; careful augmentation (Brown–Conrady) to match target domains.
- Sports/Media Analytics — Field/court metric consistency across zooms
- What: Apply per-frame intrinsics to stabilize homographies and 3D estimates in broadcast feeds where operators zoom and refocus.
- Tools/workflows: A calibration layer in vision analytics stacks to recompute mappings per frame.
- Dependencies/assumptions: Camera/lens metadata or reliable intrinsics predictions; handling rolling shutter and motion blur.
Long-Term Applications
These use cases are feasible but depend on further model accuracy, scaling, integration, or standardization.
- Mobile OEMs, Platforms (ARCore/ARKit) — On-device real-time dynamic intrinsics estimation and logging
- What: Ship lightweight models in firmware to record per-frame intrinsics and expose them via camera APIs for apps/AR engines.
- Tools/products: “DynamicIntrinsics API” in camera pipelines; on-manifold learning modules.
- Dependencies/assumptions: Efficient models; energy constraints; calibration across SKU variations; privacy and developer adoption.
- Standards/Policy — Per-frame lens-state metadata standardization
- What: Define an industry standard for logging LFL, FD, and derived intrinsics in video metadata to enable downstream geometric robustness.
- Tools/workflows: Standards body specs (e.g., SMPTE/ISO), compliance tests, vendor consortium.
- Dependencies/assumptions: OEM cooperation; legal/privacy reviews when metadata touches identifiable scenes; backward compatibility.
- 3D Reconstruction (NeRF/Gaussian Splatting) — Dynamic-intrinsics-aware recon engines
- What: Integrate priors from estimators and jointly optimize intrinsics with scene parameters to reduce reconstruction artifacts under zoom/focus changes.
- Tools/products: DI-NeRF/DI-GS toolkits with lens breathing and distortion modeling.
- Dependencies/assumptions: Training stability; robust regularization; compute requirements for large-scale videos.
- Autonomous Vehicles — Intrinsics-aware perception with variable optics
- What: Future camera stacks that support limited zoom/autofocus can retain accurate geometry for depth, detection, and mapping under changing intrinsics.
- Tools/workflows: Perception middlewares that ingest intrinsics per frame; fallback strategies when estimators are uncertain.
- Dependencies/assumptions: Automotive-grade lenses with metadata; safety certification; most current AV cameras are fixed-focus—requires hardware shift.
- Surgical/Industrial Teleoperation — Accurate overlays under zoom/focus
- What: Maintain precise 3D overlays and measurements in surgical microscopes or inspection cameras despite dynamic intrinsics.
- Tools/products: Calibration-aware AR HUDs; regulatory-compliant software updates.
- Dependencies/assumptions: High reliability/latency guarantees; validation protocols; integration with legacy systems.
- Drone Mapping/Survey — Autofocus-aware mapping and photogrammetry
- What: Compensate for intrinsics changes from autofocus to improve geo-accuracy in orthomosaics and 3D point clouds.
- Tools/workflows: Flight control software logging lens state + post-processing with intrinsics-aware bundle adjustment.
- Dependencies/assumptions: Lens metadata availability; synchronization with GPS/IMU; environmental factors (motion blur, exposure).
- Robotics (Manipulation, Wearables) — Depth and pose robustness with head/hand cameras
- What: Feed per-frame intrinsics into depth/pose estimators to mitigate errors when operators zoom or when focus shifts with distance.
- Tools/workflows: Middleware layer that re-calibrates projection matrices per frame.
- Dependencies/assumptions: Model accuracy for principal point and distortion; real-time constraints.
- Lens/Camera Design — Simulation-driven lens breathing minimization
- What: Use thin-lens modeling and synthetic scenes to quantify and optimize lens breathing and distortion characteristics in new lens designs.
- Tools/products: Opto-mechanical CAD + rendering-in-the-loop evaluation suites; firmware tuning for focus motors.
- Dependencies/assumptions: Close coupling of physics models and rendering; manufacturer data; iterative prototyping.
- Creative Software — Automatic breathing and distortion compensation in editors
- What: NLE/VFX tools that infer per-frame intrinsics to reduce apparent FOV shifts and radial distortion over time.
- Tools/products: “Breath-Correct” filters; timeline-aware calibration curves.
- Dependencies/assumptions: Estimator robustness across cameras; UX for user overrides.
- Education — Curricula and lab kits for photogrammetry with dynamic intrinsics
- What: Teach students modern calibration, lens modeling, and reconstruction workflows using InFlux++ datasets and tooling.
- Tools/workflows: Assignments, dockerized labs, notebooks with synthetic/real splits.
- Dependencies/assumptions: Access to datasets; institutional compute; instructor training.
- Real Estate/Retail Scanning — Consumer-grade 3D capture with autofocus cameras
- What: Improve 3D scans taken with phones/gimbals by correcting intrinsics per frame for more accurate dimensions and visual fidelity.
- Tools/products: “Intrinsics-Aware Scan” mobile pipeline.
- Dependencies/assumptions: On-device inference or batch cloud processing; consistent camera behaviors.
- Governance/Compliance — Privacy-by-default pipelines for public video datasets
- What: Institutionalize automated face/license plate blurring and metadata governance when releasing dynamic-intrinsics datasets.
- Tools/workflows: Policy checklists, reproducible blurring scripts, audit trails.
- Dependencies/assumptions: Detector performance in diverse contexts; evolving regulations; community buy-in.
Glossary
- AprilTags: Square fiducial markers used for camera calibration and pose estimation. "the calibration target we use is an array of AprilTags~\cite{apriltag} projected onto a rigid screen."
- Bézier interpolation: A smooth curve interpolation method often used to generate continuous trajectories between keyframes. "connects the resulting pose keyframes using Bézier interpolation."
- Board-based calibration: Calibration approach using printed calibration boards moved through the camera’s view to estimate lens and camera parameters. "Obtaining high-quality board-based calibration data requires moving the calibration board throughout the camera's FOV while ensuring that the pattern remains visible, in focus, and observed under a wide range of rotations."
- Bounded random walk: A stochastic process constrained within limits, used here to vary parameters smoothly over time without exceeding bounds. "To vary LFL, we use a bounded random walk."
- Brown–Conrady model: A classical polynomial distortion model accounting for radial and tangential lens distortions. "For lens distortion, we adopt the BrownâConrady~\cite{brown} model."
- Camera intrinsics: Parameters defining the mapping from 3D coordinates to 2D image coordinates (e.g., focal length, principal point, distortion). "Camera intrinsics are fundamental to many real-world 3D systems, as they define the geometric mapping between 3D coordinates and the 2D image."
- Camera pose: The position and orientation of the camera in 3D space relative to the scene. "A subset of the dataset also provides camera pose, depth, and surface normals."
- COLMAP: A structure-from-motion and multi-view stereo pipeline widely used to reconstruct scenes and estimate camera parameters. "\cite{megadepth} provides COLMAP-derived intrinsics for Internet photo collections~\cite{colmap}, but lacks temporal video structure and relies on reconstruction-derived pseudo ground truth."
- Defocus blur: Optical blur that occurs when objects are out of focus due to lens and focus settings. "realistic optical effects such as lens distortion and defocus blur."
- Endpoint error (EPE): A metric measuring the 2D discrepancy between projected points under predicted vs. ground-truth intrinsics. "When measuring prediction error via endpoint error (EPE), which quantifies the 2D distance between projections of the same 3D points under ground truth and predicted intrinsics, only 34.1\% of projected points fall within 50 pixels of their ground truth locations."
- Field of view (FOV): The angular extent of the observable scene captured by the camera. "InFlux divides these experiments into two categories based on the FOV spatial footprint (FSF) of the camera, defined as the 3D region within the camera's FOV that is in focus."
- FOV spatial footprint (FSF): The 3D region within the FOV that is in focus, used to categorize calibration scenarios. "InFlux divides these experiments into two categories based on the FOV spatial footprint (FSF) of the camera, defined as the 3D region within the camera's FOV that is in focus."
- Infinigen: A procedural scene generator and renderer used to create diverse synthetic environments. "We build InFlux++ Synth on top of Infinigen~\cite{infinigen, infinigen2024indoors}, a procedural scene generator and renderer"
- Intrinsics lookup table (LUT): A precomputed mapping from lens metadata (e.g., LFL, FD) to camera intrinsics for each frame. "a one-time calibration process can be used to construct a lens-specific lookup table (LUT) mapping pairs to camera intrinsics."
- Layout optimizer: An automated method to arrange scene elements to ensure realistic layouts without manual design. "Its layout optimizer and tuned parameter distributions ensure scene realism without any manual design"
- Leave-one-out (LOO) validation: A cross-validation technique where one sample is left out during fitting to assess interpolation or model reliability. "we estimate LUT interpolation reliability by performing leave-one-out (LOO) validation~\cite{influx}"
- Lens breathing: A change in apparent FOV as focus distance changes, even at fixed focal length. "this formulation naturally produces lens breathing, a visual phenomenon observed in real-world cameras where the FOV can shift slightly when the LTO changes, even with a fixed LFL."
- Lens distortion: Deviations from ideal pinhole projection due to lens optics, commonly modeled as radial and tangential effects. "featuring diverse intrinsics from changing zoom and focus, varying lens distortion, dynamic objects, and realistic optical effects such as defocus blur."
- Lens Focal Length (LFL): The focal length of the lens when focused at infinity; a physical lens parameter. "Lens Focal Length (LFL) is the CFL of a camera when it is focused at infinity~\cite{lensdef}."
- Lens to Object Distance (LTO): Distance from the camera’s optical center to the in-focus object along the optical axis. "Lens to Object Distance (LTO) is the distance from the camera's optical center to the object in focus, measured along the optical axis."
- Monotonic branch of the radial mapping: The portion of a distortion mapping where radius increases monotonically, ensuring physically visible projections. "retaining only points on the main monotonic branch of the radial mapping, cutting off the mapping at a radius ."
- Non-Lambertian surfaces: Materials that do not reflect light uniformly; include glossy or reflective surfaces challenging for vision algorithms. "Indoor scenes feature varied room structures and non-Lambertian surfaces"
- Panorama stitching artifacts: Geometric or photometric errors introduced when assembling 360° panoramas from multiple images. "their accuracy can be affected by manufacturing imperfections in cameras or stitching artifacts introduced during panorama construction."
- Parallax: Apparent displacement of scene points due to camera translation, crucial for depth and structure estimation. "This produces more parallax and viewpoint changes"
- Pinhole camera model: An idealized projection model without lens effects, often used as a simplification in rendering or calibration. "but it only uses the simple pinhole camera model for rendering."
- Principal point offset: Displacement of the principal point from the image center, affecting the intrinsics matrix. "One of our lenses has a large principal point offset, expanding the range of and values in InFlux++ Real compared to InFlux."
- Procedural generation: Algorithmic creation of diverse content (scenes, objects, materials) using controllable randomization. "a large-scale procedurally generated synthetic video dataset"
- Rapidly-Exploring Random Trees (RRT): A sampling-based motion planning algorithm used here to generate smooth, collision-free camera trajectories. "we build on the Rapidly-Exploring Random Trees (RRT) camera trajectory generator provided by Infinigen~\cite{infinigen}."
- Ray tracing: A physically based rendering technique that simulates light paths through a scene and lens. "Blender's Cycles rendering engine, which performs ray tracing based on the thin lens equation."
- Surface normals: Vectors perpendicular to surfaces, used for shading, geometry analysis, and supervision. "A subset of the dataset also provides camera pose, depth, and surface normals."
- Tangential distortion: Image distortion caused by lens assembly misalignment, modeled by parameters such as p1 and p2. "We optionally sample small tangential values."
- Thin lens equation: Optical relation linking object distance, image distance, and focal length, used in physically accurate rendering. "performs ray tracing based on the thin lens equation."
- Thin lens model: An optical model treating the lens as infinitely thin to relate LFL, LTO, and effective focal length. "we use the thin lens model to compute CFL as a function of these two parameters."
Collections
Sign up for free to add this paper to one or more collections.
