OFDataPip: Online-Offline Data Pipeline

• OFDataPip is the online-offline data pipeline that ingests real-world sensor data and converts it into standardized object states for digital twin simulation in CDA-SimBoost.
• It employs object detection, tracking, and model mapping techniques to synchronize dynamic traffic data with static simulation environments through a ROS bridge.
• The module operates in online and offline modes, enabling both real-time replay of traffic scenes and high-quality log refinement for synthetic scenario fusion.

Searching arXiv for the specified paper and related context. arXiv search query: (Zheng et al., 25 Jul 2025) OFDataPip is the Online-Offline Data Pipeline in CDA-SimBoost, where it functions as the “glue” layer that converts raw real-world sensing into simulator-consumable dynamic state. Within the CDA-SimBoost triad—Digital Twin Builder, OFDataPip, and OpenCDA-InfraX—it is the data ingestion and processing engine for real-world information: it collects data from roadside infrastructure and connected automated vehicles (CAVs), processes those inputs into object-level tracks with unified attributes, and publishes the resulting object states into OpenCDA-InfraX through a ROS bridge. In this arrangement, the Digital Twin Builder provides the static environment, OFDataPip provides the dynamic state of traffic, and OpenCDA-InfraX combines both for infrastructure-driven cooperative driving automation (CDA) simulation (Zheng et al., 25 Jul 2025).

1. Position within CDA-SimBoost

CDA-SimBoost is organized around three main components. The Digital Twin Builder turns HD maps and point clouds into simulator maps and assets for CARLA and Unreal. OFDataPip ingests, processes, and standardizes real-world data streams. OpenCDA-InfraX is the infrastructure-centric CDA simulation platform that consumes both the generated assets and the processed dynamic data. OFDataPip therefore occupies the boundary between real sensing and simulation, rather than the boundary between map construction and rendering (Zheng et al., 25 Jul 2025).

The paper states that “the purpose of OFDataPip is to collect and generate real-time data streams or high-quality offline data for OpenCDA-InfraX.” That purpose is conceptually narrow but operationally central. It does not generate the static digital twin, and it does not itself constitute the simulation platform. Instead, it supplies the evolving traffic state—vehicles, pedestrians, and other road users—so that the digital twin can either replay the real world or remain synchronized with it.

A recurrent misconception is to treat OFDataPip as a generic perception stack or as a simulator plugin. The description in CDA-SimBoost is more specific. OFDataPip is the online-offline data pipeline that standardizes heterogeneous sensing outputs into a unified object-state interface for the digital twin. This makes it both an ingestion layer and a translation layer.

2. Internal architecture and processing flow

In the overall workflow, OFDataPip takes in real-world data collected by the data collection platform—specifically from CAVs and smart infrastructure—and outputs converted data to OpenCDA-InfraX. The online pipeline is described through a flowchart, while the offline side is described textually, but both converge on the same operational goal: standardized object-state sequences that the simulator can consume (Zheng et al., 25 Jul 2025).

For online operation, the first stage is data ingestion from “raw data streams from infrastructure-mounted devices, connected autonomous vehicles (CAVs), and other real-time sources.” The paper context indicates support for LiDAR, cameras, radars, and potentially V2X and telemetry-like inputs, although OFDataPip’s own description emphasizes the pipeline role rather than enumerating every modality. The next stage is an object detection and tracking module. Incoming data is processed by object detection and tracking algorithms to estimate position, orientation, bounding dimensions, object type, and a unique tracking ID. Multi-frame association algorithms are used to ensure consistent labeling across frames, thereby supporting trajectory reconstruction.

After tracking, OFDataPip performs model selection or actor mapping. To maintain consistency within the digital twin, vehicle models are selected based on the closest size match from the simulation asset library. This means that real traffic participants are not only represented as abstract tracks; they are mapped to CARLA actors whose dimensions approximately match the detected objects.

The subsequent stage is standardization and message formation. Object states—including position, orientation, type, and tracking ID—are standardized into a unified message format using the ROS bridge package. The simulator-facing interface is therefore object-centric rather than sensor-centric. At each time step, newly observed objects cause actor creation in CARLA, while existing objects trigger state updates such as position and orientation updates. Continuous arrival of new object states sustains a real-world data stream within OpenCDA-InfraX.

The offline pipeline preserves the same endpoint but changes the intermediate workflow. Saved logs from the data collection platform are loaded, pre-labeled through model-assisted annotation, corrected by human annotators, and then refined by a trajectory-consistency algorithm. The stated result is “more diverse labels, more accurate bounding boxes, and improved consistency in tracking information across frames.”

3. Online and offline modes

The central operational distinction in OFDataPip is between real-time streaming and non-real-time log refinement. The same pipeline family serves both purposes, which is why CDA-SimBoost presents OFDataPip as a mechanism for bridging real data and simulation (Zheng et al., 25 Jul 2025).

Mode	Input and operation	Output and use
Online	Continuous streams from real infrastructure sensors and CAVs; frame-by-frame processing in real time	Real-time object states published to CARLA/OpenCDA-InfraX through the ROS bridge
Offline	Recorded sensor and vehicle logs processed with model-assisted pre-labeling, manual annotation, and trajectory refinement	High-quality standardized trajectories exported for playback through ROS/ROSbag or for dataset use

In online mode, OFDataPip processes frames as they arrive and publishes object states in real time to CARLA through the ROS bridge. The paper emphasizes “real-time data processing,” an “online pipeline that integrates raw data streams,” and the ability to “maintain a real-world data stream within OpenCDA-InfraX.” This implies constraints typical of live robotic and cyber-physical pipelines, including latency sensitivity and robustness to imperfect delivery, although the paper does not provide explicit throughput or latency measurements for OFDataPip in isolation.

In offline mode, the priorities shift from latency to quality. The paper describes a hybrid strategy combining model-assisted pre-labeling with manual annotation. Human annotators review and correct initial labels, add missing objects, and refine bounding boxes and classes. A refinement algorithm based on trajectory consistency then adjusts the annotations to ensure coherent object trajectories and consistent tracking IDs across frames. These standardized sequences can be imported via the ROS bridge and replayed synchronously using ROSbag, allowing the simulator to treat log playback as if it were a live stream.

4. Integration with the digital twin and scenario workflows

OFDataPip provides the dynamic layer of the digital twin. The Digital Twin Builder generates HD point cloud maps, vector maps, and CARLA assets in Unreal Engine, while OFDataPip supplies real-world traffic states mapped into that environment. OpenCDA-InfraX then operates over the combined result. In this sense, the framework’s “real-time digital twin” consists of static geometry from the Digital Twin Builder and dynamic actors from OFDataPip (Zheng et al., 25 Jul 2025).

The primary middleware interface is the ROS bridge. OFDataPip publishes standardized object-state messages, and OpenCDA-InfraX subscribes to them and translates them into CARLA actors. The workflow described in the experiments section is explicit: real-world sensor streams collected from the target location are first processed through OFDataPip and projected into the simulation environment to replicate actual traffic conditions. The simulation can therefore display the same traffic scene that was detected in the physical intersection.

This dynamic background is also the substrate for synthetic scenario construction. OpenCDA-InfraX’s Synthetic Data Generator can introduce additional simulated vehicles, CDA-controlled ego vehicles, and NHTSA-based rare or critical scenarios on top of the OFDataPip-fed traffic state. OFDataPip therefore supports realistic scenario construction not by generating rare events itself, but by providing the realistic background traffic and temporal context into which those events are embedded.

5. Heterogeneous sensors, unified schema, and extensibility

The broader framework is designed for cameras with multiple field-of-view layouts, LiDARs with 16-, 64-, and 128-line configurations, radars, V2X or V2V or V2I or I2I communication messages, and HD maps. OFDataPip’s role is to hide this raw heterogeneity behind a unified object-state representation. The paper characterizes the module as highly modular, with detection algorithms and data sources decoupled from the publishing process to the digital twin environment (Zheng et al., 25 Jul 2025).

The common schema is centered on object-level state. The explicitly listed attributes are position, orientation, bounding box or bounding dimensions, type, and tracking ID. Regardless of whether the upstream source is infrastructure-mounted sensing, CAV sensing, or offline annotation, the downstream simulator sees the same normalized representation. A plausible implication is that this design reduces coupling between perception implementations and simulator interfaces, because the simulator subscribes to a stable message contract rather than to raw modality-specific streams.

ROS and ROSbag are the primary external interfaces. ROS messages and the ROS bridge connect OFDataPip to CARLA and OpenCDA-InfraX, while ROSbag supports synchronized playback of offline outputs. The framework also references scenario, sensor, and environment configuration files, though OFDataPip’s own configuration syntax is not detailed in the paper text.

The extensibility model follows directly from this decoupling. Adding a new sensor or algorithm involves subscribing to the source, running detection and tracking, converting results into the unified object message format, and publishing to the same topics already consumed by the simulator. Similarly, extending deployment to new infrastructure layouts or additional intersections requires updated sensor configurations and mapping outputs into the relevant digital-twin coordinate frames.

6. Real-synthetic fusion, constraints, and open issues

A principal contribution of OFDataPip within CDA-SimBoost is enabling real-synthetic fusion for long-tail CDA evaluation. Without it, the simulator would operate only over synthetic traffic. With it, real intersections can be replayed, natural behavioral distributions can be preserved in the background, and rare synthetic events can be inserted into realistic traffic context. The paper states that the system “supports realistic scenario construction, rare event synthesis, and scalable evaluation,” and OFDataPip is the mechanism that imports the real side of that hybridization (Zheng et al., 25 Jul 2025).

Its contribution to long-tail analysis is therefore indirect but structural. OFDataPip aligns real traffic data with the simulator’s temporal and spatial frame, while the Synthetic Data Generator introduces pre-crash and hazardous scenarios such as red-light violations, left-turn-across-path cases, and cut-ins. The resulting workflow makes real-world replay, synthetic stress testing, and reproducible scenario playback part of one pipeline.

The paper does not present a dedicated limitations section for OFDataPip, but several constraints are implicit. Scalability is one: as more intersections and agents are added, CARLA’s FPS drops, so OFDataPip must remain efficient enough not to become an additional bottleneck. Sensor dependency is another: effectiveness depends on the sensor layout, so poor coverage will degrade the quality of the dynamic state. The framework also assumes workable calibration and synchronization, but does not describe tools for automatic calibration or clock correction inside OFDataPip. Finally, multi-sensor fusion within OFDataPip is not described in fine-grained algorithmic detail; detection and tracking are largely treated as a black box.

These omissions delimit the current scope rather than invalidate the design. OFDataPip is not presented as a complete theory of infrastructure perception or multi-sensor fusion. It is presented as the standardized online-offline pipeline that ingests heterogeneous real-world sensing, produces coherent object-level trajectories, and delivers them to an infrastructure-centric digital twin for CDA research.