COBOT Magic Platform

• COBOT Magic Platform is a hierarchical multisensory edge-cloud system that passively and anonymously monitors workers in cobot environments using radar, sub-THz, LWIR, and WiFi CSI.
• It employs multi-stage signal processing and fusion techniques, combining Bayesian inference and deep learning to achieve low latency and high accuracy under ISO/TS 15066 standards.
• The platform demonstrates practical industrial use with validated latencies (37–90 ms) and sub-meter localization, dynamically orchestrating sensor pipelines for enhanced human–robot collaboration safety.

The COBOT Magic Platform is a hierarchical multisensory edge-cloud system enabling passive, anonymous, and real-time worker monitoring in collaborative robot (cobot) environments through the fusion of heterogeneous IoT sensor modalities. By leveraging radar, sub-THz imaging, long-wave infrared (LWIR), and WiFi channel state information (CSI), the platform ensures precise spatiotemporal perception of human workers required for safety-critical human–robot collaboration (HRC). The architecture is distinguished by its multi-stage signal processing, feature extraction, and joint Bayesian and deep learning-based fusion, achieving low latency, high accuracy, and compliance with ISO/TS 15066 safety specifications (Kianoush et al., 2021).

1. System Architecture and Data Flow

The platform is organized as a three-tier hierarchy:

1. Sensing Tier (“Pipelines”): Each pipeline aggregates a set of homogeneous sensors, including sub-THz FMCW radars (122 GHz, ~6 units), a 100 GHz sub-THz imaging camera (32×32 array), multiple 8×8 LWIR thermopile arrays, and a multi-antenna WiFi receiver (2.4–5 GHz, 4 chains). Each sensor group continuously forwards time-stamped raw signals to corresponding edges.
2. Edge Tier (“Micro-Edges”): Each pipeline is serviced by a dedicated micro-edge unit (Intel NUC or ARM), implementing real-time preprocessing tailored to the modality—such as denoising, background subtraction, covariance normalization, and beamforming. These micro-edges reduce raw signals to reduced-dimensionality statistical features and publish results using MQTT or REST APIs.
3. Cloud Tier: Feature vectors $V_i$ from each pipeline are collected by a fusion-and-analytics engine. The cloud dynamically selects and fuses subsets of features according to task requirements, employing Bayesian inference or deep neural models (CNN, LSTM), and exposes high-level outputs (e.g., worker count, position, alarms) to SCADA/HMI systems for safety enforcement.

The canonical data flow is:

$\text{Raw sensor frames} \rightarrow \text{Edge preprocessing} \rightarrow \text{Feature vectors } V_i \rightarrow \text{Cloud fusion %%%%1%%%% ML} \rightarrow \text{Real-time HRC services}$

2. Sensing Modalities and Signal Processing Chains

2.1 Opportunistic Passive Sensing

All sensory pipelines operate passively and device-free, exploiting ambient field perturbations caused by worker motion without tags or markers, thus preserving anonymity and minimizing operational disruption. Sensors function as “virtual” presence detectors, interpreting changes in signal space induced by body interference or movement.

2.2 Pipeline-Specific Preprocessing

Each sensor $k$ in pipeline $i$ produces a time series $X_{k,i}(t)$. Preprocessing transforms this into a denoised, standardized representation:

$$\widetilde{X}_{k,i}(t) = f_i\left(X_{k,i}(t) \mid \phi_i\right)$$

where $\phi_i$ encapsulates relevant preprocessing parameters (such as background, covariance, beamforming weights).

Modalities and their operators:

• Radar ($i=1$): $X_{k,1}(t)\in\mathbb{C}^{N_{FFT}}$, with 512-point FFTs applied to beat signals; background subtraction and covariance whitening:

$$\widetilde{X}_{k,1}(t) = C_k^{-1/2}(X_{k,1}(t) - \overline{X}_k(\emptyset)),\quad \overline{X}_k(\emptyset) = \mathbb{E}[X_{k,1}(t) \mid \text{no worker}]$$

• Sub-THz Camera ($\text{Raw sensor frames} \rightarrow \text{Edge preprocessing} \rightarrow \text{Feature vectors } V_i \rightarrow \text{Cloud fusion %%%%1%%%% ML} \rightarrow \text{Real-time HRC services}$0): Frame model $\text{Raw sensor frames} \rightarrow \text{Edge preprocessing} \rightarrow \text{Feature vectors } V_i \rightarrow \text{Cloud fusion %%%%1%%%% ML} \rightarrow \text{Real-time HRC services}$1, with $\text{Raw sensor frames} \rightarrow \text{Edge preprocessing} \rightarrow \text{Feature vectors } V_i \rightarrow \text{Cloud fusion %%%%1%%%% ML} \rightarrow \text{Real-time HRC services}$2; standardization:

$\text{Raw sensor frames} \rightarrow \text{Edge preprocessing} \rightarrow \text{Feature vectors } V_i \rightarrow \text{Cloud fusion %%%%1%%%% ML} \rightarrow \text{Real-time HRC services}$3

• LWIR Arrays ($\text{Raw sensor frames} \rightarrow \text{Edge preprocessing} \rightarrow \text{Feature vectors } V_i \rightarrow \text{Cloud fusion %%%%1%%%% ML} \rightarrow \text{Real-time HRC services}$4): 8×8 thermopile frames, background subtraction:

$\text{Raw sensor frames} \rightarrow \text{Edge preprocessing} \rightarrow \text{Feature vectors } V_i \rightarrow \text{Cloud fusion %%%%1%%%% ML} \rightarrow \text{Real-time HRC services}$5

• WiFi CSI ($\text{Raw sensor frames} \rightarrow \text{Edge preprocessing} \rightarrow \text{Feature vectors } V_i \rightarrow \text{Cloud fusion %%%%1%%%% ML} \rightarrow \text{Real-time HRC services}$6): $\text{Raw sensor frames} \rightarrow \text{Edge preprocessing} \rightarrow \text{Feature vectors } V_i \rightarrow \text{Cloud fusion %%%%1%%%% ML} \rightarrow \text{Real-time HRC services}$7 (LOS, interference, noise). Estimated LOS signal $\text{Raw sensor frames} \rightarrow \text{Edge preprocessing} \rightarrow \text{Feature vectors } V_i \rightarrow \text{Cloud fusion %%%%1%%%% ML} \rightarrow \text{Real-time HRC services}$8 subtracted after beamforming:

$\text{Raw sensor frames} \rightarrow \text{Edge preprocessing} \rightarrow \text{Feature vectors } V_i \rightarrow \text{Cloud fusion %%%%1%%%% ML} \rightarrow \text{Real-time HRC services}$9

2.3 Feature Extraction

Sliding-window computation of four central moments for each denoised stream:

• Mean: $k$0
• Variance: $k$1
• Skewness: $k$2
• Kurtosis: $k$3

Hence, each feature vector $k$4 forms a compact, modality-invariant descriptor.

3. Data Fusion and Machine Learning Frameworks

3.1 Bayesian and Probability-Based Fusion

A generic feature-level Bayesian fusion for hypothesis $k$5 (e.g., worker count, pose, or location) is expressed as:

$k$6

Likelihoods $k$7 are obtained via targeted calibration. This formalism unifies decisions from heterogeneous sensors under uncertainty by maximizing joint posterior probabilities.

3.2 Deep Learning Models

Two primary model architectures are deployed:

• LSTM: Single recurrent layer, 8 hidden cells, SoftMax output over $k$8 classes (postures or counts). Cross-entropy loss:

$k$9

• CNN: 2D feature maps (e.g., 32×32, by stacking radar and THz camera data), processed using three convolutional layers (3×3 or 5×5 filters), ReLU, pooling, fully-connected layer, and SoftMax output. Also optimized under cross-entropy.

Supervised training uses 80% of scenario data, spanning 1,000–5,000 epochs with early stopping via validation accuracy. The cloud service actively orchestrates feature stream selection to balance latency and accuracy demands.

4. Real-Time Performance, Safety, and Metrics

4.1 Latency Components

Total system reaction time $i$0 is the sum of edge preprocessing and feature extraction ($i$1), edge-to-cloud network transmission ($i$2), and cloud inference ($i$3).

Empirical pilot measurements:

• For $i$4 m (operating space; pipelines 1+2): $i$5 ms
• For $i$6 m (co-presence; pipelines 1+2+3): $i$7 ms

4.2 Protective Separation Distance

Following ISO/TS 15066, the protective safety distance $i$8 is:

$i$9

where:

• $X_{k,i}(t)$0, $X_{k,i}(t)$1: worker and robot maximum directed speeds
• $X_{k,i}(t)$2: robot response latency (500 ms)
• $X_{k,i}(t)$3: robot stopping time (300 ms)
• $X_{k,i}(t)$4: mean speed in stopping phase
• $X_{k,i}(t)$5, $X_{k,i}(t)$6: localization uncertainties (measured 0.54 m for $X_{k,i}(t)$7 m, 0.28 m for $X_{k,i}(t)$8 m; $X_{k,i}(t)$9 m)

Example values with $$\widetilde{X}_{k,i}(t) = f_i\left(X_{k,i}(t) \mid \phi_i\right)$$0 m/s, $$\widetilde{X}_{k,i}(t) = f_i\left(X_{k,i}(t) \mid \phi_i\right)$$1 m/s:

• For $$\widetilde{X}_{k,i}(t) = f_i\left(X_{k,i}(t) \mid \phi_i\right)$$2 m: $$\widetilde{X}_{k,i}(t) = f_i\left(X_{k,i}(t) \mid \phi_i\right)$$3 m
• For $$\widetilde{X}_{k,i}(t) = f_i\left(X_{k,i}(t) \mid \phi_i\right)$$4 m: $$\widetilde{X}_{k,i}(t) = f_i\left(X_{k,i}(t) \mid \phi_i\right)$$5 m

Reducing $$\widetilde{X}_{k,i}(t) = f_i\left(X_{k,i}(t) \mid \phi_i\right)$$6 enhances workspace efficiency; $$\widetilde{X}_{k,i}(t) = f_i\left(X_{k,i}(t) \mid \phi_i\right)$$7 shrinks to $$\widetilde{X}_{k,i}(t) = f_i\left(X_{k,i}(t) \mid \phi_i\right)$$8 m for $$\widetilde{X}_{k,i}(t) = f_i\left(X_{k,i}(t) \mid \phi_i\right)$$9 m/s.

5. Implementation, Use Cases, and Benchmarks

5.1 Hardware and Network

Key subsystem specifications:

• Robot: Universal Robots UR10, 10 kg payload
• Radars: six 122 GHz FMCW, 1 ms ramps, 6 GHz sweep bandwidth
• THz Imaging: 32×32 array, 100 GHz, NEP 1 nW/$\phi_i$0
• IR Arrays: three 8×8 boards, 0.08 °C sensitivity
• WiFi/CSI: USRP X300 + UBX-160, 4× Rx beam steering
• Edge: four micro-edges (Intel NUC/ARM), Python/C++ stack
• Networking: MQTT over 802.11n/802.15.4e; REST APIs
• Cloud: Kubernetes cluster, TensorFlow/PyTorch, MQTT/REST API

5.2 Operational Scenarios and Performance

Use Case	Sensor Pipelines	Accuracy	Latency
Worker Counting (5.5×4 m)	WiFi CSI + beamsteering	>90% (0–3 persons, off-by-1 in 1–2)	Not specified
Operating-Space Occupancy	Radar+THz CNN	Fused: 96.9%	37 ms
Co-presence Monitoring	Radar+THz+IR CNN	Fused: 97.5%	90 ms

Standalone modality performance: radar CNN ($\phi_i$1), THz-alone ($\phi_i$2), IR-alone co-presence ($\phi_i$3).

5.3 Dynamic Feature Orchestration

Task-driven orchestration by the cloud data controller enables the system to select, at runtime, the relevant sensing pipelines (e.g., radar+THz for minimal $\phi_i$4; radar+THz+IR for high-precision co-presence). This adaptivity ensures workload and safety requirements are balanced dynamically.

6. Significance and Context

The COBOT Magic Platform exemplifies a modular, data-driven approach to industrial HRC safety enforcing passive and anonymous worker detection by leveraging commodity IoT radio and thermal hardware. The hierarchical edge-cloud design, robust statistical feature pipeline, and multimodal fusion frameworks directly address the limitations of single-modality or wearable-tag solutions. Measured real-time latencies (37–90 ms) and sub-meter localization precision (0.3–0.6 m) translate, via ISO/TS 15066, to protective separation distances ($\phi_i$5 – $\phi_i$6 m), achieving both regulatory compliance and operational efficiency. The design demonstrates the effectiveness of edge-cloud industrial IoT for complex HRC scenarios and is validated in industrial pilot deployments (Kianoush et al., 2021).