EquiBot: Equivariant Robotic Vision & Manipulation

• EquiBot is a robotic and computer vision system that employs SIM(3)-equivariant neural architectures, ensuring invariance to rotation, translation, and scale in 3D space.
• Its design integrates a diffusion-based policy with real-time detection and tracking modules to achieve robust, generalizable performance in open-world tasks and equine monitoring.
• The system demonstrates high data efficiency and practical deployment with lightweight modules, achieving state-of-the-art results in both equine stall and facial behavior analysis.

EquiBot is a class of robotic and computer vision systems developed for advanced, data-efficient, and generalizable perception and manipulation in both equine behavioral monitoring and open-world robotic learning. Its design is characterized by SIM(3)-equivariant neural policy architectures, event-based stall behavior inference from visual and sensor data, and the integration of lightweight, real-time detection and action-prediction modules capable of deployment on mobile robotic hardware (Yang et al., 1 Jul 2024, Galimzianov et al., 20 Oct 2025).

1. SIM(3)-Equivariant Diffusion Policy: Mathematical and Architectural Foundations

EquiBot utilizes a SIM(3)-equivariant neural architecture, making all perception and policy functions invariant to rotation, translation, and scale in 3D space. For $g = (R, t, s) \in \mathrm{SIM}(3)$, acting on input point sets $X = \{x_i\}_{i=1}^N \subset \mathbb{R}^3$, the transformation is defined as $g \cdot X = \{ s R x_i + t \}_{i=1}^N$. A feature extractor $f$ is SIM(3)-equivariant if $f(g \cdot X) = g \cdot f(X)$. The PointNet++ encoder architecture computes equivariant codes: $\Theta_R$ (vector features), $\Theta_{\mathrm{inv}}$ (invariant scalars), $\Theta_c$ (centroid), and $\Theta_s$ (scale), with feature routing and canonicalization via vector-neuron fusion layers. Policy output combines a translation- and scale-invariant action with system-inferred scale to generate SIM(3)-equivariant commands (Yang et al., 1 Jul 2024).

The conditional diffusion policy follows a denoising diffusion probabilistic model (DDPM); actions are noised in a forward process, and a U-Net predicts the noise for each time step. The reverse process is parameterized by $\epsilon_\theta$, ensuring multi-modal and robust policy outputs. The loss is $$\mathcal{L}_{\mathrm{DDPM}} = \mathbb{E}_{(o_t,a_t^0)} \sum_{k=1}^K \mathbb{E}_\epsilon \| \epsilon - \epsilon_\theta(a_t^k,o_t,k) \|^2$$. Equivariance is enforced architecturally, which removes the need for brute-force data augmentation and supports cross-pose, cross-scale generalization (Yang et al., 1 Jul 2024).

2. Vision-Based Stall and Facial Monitoring Pipelines

In equine stall monitoring deployments, EquiBot employs real-time object detection and multi-object tracking. The primary detection backbone is YOLOv11-M, an anchor-based architecture featuring a CSPDarknet-like backbone for feature reuse, PAN for feature fusion, and three detection heads at multiple strides ($\{8,16,32\}$ pixels). Mean average precision (mAP) and Intersection over Union (IoU) metrics are used for evaluation:

$$\mathrm{IoU}(B_p, B_g) = \frac{\mathrm{Area}(B_p \cap B_g)}{\mathrm{Area}(B_p \cup B_g)}$$

$\mathrm{mAP} = \frac{1}{K} \sum_{k=1}^K P(r_k)$

Real-time detection output is processed by BoT-SORT, a tracker combining Kalman filtering, appearance re-identification, and Hungarian matching on a cost matrix mixing IoU and cosine feature distances. Tracks are resolved with class aggregation heuristics and temporal continuity checks (Galimzianov et al., 20 Oct 2025).

Event inference proceeds through localization against stall polygons, frame state aggregation, temporal merging, non-localized segment relabeling, and cross-clip event correction. Five event types are defined (E1: visible_inside, E2: visible_outside, E3: invisible_inside, E4: invisible_outside, E5: multiple_visible_inside), with temporal and spatial constraints on boundaries and blind spots (Galimzianov et al., 20 Oct 2025).

For EquiFACS-based facial monitoring, a cascade mechanism is adopted: YOLOv3-tiny for initial face and region-of-interest (ROI) detection, followed by per-action-unit binary classifiers (AlexNet and DRML variants). The pipeline achieves real-time throughput (>15 FPS on single GPU) and provides per-frame EquiFACS AU detection in the eye and lower-face regions, supporting automated welfare coding (Li et al., 2021).

3. Data Efficiency, Generalization, and Real-World Performance

EquiBot's SIM(3)-equivariant diffusion architecture demonstrates state-of-the-art generalization and data efficiency. In simulation tasks—box closing, cloth folding, object covering, push-T, can pick-and-place, nut assembly—EquiBot outperforms vanilla diffusion and data-augmented diffusion policies in out-of-distribution (OOD) tests. Specifically, on OOD (random rotation + scale + position), reward drops <15% for EquiBot, compared to 40–60% for DP+Aug (Yang et al., 1 Jul 2024).

On Robomimic data-efficiency benchmarks, EquiBot maintains robust performance (~0.75 mean test reward) as demonstration number drops to 25, while DP baselines experience severe degradation (~0.3). In real-world manipulation—pushing chairs, luggage packing/closing, laundry door closing, bimanual folding, bed-making—EquiBot achieves 60–80% success after just 5 minutes of demonstrations per task (e.g., Push Chair: 8/10 success vs. DP/DP+Aug: 0/10) (Yang et al., 1 Jul 2024).

In equine monitoring, YOLOv11 achieves [email protected] of 0.987 (horse) and 0.954 (person); [email protected]–.95 reaches 0.924 (horse), 0.816 (person). Event detection is 100% correct for horse-related events in representative clips, while person detection is less reliable in occlusion/dim scenes due to limited training data. The invisible_* event inference classifies blind-spot transitions reliably when prior localization and entrance heuristic are properly aligned. These properties suggest strong suitability for sustained field deployment (Galimzianov et al., 20 Oct 2025).

4. Dataset Creation, Annotation Strategies, and Evaluation Protocols

EquiBot deployments require extensively curated datasets. For stall monitoring, a two-stage annotation pipeline is used:

• Stage 1 ("warm start"): Stratified sampling, CLIP-embeddings-based diversity selection, automatic detection via GroundingDINO, and human correction.
• Stage 2 ("bootstrapping"): Fine-tuning YOLOv11 on Stage-1 labels, automatic inference on additional clips, and further refinement. The completed dataset contains ~12,000 annotated frames, 2,350/480 horse and 420/95 person instances (train/val splits), and >98% IoU > 0.7 overlap in manual spot-checks (Galimzianov et al., 20 Oct 2025).

For EquiFACS facial AU detection, 20,000 images (from 20,180 video clips, 8 horses) are prepared, with subject-exclusive 8-fold cross-validation and balanced per-AU sampling. Precision, recall, F1-score, and balanced accuracy are reported per AU and averaged, with overall F1 ≈ 58% (eye/lower-face AUs). The entire pipeline is implemented in Python 3.7/PyTorch 1.6 with Darknet for YOLOv3-tiny backbones, and all scripts/configs are available in the referenced repository (Li et al., 2021).

5. System Integration and Real-Time Robotic Deployment

EquiBot's algorithms are designed for deployment on power- and compute-constrained robotic systems. For mobile stall monitoring, inference runs on NVIDIA Jetson AGX Xavier (32 TOPS) or similar, with YOLOv11-M at ~20 ms/frame and BoT-SORT plus event inference at ~5 ms/frame, enabling sustained 20 fps real-time analysis. System memory is budgeted to 4 GB GPU RAM, and power draw (30 W) is compatible with long-term battery operation (Galimzianov et al., 20 Oct 2025).

Camera placement is standardized—forward-facing 720p global shutter, 2 m from stall entrance, 15° downward tilt—matching the training geometry. Video is processed locally or streamed (5 Mbps H.264) for remote inference. Recommendations for deployment include depth sensor augmentation (RealSense D435) for improved occlusion handling, trajectory-based behavior prediction via LSTM/Transformer, and field integration with ROS or MQTT for alerting and dashboarding. A plausible implication is that the platform can seamlessly transition to additional welfare applications by reusing and fine-tuning its modular architecture (Galimzianov et al., 20 Oct 2025).

6. Challenges, Limitations, and Prospects

Critical challenges in EquiBot deployments include class-imbalance and limited training data for minority classes (particularly humans in stalls), event ambiguity in occluded or poorly lit conditions, and the need for temporal modeling for dynamic or subtle behaviors such as ear movement or half-blinks. The EquiFACS pipeline identifies ROI detector error propagation, high per-AU variance, and difficulty distinguishing rapid dynamics from stills. Recommendations include dataset expansion, adoption of deeper attention-backbone networks, addition of temporal ConvLSTM or 3D CNN modules, and shifting to multi-label/focal loss formulations as data increases (Li et al., 2021).

Continuous, in-field data collection for rare or error-prone cases and real-time short-term smoothing of blink/half-blink predictions are suggested to minimize false alarms. The modular architecture supports extensibility toward ensemble detection, semantic action prediction, and adaptive perception-action pipelines across both stationary and mobile robotic platforms. Enhanced data diversity and architectural advances are expected to further increase the reliability and adaptability of EquiBot systems in varied real-world deployment contexts (Yang et al., 1 Jul 2024, Galimzianov et al., 20 Oct 2025, Li et al., 2021).