EKF-Based Depth Camera and Deep Learning Fusion for UAV-Person Distance Estimation and Following in SAR Operations

Abstract: Search and rescue (SAR) operations require rapid responses to save lives or property. Unmanned Aerial Vehicles (UAVs) equipped with vision-based systems support these missions through prior terrain investigation or real-time assistance during the mission itself. Vision-based UAV frameworks aid human search tasks by detecting and recognizing specific individuals, then tracking and following them while maintaining a safe distance. A key safety requirement for UAV following is the accurate estimation of the distance between camera and target object under real-world conditions, achieved by fusing multiple image modalities. UAVs with deep learning-based vision systems offer a new approach to the planning and execution of SAR operations. As part of the system for automatic people detection and face recognition using deep learning, in this paper we present the fusion of depth camera measurements and monocular camera-to-body distance estimation for robust tracking and following. Deep learning-based filtering of depth camera data and estimation of camera-to-body distance from a monocular camera are achieved with YOLO-pose, enabling real-time fusion of depth information using the Extended Kalman Filter (EKF) algorithm. The proposed subsystem, designed for use in drones, estimates and measures the distance between the depth camera and the human body keypoints, to maintain the safe distance between the drone and the human target. Our system provides an accurate estimated distance, which has been validated against motion capture ground truth data. The system has been tested in real time indoors, where it reduces the average errors, root mean square error (RMSE) and standard deviations of distance estimation up to 15,3\% in three tested scenarios.

• The paper develops an onboard SAR pipeline that combines YOLOv11 detection and pose keypoints, RealSense D435i depth, and an EKF with derivative-based outlier rejection for UAV-person distance estimation.
• The fused system achieves mean errors of 0.83 cm in discrete forward/backward motion and 0.61 cm during lateral movement, while reducing aggregate errors by up to 15.3% across tested scenarios.
• The approach runs at 15 FPS on a Jetson Xavier NX and supports outdoor following at a nominal 5 m distance, although height-dependent monocular bias and limited quantitative outdoor validation remain important limitations.

Motivation and problem setting

In search and rescue (SAR) missions, a UAV that has detected and recognized a target person must then track and follow that person while maintaining a safe, accurately known camera-to-body (C-B) distance. Depth cameras such as the Intel RealSense D435i provide accurate direct range measurements within an optimal working range (0.4–4 m), but their error grows exponentially beyond this range, and they are prone to outliers caused by reflective surfaces, poor illumination, off-axis viewing angles, and sudden camera motion—conditions typical of a flying, non-gimbalized platform. Conversely, monocular estimation is noisier but stable with respect to orientation and lighting. The paper proposes fusing these two complementary modalities in real time using an Extended Kalman Filter (EKF), embedded in a full detection–recognition–tracking–following pipeline running on board the aircraft.

System architecture

The platform is a Hexsoon EDU450 with an Orange Cube flight controller and an NVIDIA Jetson Xavier NX onboard computer running Ubuntu 20.04 with ROS2; deep learning models execute via PyTorch in Docker containers. The vision payload is a single RealSense D435i providing both aligned monocular RGB (1920×1080 @ 30 FPS) and depth (1280×720 @ up to 90 FPS) streams. The pipeline proceeds as follows:

• Detection: YOLOv11 produces person bounding boxes in real time.
• Recognition: A Dlib face recognition model matches detections against a reference template; if none exists, the operator confirms the identity of a captured candidate.
• Tracking: YOLOv11-based joint detection and embedding with re-identification (in the style of YOLO11-JDE combined with ByteTrack) assigns persistent IDs robust to occlusions.
• Keypoint estimation and distance sensing: YOLOv11-pose extracts 17 body keypoints; only shoulder and hip keypoints are used. The midpoint-to-midpoint shoulder–hip (S-H) distance/line serves two purposes: (i) on the RGB image, the pixel S-H distance feeds a piecewise logarithmic model approximating C-B distance, and (ii) on the registered depth image, depth values are sampled along the S-H line (50–350 pixels) and averaged to yield a direct C-B measurement.

The choice of S-H geometry is deliberate: it is invariant to body orientation relative to the camera, addressing a known weakness of prior face-centric approaches that rely on frontal face capture or facial area heuristics.

Monocular distance model

Assuming an average male height of 1.80 m, the mapping from S-H pixel distance $x$ to approximate C-B distance in centimeters is modeled as two non-linear regimes:

$$f(x) = \begin{cases} -48.03 \ln(x - 179.4) + 401, & x < 200 \ -240.2 \ln(x - 47.3) + 1457, & x \geq 200 \end{cases}$$

The authors concede that fixing the height a priori can produce large approximation errors; however, the model preserves distance-change dynamics, which suffices for its role as the EKF prediction input rather than a standalone estimate. Leaning toward the camera—a failure mode of this anthropometric proxy—is rejected by thresholding the derivative of the S-H signal, preventing such events from corrupting the control loop.

EKF fusion and outlier rejection

The state vector comprises the estimated C-B distance and its derivative, propagated under a nearly constant velocity model. The prediction step uses the keypoint-based approximation; the correction step uses the mean depth measurement along the S-H line. Process noise variances are $\sigma_p^2 = 0.02$ and $\sigma_{\dot{p}}^2 = 0.8$, with measurement noise $\sigma_z^2 = 0.018$. These settings indicate the filter trusts the direct depth measurement predominantly, using the keypoint estimate to stabilize oscillations—an asymmetric fusion consistent with the sensors' characteristics within their respective competence ranges.

Outlier gating is derivative-based: if the magnitude of the inter-sample change in measured C-B distance exceeds 1.25 times the mean deviation over the previous ten samples, the sample is flagged as an outlier and the filter executes prediction only until valid measurements resume. This mechanism targets exactly the failure modes of stereo depth on a moving UAV: reflections, FOV-edge degradation, and abrupt attitude changes. The authors note the D435i's accuracy degrades sharply beyond 4 m (its maximum detection range being about 10 m); a stated claim of the fusion is that usable estimation extends to an experimentally tested 7 m.

Experimental evaluation

Ground truth was obtained indoors with an 8-camera OptiTrack system over a 40 m² workspace. Marker patterns defined the depth sensor origin and a body frame whose origin lies at the S-H midpoint (seven markers on a helmet, one on the abdomen). Three motion scenarios were evaluated: discrete stepping toward/away from the camera, continuous approach/recede movements, and lateral movement across the FOV. Reported errors are:

Scenario	Method	Mean error [cm]	RMSE [cm]	σ [cm]
Discrete fwd/back	Keypoint	10.45	20.24	17.34
Discrete fwd/back	Depth	13.01	34.35	31.79
Discrete fwd/back	Fused	0.83	17.16	17.14
Continuous fwd/back	Keypoint	9.68	20.41	17.97
Continuous fwd/back	Depth	8.55	26.80	25.40
Continuous fwd/back	Fused	6.02	21.48	20.61
Lateral	Keypoint	4.09	22.90	22.53
Lateral	Depth	7.22	30.80	29.94
Lateral	Fused	0.61	20.07	20.06

Two results deserve emphasis. First, in the discrete and lateral scenarios the fused mean error drops dramatically—to 0.83 cm and 0.61 cm respectively—because the fusion suppresses depth-camera outliers while retaining close-range depth precision. Second, the continuous-motion scenario is a partial contradiction of the fusion narrative: there the fused RMSE (21.48 cm) and standard deviation (20.61 cm) exceed those of the keypoint-only method (20.41 cm / 17.97 cm), which the authors attribute to the keypoint method being the most stable under fast body position changes. Overall, the paper reports reductions of average error, RMSE, and standard deviation of up to 15.3% across the three scenarios—a figure that reflects the aggregate benefit rather than uniform superiority in every metric.

Outdoor experiments were conducted on the EDU450 in unstructured environments during simulated SAR missions initiated by a GPS-enabled distress request. After GPS-guided localization, the UAV performed visual detection, recognition, tracking, and following at a nominal safe C-B distance of 5 m, within an operating range set to 3–10 m based on the indoor findings. Targets walking, running, crouching, and lying down were followed under varying illumination, wind disturbance, and GPS inaccuracy, with the pipeline sustaining 15 FPS on the Jetson Xavier NX. These outdoor tests are qualitative—no ground-truth distance accuracy is reported for them—which limits the strength of the robustness claims made on their basis.

Limitations and open questions

The paper acknowledges several constraints. The monocular model hard-codes a 1.80 m reference height, so taller or shorter subjects will bias the keypoint-based approximation, and the model fails entirely when a subject is very close to the camera. Outlier rejection relies on a hand-tuned derivative threshold (1.25×) and noise parameters fixed by design; no sensitivity analysis or adaptive tuning is provided. Indoor validation covers distances relevant to the depth camera's optimal range, whereas the claimed extension to 7 m rests on experimental plots without independent ground truth at that range, and outdoor following performance is not quantified against motion capture. Whether the derivative-gated EKF remains stable under the more aggressive dynamics of a gimbal-less UAV in wind is examined only qualitatively.

Conclusion

This work presents a complete, edge-deployable pipeline in which YOLOv11-pose keypoints serve simultaneously as the substrate for monocular C-B distance prediction and as the sampling locus for stereo depth measurement, fused by an EKF with derivative-based outlier gating. Validated against OptiTrack ground truth, the fusion achieves sub-centimeter mean errors in two of three indoor scenarios and extends effective depth sensing beyond the depth camera's optimal range, while running at 15 FPS on an embedded GPU during outdoor UAV following. The results support multi-modal keypoint-centered fusion as a practical distance-estimation subsystem for SAR-grade person following, with quantitative outdoor validation and height-adaptive monocular modeling remaining open questions for subsequent work.