Papers
Topics
Authors
Recent
Search
2000 character limit reached

Whole Body Model Predictive Control for Spin-Aware Quadrupedal Table Tennis

Published 9 Oct 2025 in cs.RO, cs.SY, and eess.SY | (2510.08754v1)

Abstract: Developing table tennis robots that mirror human speed, accuracy, and ability to predict and respond to the full range of ball spins remains a significant challenge for legged robots. To demonstrate these capabilities we present a system to play dynamic table tennis for quadrupedal robots that integrates high speed perception, trajectory prediction, and agile control. Our system uses external cameras for high-speed ball localization, physical models with learned residuals to infer spin and predict trajectories, and a novel model predictive control (MPC) formulation for agile full-body control. Notably, a continuous set of stroke strategies emerge automatically from different ball return objectives using this control paradigm. We demonstrate our system in the real world on a Spot quadruped, evaluate accuracy of each system component, and exhibit coordination through the system's ability to aim and return balls with varying spin types. As a further demonstration, the system is able to rally with human players.

Summary

  • The paper demonstrates a novel whole-body MPC approach that integrates model-based and learned spin estimation to enable agile table tennis play by quadrupedal robots.
  • It employs synchronized high-speed perception and an optimization-based aiming planner to accurately predict ball trajectories and compute precise paddle motions.
  • Experimental results show a 90.1% return rate and robust performance under varied spin conditions, marking significant advancements in robotic agility and control.

Whole-Body Model Predictive Control for Spin-Aware Quadrupedal Table Tennis

Introduction and Motivation

This work presents a comprehensive system for enabling a quadrupedal robot, specifically a Boston Dynamics Spot equipped with a 6-DoF arm, to play table tennis with spin-aware capabilities. The system integrates high-speed perception, model-based and learned spin estimation, trajectory prediction, a real-time aiming planner, and a novel whole-body model predictive control (MPC) formulation. The primary technical challenge addressed is the coordination of agile, whole-body motion with precise paddle control to return balls with varying spin and speed, a task that has previously been limited to fixed-base or gantry robots. Figure 1

Figure 1: Dynamic whole-body quadruped swinging. Robot states are shown in gray with the paddle trajectory in blue.

System Architecture

The system is composed of four main subsystems: perception, prediction, aiming, and control. The architecture is designed to operate in real time, with low-latency feedback loops and robust integration between modules. Figure 2

Figure 2: System diagram with RGB cameras (wire frame pyramids) for ball detection, predicted trajectory (green), strike plane (blue), motion capture cameras (black), and target landing location (red).

Perception

  • Hardware: Two ceiling-mounted PoE RGB cameras (1400x1080 @ 165 fps) provide stereo vision.
  • Calibration: Intrinsic parameters from manufacturer; extrinsic parameters via semi-automated PnP using table corners.
  • Detection Pipeline:
    • Background subtraction isolates moving objects.
    • A fine-tuned YOLO CNN detects the ball in cropped patches, reducing inference time to 1.5 ms per frame.
    • Stereo triangulation yields 3D ball position with median error <1 cm (see Figure 3).
    • Figure 3

    • Figure 3: Distribution of the RGB position measurement error, calculated against data from a Vicon motion capture system.

Trajectory Prediction and Spin Estimation

  • Model-Based Estimation: Ball velocity and spin are estimated by fitting polynomials to position history and solving for spin via least squares, using aerodynamic models (drag and Magnus effects).
  • Learned Residuals: A neural network is trained on 650 trajectories to correct for unmodeled effects, improving R2R^2 from 0.42 (model-only) to 0.70 (model+residual).
  • Performance: Spin estimation error converges to <55 rad/s within 150 ms of strike (see Figure 4). Figure 5

    Figure 5: Spin prediction performance: model-based estimate (left), residual network (right).

    Figure 4

    Figure 4: Prediction output errors through ball flight with median and IQR. The 7.5 cm threshold corresponds to the paddle radius.

Aiming Planner

  • Formulation: Given predicted ball state, the planner solves for paddle position, velocity, and orientation to achieve a desired landing location and outgoing spin.
  • Optimization: SQP with OSQP as the local QP solver, leveraging CasADi for code generation. The cost function penalizes paddle velocity and enforces constraints on ball landing, contact dynamics, and aerodynamics.
  • Convergence: Achieves accurate solutions for diverse shot types (drive, loop, chop) within 1 ms and four SQP iterations (see Figure 6). Figure 6

    Figure 6: Convergence of aiming planner to a given pland\mathbf{p}_\text{land}, ω+\omega^+, and tlandt_\text{land}.

Whole-Body Model Predictive Control

  • Kinematic Planner: Uses parametric Bezier curves for joint trajectories, enabling continuous enforcement of strike constraints. The cost function regularizes acceleration and deviation from rest pose.
  • Whole-Body Controller: Solves a QP for feedforward torques subject to full robot dynamics and friction cone constraints, using OSQP.
  • Execution: The combined planner/controller runs at 100 Hz, with five SQP iterations per cycle.
  • Emergent Behavior: Diverse human-like stroke strategies (loop, drive, chop) emerge from the optimization (see Figure 7). Figure 7

Figure 7

Figure 7

Figure 7

Figure 7

Figure 7: High Loop swing type, illustrating the system's ability to generate top spin.

Experimental Evaluation

Perception and Prediction

  • Ball Localization: Median 3D error <1 cm compared to Vicon ground truth.
  • Spin Estimation: Residual network improves return rates and prediction accuracy, especially for high-spin balls.

Swing Controller

  • Workspace Analysis: Evaluated over 3750 simulated strikes; position error <7.5 cm, velocity error <1 m/s, orientation error <20° in the core workspace (see Figure 8). Figure 8

    Figure 8: Position, velocity, and orientation error of the paddle at strike for loop, chop, and drive shots.

System-Level Performance

  • Aiming: 90.1% return rate over 150 trials, with clear spatial separation of landing locations for different targets (see Figure 9).
  • Spin Handling: System can return incoming spin up to 280 rad/s and generate outgoing spin up to 200 rad/s, exceeding prior work.
  • Ablation: Full residual spin estimation yields a mean return rate of 75.2%, compared to 27.2% (no spin estimation) and 52.0% (model-only).
  • Spin Generation: Achieves target outgoing spin within 20 rad/s (see Figure 10). Figure 9

    Figure 9: Landing locations when aiming to the right, center, and left. Shots originated from Spot on the right.

    Figure 10

Figure 10

Figure 10: Back spin (top) and top spin (bottom) hardware swing examples with the exiting ball trajectory shown.

Human-Robot Rally

  • The system is capable of sustaining rallies with human players, handling both incoming and outgoing spin dynamically.

Implications and Future Directions

Practical Implications

  • Generalization: The system demonstrates that legged robots can achieve agile, spin-aware table tennis play without reliance on fixed bases or gantries, broadening the applicability of robotic sports systems.
  • Emergent Control: The MPC formulation enables a continuous spectrum of stroke strategies, reducing the need for hand-crafted motion primitives or shot categorization.
  • Real-Time Performance: The architecture achieves low-latency, high-frequency control suitable for fast-paced, dynamic tasks.

Theoretical Implications

  • Integration of Model-Based and Learned Components: The combination of physics-based models with learned residuals for spin estimation provides a robust approach to handling complex, partially modeled dynamics.
  • Optimization-Based Whole-Body Control: The use of Bezier-parameterized trajectories and QP-based whole-body control demonstrates a scalable method for dynamic, contact-rich manipulation tasks.

Limitations and Future Work

  • Perception: Current reliance on off-board cameras; onboard perception for high-speed tasks remains an open challenge.
  • Prediction: Incorporating opponent motion for anticipatory spin estimation is a promising direction.
  • Control: The current controller does not include stepping; integrating RL-based footwork with MPC swing planning is a key next step.
  • Strategy: More sophisticated, adaptive strategies are needed for competitive play against advanced opponents.

Conclusion

This work establishes a new benchmark for agile, spin-aware table tennis on legged robots, integrating high-speed perception, hybrid model-based/learned prediction, real-time aiming, and whole-body MPC. The system achieves high accuracy in ball localization, spin estimation, and shot execution, with emergent human-like stroke diversity and robust performance in both isolated and interactive settings. The architecture and methods presented are extensible to other dynamic, contact-rich manipulation tasks in robotics.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Explain it Like I'm 14

Whole-Body Model Predictive Control for Spin-Aware Quadrupedal Table Tennis — Explained Simply

1. What is this paper about?

This paper shows how a four-legged robot (a Boston Dynamics Spot with a robot arm) can play table tennis in a fast and realistic way. The robot uses cameras to spot the ball, predicts where the ball will go (including how it spins), decides how to hit it, and moves its whole body to swing a paddle accurately—fast enough to rally with a human.

2. What questions did the researchers ask?

In simple terms, they asked:

  • How can a robot “see” a tiny, super-fast ping pong ball in real time?
  • How can it predict the ball’s path when the ball is spinning?
  • How should the robot aim its paddle to send the ball to a chosen spot with the right spin?
  • How can a legged robot move its whole body quickly, stay balanced, and swing accurately—like a human player?

3. How did they do it?

To make all this happen, the team built a system with four main parts. Here’s a short overview before we explain each part:

  • High-speed cameras to locate the ball in 3D
  • A prediction model that figures out the ball’s future path and spin
  • An aiming planner that decides the best paddle position, speed, and angle
  • A whole-body controller that moves the robot smoothly and powerfully

Now, let’s break down the key ideas with everyday analogies:

  • Seeing the ball: Two fast ceiling cameras act like the robot’s “eyes.” They take many pictures per second (165 fps) and use a small, speedy AI detector (YOLO) to find the ball in each image. Using both cameras together lets the robot calculate the ball’s 3D position, like how your two eyes help you judge distance.
  • Predicting the ball’s path and spin: Spin makes a ping pong ball curve in the air (this is the “Magnus effect”). The robot uses a physics model to estimate how air and spin change the ball’s motion. Then, it adds a small learned correction (“residual network”) to handle messy real-world effects. Think of it like following the rules of a game (physics) but also learning shortcuts from experience to be more accurate.
  • Aiming the hit: The robot chooses how to hold and move the paddle—where the paddle should be, how fast it should move, and the direction the paddle should face—so the ball lands at a specific spot with the desired spin. This is solved with a fast math tool that tries different options and quickly improves them (called SQP—like making a plan, checking it, then refining it a few times).
  • Moving the whole body to swing: The robot doesn’t just swing its arm; it coordinates its whole body to stay balanced and powerful, like a human player shifting weight during a stroke. They use “Bezier curves” (smooth path shapes) to plan the motion, which helps hit the ball at exactly the right moment. A “whole-body controller” then calculates the torques (the robot version of muscle forces) for all joints so the movement obeys physics and the robot doesn’t slip.

The cool part: Different human-like strokes (drive, loop for topspin, chop for backspin) naturally appear when the robot aims for different landing spots and spins. The system doesn’t need a separate controller for each shot type—those strategies emerge from the planning.

4. What did they find, and why does it matter?

Here are the main results, explained simply:

  • Accurate ball tracking: The cameras locate the ball with a median error under 1 cm. That’s precise enough to hit a 4 cm ball with a 15 cm paddle at high speed.
  • Better spin prediction with learning: Adding the small learned correction (“residual”) made spin prediction much more accurate than using physics alone. This helps the robot plan smarter swings because spin strongly affects flight and bounce.
  • Hitting the ball to chosen targets: In 150 test shots aimed at three different spots, the robot returned 90% of balls, showing reliable aim.
  • Handling and creating spin like a player:
    • It successfully returned incoming spins up to about 280 rad/s (very fast).
    • It could add outgoing spin up to about 200 rad/s.
    • When they compared systems, not estimating spin led to poor returns (average ~27%), while using physics-only was better (~52%), and the physics + learned residual was best (~75%). This proves spin estimation really matters.
  • Human-like stroke variety: The robot performed loops (topspin), drives (flatter shots), and chops (backspin), and could rally with people—including dealing with opponents’ spin.

Why this is important: Making a legged robot play such a fast sport requires super quick “see-think-act” skills. This system shows how to tie together vision, physics, learning, planning, and full-body control to handle a real-world, high-speed task—something useful far beyond table tennis.

5. What’s the impact and what comes next?

This research shows a path to more athletic, agile robots that can react quickly and precisely. That could help in:

  • Sports training robots,
  • Home and workplace assistants that can handle fast, unpredictable motions,
  • Better robot control methods for other dynamic tasks (like catching, throwing, or using tools).

The authors mention a few next steps:

  • Onboard vision: Right now, the cameras are external. Putting cameras on the robot itself would be harder—but more practical.
  • Reading the opponent: Humans watch the other player’s body to guess spin. Teaching robots to use opponent motion could make them smarter and more strategic.
  • Footwork: This version doesn’t include stepping during swings. Combining learning-based motion with their planner could add agile footwork like a human player.
  • Game strategy: The system uses simple aiming rules today. More advanced strategies could help the robot compete at higher levels.

Overall, this paper is a big step toward legged robots that can play fast, strategic sports by seeing, predicting, and moving their whole bodies with skill.

View Paper Prompt  View All Prompts 

Knowledge Gaps

Knowledge gaps, limitations, and open questions

Below is a single, concrete list of gaps and open questions that remain unresolved and are actionable for future research:

  • Reliance on off-board RGB cameras: No onboard perception was attempted; assess feasibility of head-mounted/event cameras and radar/LiDAR for high-speed ball tracking under robot-induced motion, occlusions, and variable lighting.
  • Camera time synchronization and latency: Stereo triangulation accuracy depends on frame alignment; the paper does not quantify inter-camera sync, timestamping, or end-to-end perception latency/jitter and their impact on strike timing.
  • Extrinsic calibration robustness: Extrinsics are derived from manual table corner annotations; explore self-calibrating, online methods (factor graphs, AprilTags) that adapt to camera drift, table movement, or reconfiguration without manual intervention.
  • Occlusion handling: No strategy for ball occluded by the robot, paddle, opponent, or net; develop multi-view fusion, predictive tracking, or occlusion-aware filters to maintain accurate 3D ball state.
  • Detection domain generalization: YOLO was fine-tuned on composite patches; evaluate generalization to different venues, backgrounds, ball colors/branding, lighting, and motion blur, and investigate lightweight continual adaptation.
  • End-to-end latency budget: The paper reports partial module runtimes but not the full pipeline delay from photons to torques; quantify and reduce system-level latency and jitter, and analyze sensitivity of return accuracy to delays.
  • Spin estimation assumption (constant spin): The inference assumes ω\omega is constant during flight; investigate models that allow time-varying spin (e.g., due to air drag or micro-turbulence) and quantify error reduction vs. added complexity.
  • Limited spin dimensionality: Most experiments focus on topspin/backspin (ωy\omega_y); extend estimation, aiming, and control to full 3D spin including sidespin and compound spin, and evaluate accuracy and returns under mixed spin.
  • Aerodynamic model fidelity: CDC_D, CMC_M are treated as lumped constants; incorporate Reynolds-number dependence, ball-to-ball variability, and environmental factors (humidity, air currents) and measure gains in trajectory prediction.
  • Residual network dataset scale and diversity: Spin residual was trained on ~650 trajectories with z-axis rotations; assess generalization across higher speeds/spins (approaching 600 rad/s), different balls, paddles, venues, and opponents, and identify minimum data size for reliable performance.
  • Ground-truth spin acquisition mismatch: Spinsight uses custom marked balls for ground-truth; evaluate whether models trained with markings generalize to regulation unmarked balls and quantify any domain gap.
  • Table rebound/contact modeling: Contact uses a modified Nonomura model; calibrate and validate against the robot’s specific paddle (rubber type, compliance, friction anisotropy) and different table surfaces; incorporate uncertainty bounds.
  • Fixed strike plane: The strike plane is set 0.5 m in front of the base; explore adaptive strike plane selection coupled with body repositioning to maximize workspace and timing flexibility.
  • Choice of landing time (tlandt_\text{land}): The aiming planner takes tlandt_\text{land} as an input but does not describe how it is selected; develop principled selection (e.g., optimization over arrival time) and evaluate effects on success rate and rally tempo.
  • Planner robustness to state uncertainty: The aiming solver assumes accurate terminal ball state; add chance constraints or robust MPC that explicitly models ball state uncertainty and perception errors, and measure improvements in return reliability.
  • Bezier-based kinematic MPC sampling: Joint limits and foot constraints are enforced at sampled times; quantify risk of constraint violations between samples and design adaptive sampling or continuous-time certificates for safety.
  • Paddle orientation constraint stability: Orientation tracking shows non-uniform errors; investigate alternative orientation parameterizations (quaternion cost, geodesic distances), solver warm-starts, trust-region SQP, or convex relaxations to improve reliability.
  • Whole-body controller omissions: No explicit actuator torque/rate limits or motor thermal models; incorporate realistic bounds and evaluate tracking degradation, saturation events, and safety margins on hardware.
  • Contact-implicit dynamics at strike: The controller does not model contact shock, paddle compliance, or energy transfer; instrument the paddle (force/IMU) and build closed-loop impact models to reduce post-contact error and improve spin/velocity control.
  • Footwork and stepping: Feet are held stationary, limiting workspace and recovery; design and evaluate hybrid stepping controllers (RL+MPC), contact scheduling, and gait changes synchronized with swing to expand playable regions.
  • Robot state estimation: Figure suggests motion capture for robot pose; clarify reliance on Vicon for control vs. evaluation and develop fully onboard state estimation (IMU/kinematics/vision) with quantified accuracy during fast swings.
  • Strategy learning: Returns and spins are selected via simple heuristics; build adaptive, opponent-aware strategy (e.g., reinforcement learning, game-theoretic planning) and quantify match outcomes against varied human skill levels.
  • Rally-level benchmarking: Provide comprehensive rally metrics (rally length distributions, unforced errors, forced errors, serve/receive performance, placement accuracy under pressure) against human baselines to contextualize system competence.
  • Extreme operating envelope: Demonstrations peak at ~280 rad/s incoming and ~200 rad/s outgoing spin; evaluate performance near human elite levels (10 m/s, ~600 rad/s) and characterize failure modes and needed hardware/software upgrades.
  • Environment robustness: Assess performance under changing lighting, reflective surfaces, moving backgrounds (spectators), and airflow; develop methods to detect and compensate for environmental disturbances.
  • Safety and failure recovery: No discussion of mis-hit detection, fail-safe behaviors, or human-safe control during unexpected contacts; design impact sensing, rapid abort strategies, and safety envelopes for human-in-the-loop play.
  • Multi-ball/multi-object scenarios: The perception pipeline does not address spurious detections or multiple balls; develop robust association, false-positive suppression, and fallbacks when the paddle or opponent confuses the detector.
  • Transferability to other platforms: The approach was shown on Spot with a 6-DoF arm; specify requirements and adaptations for different legged/bimanual platforms, including controller portability and hardware constraints.
  • Computational profiling: MPC and QP runtimes are partially reported; provide full profiling (CPU/GPU load, worst-case runtime, solver failures), and explore faster solvers or model reductions for higher control rates.
  • Net and table-edge interactions: The model ignores collisions with the net or table edges; integrate detection/prediction for near-net trajectories and develop aiming/planning to avoid net hits while maintaining aggressive play.
  • Energy efficiency and endurance: No analysis of battery usage, thermal limits, or long-run reliability during continuous rallies; benchmark endurance and implement energy-aware planning to sustain play.
View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

Below is a concise list of deployable applications that leverage the paper’s demonstrated capabilities—high-speed perception (165 fps stereo RGB), spin-aware trajectory prediction (model + learned residual), real-time aiming via SQP, and whole-body MPC with Bezier-curve kinematic planning and a QP-based dynamic controller. Each item includes sector alignment, potential tools/workflows, and key assumptions/dependencies.

  • Sports training partner for table tennis clubs and universities
    • Sector: sports tech, education
    • What it looks like: A turnkey training robot (Spot or similar legged manipulator with paddle end-effector) that returns balls to user-selected landing zones with programmable outgoing spin (e.g., loop/chop/drive), supports rallying, and logs performance metrics for coaching.
    • Tools/workflows: Stereo RGB camera rig + PnP-based calibration; spin-aware prediction module; aiming planner (CasADi + OSQP); whole-body MPC controller; operator UI to set targets (p_land, ω⁺, t_land).
    • Assumptions/Dependencies: External high-speed cameras; instrumented environment with known table geometry; fixed-foot swing (no stepping); sufficient safety perimeter; availability of a legged platform with 6-DoF arm.
  • Spin-aware ball analytics module (software component)
    • Sector: sports analytics, software
    • What it looks like: A library that estimates ball spin from stereo trajectories by combining an analytical model (drag + Magnus) with a learned residual network (improved R² from 0.42 to 0.70), enabling accurate flight prediction and spin tagging during drills or matches.
    • Tools/workflows: YOLO-based detection on cropped patches; stereo triangulation; parameter identification for C_D/C_M; residual network trained on ball trajectory curvature; plug-ins for coaching dashboards.
    • Assumptions/Dependencies: Calibrated cameras; consistent ball properties and aerodynamic coefficients; adequate trajectory samples (≥30 points before spin estimation) for robust inference.
  • Bezier-curve kinematic MPC for continuous strike constraints
    • Sector: robotics software (general manipulation)
    • What it looks like: A generic kinematic planning module for tasks that require a constraint at a specific time along a horizon (e.g., timed impacts, brushing/painting strokes, precise tapping), avoiding discrete-node limitations of standard collocation/shooting formulations.
    • Tools/workflows: CasADi-generated optimization; SQP with sampled constraint enforcement along Bezier curves; acceleration and posture regularization costs.
    • Assumptions/Dependencies: Accurate robot kinematics; fixed contact points during motion (e.g., feet stationary); pre-specified strike plane/time; sufficient CPU for 100 Hz re-planning.
  • Whole-body QP controller for fast arm swings on legged manipulators
    • Sector: industrial robotics, legged manipulation
    • What it looks like: Feedforward torque solver that enforces dynamics and friction-cone constraints during high-speed arm motions without stepping (e.g., rapid tool strokes, tapping inspection tasks, short transient contact maneuvers).
    • Tools/workflows: QP (OSQP) with mass/Coriolis/gravity terms, pyramidal friction cone approximation; end-effector acceleration tracking.
    • Assumptions/Dependencies: Reliable ground contact; friction parameters known; no large base motion; actuator bandwidth adequate.
  • High-speed stereo RGB detection pipeline for small fast objects
    • Sector: computer vision, manufacturing QA, research instrumentation
    • What it looks like: Low-latency (≈4 ms) stereo detection/tracking pipeline for fast-moving spherical objects; can trigger devices (e.g., launchers, gates), collect trajectory data for labs, or monitor drills in sports facilities.
    • Tools/workflows: Background subtraction → composite patch → fine-tuned YOLO; triangulation; lightweight compute; ceiling-mounted PoE RGB cameras.
    • Assumptions/Dependencies: Stable lighting; minimal occlusion; adequate ceiling height and FOV; reliable calibration via known landmarks (e.g., table corners).
  • Integrated research benchmark for perception–prediction–control
    • Sector: academia
    • What it looks like: A reproducible system architecture and metrics (e.g., <1 cm median localization error; ability to handle incoming spin up to 280 rad/s and generate up to 200 rad/s) to stress-test end-to-end robotics stacks under tight latency and accuracy constraints.
    • Tools/workflows: Component-wise evaluation (Vicon, Spinsight Elite for GT), ablation of spin modules, grid sweeps for paddle pose/velocity/orientation error.
    • Assumptions/Dependencies: Access to validation hardware; reproducible calibration; compatible robot platforms.
  • Live entertainment and museum demos
    • Sector: entertainment, public outreach
    • What it looks like: Interactive exhibits where a legged robot rallies with visitors, demonstrating agile perception and control with configurable difficulty via spin and placement.
    • Tools/workflows: Safety barriers and supervised mode; pre-set rally scripts; operator console.
    • Assumptions/Dependencies: Venue safety approvals; reliable external sensing; robust operator procedures and emergency stops.

Long-Term Applications

The following applications are feasible with further research, scaling, or development—especially in onboard perception, agile footwork (stepping), generalized object dynamics, and adaptive game strategy.

  • Fully mobile table tennis opponent with agile footwork and onboard perception
    • Sector: consumer robotics, sports tech
    • What it looks like: A legged robot that repositions (steps) dynamically, senses the ball with onboard cameras/event sensors under body occlusions, and plays full-table rallies against humans at competitive levels.
    • Tools/workflows: Hybrid RL + MPC (RL for footwork, MPC for swing); onboard multi-modal sensing (event cameras, depth); contact-implicit planning or learned stepping policies.
    • Assumptions/Dependencies: Robust real-time contact planning; high-bandwidth onboard compute and sensors; safe human–robot interaction standards; cost reductions for wider adoption.
  • Cross-sport expansion (badminton, tennis)
    • Sector: sports training, robotics
    • What it looks like: Extending spin-aware prediction, aiming, and whole-body MPC to sports with different aerodynamics (e.g., shuttlecock drag, tennis ball mass), larger courts, and higher speeds, including footwork and strategy.
    • Tools/workflows: New aerodynamic/contact models; expanded perception coverage; mobile base planning; sport-specific paddles/rackets and safety tooling.
    • Assumptions/Dependencies: Accurate sport-specific dynamics; larger sensing area; integrated locomotion control; regulatory compliance for public venues.
  • Industrial interceptive manipulation (catching/redirecting fast, possibly spinning items)
    • Sector: manufacturing, logistics
    • What it looks like: Robots that intercept and redirect items in high-throughput lines (e.g., defect removal, dynamic buffering), leveraging spin-aware prediction for robust timing and contact control.
    • Tools/workflows: Generalized dynamics models for non-spherical items (tumbling, non-uniform mass); robust grippers/end-effectors; perception systems adapted to factory conditions.
    • Assumptions/Dependencies: Safety and reliability certifications; domain-specific training data; environmental constraints (lighting, occlusions); integration with MES/SCADA.
  • Assistive therapy and neuro-rehabilitation via exergaming
    • Sector: healthcare
    • What it looks like: Therapeutic protocols where patients interact with a robot that can tailor spin, speed, and placement to train reaction time, prediction, coordination, and cognitive planning.
    • Tools/workflows: Clinical-grade safety systems; adaptive difficulty models; outcome tracking; clinician interfaces.
    • Assumptions/Dependencies: Clinical validation trials; human factors research; reimbursement frameworks; soft robotics and risk mitigation around patients.
  • Human–robot strategy learning and opponent motion-based spin prediction
    • Sector: AI/ML, sports analytics
    • What it looks like: Controllers that infer opponent spin and tactics from body motion (pose estimation) and deploy game-theoretic strategies (placement, spin mix, temporal cues) to optimize rally outcomes.
    • Tools/workflows: Multi-camera pose estimation; differentiable factor graphs; policy learning with opponent modeling; online adaptation.
    • Assumptions/Dependencies: Robust, low-latency human pose tracking under occlusions; labeled datasets; privacy and consent protocols in public play.
  • Autonomous sports officiating/analytics for coaching and broadcast
    • Sector: sports analytics, media
    • What it looks like: Systems that estimate spin and trajectory to annotate rallies (e.g., spin rates, curve types), assist officiating (net touch, off-table trajectories), and inform coach/player feedback.
    • Tools/workflows: Edge analytics; standardized camera rigs; integration with scoring systems and broadcast overlays.
    • Assumptions/Dependencies: Verified accuracy standards; league adoption; latency constraints; data governance.
  • Educational curricula and simulation toolkits for modern MPC/optimization
    • Sector: education
    • What it looks like: Course modules and sandbox simulators teaching Bezier-based kinematic MPC, SQP, QP whole-body control, and latency-aware perception pipelines applied to interceptive tasks.
    • Tools/workflows: ROS packages; simulated camera pipelines; open datasets; reproducible labs with low-cost platforms (wheeled bases + arms).
    • Assumptions/Dependencies: Open-source releases; accessible hardware; instructor training.
  • Safety standards and policy for high-speed legged robots interacting with humans
    • Sector: policy/regulation
    • What it looks like: Guidelines for sensing redundancy, speed caps, emergency stops, crowd separation, and certification pathways for public interaction.
    • Tools/workflows: Hazard analysis frameworks; test protocols; compliance documentation.
    • Assumptions/Dependencies: Multi-stakeholder collaboration; empirical safety data; clear regulatory paths.
  • Consumer-grade sports robots (cost-reduced platforms)
    • Sector: consumer robotics, sports tech
    • What it looks like: Affordable, compact mobile robots (likely wheeled base + arm) that offer spin-aware returns at home or in clubs with minimal setup.
    • Tools/workflows: Camera-in-a-box kits, simplified calibration workflows; cloud updates for models; user-friendly mobile apps.
    • Assumptions/Dependencies: Hardware cost reduction; robust onboard perception; simplified safety features; reliable maintenance networks.
View Paper Prompt  View All Prompts 

Glossary

  • Aerodynamics: The study of air-related forces affecting a moving ball, modeled in flight integration. " faerof_\text{aero} corresponds to the discrete aerodynamics in \eqref{eq:finite_difference_ball_dynamics} "
  • Bezier curves: Parametric curves defined by control points, used to create smooth, time-parameterized joint trajectories. "our kinematic planner uses parametric Bezier curves rather than discrete nodes."
  • CasADi: A software framework for fast symbolic and numeric optimization and optimal control. "All optimization formulations in this paper were generated using CasADi for fast function evaluation \cite{casadi}."
  • Coriolis terms: Velocity-dependent forces in robot dynamics that arise from motion in a rotating or curvilinear coordinate system. "where M\mathbf{M}, C\mathbf{C}, and τg\boldsymbol{\tau}_\text{g} are the mass matrix, Coriolis terms, and gravity terms respectively of the general robotic manipulator equations."
  • Direct collocation: An optimal control discretization method enforcing dynamics at nodes, challenging for continuous constraints. "This proves challenging for MPC formulations like multiple shooting or direct collocation because of their discrete dynamics"
  • Feedforward torques: Torques computed from a model to achieve desired accelerations without relying on feedback. "This can be accomplished using a whole-body controller which finds feedforward torques u\mathbf{u} given dynamics constraints"
  • Forward kinematics: The mapping from joint angles to end-effector positions in space. "KfK_f represents the forward kinematics of each foot in equation \eqref{eq:kin_foot} which keeps them stationary throughout the swing."
  • Friction cone: The set of feasible contact forces constrained by friction limits, often approximated for computation. "To keep the constraints linear, the friction cone FC\mathcal{FC} is approximated using a pyramidal approach."
  • Ground reaction forces (GRF): Contact forces exerted by the ground on the robot’s feet. "The ground reaction forces are also solved for and denoted by λ∈R12\boldsymbol{\lambda} \in \mathbb{R}^{12}"
  • Jacobian: A matrix of partial derivatives relating joint velocities to Cartesian velocities/forces. "while J\mathbf{J} is the Jacobian of the robot feet with respect to qq."
  • Magnus effect: Lift force produced by a spinning ball moving through air, altering its trajectory. "where CDC_D and CMC_M represent the lumped coefficients of the drag and Magnus effects respectively."
  • Mass matrix: The inertia matrix in robot dynamics that maps accelerations to forces. "where M\mathbf{M}, C\mathbf{C}, and τg\boldsymbol{\tau}_\text{g} are the mass matrix, Coriolis terms, and gravity terms respectively of the general robotic manipulator equations."
  • Model predictive control (MPC): Optimization-based control that plans over a prediction horizon to achieve goals under constraints. "a novel model predictive control (MPC) formulation for agile full-body control."
  • Multiple shooting: An optimal control method that discretizes trajectories into segments with continuity constraints. "MPC formulations like multiple shooting or direct collocation"
  • OSQP: An operator-splitting solver for quadratic programs used for fast QP solutions. "we utilize OSQP~\cite{osqp} and perform four SQP iterations before using the solution."
  • Perspective-n-Point (PnP): A computer vision method to estimate camera pose from known 3D points and their 2D projections. "the extrinsic calibration (rotation and translation) for each camera is calculated by solving the Perspective-n-Point (PnP) problem~\cite{pnp}."
  • Quadratic Program (QP): An optimization problem with a quadratic objective and linear constraints. "Together, this problem is a simple Quadratic Program and easily solved with off-the-shelf solvers, in this case OSQP \cite{osqp}."
  • Residual network: A learned model that predicts corrections to an analytical or model-based estimate. "we trained a residual network on 650 unique ball trajectories with varying spin."
  • Sequential Quadratic Programming (SQP): An iterative method that solves nonlinear optimization via a sequence of quadratic subproblems. "we implemented Sequential Quadratic Programming (SQP) where we take a quadratic approximation of our cost with linearized constraints"
  • Skew-symmetric matrix: A matrix A with A⊤=−AA^\top = -A, used to represent cross products in linear algebra. "By rearranging using the skew-symmetric matrix into equation \eqref{eq:skew_sym_simplification}"
  • Stereo triangulation: Recovering 3D position by intersecting rays from two calibrated camera views. "its 3D position P=[X,Y,Z]TP = [X, Y, Z]^T in the world coordinate frame is calculated using stereo triangulation"
  • Strike plane: A fixed plane in front of the robot where the ball is expected to be struck. "The integration stops when the ball reaches a fixed strike plane located \SI{0.5}{\meter} in front of the quadruped base."
  • Vicon motion capture system: An optical tracking system used to obtain precise ground-truth positions. "We evaluate the accuracy of our ball localization and prediction systems using ground truth position data from a Vicon motion capture system"
  • Whole-body controller: A controller that computes joint torques for the entire robot subject to full dynamics and contact constraints. "This can be accomplished using a whole-body controller which finds feedforward torques u\mathbf{u} given dynamics constraints"
  • YOLO (You Only Look Once): A fast convolutional neural network architecture for real-time object detection. "A fine-tuned YOLO convolutional neural network (CNN)~\cite{yolov3,darknet_yolo} is applied to this composite patch to detect the ball's 2D pixel coordinates."
View Paper Prompt  View All Prompts 

Open Problems

We found no open problems mentioned in this paper.

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 107 likes about this paper.

Sign Up for Free