Rapid Learning of Dexterous In-Hand Pen Writing through Real-Time Jacobian Estimation
Abstract: Dexterous in-hand manipulation of a grasped object with an anthropomorphic hand is an unsolved frontier for robot dexterity. The contact-richness and highly dynamic nature of object-hand interactions tend to require extensive modeling or data-collection efforts for learning-based approaches. Modern simulators used for reinforcement learning (RL) cannot fully replicate the required contact complexity, while collecting dexterous demonstrations for imitation learning (IL) remains an open problem. In this research, we present an embodied control approach based on real-time task Jacobian estimation of the combined hand and object system on the physical robot. Using only the CPU on a laptop, the proposed controller begins in-hand pen writing after approximately 18 s of initialization and continues to adapt online, without an analytic hand--object kinematic/contact model, simulation training, or precollected task demonstrations. We demonstrate that the same estimator/controller formulation works on three anthropomorphic robotic hand systems (one physical, two simulated) to show human-like, in-hand articulation of a grasped pen by an embodiment-independent formulation. Sub-millimeter in-plane precision (mean 0.6 mm across runs) is achieved across letters and shapes written in the air and on paper on a physical robot. To our knowledge, this is the first demonstration of an anthropomorphic hand writing arbitrary single-stroke trajectories with a grasped pen through purely in-hand motion, and it showcases an alternative to compute- and data-heavy approaches such as RL and IL for achieving dexterous manipulation through computationally simple and data-efficient algorithms.
Summary
- The paper demonstrates a model-free, data-light method for achieving precise in-hand pen writing by a robotic hand using real-time estimation of a low-dimensional Jacobian 2. The experiment involves the 17-DoF ORCA anthropomorphic hand and tests over 38 configurations
- tracking various shapes, letters, and a 31-minute run to accurately write the alphabet despite disturbing factors.
- Approximately $0.64 ext{mm}$ pooled in-plane tracking error across 38 evaluations on planar surfaces and successful in-hand writing make a successful demonstration of this method
Problem setting and contribution
“Rapid Learning of Dexterous In-Hand Pen Writing through Real-Time Jacobian Estimation” (2609.11775) addresses a specific but demanding instance of dexterous manipulation: articulating a grasped pen through finger motion alone while maintaining a stable contact configuration. The paper’s central claim is that this task can be solved without an analytic hand–object model, simulation-based policy training, or precollected demonstrations. Instead, the controller estimates the local differential mapping from commanded finger motion to observed pen-tip motion directly on the physical system.
The experimental platform is the 17-DoF tendon-driven ORCA anthropomorphic hand. Ten joints belonging to the thumb, index, and middle fingers are controlled; the wrist, ring finger, and little finger remain fixed. A compliant TPU sleeve enlarges the effective pen diameter and improves grasp stability. During writing, the supporting robot arm does not generate the trajectory: it only repositions the hand between letters. The pen is tracked by a webcam using an ArUco marker and a planar marker board, with the resulting trajectory filtered by a constant-velocity Kalman filter.
The evaluation deliberately emphasizes continuous single-stroke trajectories, including geometric shapes, SVG outlines, alphabetic glyphs, and physical writing on paper. The work therefore tests more than isolated waypoint accuracy: the hand must maintain a grasp, adapt to configuration-dependent contact dynamics, and track paths over extended periods.
Figure 1: The ORCA hand writes with a grasped pen using finger motion alone while the arm only repositions the hand between characters.
The principal contribution is an embodied, online alternative to RL and IL for a contact-rich manipulation problem. The paper also releases the implementation and evaluates the same estimator/controller structure on Shadow Hand and Wuji Hand 2 simulations, providing an initial test of embodiment independence.
Online Jacobian estimation and control architecture
The controller estimates a command-space task Jacobian relating commanded finger motion to planar pen-tip motion. For the physical experiments, the task dimension is , while the number of commanded joints is . Locally, the system assumes that pen-tip velocity can be approximated by the current differential relation between joint-command velocity and task-space velocity, even though the underlying hand–object system is nonlinear, contact-dependent, and time-varying.
The Jacobian is estimated using a recursive least-squares update with a forgetting factor. The implementation stores only diagonal covariance terms, reducing computational and memory requirements. Crucially, the regressor is formed from the previous commanded joint increment rather than measured joint velocity. This choice makes the estimate represent the operational command-to-motion mapping, including actuator and tendon-transmission effects; the authors report that measured velocities were too noisy and destabilized estimation in preliminary experiments. Updates are gated so that the estimator does not adapt to observation noise when the fingers are nearly stationary.
The method begins with an approximately 18-second excitation phase. Six manually selected grip poses are connected by a Catmull–Rom spline, generating smooth joint motion while preserving the grasp. This phase is necessary because arbitrary excitation can be uninformative in a closed-chain hand: joint motion may perturb the grasp without sufficiently exciting the pen-tip directions. The excitation is therefore not a generic calibration procedure but a task-specific mechanism for obtaining a well-conditioned initial estimate.
Figure 2: The controller closes a vision-based perceive–estimate–act loop, combining recursive Jacobian estimation, damped inversion, and nullspace grip stabilization.
After excitation, the desired pen-tip motion is generated from a PID-plus-feedforward task-space controller. The estimated Jacobian is inverted with a damped right pseudoinverse. A nullspace posture term pulls the hand toward its initial grip configuration while approximately preserving the task motion. This term is particularly important in a redundant hand: the task has only two controlled dimensions, leaving substantial joint-space freedom that could otherwise cause the fingers to drift and destabilize the pen.
The controller operates at approximately 15 Hz, limited by the perception pipeline rather than the nominal 30 Hz camera rate. Joint increments are velocity-clipped and low-pass filtered before being integrated into position commands. The resulting architecture is computationally lightweight and runs on a laptop CPU.
Experimental accuracy and writing repertoire
The fully configured physical system achieves sub-millimeter in-plane tracking across both free-space and on-paper trials. Across 22 post-excitation air runs, the mean error is approximately mm, with a run-to-run range of $0.39$–$0.86$ mm and a mean 95th-percentile error of approximately $1.4$ mm. The best reported air run reaches approximately $0.39$ mm mean error.
On paper, 16 runs yield approximately mm mean error, with a range of $0.57$–0 mm and a similar 95th-percentile error of approximately 1 mm. The pooled result over 38 fully configured runs is approximately 2 mm. Thus, within the controlled planar action space, paper contact does not substantially degrade lateral tracking.
The paper’s physical writing demonstration includes the word “hello,” with per-letter planar RMSE between 3 and 4 mm. This is a meaningful result because the pen deposits ink while the hand maintains the grasp through finger motion; the arm does not provide assistance during each stroke. However, the paper is deliberately folded or elevated to absorb an uncontrolled vertical drift of approximately 5–6 mm. Consequently, the result establishes planar writing on a compliant surface rather than fully constrained writing on a rigid plane.
Figure 3: On-paper “hello” writing shows consistent sub-millimeter in-plane tracking while the fingers generate every stroke.
The repertoire extends beyond isolated letters. In one uninterrupted run lasting approximately 31 minutes, the hand writes all 26 letters as continuous cubic Bézier glyphs. Across six equal temporal windows, the mean in-plane error remains between 7 and 8 mm, while the 95th percentile remains between 9 and 0 mm. There is no upward accuracy trend despite 26,600 post-excitation control steps.
The run includes a substantial transient during the letter “K”: a sudden finger motion produces an in-plane excursion of approximately 20 mm, exceeding 5 mm for roughly one second. The controller recovers without re-excitation or manual intervention. This result supports the paper’s claim that continuous adaptation is useful not merely for initial identification but also for recovering from disturbances and slow grip changes.
Figure 4: A single approximately 31-minute run covers the full alphabet without recalibration, with stable error apart from a recoverable transient during “K”.
Ablation evidence
The ablations identify two essential components: continued Jacobian adaptation and nullspace grip regularization.
Freezing the Jacobian immediately after excitation performs poorly. Two of three runs diverge within two minutes, including one failure after approximately 40 seconds; the surviving run reaches approximately 1.0 mm mean error. Freezing after only 12 seconds of tracking is also inadequate: one of two runs diverges and the other degrades to approximately 1.1 mm. In contrast, freezing after approximately 30 seconds of online tracking produces 1 mm error with all four runs completing, and the frozen map transfers from a learned circle to unseen letters with errors of 2–3 mm.
This result qualifies the paper’s strongest adaptation claim. Continuous updating is not strictly required once a sufficiently informative task-space map has been learned, but the initial excitation phase alone is insufficient. Continuous adaptation is valuable because it supports operation under disturbances and contact changes that were not present during the brief identification interval.
Removing the nullspace posture term is more damaging to reliability. Across ten runs without grip regularization, four fail outright, two degrade severely, and only four complete near baseline. Catastrophic cases exhibit mean errors above 12 mm and 95th-percentile errors up to approximately 140 mm. The nullspace term therefore functions as a grasp-maintenance mechanism rather than a minor secondary optimization.
Figure 5: Ablation conditions distinguish the effects of freezing the learned Jacobian and disabling nullspace-based grip stabilization.
The speed experiments expose a substantial throughput limitation. The nominal writing speed is only 4 m/s. At twice that speed, one run achieves approximately 0.8 mm error while a repeat reaches 2.3 mm with a 6.6 mm 95th percentile. At three times the nominal speed, the error is approximately 2.1 mm; a four-times run nevertheless achieves approximately 1.1 mm. Progressive speed ramps cause large transients or divergence, with excursions reaching approximately 57 mm. The controller therefore demonstrates precision at deliberately slow motion, not human-like writing speed.
Cross-embodiment simulation results
The same estimator/controller formulation is evaluated in MuJoCo on Shadow Hand and Wuji Hand 2. Unlike the physical experiments, the simulations control all three pen-tip coordinates, including 5, because the simulated setup lacks the compliant TPU sleeve used to stabilize the physical grasp. The simulations also use noise-free ground-truth states, making them more demanding in task dimension but less realistic in sensing.
Shadow Hand achieves a reported RMSE of 6 mm, whereas Wuji Hand 2 achieves 7 mm. The difference indicates that the method is not uniformly insensitive to embodiment: kinematic structure, grasp geometry, and controllability still influence performance. Nevertheless, the same estimator/controller structure operates without an analytic, hand-specific Jacobian on substantially different hand models.
Figure 6: Simulation results extend the method to three-dimensional pen-tip tracking on Shadow Hand and Wuji Hand 2.
The cross-platform evidence is therefore appropriately interpreted as initial portability rather than complete embodiment independence. The physical and simulated evaluations differ in sensing, contact mechanics, task dimension, and pen compliance, so their numerical errors should not be compared as equivalent measures.
Limitations and open questions
The principal physical limitation is that only planar pen-tip position is controlled. Vertical motion remains uncontrolled and reaches approximately 8–9 mm during writing. The on-paper result depends on a deliberately compliant or elevated surface that absorbs this drift; the system cannot currently write reliably on a rigid plane. It also does not support multi-stroke glyphs such as “i” through in-hand motion alone. The robot arm supplies hardcoded repositioning between letters, so coordinated hand–arm writing remains outside the demonstrated control problem.
The formulation assumes a continuous task with an unchanged contact state. It cannot intentionally break and re-establish contact, regrasp the pen, or perform finger gaiting. Such motions would require switching among distinct local mappings and planning across contact discontinuities, which the current RLS estimator does not address.
The reported accuracy is also not independently validated against the deposited ink. Both control and evaluation use the same webcam-based marker-tracking pipeline, so camera calibration error and ArUco jitter are included in the measured trajectory and may bias the absolute accuracy estimate. The physical experiment additionally relies on a custom TPU sleeve and manually selected excitation poses, which reduces the amount of calibration but does not eliminate task-specific setup.
Finally, the comparison with RL-based systems should be interpreted cautiously. The paper reports stronger physical-world evidence than several simulation-only baselines, but the compared systems differ in hand morphology, sensing, trajectory complexity, evaluation protocol, and task definition. The results support the efficiency of the proposed method for this continuous writing task; they do not establish a general superiority over RL or IL across dexterous manipulation.
Conclusion
The paper demonstrates that recursive online estimation of a low-dimensional command-space Jacobian can control a high-DoF anthropomorphic hand during contact-rich in-hand pen manipulation. With approximately 18 seconds of excitation and continued adaptation, the ORCA hand achieves approximately 0 mm pooled in-plane error across 38 physical runs, writes on paper through finger motion alone, and maintains sub-millimeter accuracy over a 31-minute full-alphabet run. The ablations show that informative excitation, online adaptation, and nullspace grip stabilization are all central to reliable operation.
The result is strongest as a carefully constrained demonstration that model-free, data-light local feedback can support precise continuous in-hand manipulation on real hardware. The unresolved questions concern rigid-surface writing, three-dimensional hand-only control, higher speeds, independent physical accuracy measurement, and contact-discontinuous manipulation.
Paper Prompts
Sign up for free to create and run prompts on this paper.
Top Community Prompts
Explain it Like I'm 14
1. What is this paper about?
This paper describes a robot hand that can write with a pen by moving the pen inside its fingers, rather than moving its wrist or arm.
The researchers wanted to solve a difficult robotics problem: getting a robot to move an object held in its hand. Humans do this naturally—for example, we can turn a pencil, adjust a key, or spin a coin using only our fingers. For robots, this is hard because the fingers are constantly touching and pushing the object in complicated ways.
The robot hand in this study learned to write letters and shapes after about 18 seconds of preparation. It used a webcam and an ordinary laptop CPU. It did not need:
- A detailed mathematical model of the hand and pen
- Practice in a computer simulation
- A large collection of human demonstrations
- A powerful computer
2. What questions did the researchers ask?
The main research questions were:
- Can a robot hand learn how its finger movements affect a pen’s movement while it is working?
- Can it use this knowledge to write letters and shapes accurately?
- Can it keep writing even when the way the pen moves changes slightly?
- Can the same method work with different kinds of robotic hands?
- Which parts of the control system are necessary for successful writing?
The researchers were especially interested in whether a simple controller could replace more complicated approaches such as:
- Reinforcement learning, where a robot practices many times, often in simulation, to discover what actions work
- Imitation learning, where a robot learns by copying many human demonstrations
3. How did the researchers do it?
The robot hand and pen
The main robot was the ORCA hand, which has 17 movable joints and is shaped somewhat like a human hand. The researchers controlled 10 joints in the thumb, index finger, and middle finger. The other fingers and the wrist stayed mostly still.
The robot held a pen inside a soft sleeve. This made the pen thicker and easier to grip.
A webcam watched a special visual marker attached to the pen. Other markers on the table helped the camera understand the position of the paper. From the camera images, the computer calculated where the pen tip was.
What is a Jacobian?
A key idea in the paper is the Jacobian. In this study, it means a map showing how small movements of the robot’s finger joints affect the movement of the pen tip.
For example, the robot might discover:
- Moving one finger slightly to the left moves the pen tip upward.
- Moving two fingers together turns the pen.
- Moving another joint has almost no effect on the pen tip.
The researchers did not calculate this map ahead of time. Instead, the robot learned it by moving its fingers and watching how the pen moved.
An everyday analogy is learning to steer a bicycle. At first, you may not know exactly how much to turn the handlebars to move in a certain direction. You try small movements, observe what happens, and gradually improve your understanding.
The preparation phase
Before writing, the robot moved its fingers through six different gripping positions for about 18 seconds. This gave it enough information to begin understanding how its movements affected the pen.
This was called an excitation phase. It is similar to testing a new video game controller by pressing different buttons to see what each one does.
Learning while writing
The robot continued updating its movement map as it wrote. This was important because the pen could shift slightly, the grip could change, or the finger movements could behave differently over time.
The computer compared:
- Where the pen tip was supposed to be
- Where the camera showed that it actually was
It then adjusted the finger movements to reduce the error.
This is called closed-loop control. It is like a person correcting their handwriting while looking at the letters they are drawing.
Keeping the pen secure
The robot had more finger joints than were strictly needed to move the pen in a flat, two-dimensional path. The researchers used these extra movements to keep the hand close to its original gripping position.
This helped stop the fingers from slowly losing their grip on the pen. The researchers called this nullspace grip stabilization. In simple terms, the robot moved some joints in ways that helped hold the pen without greatly changing the pen’s intended path.
Testing the system
The researchers asked the robot to:
- Draw circles, hearts, stars, flowers, and other shapes
- Write individual letters
- Write the word “hello”
- Write all 26 letters of the alphabet in one long run
- Write both in the air and on paper
They also tested the method in simulation with two other robotic hands: the Shadow Hand and the Wuji Hand 2.
4. What did the researchers find?
The robot wrote accurately
Across its tests, the robot’s average error in the flat writing direction was about 0.64 millimeters. That is less than the width of a small pencil mark.
The results were similar when writing:
- In the air: about 0.62 mm average error
- On paper: about 0.67 mm average error
This shows that touching the paper did not seriously reduce the robot’s accuracy.
The robot also completed a continuous writing run lasting about 31 minutes, including all 26 letters. Its accuracy did not get worse over time.
The robot could recover from mistakes
During the alphabet test, the robot made a large mistake while writing the letter K. The pen moved far away from the planned path for a short time, but the controller corrected itself and continued writing.
This is important because real-world robots will sometimes experience unexpected movements or small slips.
Continuous learning was important
The researchers tested what happened when the robot stopped updating its movement map.
If the map was frozen too early:
- Some tests failed completely.
- Other tests became less accurate.
- The robot sometimes moved away from the writing path.
When the robot continued learning while writing, it was more reliable and could adjust to changes such as a small shift in the grip.
Grip stabilization was important
When the researchers removed the part of the controller that helped maintain the grip, the robot often lost the pen or became much less accurate.
Out of 10 tests without grip stabilization, several failed or became badly inaccurate. This showed that moving the pen correctly is not enough—the robot must also keep holding the pen securely.
The robot was slow
The robot wrote carefully at a slow speed. When the researchers tried to make it write two or three times faster, its accuracy became worse and some tests failed.
Therefore, the system was precise but not yet fast enough to write like a human.
The method worked with other robot hands
In simulations, the same basic method worked with two different robotic hands. However, their accuracy was different:
- Shadow Hand: about 0.17 mm error
- Wuji Hand 2: about 1.48 mm error
This suggests that the method can adapt to different hand designs without requiring a completely new mathematical model for each one.
5. Why are these findings important?
Robotic hands usually need a lot of information before they can perform difficult tasks. Researchers may have to:
- Build a detailed model of the robot and object
- Create a realistic simulation
- Collect many examples of people performing the task
- Use powerful computers for training
This paper shows another possibility: a robot can learn the important movement relationship directly from its own actions and keep improving while it works.
The approach is useful because it is:
- Fast to start: the robot needs only about 18 seconds of preparation
- Data-efficient: it does not need many demonstrations
- Computationally simple: it runs on a laptop CPU
- Adaptable: it can respond to changes in the grip and pen movement
- Accurate: it achieved sub-millimeter average error in two dimensions
6. Limitations and possible future impact
The system still has important limitations.
The robot mainly controlled the pen’s movement across the paper. It did not fully control how high or low the pen moved. To deal with this, the researchers used slightly raised or flexible paper. Because of this, the robot could not yet write reliably on a completely rigid, flat surface.
The robot also needed an arm to move from one letter to the next. The fingers created each individual stroke, but the arm repositioned the hand between letters. The system could not yet easily perform tasks where it releases and re-grasps the pen.
The writing speed was also much slower than human handwriting. Future versions might need better cameras, faster control, improved friction handling, and control of the hand and arm together.
Overall, the paper suggests that robots may be able to learn difficult hand skills without huge amounts of training data or complicated models. This could be useful for robotic assistants, prosthetic hands, and other machines that need to manipulate objects carefully using their fingers.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
The paper demonstrates a capable but narrowly defined in-hand writing system. The following issues remain unresolved:
- Generalization across physical robot embodiments is not established. The method is tested on only one physical hand; the Shadow Hand and Wuji Hand 2 results are simulation-only and benefit from noise-free state measurements.
- Sim-to-real transfer for the additional hand platforms is unverified. It remains unknown whether the reported performance on the simulated Shadow and Wuji hands persists under actuator backlash, tendon compliance, sensing noise, contact uncertainty, and hardware wear.
- The method’s dependence on a manually designed excitation trajectory is unclear. The six grip-pose waypoints are selected manually and reused for the ORCA hand, but the paper does not determine how excitation should be generated automatically for a new hand, grasp, object, or workspace.
- The minimum excitation required for reliable Jacobian identification is not quantified. The effects of excitation duration, waypoint number, trajectory conditioning, grasp configuration, and initialization on convergence and failure probability remain unexplored.
- The validity region of the estimated Jacobian is not characterized theoretically or experimentally. The paper reports successful tracking but does not establish bounds on Jacobian error, configuration change, contact variation, or task velocity under which the controller remains stable.
- The estimator’s simplifying assumptions are not systematically evaluated. The diagonal covariance approximation, fixed observation-noise variance, covariance floor, forgetting factor, and use of commanded rather than measured joint increments may substantially affect performance, but their sensitivity is not reported.
- The relationship between command-space and physical joint-space dynamics remains unresolved. Because the estimator uses commanded increments as regressors, it may absorb actuator and transmission effects without distinguishing them from changes in hand–object contact mechanics; the consequences for transfer and interpretability are unknown.
- The controller does not control the pen’s out-of-plane position on the physical robot. The approximately $2$–$3$ mm drift prevents reliable writing on rigid planar surfaces and limits the method to compliant or specially prepared paper.
- Three-dimensional in-hand manipulation on hardware has not been demonstrated. Although 3-D tracking is evaluated in simulation, the paper does not show whether real-time estimation can control position and orientation of a tool in all relevant Cartesian dimensions under physical contact.
- Pen orientation and rotational dynamics are not measured or controlled. The evaluation focuses on planar pen-tip position, leaving unresolved whether the pen maintains the required writing angle, roll, pitch, and yaw during motion.
- The writing surface is highly specialized. The folded or raised paper is used to absorb vertical drift, so performance on rigid paper, rough surfaces, inclined surfaces, curved surfaces, or surfaces with varying friction is unknown.
- The role of the compliant TPU pen sleeve is not isolated. It is unclear how much of the robustness comes from the proposed Jacobian controller versus the enlarged, compliant grip interface, and whether comparable results are possible with ordinary pens or objects of different sizes and materials.
- Robustness to grasp variation is not established. The experiments use a fixed power-precision grip and a manually recorded initial posture; performance under different grasp poses, grip forces, object placements, or intentional perturbations is not evaluated.
- Contact changes and discontinuous manipulation remain unsupported. The method cannot currently handle pen regrasping, finger gaiting, object repositioning, contact breaking, or re-establishing contact, leaving open how Jacobian estimation should be reset, blended, or made hybrid across contact modes.
- The speed–accuracy–stability trade-off is insufficiently characterized. The speed ablation uses few trials at selected speed multipliers and produces inconsistent outcomes; the maximum reliable speed, causes of failure, and dependence on trajectory curvature and contact friction remain unknown.
- Human-level writing performance is not assessed. The paper does not compare writing speed, legibility, stroke smoothness, force, or completion time against human handwriting or established robotic writing systems.
- The evaluation is limited in statistical power and reproducibility. Many ablations contain only two to four trials, and the reported means exclude diverged runs; confidence intervals, randomization procedures, trial-to-trial variability, and independent repetitions across days or operators are not provided.
- Failure modes are not systematically modeled. The paper reports divergence, grip loss, and large tracking excursions, but does not identify their causal contributions from perception errors, actuator saturation, friction, contact transitions, estimator instability, or controller tuning.
- The effect of perception quality is not quantified. ArUco jitter is acknowledged as an accuracy bottleneck, but the paper does not measure how camera resolution, frame rate, lighting, occlusion, marker placement, filtering delay, or calibration error affect tracking and stability.
- Independent ground-truth validation is missing. Accuracy is measured through the same vision pipeline used for control, so the reported sub-millimeter errors may include correlated sensing and evaluation errors; independent motion capture, instrumented pen-tip tracking, or ink-deposition measurements are needed.
- Written-output quality is not evaluated directly. The study does not quantify ink continuity, stroke width, pressure consistency, line quality, legibility, or deviations between the commanded trajectory and the deposited physical trace.
- The method’s workspace and trajectory-complexity limits are unclear. The experiments use trajectories within a manually selected reachable range, but do not identify the maximum workspace, minimum feature size, curvature limits, self-intersection behavior, or performance on discontinuous and highly irregular paths.
- Multi-stroke writing and full handwriting remain unresolved. The arm provides hardcoded repositioning between letters, while the hand cannot independently control vertical lift and placement; coordinated hand–arm control for words, punctuation, dots, crossings, and arbitrary stroke sequences is left for future work.
- The controller’s posture regularization may restrict dexterity. Pulling the joints toward a fixed initial grip could prevent useful posture changes or cause failure when the task requires large configuration changes; adaptive posture objectives and contact-force-aware nullspace control are not investigated.
- Contact forces and grip stability are not measured. The controller maintains the grasp indirectly through posture regularization, but no tactile, force, or torque measurements establish whether the pen is close to slipping, damaging the surface, or exceeding safe contact loads.
- Parameter transfer across tasks is not demonstrated. It is unknown whether the same PID gains, damping, forgetting factor, covariance settings, velocity limits, and posture gain work across different pens, surfaces, writing speeds, trajectories, and hand embodiments.
- The computational and latency claims are incomplete. Although the method runs on a laptop CPU, the paper does not report end-to-end sensing, filtering, estimation, communication, and actuation latency or evaluate how performance changes at different control rates.
- Long-term reliability and hardware wear are untested. A single approximately 31-minute run does not establish robustness over repeated sessions, tendon stretch, joint backlash, sleeve deformation, marker displacement, battery variation, or mechanical degradation.
- Comparisons with competing approaches are not controlled. The discussion compares results from different robots, sensing modalities, task definitions, and simulation or hardware settings, making it difficult to attribute performance differences to the control method itself.
- The claimed embodiment independence is not fully supported. The formulation is shared across platforms, but the physical setup, controlled dimensions, sensing assumptions, grasp mechanics, and parameter choices differ; a systematic cross-platform benchmark is still needed.
- The applicability beyond pen writing is unresolved. It remains unknown whether the approach can manipulate objects with different shapes, masses, inertias, friction coefficients, compliance, or visual observability, and whether it can perform tasks involving forces rather than only tool-tip position tracking.
- No formal safety or recovery mechanism is provided. The controller can produce large excursions, including reported deviations of tens of millimeters, but does not include explicit collision avoidance, force limits, grasp-loss detection, emergency recovery, or safe shutdown criteria.
Practical Applications
Immediate Applications
- Low-cost deployment of dexterous manipulation research platforms — robotics and academia. Laboratories can use the released Jacobian-estimation controller as a baseline for in-hand manipulation without building a detailed hand–object model, collecting demonstrations, or training an RL policy. A webcam, visual marker, CPU laptop, and an anthropomorphic or tendon-driven hand are sufficient for initial experiments. Potential workflow: attach a visual marker to a grasped tool, run an approximately 18-second excitation sequence, estimate the command-space Jacobian online, and execute planar trajectories with closed-loop visual feedback. Dependencies: the robot must expose sufficiently reliable joint commands and measurements; the object must remain continuously grasped; the camera must provide stable tracking; and the excitation poses must be manually selected so that both task-space directions are adequately excited.
- Robotic pen plotting, labeling, and template tracing — manufacturing and office automation. The system can immediately support slow, planar drawing tasks such as writing labels, tracing inspection marks, producing signatures or symbols, and drawing single-stroke shapes. The paper reports approximately $0.64$ mm mean in-plane error across air and paper experiments, including continuous alphabet writing. Potential products: a compact robotic plotter based on a dexterous hand, automated marker or stylus systems, and tools for tracing SVG outlines onto paper or workpieces. Dependencies: current demonstrations are slow, primarily two-dimensional, and use a compliant surface to absorb approximately $2$–$3$ mm of out-of-plane drift. Rigid-surface writing, high-speed handwriting, and independently verified ink accuracy remain unresolved.
- Tool-tip control for simple in-hand tool operations — industrial robotics. The same control architecture can be adapted to markers, styluses, brushes, probes, screwdrivers, or other continuously grasped tools when the desired task is expressed as a visible tool-tip trajectory. Rather than requiring a calibrated hand–tool model, the controller can learn the local relationship between finger commands and tool motion during operation. Potential workflow: use vision to track the tool tip, define a planar or three-dimensional path, initialize the estimator through safe grip excitation, and maintain grasp stability with nullspace posture regularization. Dependencies: this is an extrapolation from pen writing rather than a demonstrated industrial deployment. Tool geometry, friction, visibility, contact forces, and payload must be compatible with stable continuous grasping.
- Rapid prototyping of robot-specific controllers — robotics software. Engineers can use the open-source implementation as a lightweight alternative to simulation-heavy RL or imitation-learning pipelines when adapting manipulation control to a new hand. The method is embodiment-independent in formulation and was demonstrated in simulation on Shadow Hand and Wuji Hand 2 in addition to the physical ORCA hand. Potential tools: ROS or MuJoCo packages for online recursive-least-squares Jacobian estimation, damped task-space control, automatic excitation, and nullspace grip stabilization. Dependencies: the reported cross-embodiment evidence is limited; simulated systems use noise-free state information, whereas physical deployment requires robust perception and actuator interfaces.
- Teaching and benchmarking dexterous manipulation — academia and education. Single-stroke letters, geometric shapes, and SVG trajectories provide interpretable benchmarks for evaluating grasp stability, online adaptation, path tracking, and long-horizon robustness. Researchers can compare controllers using mean error, 95th-percentile error, recovery from disturbances, initialization time, and computational requirements. Dependencies: comparisons should control for camera quality, marker tracking, writing speed, hand morphology, task dimensionality, and whether errors are measured independently of the controller’s own perception system.
- Onboard control in compute-constrained robots — embedded robotics. Because the controller runs on a laptop CPU and does not require a learned policy or GPU-scale training, it can be considered for small mobile manipulators, educational robots, or embedded dexterous systems with limited computational resources. Dependencies: the current approximately 15 Hz perception-limited loop may be inadequate for fast manipulation or highly dynamic contact. Embedded deployment would require validating timing, numerical stability, safety limits, and camera performance under real operating conditions.
- Robotic assistance for repetitive planar drawing or writing — accessibility and daily-life automation. A dexterous hand could execute predefined symbols, simple notes, diagrams, or tactile/visual markings while the arm provides gross positioning and the fingers provide fine motion. This could support assistive writing devices or household robots that need to use pen-like tools. Dependencies: the paper does not demonstrate user-specific handwriting, multi-stroke characters, autonomous paper handling, or safety around people. Arm–hand coordination and reliable -axis control would be necessary for practical products.
Long-Term Applications
- General-purpose dexterous tool use — manufacturing, logistics, and service robotics. Extending the estimator from pen-tip tracking to task-relevant tool features could enable robots to manipulate brushes, cutters, inspection probes, utensils, and hand tools without a hand-specific analytic model. The approach is particularly promising for tasks where the contact state remains continuous but the exact hand–object mechanics are difficult to model. Required development: multi-object and multi-tool initialization, automatic excitation planning, force or tactile feedback, uncertainty-aware estimation, and recovery policies for slipping or partial loss of grasp. The present method cannot handle contact discontinuities such as regrasping or finger gaiting.
- Full hand–arm coordinated writing and drawing — office, education, and assistive robotics. A future system could use the arm for coarse placement and paper-to-paper transitions while the fingers execute detailed strokes. This would remove the current restriction to single-stroke, in-plane motion and support characters such as “i,” diagrams, signatures, and arbitrary documents. Required development: hierarchical allocation of motion between arm and hand, three-dimensional pen-tip control, rigid-surface contact regulation, automatic lifting and regrasping, and coordination during page repositioning. Human-level speed and sustained accuracy remain unvalidated.
- Adaptive grasping and manipulation of unknown objects — general-purpose robotics. The online Jacobian concept could provide a model-light mechanism for learning how a newly grasped object responds to finger commands. This could reduce dependence on object-specific CAD models, demonstrations, or extensive simulation. Required development: object recognition and feature tracking, online detection of changes in contact mode, safe exploration, force/tactile sensing, and explicit handling of discontinuous transitions. The current estimator assumes a locally continuous mapping and a fixed grasp configuration.
- Prosthetic and wearable hand control — healthcare and rehabilitation. The paper explicitly identifies onboard prosthetics as a possible application. A prosthetic or assistive hand could learn the local mapping between motor commands and the motion of a held object, potentially compensating for uncertain socket alignment, tendon transmission, or user-specific mechanics without a full analytic model. Required development: human-safe adaptation, low-power hardware, user-intent interfaces, failure detection, individualized calibration, and extensive clinical testing. The present experiments do not establish suitability for biological users, skin-contact safety, or medical reliability.
- Adaptive robotic surgery and minimally invasive tool manipulation — healthcare. In principle, online task-Jacobian estimation could help control tool tips when instrument mechanics, compliance, or contact conditions are difficult to model. A model-light controller may be useful for specialized continuously grasped instruments. Required development: force-limited control, sterilizable sensing, deterministic safety guarantees, redundancy management, very low latency, independent sensing of tool position, and regulatory validation. The slow planar pen task is not evidence that the method is ready for surgical use.
- High-speed handwriting and human-like manipulation — consumer robotics and human–robot interaction. Improving estimation, perception, and friction compensation could lead to robots that write, sketch, sign, or draw at practical speeds. The experiments indicate that increasing speed produces nonlinear degradation: even a twofold increase sometimes raised error substantially, while progressive speed ramps caused large transients. Required development: higher-frame-rate vision, better markerless or marker-based tracking, predictive dynamics, friction and contact-force compensation, adaptive speed scheduling, and robust disturbance recovery.
- Markerless, multimodal dexterous control — field robotics and consumer products. Replacing ArUco markers with natural visual features, tactile sensing, or sensor fusion could make the controller usable with ordinary objects and tools. Tactile feedback could also reveal grip pressure, slip, and contact changes that are invisible to the current webcam-based system. Required development: robust feature tracking under occlusion, calibration-independent 3D perception, tactile–visual fusion, uncertainty estimation, and validation under changing lighting, backgrounds, and object materials.
- Policy and funding guidance for efficient robot learning — public-sector research and industrial strategy. The results support evaluating lightweight adaptive control alongside RL and imitation learning, particularly when compute, demonstrations, or accurate simulation are unavailable. Research programs could prioritize algorithms that learn online from embodied interaction and report initialization time, energy use, hardware requirements, and failure recovery—not only final task accuracy. Assumptions and caveats: the evidence comes from a specialized pen-writing benchmark, a small number of runs, one physical hand, and simulations with privileged state information. Policy conclusions should therefore treat the method as a promising low-resource research direction rather than proof that model-free adaptive control broadly outperforms RL or imitation learning.
Glossary
- Ablation: An experiment that removes or changes one component of a system to measure its effect. “Ablation tests that compare the in-plane tracking metrics and robustness”
- Anthropomorphic: Designed to resemble the structure or function of a human body part. “an anthropomorphic, tendon-driven hand”
- ArUco marker: A visually detectable fiducial marker used for camera-based pose estimation. “A webcam tracks an ArUco marker attached to the pen”
- Catmull–Rom spline: A smooth interpolating curve that passes through specified control points. “the hand is swept through six grip poses along a Catmull--Rom spline”
- Closed kinematic chain: A mechanical structure containing interconnected joints that form one or more loops. “the fingers and grasped object form multiple closed kinematic chains”
- Command-space task Jacobian: A Jacobian that maps commanded joint changes to changes in task-space variables. “the command-space task Jacobian estimation”
- Contact-rich manipulation: Manipulation involving numerous or continuously changing physical contacts. “contact-rich in-hand writing”
- Covariance matrix: A matrix representing the uncertainty and correlations of estimated quantities. “We approximate the covariance matrix by a diagonal matrix”
- Damped pseudo-inverse: A regularized approximation of a matrix inverse used to improve numerical stability. “invert the current estimate with a damped (Tikhonov) right pseudo-inverse”
- Degree of freedom (DoF): An independent variable of motion available to a mechanical system. “the $17$-DoF, tendon-driven, anthropomorphic ORCA hand”
- Domain randomization: A simulation technique that varies environmental or model parameters to improve transfer to reality. “relies on heavy domain randomization to close the residual sim-to-real gap”
- Egocentric: Viewed or recorded from the perspective of the acting person or agent. “the egocentric human video increasingly used to supplement robot data”
- Embodiment: The particular physical form, hardware, or morphology through which a robot performs a task. “by an embodiment-independent formulation”
- Excitation phase: A deliberate period of input motion intended to generate informative data for system identification. “We therefore begin each run with a short excitation phase”
- Feedforward: A control action based on the desired motion rather than only on current error. “The task-space command ... is a PID-plus-feedforward law”
- Forgetting factor: A factor in online estimation that reduces the influence of older observations. “the covariance is first inflated by a forgetting factor”
- Forward Jacobian: A Jacobian that maps actuator or joint motion to task-space motion. “An online forward Jacobian estimated from observed joint and pen-tip motion”
- Grasp regularization: A control mechanism that constrains a robot’s configuration to maintain a stable grasp. “The writing result rests on two controller choices ... and a nullspace grip regularization”
- Image Jacobian: A Jacobian relating robot motion or camera motion to changes in image features. “estimated the image Jacobian online with a Kalman filter”
- In-hand manipulation: Repositioning or articulating an object using the fingers while holding it. “contact-rich in-hand manipulation”
- Innovation-based outlier rejection: A filtering method that discards observations whose prediction errors are unusually large. “a constant-velocity Kalman filter with innovation-based outlier rejection”
- Kalman filter: A recursive estimator that combines predictions and noisy measurements to infer a changing system state. “The resulting pen-tip trajectory is smoothed using a constant-velocity Kalman filter”
- Kinematic chain: A sequence of rigid links connected by joints. “forming multiple closed kinematic chains in a redundant structure”
- Least-squares estimator: An estimator that minimizes the squared discrepancy between observed and predicted values. “a recursive least-squares (RLS) / Kalman-filter-style problem”
- Low-pass filter: A filter that attenuates rapid changes while preserving slower variations. “before being integrated into the absolute joint-position command”
- Nullspace: The set of inputs that produce no motion in the controlled task variables. “the posture can be adjusted while minimizing interference with the task”
- Nullspace operator: A matrix that projects commands into a task Jacobian’s nullspace. “adding a posture stabilization term projected through an approximte nullspace operator”
- Online estimation: Estimation performed continuously as new data become available during system operation. “The online task Jacobian estimation follows the formulation”
- Recursive least squares (RLS): An online least-squares method that updates parameter estimates recursively as observations arrive. “the canonical recursive-least-squares (RLS) estimator”
- Redundant structure: A robotic system with more controllable degrees of freedom than task dimensions require. “a redundant, tendon-driven in-hand system”
- Regularization: The addition of constraints or penalty terms to improve numerical stability or prevent undesirable solutions. “The damping regularizes the inverse”
- Sim-to-real gap: The difference between performance in simulation and performance on physical hardware. “to close the residual sim-to-real gap”
- Synergy space: A lower-dimensional representation of coordinated joint or tendon motions. “control in task-specific, tendon-derived synergy spaces”
- Task Jacobian: A matrix describing the local differential relationship between joint or actuator velocities and task-space velocities. “Locally, joint and task velocities are related by the task Jacobian”
- Task space: The space of variables describing the desired behavior or pose of an object or robot endpoint. “a direct map from task space to command (joint) space”
- Tendon-driven: Actuated by tensioned cables or tendons rather than by motors located directly at each joint. “an anthropomorphic, tendon-driven hand”
- Tikhonov regularization: A form of regularization that stabilizes an ill-conditioned inverse by adding a weighted identity term. “a damped (Tikhonov) right pseudo-inverse”
- Trajectory retargeting: Adapting motion data generated for one body or robot to another with different geometry or kinematics. “The human motion data must also be retargeted to robot hands with different kinematics”
- Visuomotor Jacobian: A Jacobian relating visual observations to robot motor or actuator commands. “a visuomotor Jacobian field defined on all 3D points along the robot”
- Visual servoing: Robot control that uses visual measurements as feedback. “visual servoing of robot arms”
- Zero-shot calibration: Operation without a prior calibration procedure tailored to the specific task or system. “without an analytic hand--object kinematic/contact model, simulation training, or precollected task demonstrations”