TeleSim: Modular Teleoperation Systems

• TeleSim is a modular teleoperation framework that uses a digital-twin interface and hardware-in-the-loop benchmarking to transform operator intent into precise robot motions.
• It supports diverse controller modalities—such as VR controllers and finger-mapping devices—with ROS 2 integration and RMPflow-based motion planning for various robotic arms.
• Quantitative evaluations reveal that system performance is highly sensitive to network parameters and ergonomic configurations, underscoring its role in advancing teleoperation research.

Searching arXiv for TeleSim-related papers to ground the article in the latest relevant literature. arxiv_search(query="TeleSim teleoperation digital twin robotic arm network-aware testbed", max_results=10) arXiv search results retrieved. Proceeding with the supplied TeleSim papers and the matched arXiv records. TeleSim—capitalized as TELESIM in the digital-twin teleoperation papers and as TeleSim in the network-aware benchmark paper—denotes a set of telerobotic systems organized around an explicit mediation layer between operator intent and robot execution. In one usage, it is a modular and plug-and-play framework for direct teleoperation of a robotic arm using a digital twin as the interface between the user and the robotic system. In another, it is a hardware-in-the-loop teleoperation platform and benchmark dataset designed to assess performance under controlled network degradation. Across these usages, the recurring technical motifs are modularity, real-time mediation, and instrumented evaluation of teleoperation performance (Audonnet et al., 2023, Audonnet et al., 2024, Deng et al., 6 Jul 2025).

1. Scope and nomenclature

In the cited arXiv literature, the name has been used for two related but non-identical research artifacts: a digital-twin-mediated teleoperation framework for robotic manipulators, and a network-aware testbed plus dataset for evaluating teleoperation under adverse communication conditions. The overlap is substantive rather than merely nominal: both systems treat teleoperation as a pipeline in which user action is transformed, mediated, and evaluated before or during execution on physical hardware (Audonnet et al., 2023, Audonnet et al., 2024, Deng et al., 6 Jul 2025).

arXiv id	Object	Core components
(Audonnet et al., 2023)	Digital-twin teleoperation framework	Isaac Sim, RMPFlow, ROS 2, Baxter, UR3
(Audonnet et al., 2024)	Extended empirical analysis of the framework	Baxter, UR3, UR5e, NASA-TLX, SEQ, partial NARS
(Deng et al., 6 Jul 2025)	Network-aware teleoperation testbed and dataset	Lite 6, RealSense D435i, OMNeT++, 300 trials

The earlier line of work is centered on non-expert robotic-arm teleoperation through a high-fidelity digital twin. The later benchmark-oriented line is centered on communication impairments, especially bandwidth, latency, jitter, and packet loss. A plausible implication is that the shared name marks a broader research program concerned with end-to-end teleoperation rather than a single fixed software stack.

2. Digital-twin architecture and motion generation

The digital-twin framework is organized as three main layers. The Controller Layer accepts any device that can output a 3D pose $X_{\mathrm{cont}}(t)$ in user space; the reported instances are a Valve Index VR controller and a SenseGlove plus Vive tracker. The Digital-Twin Motion Planning Layer is implemented in NVIDIA Isaac Sim, a 3D ray-tracing simulator running RMPFlow, which takes the controller pose as a goal for the end-effector and computes collision-aware joint commands. The Robot Control Layer is a generic ROS 2 control interface that streams these commands to the physical manipulator, reported as a UR3 or Baxter in the original study (Audonnet et al., 2023).

A later architectural description presents the same pipeline as Telecontrollers $\rightarrow$ Digital Twin $\rightarrow$ Real robot. In that formulation, TeleSim runs in NVIDIA Isaac Sim as a plug-and-play digital twin; it subscribes to controller pose topics in ROS 2, updates a virtual “target cube” at the commanded 6D pose, and invokes an RMPflow-based real-time motion planner with collision avoidance. A ROS 2-Control driver then sends the resulting joint commands to the real arm, with Baxter interfaced via ros1_bridge and UR3/UR5e via the Universal Robots ROS 2 driver. The framework is explicitly modular: controllers, digital twin, planner, and robot driver are independent ROS 2 nodes, and new arms or controllers are introduced by changing URDFs and remapping topics (Audonnet et al., 2024).

The motion-generation step is described in optimization form as

$$\min_{\ddot{\mathbf q}} \tfrac12\,\ddot{\mathbf q}^\top \mathbf W\,\ddot{\mathbf q} \quad\text{s.t.}\quad \mathbf J(q)\,\ddot{\mathbf q} + \dot{\mathbf J}(q,\dot q)\,\dot{\mathbf q} = \ddot{\mathbf x}_\text{goal}, \quad \text{collision-free},$$

where $\mathbf J$ is the Jacobian mapping joint accelerations to end-effector accelerations and $\mathbf W$ encodes actuator weights. The later paper also gives a schematic RMPflow relation,

$$\dot q = H(q)^{-1} \bigl(J(q)^\top \Lambda (u - f(q))\bigr),$$

with $H(q)$ the Riemannian metric on joint space, $\Lambda$ the task-space metric, $J(q)$ the Jacobian, and $\rightarrow$0 the forward-kinematic pose. In that presentation, the mapping $\rightarrow$1 is effected implicitly by the RMPflow planner in Isaac Sim (Audonnet et al., 2023, Audonnet et al., 2024).

3. Controllers, mappings, and end-effectors

Two controller modalities define the core interaction strategies in the digital-twin framework. The first is a VR Controller, specifically a Valve Index tracked 6-DoF wand whose pose in the headset frame,

$\rightarrow$2

is transformed by a user-defined offset into the target pose for the digital twin. In the Baxter configuration, the wand’s main trigger closes the gripper. The later cross-study description generalizes this configuration to Valve Index VR hand-controller pairs for both Baxter and UR5e (Audonnet et al., 2023, Audonnet et al., 2024).

The second modality is a Finger-Mapping Hardware Controller, implemented with a SenseGlove and an HTC Vive Tracker. In the original account, the SenseGlove has two active DoFs, thumb and index, and each finger’s metacarpal rotation $\rightarrow$3 is linearized into a single-DoF command for the modified Yale OpenHand T42 gripper. In the later description, thumb–index closure is mapped to the two fingers of the T42 gripper, while hand pose is mapped to the UR3 end-effector pose (Audonnet et al., 2023, Audonnet et al., 2024).

The physical actuation path for the finger-mapping controller is unusually explicit. Two Arduino UNOs relay position commands via I²C to MX-28 Dynamixels and report current finger loads back to ROS 2, with torque limited to prevent cable breakage. The protection constraint is stated as

$\rightarrow$4

The grasping systems compared in the original study are Baxter’s parallel two-finger gripper and UR3’s underactuated T42 gripper with one motor per finger driven by SenseGlove angles $\rightarrow$5. Haptic feedback was deliberately omitted to ensure fairness across controllers (Audonnet et al., 2023).

These mappings also expose a recurrent design trade-off. The VR controller provides direct 6-DoF spatial command for gross arm motions, while the SenseGlove provides additional hand-level DoF for gripper control. The empirical record suggests that the added granularity of finger mapping came with ergonomic and grasp-strength costs in the reported UR3 configuration (Audonnet et al., 2023, Audonnet et al., 2024).

4. Task design and quantitative performance on robotic manipulation

The principal behavioral evaluation of the digital-twin framework is a block-stacking task. In the original study, 37 non-expert participants teleoperated the robot to pick and place 3 cubes in a tower and repeated the task as many times as possible in 10 minutes after only 5 minutes of training. The cubes were arranged in an isosceles pattern and the tower was to be built at a central red square of side length 4 cm. The recorded variables were the number of towers $\rightarrow$6, time per tower $\rightarrow$7, and action-level outcomes such as place, drop, and collapse; after each robot session, subjects gave a Single-Ease Question score $\rightarrow$8, where higher values indicated greater ease (Audonnet et al., 2023).

The first reported results showed that most users were able to build at least a tower of 3 cubes regardless of control modality or robot. For Baxter with VR, 85% of participants built at least one tower, with final $\rightarrow$9. For UR3 with finger mapping, 50% built at least one tower, with $\rightarrow$0. Average time per tower decreased over successive attempts, asymptoting to approximately 100 s on Baxter and approximately 240 s on UR3. Action-level outcomes further separated the two conditions: Baxter achieved a placing rate of $\rightarrow$1, dropping rate of $\rightarrow$2, collapse rate of $\rightarrow$3, and still-in-place rate of $\rightarrow$4; UR3 achieved a placing rate of $\rightarrow$5, dropping rate of $\rightarrow$6, collapse rate of $\rightarrow$7, and still-in-place rate of $\rightarrow$8. The corresponding SEQ ratings were $\rightarrow$9 and $$\min_{\ddot{\mathbf q}} \tfrac12\,\ddot{\mathbf q}^\top \mathbf W\,\ddot{\mathbf q} \quad\text{s.t.}\quad \mathbf J(q)\,\ddot{\mathbf q} + \dot{\mathbf J}(q,\dot q)\,\dot{\mathbf q} = \ddot{\mathbf x}_\text{goal}, \quad \text{collision-free},$$0 (Audonnet et al., 2023).

The later empirical paper enlarged the comparison to three robot/controller combinations and two national cohorts. In the UK study the design was within-subjects; in Japan it was between-subjects under similar conditions. The three reported conditions were Baxter with VR controller and default grippers, UR3 with SenseGlove and Yale T42 gripper, and UR5e with VR controller and Robotiq 2F-140 gripper (Audonnet et al., 2024).

Condition	Mean towers built	SEQ (1–7)
Baxter + VR	3.25	$$\min_{\ddot{\mathbf q}} \tfrac12\,\ddot{\mathbf q}^\top \mathbf W\,\ddot{\mathbf q} \quad\text{s.t.}\quad \mathbf J(q)\,\ddot{\mathbf q} + \dot{\mathbf J}(q,\dot q)\,\dot{\mathbf q} = \ddot{\mathbf x}_\text{goal}, \quad \text{collision-free},$$1
UR3 + SenseGlove	1.03	$$\min_{\ddot{\mathbf q}} \tfrac12\,\ddot{\mathbf q}^\top \mathbf W\,\ddot{\mathbf q} \quad\text{s.t.}\quad \mathbf J(q)\,\ddot{\mathbf q} + \dot{\mathbf J}(q,\dot q)\,\dot{\mathbf q} = \ddot{\mathbf x}_\text{goal}, \quad \text{collision-free},$$2
UR5e + VR	10.50	$$\min_{\ddot{\mathbf q}} \tfrac12\,\ddot{\mathbf q}^\top \mathbf W\,\ddot{\mathbf q} \quad\text{s.t.}\quad \mathbf J(q)\,\ddot{\mathbf q} + \dot{\mathbf J}(q,\dot q)\,\dot{\mathbf q} = \ddot{\mathbf x}_\text{goal}, \quad \text{collision-free},$$3

The UR5e condition dominated the later study’s tower metric: all users built at least 3 towers, and the fastest users built more than 20. UR5e also exceeded Baxter and UR3 in placing and still-in-place rates; its stability was significantly better than Baxter at $$\min_{\ddot{\mathbf q}} \tfrac12\,\ddot{\mathbf q}^\top \mathbf W\,\ddot{\mathbf q} \quad\text{s.t.}\quad \mathbf J(q)\,\ddot{\mathbf q} + \dot{\mathbf J}(q,\dot q)\,\dot{\mathbf q} = \ddot{\mathbf x}_\text{goal}, \quad \text{collision-free},$$4, while UR3 was significantly worse than both at $$\min_{\ddot{\mathbf q}} \tfrac12\,\ddot{\mathbf q}^\top \mathbf W\,\ddot{\mathbf q} \quad\text{s.t.}\quad \mathbf J(q)\,\ddot{\mathbf q} + \dot{\mathbf J}(q,\dot q)\,\dot{\mathbf q} = \ddot{\mathbf x}_\text{goal}, \quad \text{collision-free},$$5. Time per tower also separated the systems: UR5e users averaged less than 100 s per tower and improved over time, reaching 40 s per tower for experts, whereas Baxter and UR3 rarely reached less than 100 s except for top performers. The SEQ scores correlated with the number of towers built (Audonnet et al., 2024).

Qualitative comments from the original study complement these metrics. Most users described teleoperation via the digital twin as “intuitive” with the VR controller, particularly for gross arm motions. Some reported that a 500 ms end-effector lag was initially disconcerting but adapted by moving more slowly. SenseGlove users valued the extra DoF but also reported “accidental drops” linked to limited gripper strength and the lack of direct haptic feedback (Audonnet et al., 2023).

5. Workload, trust, and cross-cultural evaluation

The later TeleSim study extended the evaluation from task completion to workload and trust. It used the standard raw-NASA-TLX, the Single-Ease Question, and a partial NARS S1 questionnaire covering negative attitudes to robot-interaction situations. The six NASA-TLX subscales were Mental Demand, Physical Demand, Temporal Demand, Performance, Effort, and Frustration (Audonnet et al., 2024).

The reported workload pattern was strongly condition-dependent. Mental demand was lowest for UR5e and significantly less than for Baxter and UR3 at $$\min_{\ddot{\mathbf q}} \tfrac12\,\ddot{\mathbf q}^\top \mathbf W\,\ddot{\mathbf q} \quad\text{s.t.}\quad \mathbf J(q)\,\ddot{\mathbf q} + \dot{\mathbf J}(q,\dot q)\,\dot{\mathbf q} = \ddot{\mathbf x}_\text{goal}, \quad \text{collision-free},$$6. Physical demand was highest for UR3 and significantly greater than for Baxter and UR5e at $$\min_{\ddot{\mathbf q}} \tfrac12\,\ddot{\mathbf q}^\top \mathbf W\,\ddot{\mathbf q} \quad\text{s.t.}\quad \mathbf J(q)\,\ddot{\mathbf q} + \dot{\mathbf J}(q,\dot q)\,\dot{\mathbf q} = \ddot{\mathbf x}_\text{goal}, \quad \text{collision-free},$$7. Temporal demand showed no significant difference, which was attributed to the uniform time limit. Perceived performance was lowest for UR3 at $$\min_{\ddot{\mathbf q}} \tfrac12\,\ddot{\mathbf q}^\top \mathbf W\,\ddot{\mathbf q} \quad\text{s.t.}\quad \mathbf J(q)\,\ddot{\mathbf q} + \dot{\mathbf J}(q,\dot q)\,\dot{\mathbf q} = \ddot{\mathbf x}_\text{goal}, \quad \text{collision-free},$$8, effort was highest for UR3 at $$\min_{\ddot{\mathbf q}} \tfrac12\,\ddot{\mathbf q}^\top \mathbf W\,\ddot{\mathbf q} \quad\text{s.t.}\quad \mathbf J(q)\,\ddot{\mathbf q} + \dot{\mathbf J}(q,\dot q)\,\dot{\mathbf q} = \ddot{\mathbf x}_\text{goal}, \quad \text{collision-free},$$9, and frustration was also highest for UR3, spanning the full range of scores. The authors therefore recommend caution in using frustration alone as a system metric (Audonnet et al., 2024).

The trust findings were framed through partial NARS S1, where lower scores indicate greater trust. Japanese participants had a lower mean than UK participants. The difference was statistically significant for Q5—“I would hate the idea that robots…were making judgments”—at $\mathbf J$0, and the overall NARS S1 total also indicated that the Japanese cohort was more trusting than the British cohort at $\mathbf J$1. The paper notes that this result contradicted some prior surveys, specifically Bartneck 2007 (Audonnet et al., 2024).

Experience variables introduced additional nuance. Japanese participants reported more experience with robots, $\mathbf J$2 versus $\mathbf J$3 in the UK, with $\mathbf J$4. UK participants reported more experience with wearables than the Japanese group, with $\mathbf J$5. VR experience was comparable across cohorts, with Japan at $\mathbf J$6 and the UK at $\mathbf J$7. The study further states that VR experience did not affect performance, whereas prior robot experience correlated with top-end performance. This finding narrows a common misconception that teleoperation performance is determined primarily by headset or controller familiarity; in the reported experiments, robot capability and ergonomic configuration were at least as consequential (Audonnet et al., 2024).

6. Network-aware TeleSim as testbed and benchmark dataset

A distinct 2025 usage of the name defines TeleSim as a hardware-in-the-loop teleoperation platform coupled to a benchmark dataset for studying network effects in telerobotics. The physical hardware comprises a 6-DoF Lite 6 robotic arm controlled via ROS, an Intel RealSense D435i RGB camera at 640×480 px and 30 fps mounted above the work area, an HTC VIVE Focus 3 headset for video rendering, and a wireless VR controller for operator motion commands. Network simulation is implemented in OMNeT++ with the INET framework, using two software-defined switches and one router to enforce bandwidth, one-way latency, jitter, and packet loss on RTP/UDP video streams. Control commands are routed over a dedicated Gigabit LAN without impairment so that only the video feedback channel is stressed (Deng et al., 6 Jul 2025).

The data-acquisition workflow starts at the first camera frame, which triggers simultaneous recording of video, network logs, and a high-precision timer. At task end, defined as successful block placement, recording stops and timestamps, network statistics, and success or failure flags are logged. Video is post-processed with MSU VQMT to compute frame-level PSNR and SSIM. The dataset comprises 300 trials of a standardized fine-manipulation pick-and-place task, evenly distributed across three network-quality tiers; the task object is a plastic block measuring $\mathbf J$8 cm, initially 20 cm from the robot base, and the goal is to place it into a receptacle measuring $\mathbf J$9 cm (Deng et al., 6 Jul 2025).

Tier	$\mathbf W$0	Trials
High	$\mathbf W$1	100
Medium	$\mathbf W$2	100
Low	$\mathbf W$3	100

Fifteen synchronized parameters are recorded in four groups: network configuration, measured QoS, video quality, and task performance. The measured QoS variables are mean and max throughput, mean and max one-way latency, mean and max jitter, and measured packet loss ratio. Task-level outputs include completion time and a success flag, while video quality is summarized with PSNR and SSIM (Deng et al., 6 Jul 2025).

The key quantitative finding is a severe degradation from High to Low tier. Completion time rises from 117.7 s to 378.8 s, a change of $\mathbf W$4. Success rate falls from 92% to 28%, a drop of 64 percentage points. PSNR falls from 24.23 dB to 12.59 dB, and SSIM falls from 0.930 to 0.741. The Medium tier occupies an intermediate regime, with completion time 205.9 s, success rate 72%, PSNR 25.03 dB, and SSIM 0.949. The paper characterizes the resulting behavior as a non-linear performance collapse and argues that there is a hard threshold beyond which manual teleoperation becomes infeasible (Deng et al., 6 Jul 2025).

7. Limitations, design implications, and relation between the variants

The digital-twin framework and the network-aware benchmark emphasize different constraints, but both identify concrete limits on human-in-the-loop teleoperation. In the digital-twin system, the principal reported limitations are latency, omission of haptics, a fixed VR origin, and incomplete decoupling between robot-specific inverse kinematics and controller modalities. The end-to-end path-planning plus actuation delay is reported as approximately 500 ms peak; the proposed mitigations are lightening collision checking or using predictive controllers. Future directions include calibrated haptics, free roaming and AR overlays of the digital twin, incorporation of high-level task planners such as pickup–place sequences, and further control decoupling so that any 3D pose source can drive robot-specific inverse kinematics (Audonnet et al., 2023).

The extended user-study paper turns those observations into broader design recommendations. It states that hardware matters: combined reach and stability, exemplified by UR5e, produced the best performance and lowest mental load. It further states that controller posture matters: VR hand controllers allowed a relaxed arm posture and lower physical demand and frustration, whereas the SenseGlove on UR3 forced the arm away from the body and increased physical workload and frustration. Mental effort depended on perceived robot capabilities, and trust differed across cultures, implying that trust should be measured per demographic when teleoperation interfaces are deployed (Audonnet et al., 2024).

The network-aware TeleSim adds recommendations at the communication layer. It proposes dynamic bitrate and foveated streaming to maintain target PSNR and SSIM, predictive motion compensation to mask latency above 200 ms, quality-triggered task handoff to semi-autonomous subroutines when $\mathbf W$5 ms or $\mathbf W$6, forward error correction plus jitter buffers, and multimodal fusion with local autonomy to compensate for video outages (Deng et al., 6 Jul 2025).

Taken together, the papers support a layered reading of TeleSim. One layer concerns embodied interface mediation through a digital twin, RMP-based planning, ROS 2 modularity, and user-centered evaluation. Another concerns network mediation, where the bottleneck is not kinematic mapping but degradation of the video feedback loop. This suggests that TeleSim is best understood not as a single appliance-like platform, but as a research designation for modular teleoperation infrastructures in which the intermediary layer—simulation, planner, driver, or virtual network—is itself the central experimental object (Audonnet et al., 2023, Audonnet et al., 2024, Deng et al., 6 Jul 2025).