Be My Eye: Learning to See Beyond a Robot's Own Perspective
This presentation explores how robots can share compact, task-relevant visual information to collaborate effectively when one robot sees what another cannot. The work introduces active context sharing, a framework that compresses multi-camera observations into small sets of learnable tokens optimized both for preserving task-critical information and directly improving a partner robot's actions, achieving performance comparable to high-bandwidth methods while using 80% fewer transmitted tokens.Script
When robots work together, what one robot sees can be exactly what another robot needs to act correctly. This paper tackles a fundamental challenge: how can robots share compact, task-relevant information that directly improves a partner's decisions, especially when occlusions and limited camera views leave critical gaps in perception?
The authors introduce active context sharing, a framework that encodes observations into compact latent tokens conditioned on the task itself. Instead of transmitting raw images or 160 generic visual tokens, their method uses just 32 learnable tokens that preserve what matters for collaborative action while cutting communication bandwidth by 80%.
Training happens in two stages. First, the context encoder learns to compress observations by predicting task-relevant future visual states and organizing representations according to task identity and execution progress. Then, the receiver robot's action supervision backpropagates through the shared tokens, jointly optimizing both the representation and the collaborative policy.
In the occluded target selection task, Robot A observes which box receives a plate, but Robot B cannot see the placement action. Using shared context, Robot B achieved 89% target accuracy compared to just 39% without any sharing, despite never directly observing the critical moment.
The method matched or exceeded high-bandwidth baselines across both tasks. In alternating leader-follower marker retrieval, both Generic SigLIP and the proposed approach achieved perfect 100% success on all metrics, but the compact representation used only 32 tokens instead of 160, proving that task-conditioned compression preserves exactly what collaborative control requires.
This work shows that effective multi-robot communication should optimize for collaborative action, not just visual reconstruction. The framework opens pathways for larger robot teams and multimodal sharing, and you can explore the full details and create your own video at EmergentMind.com.
