Real-Time Collaborative Robot Simulation Lab

Explore Labs

Description

Robot simulation is usually a single-user affair: one person, one workstation, one copy of the world. This laboratory turns it into a shared space. It is one MuJoCo physics world, running on a server such as a Binder session and streamed in real time to everyone who opens the same link. From a laptop, a phone or a VR headset, participants spawn robots from a model library of arms, mobile manipulators, quadrupeds and humanoids, drive them with a keyboard, gamepad or VR controllers, and pull objects around by hand. Every change appears for all participants the moment anyone makes it, so one person can set up a scene while another drives a robot through it and a third takes over the arm in VR. Walking robots move under pretrained neural policies and arms follow inverse kinematics, so even newcomers can steer them. Nothing has to be installed, and without a server the same page runs the physics in the browser for working alone. Through a ROS 2 bridge, the simulated robots can also drive real ones: a Franka Panda follows its simulated twin while an operator teleoperates it in VR, and sessions are recorded as demonstrations. The same shared world therefore serves classroom teaching, remote collaboration, quick experiments and the collection of training data.
 

Example Videos:

VR Teleoperation in Digital Twin Environment:
 
Live demo: Garmi setting table
 
 
 
 
 
Sim2Real Transfer & VR Teleoperation:
 
   
 
 
Multi-Robot Collaboration & Competition: 3v3 MicroDuck Soccer
 

Live demo: Microduck football

Source Code

The lab is open source: github.com/yxzhan/robot-lobby.

The real-robot side is in github.com/yxzhan/robot-station.

Software Components

MuJoCo: The physics engine, running in the browser through the official WebAssembly bindings or in Python on the server.

Filament: A physically based renderer, here in WebGL2, with sun shadows and ambient occlusion.

WebXR: The browser standard for VR and AR headsets, used for teleoperation with Quest controllers.

ROS 2 and rosbridge: Joint states, TF, commands and camera images exchanged with ROS 2, and through it with real robots.

Pretrained policies: ONNX locomotion policies (GR00T for the Unitree G1, MuJoCo Playground for Spot and Go2, Microduck) running in the browser or on the server.

 
 
Back to top