Explore Labs

High-Fidelity Digital Twin Lab

Explore Labs

Enter Lab Back to overview

Simulation is usually a way of making a problem easier: strip out the noise, hand the planner a clean world, study the algorithm in isolation. This laboratory uses it the other way round — at a fidelity that puts the difficulty back, so that a plan which succeeds here has had to earn it. What is tested is not whether the reasoning is right in principle, but how much of it survives an encounter with a body.

It is also a laboratory anyone can walk into. High-fidelity embodied research is normally gated by hardware and by weeks of integration; here it is a link. A GPU-backed session opens in your browser with everything already assembled, so an experiment becomes something another group can rerun rather than rebuild.

Inside, you are equipped with high-fidelity digital-twin assets — the robots and the domestic environments they work in — and with open-source CRAM plans that use them to carry out a range of everyday household tasks.

Example Videos

Kitchen Manipulation with GARMI

GARMI opens a drawer, perceives the worktop, transports an object into the drawer, and opens and closes a cabinet door under whole-body control.

Multiple Robots in the AICOR Apartment

Different robots at work in the same AICOR apartment, each paired with the environment at run time.

Generating Demonstration Data at Scale

The twin is a source of training data as well as a place to run plans: 100 episodes of "put the cup into the sink" are generated with the cup placed at random and recorded from four cameras in LeRobot format. An ACT policy trained on them succeeds in 14 of 50 evaluation rollouts, which is what makes the data worth generating.

Browse the episodes in the LeRobot dataset visualiser.

Run this Lab Locally

  1. Preparation.

    What Needed Check with Install guide
    OS Ubuntu 20.04 or newer, x86_64 lsb_release -d —
    Hardware NVIDIA RTX 2070 or better, ~16 GB RAM, ~50 GB free disk free -g, df -h ~ —
    NVIDIA driver 580.65.06 or newer nvidia-smi Isaac Sim requirements
    Docker Engine any current release docker run --rm hello-world Install Docker Engine
    NVIDIA Container Toolkit needed for --gpus all docker run --rm --gpus all ubuntu nvidia-smi Installing the NVIDIA Container Toolkit

  2. Run in Terminal:

    bash <(curl -fsSL https://raw.githubusercontent.com/yxzhan/cram-vrb-lab/dev/install.sh)

     

    The lab docker image is about 16 GB to download and ~50 GB unpacked, so the first run takes a while.

  3. Once the 1st step finish, it will enter the container shell (e.g., root@legion-34:~/cram-vrb-lab), now run the Garmi demo:

    binder/cram_python_wrapper.sh demos/garmi_demo.py

     

    The first Isaac Sim start compiles shaders and can take 10 minutes or more. The compiled cache is written to ~/isaac_cache on the host, so later runs start in a few tens of seconds.

  4. Open the NeemHub web UI: http://localhost:8711

    NeemHub shows the running demo in the browser: the live 3D scene, the plan tree as it executes, and a question console that answers questions about the episode.

     

  5. To shutdown everything, "ctrl +c" to kill the running demo and run "exit".

       

Source Code

The laboratory is open source and developed at github.com/yxzhan/cram-vrb-lab.

 

Architecture

The plan, the world model and the controller sit above a ROS 2 interface; the Isaac Sim scene sits below it, in the position a physical robot would occupy.

Software Components

CRAM: A software toolbox for implementing autonomous robots.

Semantic Digital Twin: A unified world model bridging geometry, kinematics and asserted meaning, shared by the planner and the perception pipeline.

GISKARD: A framework for constraint- and optimization-based robot motion and planning control.

ROBOKUDO: A perception framework targeted for robot manipulation tasks.

NVIDIA Isaac Sim: A GPU-accelerated robotics simulator providing the photorealistic rendering and PhysX contact simulation this laboratory is built on.

Back to top