Evaluation for embodiment tasks

We provide a simple communication mechanism like ROS, for you to run robots, or virtual robots in one machine or virtual environment (venv), send the images and language instructions to SpatialBot by ssh and sftp, run SpatialBot in another machine or venv, and send robot actions back to the robot.

Calvin

Refer to Calvin GitHub to prepare data and venv.
In Calvin venv, run calvin/calvin_models/calvin_agent/evaluation/evaluate_policy_local_spatialbot.py or calvin/calvin_models/calvin_agent/evaluation/evaluate_policy_remote_spatialbot.py (We will provide the scripts soon).
In SpatialBot venv, run SpatialBot to generate robot actions by

sh script/eval/lora/calvin_infer.sh

Open X-Embodiment (RTX)

We use datasets downloaded from unofficial RTX HuggingFace repo.
Prepare images and data json with our script. (Available soon)
Evaluate model by

sh script/eval/lora/remote_rtx_infer.sh

If you would like to see open-loop results by evaluating model on a validation episode, run

sh script/eval/lora/local_rtx_infer.sh

Results will be written to control.png

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!