View episode

DexBench New Early access

DexBench is RLWRLD's benchmark for bimanual dexterous manipulation, built on Isaac Lab Arena with the Newton physics backend. In the early access version, DexBench evaluates policies in four industrial scenes: a retail store, a logistics center, an assembly line and a hotel banquet hall. Policies will receive the stereo head camera feed, 62 joint positions and a language instruction, and will drive all 62 joints with absolute targets at 30 Hz.

The early access version is available today on Manifold, while the full version will be made available in a few days time.

Run DexBench on Manifold

Manifold helps you evaluate robot policies on DexBench faster:

• Fast, multi GPU accelerated runs
• Experiment tracking to compare scores across policies and versions
• Failure reports to see where and why your policy fails
• Detailed replays of the episodes

DB-R2 · Drink from a fridge shelf

DB-L2 · Box from cage to conveyor

DB-M1 · Drawer into a washer

DB-H1 · Plate stack to a cart

Benchmark	Tasks	Total episodes
dexbench-lite	4	80
dexbench-ea	4	1680

Start with dexbench-lite. It has the same tasks and cells as dexbench-ea, with 80 episodes instead of 1680. It gives you a first result in well under an hour, and it is available only on Manifold. When you're ready, run dexbench-ea. It runs all 1680 episodes of the early access evaluation, which are the episodes that RLWRLD reports its results on.

A dexbench-lite run takes about 5 GPU hours, so the 12.5 GPU hours of free alpha credits cover two runs.

Tasks

Task	Scene	Instruction	Success	Time cap
DB-R2	Retail	Take the drink out of the open refrigerator's second shelf and put it in the basket without disturbing the drinks next to it.	The drink rests inside the basket for 1 s.	35 s
DB-L2	Logistics	Lift the package off the cage with both hands and place it on the conveyor.	The box rests upright on the conveyor for 0.5 s.	25 s
DB-M1	Manufacturing	Push the detergent drawer you are holding into the washer's slot until it is seated.	The drawer stays seated for 0.5 s.	70 s
DB-H1	Hotel	Lift the stack of plates from the banquet table and seat it on the serving cart stack.	All dishes are nested on the cart for 0.25 s.	50 s

The Logistics, Manufacturing and Hotel tasks send the same instruction in all of their episodes. In the Retail task, drink changes with the episode. For example, an episode can say "Take the Coca-Cola 355 mL can out of the open refrigerator's second shelf…".

Scoring

DexBench splits the episodes of each task into cells.

Cell	Objects	Start positions
Train	objects from training	inside the training range
GA-O	new objects	inside the training range
GA-S	objects from training	beyond the training range
GA-OS	new objects	beyond the training range

In early access, the Manufacturing task uses only the drawer from training. For this reason, the Manufacturing task has only the Train and GA-S cells.

A task's score is the average success rate of its cells. Manifold shows each cell's performance individually, and additionally shows a success rate for the whole run.

Baseline results

RLWRLD fine-tuned three models on 240 demonstrations of each task. RLWRLD trained one checkpoint on each task alone, and one checkpoint on all four tasks together. The table shows the scores of the checkpoints that trained on all four tasks, in percent.

Model	DB-R2	DB-L2	DB-M1	DB-H1
GR00T N1.7	0.4	49.0	76.3	29.4
Cosmos3-Nano	0.8	48.8	72.1	8.5
π0.5	3.3	74.4	59.6	37.7

For the checkpoints trained on one task, the ablations and the Astra agent trials, see the DexBench report.

Failure analysis Manifold exclusive

DexBench splits each task into four stages for partial credit. Manifold shows the stages that each episode reached, so you can see where your policy stops.

Task	Stage 1	Stage 2	Stage 3	Stage 4
DB-R2	grasp and lift the drink	pull the drink out of the shelf	bring the drink over the basket	place the drink in the basket
DB-L2	lift the box	clear the box of the cage	bring the box over the conveyor	place the box on the conveyor
DB-M1	align the drawer at the slot	engage the slot's rails	push the drawer past half depth	seat the drawer in the slot
DB-H1	lift the plates	raise the plates to the cart's height	bring the plates over the cart's stack	seat the plates on the stack

Manifold also reads the contacts of each hand link and marks failure moments on the timeline of an episode. For example, Manifold marks a moment when the robot knocks into the scene, when a hand drops the object that it carries, or when the robot grasps an object that the task does not use.

To see the stages and the failure moments of a run, open the sample episode.

Contract

Observation

At each step, the benchmark sends an observation with these fields. If Manifold has adapters for the difference, Manifold converts the observation to the form that your policy signature declares. Then it passes the observation to your policy.

Field	Values	Meaning
state["joint_pos"]	62 values	joint angles in radians: arms, torso, head and both hands
instruction		the task prompt, a string
sensors["head_left"]		640x360 color image, uint8
sensors["head_right"]		640x360 color image, uint8

RLWRLD plans to add wrist cameras at general availability.

Action

Return 62 absolute joint targets. The benchmark runs 16 actions from your chunk and then queries your policy again. The benchmark does this whatever the length of your chunk.