RoboCasa365

RoboCasa365 tests a Franka Panda on a mobile base in simulated kitchens, with a language instruction for each task. Behind the scenes, Manifold runs robocasa/robocasa v1.0.

Manifold runs the three task sets of RoboCasa's multitask learning benchmark. Every episode samples its kitchen and objects from RoboCasa's pretrain split.

• robocasa365-atomic-seen runs short tasks with a single skill, such as opening a drawer
• robocasa365-composite-seen runs tasks that chain several skills, from the pretraining task set
• robocasa365-composite-unseen runs composite tasks that are not in the pretraining task set

Benchmark	Tasks	Episodes per task	Total episodes
robocasa365-atomic-seen	18	50	900
robocasa365-composite-seen	16	50	800
robocasa365-composite-unseen	16	50	800

Contract

Observation

Each step, the benchmark sends an observation with these fields. If Manifold has adapters for the difference, it converts the observation to the form that your policy signature declares. Then it passes the observation to your policy.

Field	Values	Meaning
state["ee_pose"]	x, y, z	end effector position relative to the robot's base, in meters
	qx, qy, qz, qw	end effector rotation relative to the robot's base, as a quaternion, scalar last
	f0, f1	finger joint positions in meters; the second finger's value is negative
state["base_pose"]	x, y, z	base position in the world, in meters
	qx, qy, qz, qw	base rotation in the world, as a quaternion, scalar last
instruction		the task prompt, a string
sensors["agentview_left"]		256x256 color image, uint8, upright
sensors["agentview_right"]		256x256 color image, uint8, upright
sensors["wrist"]		256x256 color image, uint8, upright

The benchmark sends no depth images and no camera calibration.

Action

The benchmark accepts 12 values per step, in this order.

Index	Values	Meaning
0 to 2	dx, dy, dz	end effector position change, in the robot base's frame
3 to 5	rx, ry, rz	end effector rotation change, axis-angle, in radians
6	gripper	below 0.5 opens the gripper, 0.5 and above closes it
7 to 9	vx, vy, vyaw	mobile base velocity, in the robot base's frame
10	torso	torso height change
11	mode	below 0.5 applies the arm's change to its measured pose, 0.5 and above applies it to its previous target, so the arm keeps up with the moving base

Return actions in the exact format that your policy signature declares. Manifold will attempt to convert your policy's output to match what the benchmark accepts.

Compatible policy signatures

Change the values to match your policy. If the policy signature below matches what your policy natively requires, then it is guaranteed to be compatible with RoboCasa365.

ProprioceptionFormat

End effector pose

Base pose

CameraOrientationSize

widthheight

widthheight

widthheight

Show policy signature

POLICY_SIGNATURE = PolicySignature(
    action_space=CompositeActionSpace(
        groups=(
            EEActionSpace(
                rotation=RotationFormat.AXIS_ANGLE,
                gripper=GripperFormat.SIGNED_OPEN_LOW,
                frame=Frame.BASE,
                delta=True,
            ),
            BodyActionSpace(
                part=BodyPart.MOBILE_BASE,
                dof=3,
                command=Command.VELOCITY,
                frame=Frame.BASE,
            ),
            BodyActionSpace(part=BodyPart.TORSO, dof=1, command=Command.DELTA),
        ),
        mode_slot=ModeCut.BINARY_0_5,
    ),
    proprioception=Proprioception(
        ee_pose=EEObservationSpec(
            rotation=RotationFormat.QUATERNION,
            gripper=GripperObservationSpec(dim=2),
            frame=Frame.BASE,
        ),
        base_pose=EEObservationSpec(rotation=RotationFormat.QUATERNION, frame=Frame.WORLD),
    ),
    cameras=(
        Camera(
            name="agentview_left",
            shape=(256, 256, 3),
            mount=Mount.SCENE,
            orientation=CameraOrientation.UPRIGHT,
        ),
        Camera(
            name="agentview_right",
            shape=(256, 256, 3),
            mount=Mount.SCENE,
            orientation=CameraOrientation.UPRIGHT,
        ),
        Camera(
            name="wrist",
            shape=(256, 256, 3),
            mount=Mount.WRIST,
            orientation=CameraOrientation.UPRIGHT,
        ),
    ),
    instruction=True,
)

Debugging and testing

• The gripper and motion mode values have a threshold of 0.5. A policy that expects a threshold value of 0 can misbehave. For example, if it outputs 0.3 to close the gripper, the gripper opens instead.
• Observation quaternions are scalar last (x, y, z, w). Check the order your rotation library uses before converting.

To check your policy end to end before a full run, submit it against debug-robocasa365, which runs the first atomic task, CloseBlenderLid, for 2 episodes.

For the full walkthrough, see Serve your policy.