LLM as a Policy

To complete a task, a policy has to find the objects, work out where they are in 3D, plan the order of steps, and move the robot. A harness around the LLM can take over some of these jobs.

LLMharness

Seefind the objects

Locatewhere they are in 3D

Planthe order of steps

Actmove the robot

Direct control

LLM

Motion primitives

LLM

script

Code as policy

LLM

depth camera

LLM code

script

VLA supervision

VLA

LLM

VLA

Next

Start with direct control, where the LLM does every job. Then try motion primitives, where the LLM picks named moves, and code as policy, where the LLM writes a program that calls functions to find objects and move the robot.