Evaluate your policy

Submit a run to score your policy against a benchmark. Manifold runs the benchmark on its own GPUs, split across many machines in parallel, and records each episode.

Before you start

Serve your policy, or ensure that your Manifold cloud policy has been registered first. Then check the names of your policy and of the benchmark you want:

manifold policy list
manifold benchmark list

Submit a run

manifold run submit POLICY:VERSION BENCHMARK --name NAME

Argument or option	What it does
POLICY:VERSION	The policy version to evaluate, for example my-policy:0.0.1. For a self-hosted policy, use the version that you started with manifold policy serve. For a Manifold cloud policy, you can leave out :VERSION to use the newest version.
BENCHMARK	The benchmark to run against, as BENCHMARK:VERSION. Leave out :VERSION to use the newest version.
--name	Required. A label for the run in the runs list and in the app.
--shards	Splits the episodes into this many parallel shards. Leave it off to use the benchmark's default.
--org	The organization that owns the run. You need it only if you belong to more than one.

The command prints the versions that the run uses:

✓ Run queued RUN_ID
  POLICY:VERSION x BENCHMARK:VERSION, EPISODES episodes
Follow it with: manifold run watch RUN_ID
View it at: https://app.manifold.bifrost.ai/ORG/runs/RUN_ID

Self-hosted policies

The run goes to the manifold policy serve command that you started for that version. If that command is not running, run submit stops and tells you how to start it:

Run rejected: my-policy:0.0.1 is not ready yet.
Start it with: manifold policy serve my-policy:0.0.1 -- COMMAND

Follow the run

Open the link that run submit prints to follow the run in the Manifold app. You can also follow it from the command line:

Command	What it does
manifold run watch RUN_ID	Follows the run until it finishes.
manifold run get RUN_ID	Shows the run's progress and score.
manifold run list	Lists your organization's runs, newest first.
manifold run cancel RUN_ID	Stops the run. Finished episodes stay recorded.