Sovereign 8s.is mirror of standardgalactic/qplanning-code
  • Python 97.5%
  • Shell 2.5%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
varungiridhar 4b955b4230 Initial release
Value-guided action selection and self-improvement for frozen behaviour-cloning
policies. A policy proposes candidate action chunks; a Q-function scores them and
decides which to execute. The Q-function is then refined on deployment rollouts --
including failed ones -- while the policy stays frozen.

The policy is a plug-in: anything implementing the sampler interface in
qplanning/policies/base.py works, with FastWAM provided as the default.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-24 10:15:08 -04:00
configs Initial release 2026-08-24 10:15:08 -04:00
docs Initial release 2026-08-24 10:15:08 -04:00
qplanning Initial release 2026-08-24 10:15:08 -04:00
scripts Initial release 2026-08-24 10:15:08 -04:00
tests Initial release 2026-08-24 10:15:08 -04:00
.env.example Initial release 2026-08-24 10:15:08 -04:00
.gitignore Initial release 2026-08-24 10:15:08 -04:00
CITATION.cff Initial release 2026-08-24 10:15:08 -04:00
LICENSE Initial release 2026-08-24 10:15:08 -04:00
NOTICE Initial release 2026-08-24 10:15:08 -04:00
pyproject.toml Initial release 2026-08-24 10:15:08 -04:00
README.md Initial release 2026-08-24 10:15:08 -04:00

Q-Planning

Value-guided action selection and self-improvement for frozen behaviour-cloning policies.

Q-Planning pairs a behaviour-cloning policy with an off-policy Q-function over action chunks. At inference the policy proposes several candidate chunks and the Q-function decides which to execute. Over time the Q-function is refined on deployment rollouts — including failed ones, which imitation cannot use — while the policy stays frozen.

The policy is treated as a black box that can draw action chunks, so any policy satisfying a small interface can be used. FastWAM is provided as the default.

Install

git clone <this-repo> && cd qplanning_release
python -m venv .venv && source .venv/bin/activate
pip install -e ".[libero]"          # or ".[robotwin]"
cp .env.example .env && $EDITOR .env

.env holds every path the tool needs — checkpoints, datasets, an output directory. Nothing is hardcoded, and a checkpoint field accepts a local directory or a Hugging Face repo id interchangeably.

RoboTwin needs a separate environment (SAPIEN, curobo); see docs/setup.md.

Then check the install:

qplanning doctor --config configs/libero_10/qplanning_offline.yaml   # resolve paths, no GPU
qplanning probe  --config configs/libero_10/qplanning_offline.yaml   # load models, no simulator

doctor reports which paths are missing before you queue anything long. probe loads the policy and Q-function and runs one planning step on a synthetic observation, so a failure here is a setup problem rather than an environment problem.

Usage

Evaluate

# Policy alone.
qplanning eval --config configs/libero_10/baseline.yaml

# Policy under Q-guided selection.
qplanning eval --config configs/libero_10/qplanning_offline.yaml

Writes one row per episode to <output_dir>/episodes.csv — task, success, episode length in environment steps — plus a summary.json and the fully-resolved config used.

Split a long evaluation across jobs with --eval.shard=k/n; the shards write separate CSVs that the reporting step concatenates.

Self-improvement

qplanning self-improve --config configs/libero_10/self_improve.yaml

Each iteration rolls out the planner, appends every episode to a replay buffer, and updates only the Q-function. Two blocks run per task per iteration: a held-out block on fixed seeds that is never trained on and serves as the metric, and a training block on advancing seeds that feeds the buffer.

Collection can be fanned out and the update run separately:

qplanning self-improve --config configs/robotwin/self_improve.yaml --stage collect --eval.shard=0/16
qplanning self-improve --config configs/robotwin/self_improve.yaml --stage finetune

Train a Q-function

qplanning train-q --config configs/libero_10/train_q.yaml

Trains on demonstrations — the same ones the policy was trained on; no extra data is collected. Multi-GPU by default; --train_q.num_gpus=1 for a single device, --dry-run to print the command without running it.

Report

qplanning report --config configs/report.yaml --format markdown   # or latex, csv

Aggregates per-episode CSVs into a table: per-task success rate, per-task mean length over successful episodes, averaged across tasks. Columns are restricted to the tasks all of them evaluated, so a task that failed in one condition cannot change another's denominator.

Nothing requires a scheduler. scripts/run.sh is the primary entry point; scripts/slurm/ holds optional wrappers whose job script carries no site-specific settings, so adapting them means writing one small profile file.

Running a full multi-condition comparison, including sharding across jobs, is covered in docs/evaluation.md.

Configuration

One YAML describes a run. Benchmark configs extend configs/base.yaml and override only what differs. Any field can be overridden inline:

qplanning eval --config configs/libero_10/qplanning_offline.yaml \
    --planner.n_samples=16 --eval.episodes_per_task=5

The planner is the main thing to tune:

planner:
  type: q_weighted    # or `none` to run the policy unguided
  n_samples: 64       # candidates drawn per planning step
  n_elites: 16        # how many of the best are averaged (0 = all)
  temperature: 1.0    # softmax temperature over Q-values
  denoise_steps: 3    # sampler steps per candidate; low keeps candidates diverse

Full reference in docs/configuration.md; common failures and what they mean are in docs/troubleshooting.md.

Using your own policy

Implement the interface and register the class:

from qplanning.policies.registry import register_bc

@register_bc("my_policy")
class MyAdapter:
    chunk_size: int   # must match the Q-function's horizon
    action_dim: int

    def reset(self): ...              # called at the start of every episode
    def to(self, device): ...
    def parameters(self): ...         # never optimized; Q-Planning leaves the policy frozen

    def sample_chunks(self, batch, n_samples, *, denoise_steps=None, generator=None):
        """(n_samples, chunk_size, action_dim), in your policy's normalized action space.
        Encode the observation once and repeat only the sampling — otherwise planning cost
        grows linearly with n_samples."""

    def unnormalize_actions(self, actions):
        """Into raw environment units. Required: your policy and the Q-function hold
        different action statistics, and scoring un-converted actions ranks candidates
        wrongly while looking plausible."""

Set bc.type: my_policy. Third-party packages can register through the qplanning.bc_adapters entry-point group without editing this repository. The full protocol is in qplanning/policies/base.py; qplanning/policies/fastwam.py is a worked example.

Repository layout

configs/            one YAML per benchmark and stage; base.yaml holds shared defaults
scripts/
  run.sh            the entry point; scripts/slurm/ wraps it for a scheduler
qplanning/
  cli.py            subcommand dispatch
  planner.py        candidate scoring and the Q-weighted average
  policy.py         the rollout policy: sampler + planner + action queue
  policies/         the policy interface, and the FastWAM adapter
  eval.py           rollouts, one row per episode
  self_improve.py   collect, refine the Q-function, repeat
  train_q.py        Q-function training on demonstrations
  report.py         per-episode results to an aggregate table
  doctor.py         path resolution; probe.py loads the models without a simulator
  config/           schema, inheritance, interpolation, path resolution
  data/             replay buffer and the demonstration/online mixture
  envs.py           benchmark environments; benchmarks/ holds the task inventories
  *_qfunction.py    configuration, model and processor for the Q-function
docs/               setup, configuration, evaluation, troubleshooting
tests/

Acknowledgements

Built on LeRobot. FastWAM provides the default policy; LIBERO and RoboTwin provide the benchmarks.

Citation

@inproceedings{giridhar2026qplanning,
  title     = {Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning},
  author    = {Giridhar, Varun and Khandelwal, Anant and Collins, Jeremy A.
               and Georgiev, Ignat and Garg, Animesh},
  booktitle = {Conference on Robot Learning (CoRL)},
  year      = {2026},
}

Apache-2.0. See LICENSE.