Learning to walk

One reinforcement-learning loop

The policy receives a vector containing body attitude, angular velocity, joint state, a velocity command, and the previous action. It outputs joint actions, physics advances, and rewards measure progress toward the objective. PPO collects a batch of interactions, updates the policy, and samples again.

1. 1Observe
2. 2Sample actions
3. 3Simulate
4. 4Compute rewards
5. 5Update PPO

The environment count is the number of simultaneous robot copies. Iterations repeat collection and optimization. More environments increase parallel sampling and memory demand. A rising training reward means the optimized objective is improving; independent behavior checks are still necessary.

Why PPO looks beyond immediate reward

Lifting a foot can briefly reduce stability while being necessary for forward progress. Reinforcement learning therefore considers return over time rather than just the current reward. A discount factor reduces the weight of more distant rewards, balancing immediate feedback with longer-term consequences.

In PPO, the actor chooses actions and the critic estimates future return. An outcome better than the critic expected receives a positive advantage estimate, encouraging a higher probability for the corresponding action; a worse outcome does the opposite. PPO limits how far an update moves from the old policy to reduce abrupt changes. It does not guarantee that every update improves real task success.

A critic may receive additional simulation information to evaluate trajectories, but the deployed actor must depend only on observations available at runtime. A simulator knowing exact body velocity does not imply that hardware has a perfect velocity sensor. Keep training or scoring ground truth out of the deployed controller.

When designing rewards, list desired behaviors and then consider shortcuts. Rewarding ball motion may encourage body collisions; rewarding stillness may discourage walking. Track both goals and constraints: velocity error, episode length, falls, joint limits, and contact type. This chapter practices the training loop; Chapter 6 tests the complete task definition.

Run a short training exercise

python3 scripts/workshop.py train-walk \
  --num-envs 64 \
  --iterations 20 \
  --seed 101 \
  --output outputs/student-walk-01

This is a workflow exercise, not a promise that 20 iterations will learn to walk. Expected result: terminal progress, a new run directory, checkpoints, and an exported policy. Observe loss, return, and episode length without committing to a long training run.

python3 scripts/workshop.py tensorboard \
  --logdir outputs/student-walk-01 \
  --port 6006

Open the local TensorBoard address printed in the terminal. Compare reward with episode length. If reward grows while episodes remain short, consider whether the objective rewards a brief action followed by a fall.

Current practice uses full-body contacts, including the head and legs during falls. New outputs are separate experiments; historical learning curves retain their original configuration. Training does not render frames; policy playback uses the shared RTX lighting and materials.

From a smoke run to a longer training run

The 20-iteration exercise checks the pipeline, not convergence. A longer run below uses 1024 environments for 2500 iterations in a new output directory. Judge convergence with curves and independent evaluation, not iteration count alone.

python3 scripts/workshop.py train-walk \
  --num-envs 1024 --iterations 2500 --seed 101 \
  --output outputs/walking-training-01

Use TensorBoard to compare reward, episode length, falls and velocity error. Use the short-run export in the next section, or explicitly select outputs/walking-training-01/exported/policy.onnx after this longer run. Neither changes the official model.

Explicitly test your own policy

Run the command below in a terminal at the repository root. If it specifies an output directory, use a new one to retain earlier results.

python3 scripts/workshop.py walk \
  --model outputs/student-walk-01/exported/policy.onnx \
  --velocity 0.30 0.00 0.00 \
  --seconds 12 \
  --seed 202 \
  --output outputs/student-walk-test-01

Here you deliberately select your own model. Compare support, drift, and falls with the official model from Chapter 1. Later lessons still use models/walk.onnx; this export does not change their default.

python3 scripts/workshop.py walk \
  --model models/walk.onnx --velocity 0.3 0 0 \
  --seconds 12 --seed 202 \
  --output outputs/official-walk-comparison-01

Match command, duration and seed. Compare falls, distance and actual velocity, then test standing at zero command. One moving demonstration is not a complete skill benchmark.

Why domain randomization matters

Real actuators do not have identical friction, delay, and backlash on every run. Varying parameters in training exposes the policy to more conditions. A narrow range can overfit; an excessively broad range can make learning difficult. Randomization can improve robustness but does not replace hardware testing.

Checkpoints, curves, and independent tests

Practice starts from random parameters rather than the course walking weights. Each PPO iteration gathers trajectories from several environments and then updates the policy. Rewards combine velocity tracking, body attitude, action smoothness, and contact-related terms. Two policies with similar total rewards may obtain them through accurate motion or simply moving less; inspect both behavior and individual reward terms.

initial.pt records initialization; model_*.pt files are checkpoints; env.yaml and agent.yaml preserve environment and optimizer settings; exported/policy.onnx is the inference export. A checkpoint can contain optimizer state, while ONNX normally preserves the inference graph. They are not interchangeable file formats.

Record the training budget, held-out test seeds, and observed behavior. The 20-iteration exercise teaches sampling, updates, saving, and export. A weak practice policy is no reason to replace downstream course defaults. For a larger budget, keep the earlier run, use a new output directory, and state whether you changed environment count, iteration count, or rewards.

Will a flat-ground skill handle a slope?

You can now collect rollouts, update PPO, export ONNX and test a policy. Will its foot clearance and balance adjustments work on a slope? Predict the outcome, then test the same policy in Chapter 4.

References

• Proximal Policy Optimization paper ↗
• Isaac Lab: Microduck task source ↗