SELF-PACED WORKSHOP
Teach GR00T from demonstrations
Adapt the SO-101 sim-to-real data workflow to Microduck: what a vision-language-action model learns and what adaptation requires.
Express the task through goals and demonstrations
A language goal such as “send the ball into the goal,” paired with successful demonstrations, offers another way to specify the task. A vision-language-action model uses images, instructions and robot state to predict a short action sequence. Data can capture decisions previously distributed across looking, alignment and approach rules.
Scroll horizontally to view the full diagram
Action prediction and whole-task planning are related but distinct. An action chunk is not a globally collision-free path. Longer tasks also need goal management, checks and recovery, which the next chapter addresses.
Use the current public GR00T N1.7
As checked on 26 October 2026, the current public release is GR00T N1.7, with base checkpoint nvidia/GR00T-N1.7-3B. It uses a Cosmos-Reason2-2B / Qwen3-VL backbone and an action head; that does not mean Cosmos 3 is already part of this model. Official workflows support custom-robot fine-tuning and policy serving.
Microduck is not a pretrained embodiment validated here. SO-101 arm actions, the model’s tensor capacity and Microduck’s fourteen controlled joints are not interchangeable. Every action needs a defined physical meaning, unit, rate, normalization and controller interface.
What transfers from the SO-101 course
The SO-101 course supplies a reusable workflow: teleoperate, record synchronized images/state/actions, prepare LeRobot data, adapt GR00T, then evaluate simulation and hardware separately. Lighting, robot color and mat variations broaden visual coverage. The transferable part is this data and evaluation method; vial-manipulation weights are not a football policy.
| SO-101 lesson | Microduck counterpart |
|---|---|
| Synchronize cameras and joints | Retain body motion, gaze and action timing |
| Demonstrate grasp, place and recovery | Demonstrate approach, contact, braking and recovery |
| Evaluate sim and real separately | Test new scenes before hardware calibration |
| Vary appearance | Cover home lighting, backgrounds and floors |
Define what the model may control
A practical first research route retains the locomotion policies and learns bounded body-motion and gaze requests, reducing hand-written observation/alignment/advance transitions. Walking and skating have different command semantics: use separate configurations or an explicit mode condition. A gliding shot remains in roller mode throughout. A higher-level controller can request a kicking skill; a continuous number is not automatically a validated discrete skill switch.
Learning joint actions is another route, with greater data and validation demands because balance and contact must also be learned. N1.7’s new-embodiment example supports non-end-effector actions. Definable fields, however, do not establish a learned fourteen-joint football policy. Validate the hierarchical interface before selecting the training target.
What makes demonstrations useful
Scroll horizontally to view the full diagram
Existing logs contain images, joint state, inertial observations and executed actions, but one successful episode is far from sufficient. Keep success, failure and corrective demonstrations explicitly labeled; ordinary imitation learning must not treat every failed action as a correct target. Distinguish raw policy outputs, actions after gaze overrides and actual motor targets.
The head camera moves, and images/actions have different sampling rates. Align timestamps and observation ages without using future images for past actions. Do not split neighboring frames from one episode across train and test. Hold out complete rooms or layouts to assess unfamiliar homes.
Readiness and the next experiment
The official custom-embodiment configuration, data format and policy interface have been reviewed. Existing roller logs provide 163 image/state frames aligned with entries in 652 action records; 97 actions differ from the raw network output after executor corrections. This checks 163 one-step pairs at 12.5 Hz camera capture and 50 Hz action recording. Official shape validators and added timing/dimension checks pass; future multi-step action chunks remain unchecked. N1.7 weights were not loaded and no inference or fine-tuning ran. Task language, action-conversion configuration and LeRobot export are still needed for training.
This probe uses 14-dimensional low-level action records. The proposed high-level velocity/skill route still needs its own labels and executor; validating one does not validate the other. The visual backbone has model-access requirements. Keep future training dependencies separate from the existing Newton environment. See the readiness scope and remaining requirements.
For the next experiment, fix one locomotion mode and one short task, collect successful and corrective demonstrations, and evaluate unseen starts and rooms. Keep robot, props, sensors and success criteria identical across baselines; measure scoring, falls, stopping and observation cycles. This establishes whether learning actually reduces repeated corrections.