SELF-PACED WORKSHOP
Cover different homes with Cosmos and Isaac
Turn lighting, floor and layout changes into data and evaluation designs; distinguish world models, physics and task orchestration.
What changes in another home
These variations matter. Warm light, backlighting and shadows change the ball’s appearance; walls, tiles and furniture change its background. Narrow passages change routes, while carpet and smooth floors change motion and braking. Visual generalization, spatial planning and dynamics adaptation need separate treatment.
| Variation | Primary effect | Useful evaluation |
|---|---|---|
| Lighting, color, background | Perception and visual policies | Test new appearance with consistent geometry/actions |
| Layout, occlusion, starts | Observation strategy and routes | Closed-loop tasks and collisions in new layouts |
| Friction, tyres, ball mass, latency | Contact, propulsion and braking | Physics ranges and hardware measurements |
| Instructions and goals | Task understanding and skill selection | Language variation and held-out task combinations |
What Cosmos 3 provides
Cosmos 3 is a public omnimodal world-model family covering language, images, video, audio and actions. Reasoner, Generator and specialized policy/simulation variants support scene understanding, candidate scenario generation and action/future-state modeling. Its scope extends beyond background replacement.
For Microduck, initial uses include visual-data expansion and failure analysis. Preserve robot, goal and ball geometry while generating candidate room appearances, then inspect shape, occlusion and temporal consistency. Where depth or segmentation conditioning is required, compare the actual interfaces of specialized models such as Cosmos-Transfer2.5 as well.
A plausible scoring video is not an executable action trajectory. Imitation data require action labels consistent with images, states and timing. A changed ball size, post position or contact time can invalidate the original actions. Cosmos policy/action variants likewise need embodiment adaptation; a DROID specialization is not a Microduck policy.
Generation, physics, learning and orchestration
Scroll horizontally to view the full diagram
Cosmos also has world-simulation and policy variants. This course retains Newton for physical evidence of contact, scoring and stopping. OSMO orchestrates workflows; it does not automatically reconcile incompatible action spaces into a common learning target.
Robot × scene × task: Isaac Lab-Arena
Isaac Lab-Arena directly addresses composition. It separates Scene, Embodiment (robot, sensors and actions), and Task (objective and predicates), then builds Isaac Lab environments. It organizes evaluation across layouts, robots and tasks and registers environments for learning/data workflows. The main branch inspected on October 26, 2026 targets Isaac Lab 3.0 / Isaac Sim 6.2 and remains early-stage software; its dependencies differ from release/0.3.1.
Arena provides policy clients including GR00T and Cosmos-related adapters and supports Newton selection. Microduck still needs its own embodiment, football task and action contract. Combinations must satisfy task capabilities. Concurrent evaluation does not itself train a single policy for every robot.
Design an evaluation matrix
This only counts combinations; it launches no simulation or training. Unchecking a factor fixes it to one condition.
Walking and skating are modes of one robot and are evaluated separately here; they are not two physical embodiments. The count describes an experiment design, not completed runs or universal compatibility.
How to test whether randomization helps
Keep a baseline with no added randomization, then separately add appearance, layout and measured dynamics variation while controlling demonstration count, training budget and evaluation criteria. Hold out rooms, layouts or combinations. Keep failed trials rather than replace them with successful retries.
Measure full scoring and settling, falls, route collisions, observation cycles and completion time, not just detection scores. Current failures also involve geometry extraction and braking. Visual randomization may improve observations without fixing stopping control. Target the observed failure before scaling training.
Next: a small, comparable experiment
Choose one locomotion mode, keep the official 10 cm ball and goal fixed, and use the two football floors, marble and wood with controlled lighting, then a small set of indoor layouts. Keep goal-frame material fixed. After aligning simulated sensor data with executed action records, select one initial experiment: GR00T adaptation, ASPIRE skill exploration or Cosmos data expansion, rather than change everything together.
The course reviews methods, interfaces and existing record fields, and illustrates combination design. No new GR00T, ASPIRE, Cosmos or Arena training/control experiment has run. Independent closed-loop results should guide subsequent data and compute investment.