What NVIDIA just shipped
NVIDIA released four new Cosmos 3 checkpoints purpose-built for interactive world simulation: two focused on autonomous driving with multi-sensor generation, two aimed at robotic manipulation and sim-to-real transfer. Each model has 16B trainable parameters and ships under the OpenMDW1.1 license, permitting both commercial and non-commercial use.
These aren't general-purpose video generators tweaked for robotics use cases. They're post-trained specializations of the Cosmos 3 Nano foundation model, each optimized for a specific control modality and inference pattern. The autonomous vehicle models handle synchronized multi-camera and LiDAR generation. The robotics models predict visual consequences from bimanual robot actions or turn physics simulator state into photorealistic observations.
What makes this release interesting isn't just the capabilities—it's the training recipe NVIDIA is publishing alongside the weights. They're showing the full four-stage path from bidirectional pre-training through causal streaming distillation, complete with open skills and notebooks for inference.
The driving models: synchronized sensors and streaming generation
Cosmos3-Nano-Transfer-Auto generates offline synthetic driving data across up to 11 camera views, including fisheyes, with optional synchronized LiDAR output. You control it using World Scenario Map (WSM) specifications—per-camera scene structure and traffic motion—plus text prompts for appearance.
The key value proposition: you can vary weather, lighting, time of day, and environmental conditions to expand training coverage without collecting real-world miles. If you need joint RGB–LiDAR generation, you provide aligned HD-map range-map controls. NVIDIA positions Transfer-Auto as the adaptation starting point for your own sensor rig and driving domain.
Cosmos3-Nano-Sim-Auto is the causally distilled streaming variant. It generates successive small chunks of synchronized camera video and LiDAR in just a few denoising steps, using rolling history and updated driving controls for fast streaming and long interactive rollouts. This is the model you'd use for closed-loop simulation where the agent's actions update the world state, which then feeds back into the next observation.
The distillation matters for deployment: fewer denoising steps means lower latency, which is essential when you're trying to run interactive loops at reasonable frame rates.
The robotics models: two paths to visual feedback
Cosmos3-Nano-Sim-Bimanual predicts how an environment evolves in response to bimanual robot actions. It supports interactive two-arm simulation for YAM and AgiBot embodiments, controllable by policy or human joystick input.
The action representation uses 59 control values padded to 64, with a detailed schema defining coordinate frames, rotations, gripper calibration, validity masks, and control domains. Start with an image, move the robot or camera, generate the next observations. One control sequence can switch between viewpoint changes and manipulation.
Crucial detail: timing must align. T transitions require T+1 observations. The training transforms raw captures into the model's action representation; released inference actions are already normalized.
Cosmos3-Nano-Sim-Transfer takes a different approach. Your physics simulator handles motion, contacts, and ground-truth state. Sim-Transfer uses spatial controls—depth, segmentation, edges, blur—plus text prompts to render photorealistic RGB observations that a policy can consume. The loop: action → simulator update → spatial controls → generated RGB → next policy action. Physics and state stay with the simulator; Sim-Transfer only handles visual rendering.
This matters because it gives you two architectural options. Sim-Bimanual generates observations directly from actions. Sim-Transfer sits downstream of a traditional simulator and just handles the visual domain gap. Different use cases, different integration points.
The streaming training recipe
NVIDIA's four-stage recipe is where the real knowledge transfer happens:
Stage 0 (Bidirectional adaptation): Learn the domain and control-to-video relationships from complete recorded clips. This is your standard video generation pre-training but on domain-specific data.
Stage 1 (Teacher forcing): Predict each chunk from earlier recorded observations and available controls, with clean answers and future targets masked out. You're teaching the model to generate causally, but still with access to ground truth history.
Stage 2 (Consistency distillation): Use a frozen teacher to train a student that generates each chunk in four denoising steps. Latency optimization via distillation.
Stage 3 (Self-forcing): Train on the student's generated history, not ground truth. A frozen teacher and auxiliary score model provide the distribution-matching signal.
Stage 3 is critical and often skipped in other approaches. Slightly mispredicted objects become context for later frames, so errors compound during rollout. Self-forcing exposes training to that generated history, making the model more robust to its own mistakes.
Architectural details that matter
Cosmos 3's Mixture-of-Transformers backbone separates text processing from visual generation, connecting them through attention. The generator predicts each short segment from prompt, available controls, and generated history using temporal-causal attention that reads available context while blocking future targets. A key/value cache retains attention information for the next prediction.
The model operates on compressed video representations (latents). After the initial image, each latent time step represents four RGB frames. Transfer predicts one step per chunk; the optimized Bimanual setup predicts two. Smaller chunks let new controls affect generation more frequently—this is the control-observation feedback loop tuning.
What this enables
For robotics researchers: a foundation for policy learning in simulation with visual feedback that doesn't require hand-crafted rendering pipelines. Sim-Bimanual lets you train policies that see realistic observations as consequences of their actions. Sim-Transfer lets you keep your physics simulator but close the sim-to-real visual gap.
For autonomous driving teams: synchronized multi-sensor generation that maintains geometric consistency between cameras and LiDAR. The ability to programmatically vary scene conditions and traffic scenarios means you can systematically probe edge cases and evaluate policy robustness across environmental conditions you haven't collected.
For world model researchers: streaming generation that connects video prediction to ongoing interaction. These models show how actions and observations unfold together rather than treating prediction as a one-shot task.
The open questions
NVIDIA is explicit about what remains unproven: "measure where the models faithfully support policy learning and evaluation, understand their limits, and improve where they fall short."
The technical reports show impressive qualitative results but don't quantify how well policies trained in these simulations transfer to real robots or vehicles. That's the actual test. Can you train a manipulation policy using Sim-Bimanual observations and deploy it on hardware? How much does the visual domain gap hurt you compared to training in a traditional rendered simulator?
For driving, the question is whether WSM control gives you enough expressiveness to meaningfully explore the scenario space you care about, and whether the generated LiDAR is geometrically consistent enough for perception systems that fuse camera and LiDAR inputs.
Distillation introduces another variable. Four-step generation is fast, but how much quality degradation comes with it? The consistency distillation + self-forcing recipe is clever, but you're still trading off between latency and fidelity.
Why open weights matter here
Public checkpoints let researchers compare methods on the same foundation. If you're developing a new streaming video architecture, you can now benchmark against NVIDIA's recipe. If you're working on robot policy learning, you have a shared baseline for synthetic data quality.
The bigger unlock is adaptation. NVIDIA provides the stage-0 adaptation path explicitly: post-train Transfer-Auto on your sensor rig and driving domain, then distill it into your own Sim-Auto variant. The same pattern works for robotics—adapt to your embodiment, then distill for deployment.
This matters because these problems are still hard and unsolved. Open weights mean the community can iterate on the same artifacts rather than everyone reimplementing from descriptions in papers.
Inference infrastructure
All four models ship with "open skills"—executable specifications for running inference through the NVIDIA Cosmos Framework. Requirements are Linux, an NVIDIA GPU, Docker, and NVIDIA Container Toolkit. The published vLLM-Omni setup was tested on GB200 (this is not a minimum spec, just what they validated on).
Each model gets both a skill definition and a Jupyter notebook. Prefer declarative automation? Point your agent at the skill. Prefer interactive exploration? Open the notebook. Both paths download weights and fetch sample inputs to get you generating quickly.
The bigger picture
World models for robotics and driving have been a research direction for years, but deployment has been blocked by the visual quality gap. Learned simulators that generate realistic observations from actions or controls change what's possible for policy learning and evaluation.
NVIDIA is making a specific architectural bet: diffusion-based video generation with explicit control inputs, distilled for streaming inference, trained with self-forcing for robustness. That's one point in the design space, but it's a well-executed one with working code and open weights.
The question now is whether this approach produces simulations faithful enough for policies to transfer. The models are out, the recipe is documented, the infrastructure is packaged. Time for the research community to find out what actually works.