NVIDIA Cosmos3-H-Dreams: Bringing Clinical Surgery to Life Through Generative Simulation

NVIDIA Cosmos3‑H‑Dreams: Generative Simulation for Clinical Surgery

Cosmos3‑H‑Dreams pushes the frontier of surgical simulation by moving from tabletop suturing demos to full‑body, real‑time generative video of clinical cholecystectomy. Built on the 16‑billion‑parameter Cosmos 3 foundation model and accelerated with FlashDreams, the system learns directly from synchronized video and robot kinematics, eliminating the need for explicit geometry or physics pipelines. The result is an interactive simulator that reacts to instrument commands with plausible tissue motion, occlusion, and lighting—all on a single NVIDIA RTX PRO 6000 GPU.

---

From Tabletop Tasks to Clinical Scenes

Tabletop environments offer clean backgrounds and limited instrument interaction, making it easier to train video‑generation models. Inside the abdomen, however, tiny motions cause complex cascades: tissue stretches, reflections shift, smoke appears, and structures become hidden. A generative simulator must therefore maintain temporal consistency and action fidelity across thousands of frames. Small visual drift can quickly corrupt instrument geometry or anatomical landmarks, so low‑latency generation and robust conditioning on the full 28‑dimensional action vector (pose, gripper state, energy controls, thumb‑stick axes/buttons) are essential.

Cosmos3‑H‑Dreams addresses these challenges by:

• Training on high‑resolution (832 × 480) cholecystectomy recordings captured on the CMR Surgical Versius platform.
• Using a bidirectional teacher model that sees future frames, then distilling it into a causal student capable of streaming inference.
• Integrating FadeMem cache consolidation to keep attention memory manageable during long rollouts.

---

What’s New in Cosmos3‑H‑Dreams?

1. A Cosmos 3 Foundation with Richer Surgical Controls

The model inherits the Cosmos 3 backbone, pretrained on the Open‑H‑Embodiment dataset, and is fine‑tuned on curated cholecystectomy video. It focuses on two instruments—a left Cadiere grasper and a right monopolar hook—while exposing the full suite of control signals needed for realistic interaction.

2. Teacher‑Student Distillation for Long Autoregressive Rollouts

Training proceeds in stages:

1. Teacher adaptation on 24‑frame horizons, then extended to 72 frames (≈7 s at 10 Hz) to capture longer tissue dynamics.
2. Causal student distillation using a two‑phase approach:

- Causal teacher forcing teaches the student to generate sequentially from recorded context. - Self‑forcing lets the student generate its own context, guided by the teacher and a learned critic, reducing error accumulation. The released checkpoints include a four‑step denoising student (higher visual fidelity) and a two‑step version (faster inference).

3. Streaming Inference with FlashDreams

FlashDreams provides an accelerated runtime that streams video blocks as new actions arrive. By consolidating the attention cache with FadeMem, the system sustains interactive frame rates without sacrificing the long‑term coherence needed for surgical simulation.

---

Exploring Unseen Trajectories

A key test of any generative simulator is its ability to handle novel instrument motions not seen during training. Cosmos3‑H‑Dreams demonstrates this by starting from the same initial frame and branching into distinct action sequences recorded via a VR headset. The resulting rollouts show coherent tissue deformation, consistent instrument appearance, and faithful adherence to the supplied control signals—highlighting the model’s capacity to extrapolate safely within the surgical domain.

---

Adapting the System to Your Data

Cosmos3‑H‑Dreams is released with both four‑step and two‑step checkpoints for cholecystectomy, but the pipeline is dataset‑agnostic. Researchers can adapt it to other procedures by:

1. Preparing synchronized video and kinematic logs matching the 28‑dimensional action schema.
2. Fine‑tuning the bidirectional teacher on the new data using the provided public training scripts.
3. Running the self‑forcing distillation recipe to produce a causal student ready for streaming inference.

All scripts, model weights, and documentation are hosted on the NVIDIA GitHub and Hugging Face repositories linked in the article.

---

Looking Ahead

Cosmos3‑H‑Dreams opens several research avenues:

• Full‑procedure modeling to cover all phases of cholecystectomy with consistent task boundaries.
• Hybrid simulation that blends learned visual realism with explicit physics for measurable deformation metrics.
• Surgeon‑in‑the‑loop evaluation to establish benchmarks for anatomical accuracy and instrument fidelity.
• Multi‑instrument support and richer controller integration for complex robotic platforms.
• Closed‑loop policy learning, where autonomous agents train against the generative world to acquire surgical subtasks.

These directions are exploratory; Cosmos3‑H‑Dreams remains a research simulator, not a clinically validated tool.

---

Get Started

• Code & examples: GitHub repository for Cosmos3‑H‑Dreams
• Model checkpoints: Hugging Face pages for Cosmos3‑H‑Dreams and Cosmos3‑H‑Embodied‑Nano
• Data & recipes: Open‑H‑Embodiment dataset and step‑by‑step fine‑tuning guide
• Runtime: FlashDreams streaming library (GitHub)

Bring your own controllers, datasets, and research questions—NVIDIA’s generative surgical world is ready for you to explore.