D\'ecoupage: Directorial Audio-Visual Control for Long Multi-Shot Video Generation

Découpage introduces a screenplay‑driven diffusion framework that separately controls visual and audio streams, enabling precise lip‑sync, silence handling, and shot transitions for long multi‑shot video generation.

D\'ecoupage: Directorial Audio-Visual Control for Long Multi-Shot Video Generation

Introduction

Generating coherent, long‑form video that matches a screenplay’s timing and dialogue is a persistent challenge for AI‑driven media creation. Traditional audio‑visual generative models bind visual frames and sound tightly together, which works for short clips but quickly breaks down when scenes span multiple shots, characters, and nuanced timing such as J‑cuts, L‑cuts, or intentional silences. Découpage tackles this gap by decoupling visual and acoustic generation while preserving a shared narrative thread through persistent character identities. The result is a system that can produce 15‑ to 60‑second multi‑shot videos that respect both visual continuity and audio pacing, including accurate lip‑sync only when a character is on‑screen.

A screenplay‑driven architecture

At the core of Découpage is a joint audio‑video diffusion model equipped with dual‑schedule temporal cross‑attention. The model receives two parallel inputs:

  1. Visual script – a sequence of shot descriptors, camera angles, and character presence flags.
  2. Audio script – dialogue lines, timing cues, and silence intervals.

These streams are processed independently but remain linked through a character identity embedding that persists across shots. This embedding ensures that the same virtual actor maintains consistent appearance, voice, and facial dynamics throughout the narrative, even when they disappear from the frame for a period.

Directorial Lip‑sync and silence handling

A key innovation is the Directorial Lip‑sync mechanism. Conventional diffusion pipelines condition facial motion on speech audio at every timestep, which leads to unwanted mouth movements during off‑screen dialogue or background silence. Découpage introduces a binary mask derived from the visual script that tells the model when to apply speech conditioning. When a character is off‑screen, the mask disables lip‑sync, allowing the audio track to continue uninterrupted while the visual stream shows other actions or cuts. Conversely, intentional silences are respected by suppressing speech conditioning entirely, preventing the model from hallucinating speech.

Multi‑shot coherence and transition control

Longer narratives require smooth transitions between shots. Découpage’s temporal cross‑attention operates on two schedules: a coarse schedule that aligns high‑level shot boundaries and a fine schedule that refines frame‑level details within each shot. This hierarchy lets the system enforce continuity constraints—such as consistent lighting or object placement—while still allowing creative variation at the micro level. Experiments on a conversational benchmark demonstrate that the model can reliably place cuts at scripted timestamps, maintain character pose continuity, and avoid the “jump‑cut” artifacts common in earlier approaches.

Practical implications

For filmmakers, game developers, and content creators, Découpage offers a workflow that mirrors traditional directing: write a screenplay, annotate shot timings, and let the model render a synchronized audio‑visual draft. The decoupled design also simplifies post‑production edits; adjusting a line of dialogue or shifting a cut does not require re‑generating the entire video, only the affected stream. Moreover, the system’s ability to respect silence and off‑screen speech opens new storytelling possibilities, such as overlapping conversations or dramatic pauses that were previously difficult for AI generators.

Conclusion

Découpage represents a significant step toward director‑level control in AI‑generated video. By separating visual and audio generation, anchoring both to persistent character identities, and providing explicit mechanisms for lip‑sync and silence, the framework delivers long, multi‑shot videos with cinematic timing and coherence. As diffusion models continue to evolve, approaches like Découpage will likely become foundational tools for scalable, script‑driven media production.

Source

Read original source

Why we picked this

Presents a new audio‑visual generative model for long video generation, significant AI advancement.

← Back to all articles