Semantically Complex Audio to Video Generation with Audio Source Separation

TL;DR

We present Maestro, a novel multi-source audio-to-video generation framework that incorporates decomposed audio sources into video generative models. Real-world audio consists of mixed sources, causing existing models to struggle with capturing all semantics. Our proposed Attention Mosaic directly maps each decomposed audio feature to its corresponding spatial attention region, ensuring accurate object placement and rendering. A Condition Injection Module (CIM) further produces natural contexts involving non-audible objects by leveraging the spatial prior knowledge of Stable Diffusion. Extensive experiments demonstrate state-of-the-art performance in both multi- and single-source audio-to-video generation tasks.

Key Contributions

• Audio Source Decomposition — Addresses existing limitations by decomposing mixed audio sources via AudioSep to accurately capture multiple semantics, enabling faithful multi-object video generation.
• Attention Mosaic for Spatial Control — Employs Object Locating (2D Gaussian weighting) and Mosaic Denoising (masked spatial attention) to directly map each audio source to its designated region, vastly enhancing instance-level audio-visual alignment.
• Diffusion-Based Context Optimization — The Condition Injection Module (CIM) is pre-trained using Stable Diffusion's denoising loss alongside MSE, achieving superior contextual expression of non-audible background objects beyond contrastive learning.

Project Design

Overview of Maestro. The model consists of two main components: (i) Multi-source audio feature extraction, producing time-dependent audio tokens via the Condition Injection Module, and (ii) Attention Mosaic, directly mapping each audio feature to its designated spatial location in the video diffusion model.

Audio input is divided into segments and processed through an audio encoder and CIM. By employing a Stable Diffusion denoising loss (λ₁ = 0.9) along with MSE, CIM learns video frame information corresponding to each audio segment, enabling natural expression of background and non-audible contexts.

At timestep t, noise is predicted via either Object Locating — using a 2D Gaussian weight function to place each object within its bounding box — or Mosaic Denoising — masking query maps so each audio condition exclusively attends to its designated region, ensuring fine-grained object separation and quality.

Experiments

Comparison with Audio-to-Video Baselines

Compared to AADiff, TPoS, and TempoToken, Maestro excels in representing semantically complex audio — maintaining object action fidelity and temporal consistency even with overlapping sound sources.

Quantitative evaluation compared with audio-to-video generation baselines on two datasets: VGGSound and Landscape. The bolded values are the best-performing results.

Comparison with Text-to-Video Baselines

Against multi-object text-to-video models (Peekaboo and TrailBlazer), Maestro achieves superior visual separation, accurate bounding-box placement, and natural object interactions.

Ablation: Attention Mosaic

Without Object Locating (OL) or Mosaic Denoising (MD), the model merges multiple audio conditions or fails to separate objects. Combining both produces perfectly separated, stable outputs.

Dynamic Sound Responsiveness

(a) Sound transitions visually shift from a crackling fire to extinguishing wind. (b) Volume changes alter the visual intensity and size of the generated fire — demonstrating dynamic audio responsiveness.

Hyperparameter Tuning

Ablation of timestep divider N_D (λ₃). A value of 6 balances clear object positioning (OL) and generation quality (MD).

Optimizing loss weight λ₁. Setting λ₁ = 0.9 (90% denoising loss) significantly improves contextual rendering over pure MSE.

Conclusion

We propose Maestro, a novel pipeline for multi-source audio-to-video generation that accurately produces multiple objects within frames by decomposing audio sources. Our Attention Mosaic technique ensures that generated objects are placed precisely where desired, enhancing fine details. Additionally, by applying an effective optimization strategy to the Condition Injection Module, we achieve the generation of better contextual expressions involving non-audible objects. Results from the user study indicate that Maestro significantly produces high-quality video from both multi-source and single-source audio, improving audio-visual alignment. We also observe that our method effectively captures basic characteristics such as volume changes and transitions.