VSSFlow: Unifying Video-conditioned Sound and Speech Generation via Joint Learning

Summary

The article introduces VSSFlow, a unified framework designed to integrate video-conditioned sound and speech generation tasks, specifically video-to-sound (V2S) and visual text-to-speech (VisualTTS). It addresses the challenges of handling heterogeneous input conditions through a novel condition aggregation mechanism. The framework utilizes distinct inductive biases from cross-attention and self-attention layers, optimizing the processing of ambiguous video conditions and deterministic speech transcripts. Contrary to traditional beliefs, VSSFlow demonstrates that joint training on these tasks can enhance performance without complex training strategies, leveraging a shared audio prior to improve convergence and stability in generation. Experimental results indicate that VSSFlow outperforms existing domain-specific models on both V2S and VisualTTS benchmarks, highlighting the potential of unified generative models in advancing multimodal machine learning applications.

Key Learnings

• 1VSSFlow effectively integrates V2S and VisualTTS tasks using a unified flow-matching framework.
• 2The framework employs a novel condition aggregation mechanism to manage distinct input signals.
• 3Cross-attention is utilized for ambiguous video conditions, while self-attention is reserved for deterministic speech transcripts.
• 4Joint training on V2S and VisualTTS tasks can enhance performance without the need for complex training stages.
• 5The learned general audio prior shared between tasks accelerates convergence and stabilizes the generation process.

Who Should Read This

Senior Machine Learning Researchers focusing on multimodal generative models and their applications in sound and speech generation.

Test Your Knowledge

Topics