Meta Launches Movie Gen, an AI-Driven Tool that Converts Text into Videos and Sound

Meta has unveiled Movie Gen, a new AI-driven tool that transforms text into videos and sounds. This innovative generator allows users to convert photos into videos and create or extend soundtracks using text prompts. With this launch, Meta aims to compete with other media generation platforms like OpenAI's Sora, which is yet to be released.

Movie Gen offers four primary features: video generation, personalized video creation, precise video editing, and audio generation. The tool uses a 30B parameter transformer model to produce videos up to 16 seconds long at 16 frames per second. It combines text-to-image and text-to-video techniques for accurate object motion and camera movements.

Empowering Creativity with AI

Meta's goal is to democratise creativity by providing tools that enhance creative expression. They believe everyone should have access to such tools, whether they're aspiring filmmakers or casual creators. According to Meta's research, Movie Gen enables users to craft custom videos and sounds from simple text inputs, outperforming other models in tests.

The personalised video feature allows users to create unique videos from individual images using text prompts. This function excels at maintaining human identity and motion. Additionally, Movie Gen supports precise video editing with both localised edits (like adding elements) and global edits (such as changing backgrounds) without altering the main content.

Technical Innovations and Future Plans

Meta's foundation models have led to advancements in architecture, training methods, and evaluation protocols. Human evaluators consistently preferred Movie Gen over other industry alternatives across its four main capabilities. A comprehensive 92-page research paper details the technical insights of Movie Gen.

The audio generation feature uses a 13B parameter model to produce audio up to 45 seconds long, including sound effects and background music. All audio syncs with the video seamlessly, with an extension feature for longer soundtracks.

Challenges and Collaborations

Despite its potential, Meta acknowledges some limitations of Movie Gen, such as lengthy generation times and the need for further optimisation. They are actively working on these areas as development progresses. Meta plans to collaborate with filmmakers and creators for feedback-based refinements.

Chris Cox, Meta's chief product officer, mentioned on Threads that Movie Gen isn't ready for public release due to high costs and slow processing times but noted promising results. CEO Mark Zuckerberg announced that Instagram will integrate Movie Gen next year, showcasing a sample video made by the tool.

The company envisions a future where users can create personalised videos for platforms like Reels or generate custom animations for apps like WhatsApp. As development continues, Meta remains committed to sharing AI research with the public while empowering users through creative tools.