Adobe Firefly Announced to Challenge OpenAI Sora in the Race to Dominate Generative AI Video Tools

Adobe is venturing into generative AI for video with its Firefly Video Model, which was first hinted at earlier this year. This model is now being integrated into new tools, including some within Premiere Pro. These tools enable users to extend footage and create videos from still images and text prompts.

The Generative Extend tool, debuting in beta for Premiere Pro, allows users to lengthen the start or end of clips that are slightly short. It can also make mid-shot adjustments like correcting eye-line shifts or unexpected movements. However, clips can only be extended by two seconds, making it ideal for minor tweaks rather than full reshoots.

New Tools and Features

Two additional video generation tools are launching online. Adobe's Text-to-Video and Image-to-Video tools, initially announced in September, are now available as a limited public beta in the Firefly web app. Text-to-Video works similarly to other generators like Runway and OpenAI's Sora, where users input a text description to generate content.

Image-to-Video offers more control by allowing users to add a reference image alongside a text prompt. Adobe suggests this could help create b-roll from photos or visualize reshoots using stills from existing videos. However, it doesn't fully replace reshoots due to visible errors like wobbling cables and shifting backgrounds.

Limitations and Comparisons

The maximum length for Text-to-Video and Image-to-Video clips is five seconds, with quality capped at 720p and 24 frames per second. In contrast, OpenAI claims Sora can produce minute-long videos while maintaining quality and adhering to prompts, though it's not publicly available yet despite being announced before Adobe's tools.

Adobe's AI-generated clips take about 90 seconds to create, but the company is developing a "turbo mode" to reduce this time. Despite limitations, Adobe asserts its tools are "commercially safe" as they are trained on content the company had permission to use.

Commercial Viability and Future Prospects

This commercial safety could be crucial for users since models from other providers face scrutiny over alleged training on scraped YouTube videos or personal content. Another advantage is that videos made or edited with Adobe's Firefly model can include Content Credentials to disclose AI usage and ownership rights when published online.

The launch of these AI video tools was announced at Adobe's MAX conference alongside other AI-powered features across its creative apps. Although it's unclear when these tools will exit beta, their public availability sets them apart from competitors like OpenAI's Sora, Meta's Movie Gen, and Google's Veo generators.

Source