Sora is an upcoming text-to-video model developed by OpenAI. The model generates short video clips based on user prompts, and can also extend existing short videos. As of December 2024, it is unreleased and not yet available to the public.[1] OpenAI has provided no official Sora release date.
A video generated by Sora of a woman walking down a Tokyo street | |
| Developer(s) | OpenAI |
|---|---|
| Platform | OpenAI |
| Type | Text-to-video model |
| Website | openai |
History
editSeveral other text-to-video generating models had been created prior to Sora, including Meta's Make-A-Video, Runway's Gen-2, and Google's Lumiere, the last of which, as of February 2024,[update] is also still in its research phase.[2] OpenAI, the company behind Sora, had released DALL·E 3, the third of its DALL-E text-to-image models, in September 2023.[3]
The team that developed Sora named it after the Japanese word for sky to signify its "limitless creative potential".[4] On February 15, 2024, OpenAI first previewed Sora by releasing multiple clips of high-definition videos that it created, including an SUV driving down a mountain road, an animation of a "short fluffy monster" next to a candle, two people walking through Tokyo in the snow, and fake historical footage of the California gold rush, and stated that it was able to generate videos up to one minute long.[2] The company then shared a technical report, which highlighted the methods used to train the model.[5][6] OpenAI CEO Sam Altman also posted a series of tweets, responding to Twitter users' prompts with Sora-generated videos of the prompts.
OpenAI has stated that it plans to make Sora available to the public but that it would not be soon; it has not specified when.[2][1] The company provided limited access to a small "red team", including experts in misinformation and bias, to perform adversarial testing on the model.[3] The company also shared Sora with a small group of creative professionals, including video makers and artists, to seek feedback on its usefulness in creative fields.[7]
On November 24th 2024, an API key for Sora access was leaked by a group of testers on Hugging Face, who posted a manifesto stating they protesting that Sora was used for "art washing". OpenAI revoked the access three hours after the leak was made public, and gave a statement that "hundreds of artists" have shaped the development, and that "participation is voluntary."[8]
Capabilities and limitations
editThe technology behind Sora is an adaptation of the technology behind DALL-E 3. According to OpenAI, Sora is a diffusion transformer[9] – a denoising latent diffusion model with one Transformer as the denoiser. A video is generated in latent space by denoising 3D "patches", then transformed to standard space by a video decompressor. Re-captioning is used to augment training data, by using a video-to-text model to create detailed captions on videos.[6]
OpenAI trained the model using publicly available videos as well as copyrighted videos licensed for the purpose, but did not reveal the number or the exact source of the videos.[4] Upon its release, OpenAI acknowledged some of Sora's shortcomings, including its struggling to simulate complex physics, to understand causality, and to differentiate left from right.[10] One example shows a group of wolf pups seemingly multiplying and converging, creating a hard-to-follow scenario.[11] OpenAI also stated that, in adherence to the company's existing safety practices, Sora will restrict text prompts for sexual, violent, hateful, or celebrity imagery, as well as content featuring pre-existing intellectual property.[3]
Tim Brooks, a researcher on Sora, stated that the model figured out how to create 3D graphics from its dataset alone, while Bill Peebles, also a Sora researcher, said that the model automatically created different video angles without being prompted.[2] According to OpenAI, Sora-generated videos are tagged with C2PA metadata to indicate that they were AI-generated.[4]
Reception
editWill Douglas Heaven of the MIT Technology Review called the demonstration videos "impressive", but noted that they must have been cherry-picked and may not be representative of Sora's typical output.[7] American academic Oren Etzioni expressed concerns over the technology's ability to create online disinformation for political campaigns.[4] For Wired, Steven Levy similarly wrote that it had the potential to become "a misinformation train wreck" and opined that its preview clips were "impressive" but "not perfect" and that it "show[ed] an emergent grasp of cinematic grammar" due to its unprompted shot changes. Levy added, "[i]t will be a very long time, if ever, before text-to-video threatens actual filmmaking."[2] Lisa Lacy of CNET called its example videos "remarkably realistic – except perhaps when a human face appears close up or when sea creatures are swimming".[3]
Filmmaker Tyler Perry announced he would be putting a planned $800 million expansion of his Atlanta studio on hold, expressing concern about Sora's potential impact on the film industry.[12][13]
See also
edit- VideoPoet – Text-to-video model by Google
- Dream Machine (text-to-video model)
References
edit- ^ a b Yang, Angela (February 15, 2024). "OpenAI teases 'Sora,' its new text-to-video AI model". NBC News. Archived from the original on February 15, 2024. Retrieved February 16, 2024.
- ^ a b c d e Levy, Steven (February 15, 2024). "OpenAI's Sora Turns AI Prompts Into Photorealistic Videos". Wired. Archived from the original on February 15, 2024. Retrieved February 16, 2024.
- ^ a b c d Lacy, Lisa (February 15, 2024). "Meet Sora, OpenAI's Text-to-Video Generator". CNET. Archived from the original on February 16, 2024. Retrieved February 16, 2024.
- ^ a b c d Metz, Cade (February 15, 2024). "OpenAI Unveils A.I. That Instantly Generates Eye-Popping Videos". The New York Times. Archived from the original on February 15, 2024. Retrieved February 15, 2024.
- ^ Brooks, Tim; Peebles, Bill; Holmes, Connor; DePue, Will; Guo, Yufei; Jing, Li; Schnurr, David; Taylor, Joe; Luhman, Troy; Luhman, Eric; Ng, Clarence Wing Yin; Wang, Ricky; Ramesh, Aditya (February 15, 2024). "Video generation models as world simulators". OpenAI. Archived from the original on February 16, 2024. Retrieved February 16, 2024.
- ^ a b Edwards, Benj (February 16, 2024). "OpenAI collapses media reality with Sora, a photorealistic AI video generator". Ars Technica. Archived from the original on February 17, 2024. Retrieved February 17, 2024.
- ^ a b Heaven, Will Douglas (February 15, 2024). "OpenAI teases an amazing new generative video model called Sora". MIT Technology Review. Archived from the original on February 15, 2024. Retrieved February 15, 2024.
- ^ Spangler, Todd (November 27, 2024). "OpenAI Shuts Down Sora Access After Artists Released Video-Generation Tool in Protest: 'We Are Not Your PR Puppets'". Variety. Retrieved December 2, 2024.
{{cite web}}: CS1 maint: url-status (link) - ^ Peebles, William; Xie, Saining (2023). "Scalable Diffusion Models with Transformers". 2023 IEEE/CVF International Conference on Computer Vision (ICCV). pp. 4172–4182. arXiv:2212.09748. doi:10.1109/ICCV51070.2023.00387. ISBN 979-8-3503-0718-4. ISSN 2380-7504. S2CID 254854389. Archived from the original on February 17, 2024. Retrieved February 17, 2024.
- ^ Pequeño IV, Antonio (February 15, 2024). "OpenAI Reveals 'Sora': AI Video Model Capable Of Realistic Text-To-Video Prompts". Forbes. Archived from the original on February 15, 2024. Retrieved February 15, 2024.
- ^ "Sora-generated video of wolves playing with some video issues". ABC News Australia. Retrieved May 16, 2024.
- ^ Kilkenny, Katie (February 23, 2024). "Tyler Perry Puts $800M Studio Expansion on Hold After Seeing OpenAI's Sora: "Jobs Are Going to Be Lost"". The Hollywood Reporter. Archived from the original on February 26, 2024. Retrieved February 26, 2024.
- ^ Edwards, Benj (February 23, 2024). "Tyler Perry puts $800 million studio expansion on hold because of OpenAI's Sora". Ars Technica. Archived from the original on February 26, 2024. Retrieved February 26, 2024.
External links
edit