Google has shown a demo of a generative AI that can create videos up to five seconds long. The tool is called Lumiere. This is still a demo and it is not yet known when Lumiere will become available to users.
Lumiere is able to generate photorealistic videos based on textual inputs. It is also possible to partially or fully animate images. Furthermore, Lumiere can mimic the style of an image, such as a drawing, and then create videos with it. It is even possible to edit videos with the program. In one of the examples that Google showschanges not only the color, but also the model of a dress that a woman is wearing, just by providing a textual input.
In a paper that on arXiv was published, Google’s research team describes how the software works. The team developed a new architecture called ‘Space-Time U-Net’. This makes it possible to generate a video in one go. This should distinguish the architecture from existing models, which generate distributed key frames in a first step, after which the intermediate frames are supplemented with temporal super-resolution. Temporal super-resolution is an image processing technique used to improve the temporal resolution of a video. The goal is to generate intermediate frames from the existing frames in a video, effectively increasing the video’s frame rate. This is not the case with Lumiere, which generates the images without that super resolution.
The generated output is currently limited to videos only five seconds long with a resolution of 1024×1024 pixels. Google itself considers that low resolution, but it is unclear whether future versions of the system will support higher resolution. Lumiere is currently a research project and therefore not yet available to the general public. It is not known when or if that will happen.