Description: A video model conditioned on any motion.
motion (1931) video generation (16)
We visualize tracks and first frame inputs on the left and the generated video on the right. The tracks have trails to indicate their trajectory, and are colored to help distinguish them. Some motion prompts are created by converting user mouse inputs, in which case we visualize the mouse motions and drags by placing a cursor where the mouse is, and a hand if the user is dragging. Please note that this does not indicate that the videos are generated real-time. In fact, it takes about 12 minutes to sample ea
All videos on this page are cherry-picked from four samples, except for those in the "uncurated" section , in which we show uncurated samples generated by our model.
We first train a video generation model conditioned on any motion. To do this, we condition our model on point trajectories [19] [20] [21] — an incredibly flexible representation of motion. This allows us to encode the motion of just single points or thousands of points, the motion of specific objects or of a global scene, and even occlusions and temporally sparse motion.