Shape of Motion: 4D Reconstruction from a Single Video
Description: DreamScene4D creates 4D Gaussian Representations from complex real videos.
3d gaussian splatting (11) dreamscene4d (1) 4d gaussian splatting (1) video-to-4d (1)
We present DreamScene4D, the first method capable of lifting multi-object monocular videos to 4D using dynamic Gaussian Splatting. It can handle large and complex motions observed in challenging real-life videos, thanks to object-scene decomposition and a motion factorization scheme.
DreamScene4D can generate arbitrary novel views for dynamic multi-object scenes across occlusions, as well as enable 2D point motion tracking by projecting the inferred 3D Gaussian trajectories to 2D, while never explicitly trained to do so.
(a) We decompose and amodally complete objects and the background in the video, then use DreamGaussian to obtain static 3D Gaussian representations. (b) Next, we factorize the object motion into multiple components and optimize them independently. (c) Finally, we re-compose the objects using monocular depth prediction guidance.
Shape of Motion: 4D Reconstruction from a Single Video