Qiao Gu - Qiao Gu’s Homepage
Zhaoyang Lv
Description: EgoLifter: Open-world 3D Segmentation for Egocentric Perception
3d mapping (54) 3d reconstruction (45) foundation models (14) egocentric perception (7) open-set (4) 3d segmentation (4) egolifter (1)
EgoLifter augments 3D Gaussians with feature vectors and encode object identity using contrastive learning. Individual 3D objects can therefore be extracted by click-query or clustering.
In this paper we present EgoLifter , a novel system that can automatically segment scenes captured from egocentric sensors into a complete decomposition of individual 3D objects. The system is specifically designed for egocentric data where scenes contain hundreds of objects captured from natural (non-scanning) motion. EgoLifter adopts 3D Gaussians as the underlying representation of 3D scenes and objects and uses segmentation masks from the Segment Anything Model (SAM) as weak supervision to learn flexible
EgoLifter solves 3D reconstruction and open-world segmentation simultaneously from egocentric videos. EgoLifter augments 3D Gaussian Splatting with instance features and lifts open-world 2D segmentation by contrastive learning, where 3D Gaussians belonging to the same objects are learned to have similar features. In this way, EgoLifter solves the multi-view mask association problem and establishes a consistent 3D representation that can be decomposed into object instances.
Qiao Gu - Qiao Gu’s Homepage
Zhaoyang Lv