upfusion3d.github.io - UpFusion: Novel View Diffusion from Unposed Sparse View Observations

home > domains > upfusion3d.github.io

Description: UpFusion is a system that can perform novel view synthesis of an object given a sparse set of reference images without corresponding pose information.

3d (9633) diffusion (262) unposed (9) novel view synthesis (9) sparse (6) upfusion (1)

Example domain paragraphs

We propose UpFusion, a system that can perform novel view synthesis and infer 3D representations for an object given a sparse set of reference images without corresponding pose information.

Current sparse-view 3D inference methods typically rely on camera poses to geometrically aggregate information from input views, but are not robust in-the-wild when such information is unavailable/inaccurate. In contrast, UpFusion sidesteps this requirement by learning to implicitly leverage the available images as context in a conditional generative model for synthesizing novel views.

We incorporate two complementary forms of conditioning into diffusion models for leveraging the input views: a) via inferring query-view aligned features using a scene-level transformer, b) via intermediate attentional layers that can directly observe the input image tokens. We show that this mechanism allows generating high-fidelity novel views while improving the synthesis quality given additional (unposed) images.

Links to upfusion3d.github.io (5)



© example3.com 2024