Points-to-3D: Structure-Aware 3D Generation with Point Cloud Priors

Jiatong Xia*, Zicheng Duan*, Anton van den Hengel, Lingqiao Liu†
Australian Institute for Machine Learning, University of Adelaide
*Equal contribution †Corresponding author
Paper Supplementary arXiv (Coming Soon) Code (Coming Soon)
Points-to-3D Teaser

We introduce explicit 3D point cloud priors into 3D generation framework. Given a pre-existing point cloud or a feed-forward point cloud prediction from image input, our model generates high-quality 3D assets that faithfully preserve the observed structure while plausibly completing unobserved regions with coherent geometry.

Method Overview

Method Overview

Overall framework. Given point cloud priors—either pre-existing or predicted by VGGT from input image—we first voxelize and VAE-encode it to obtain an SS latent, where the empty regions are filled with random noise and concatenated with an extracted mask to form the input paradigm for our model. During training, we optimize our inpainting flow transformer via conditional flow matching loss. During inference, we employ a two-stage sampling procedure: (1) structural inpainting with s steps to inpaint the global structure, and (2) boundary refinement with remaining (t-s) steps to refine the inpainting boundaries.

Abstract

Recent progress in 3D generation has been driven largely by models conditioned on images or text, while readily available 3D priors are still underused. In many real-world scenarios, visible-region point clouds are easy to obtain—from active sensors such as LiDAR or from feed-forward predictors like VGGT—offering explicit geometric constraints that current methods fail to exploit.

In this work, we introduce Points-to-3D, a diffusion-based framework that leverages point cloud priors for geometry-controllable 3D asset and scene generation. Built on the latent 3D diffusion model TRELLIS, Points-to-3D first replaces pure-noise sparse structure latent initialization with a point cloud priors tailored input formulation. A structure inpainting network, trained within the TRELLIS framework on task-specific data designed to learn global structural inpainting, is then used for inference with a staged sampling strategy (structural inpainting followed by boundary refinement), completing the global geometry while preserving the visible regions of the input priors.

In practice, Points-to-3D can take either accurate point-cloud priors or VGGT-estimated point clouds from single images as input. Experiments on both single-object and multi-object scenarios consistently demonstrate superior performance over state-of-the-art baselines in terms of rendering quality and geometric fidelity, highlighting the effectiveness of explicitly embedding point-cloud priors for achieving more accurate and structurally controllable 3D generation.

Training Data Construction

Training Data Processing

Training data processing. We preserve the visible portion of the complete point cloud and convert it into training inputs. We render depth maps from multiple viewpoints and extract visible point clouds using depth consistency checks, then encode them into sparse structure latents paired with ground-truth complete structures for training.

Point Cloud Priors

Point Cloud Examples

Input point cloud priors examples. We support two types of point cloud priors: (1) Sampled Point Cloud - partial point clouds directly captured by hardware sensors (e.g., LiDAR on an iPhone), and (2) VGGT Estimated - point clouds estimated from input images via feed-forward point-map prediction. Both types impose reliable geometric constraints that steer our model toward controllable and faithful 3D generation.

Experimental Results

Single-Object Generation on Toys4K

Method Rendering Geometry
PSNR↑ SSIM(%)↑ LPIPS↓ DINO(%)↓ CD↓ F-Score↑ PSNR-N↑ LPIPS-N↓
GaussianAnything 20.08 89.31 0.183 26.74 0.084 0.513 20.99 0.199
Real3D 19.55 90.65 0.169 27.65 0.065 0.574 21.31 0.178
LGM 20.55 89.98 0.181 23.45 0.075 0.487 20.04 0.202
VoxHammer (3D Inversion) 20.51 90.01 0.123 15.10 0.046 0.724 20.28 0.158
TRELLIS 21.94 91.46 0.105 7.82 0.034 0.832 23.81 0.105
Points-to-3D (Ours-VGGT Esti.) 22.55 92.09 0.088 7.37 0.024 0.881 24.53 0.085
Points-to-3D (Ours-P.C.Priors) 22.91 92.83 0.070 7.29 0.013 0.964 27.10 0.053

Table 1. Comparison on single-object generation on Toys4K dataset. We showcase the performance of our method in two scenarios: one where explicit point cloud priors are provided, and another where point clouds are inferred from condition images using VGGT.

Toys4K Results

Single-object generation on Toys4K. For the explicit point cloud priors results, we use point clouds extracted strictly from the visible region of input images, whereas the "VGGT-estimated" results use point clouds inferred from the condition images by VGGT. Our method consistently outperforms existing approaches across all evaluation metrics. With point cloud priors, we achieve an F-score of 0.964, demonstrating that our approach produces geometry that closely approximates the ground-truth structure. Normal maps further highlight the superior geometric quality achieved by our approach.

Multi-Object Generation on 3D-FRONT

Method Rendering Geometry
PSNR↑ SSIM(%)↑ LPIPS↓ DINO(%)↓ CD↓ F-Score↑ PSNR-N↑ LPIPS-N↓
TRELLIS 18.21 83.12 0.239 12.33 0.094 0.478 18.76 0.258
VoxHammer (3D Inversion) 19.29 84.70 0.179 18.41 0.051 0.686 20.43 0.181
SceneGen 18.32 83.35 0.231 14.43 0.086 0.485 19.08 0.229
MIDI 19.23 85.59 0.166 14.25 0.075 0.513 20.82 0.164
Points-to-3D (Ours-VGGT Esti.) 20.52 86.51 0.152 8.90 0.040 0.743 20.97 0.160
Points-to-3D (Ours-P.C.Priors) 21.63 87.73 0.124 8.29 0.025 0.886 22.38 0.124

Table 2. Comparison on multi-object generation on 3D-FRONT dataset. Points-to-3D consistently outperforms state-of-the-art multi-object generation methods across all evaluation metrics.

3D-FRONT Results

Multi-object generation on 3D-FRONT. Our method achieves significant improvements across all evaluation metrics compared to other methods in complex multi-object scenarios. Unlike MIDI or SceneGen which implicitly utilize spatial information, our framework explicitly incorporates geometric priors within the architecture, enabling more direct and effective control over 3D geometry. The rendered images and normal maps demonstrate that our results better align with the ground-truth scene geometry and visual appearance.

Visible Region Performance

Methods CD↓ F-Score↑ PSNR-N↑ LPIPS-N↓
TRELLIS-O. 0.034 0.832 23.81 0.105
TRELLIS-V. 0.032 0.854 24.77 0.093
Points-to-3D-O. 0.013 0.964 27.10 0.053
Points-to-3D-V. 0.007 0.998 29.00 0.036

Table 3. Comparison on visible and overall geometry results on Toys4K. For each method, the upper row (O.) shows the overall results, while the lower row (V.) shows the visible region results. Our method achieves F-score of 0.998 in visible regions, demonstrating strong alignment with ground truth structure.

Ablation Study: Staged Sampling Strategy

Inp. Ref. CD↓ F-Score↑ PSNR-N↑ LPIPS-N↓
50 0 0.014 0.960 25.88 0.065
40 10 0.013 0.962 26.49 0.059
30 20 0.013 0.963 26.89 0.056
25 25 0.013 0.963 27.10 0.053
20 30 0.013 0.962 27.03 0.055
10 40 0.014 0.961 26.72 0.061

Table 4. Ablation study on the number of inpainting steps (Inp.) and refinement steps (Ref.) in our sampling strategy. The best performance is achieved with 25 inpainting steps and 25 refinement steps.

Ablation Study

Ablation study. Allocating the full sampling to inpainting (Inp.) results in geometric "holes" along the inpainting edge. By setting the sampling schedule to 25 inpainting steps followed by 25 refinement steps, the geometric metrics reach their best performance, and the previously observed "holes" are effectively eliminated, yielding the overall best generation results.

Real-world Examples

Real-world Examples

Real-world examples on Pix3D. Our approach maintains robust performance on real image inputs, producing geometry that aligns more faithfully with the input images compared to the baseline method.

Text-to-3D Generation

Text-to-3D Results

Text-to-3D generation on Toys4K. Our method successfully generates geometries that are semantically consistent with the input prompts and structurally well-controlled by the given point cloud priors.

Multi-View Input Generation

Multi-view Results

Generation results with 3 input views on Toys4K. Our flow-based model can directly incorporate multi-view reference images at different denoising steps. For VGGT-estimated point clouds, multi-view inputs produce more accurate predictions. With accurate point cloud priors extracted from three views, our method produces reconstructions that are very close to the ground truth, demonstrating the robustness and effectiveness across different numbers of input images.

Additional Results

More Results

More generation examples. Our method demonstrates consistent high-quality results across diverse object categories and scene types, highlighting the effectiveness of explicitly embedding point-cloud priors for achieving accurate and structurally controllable 3D generation.

Key Contributions

  • Explicit 3D Prior Integration: Unlike existing methods that condition solely on images or text, we explicitly incorporate point cloud priors into the latent space, enabling precise geometric control over generated 3D assets.
  • Inpainting-based Framework: We formulate 3D generation as a latent inpainting problem, preserving observed structures while plausibly completing unobserved regions guided by visible-region cues.
  • Staged Sampling Strategy: Our two-stage approach (structural inpainting + boundary refinement) effectively addresses the "holes" problem at inpainting boundaries, ensuring both global consistency and local precision.
  • Flexible Input Support: The method supports both hardware-sensed point clouds (e.g., LiDAR) and feed-forward predicted point clouds (e.g., VGGT), enabling deployment in various real-world scenarios.
  • Superior Performance: Experiments demonstrate F-scores up to 0.998 in visible regions and 0.964 overall, significantly outperforming state-of-the-art baselines in both rendering quality and geometric fidelity.