=Overview

3D shapes have complementary abstractions from low-level geometry to part-based hierarchies to languages, which convey different levels of information. This paper presents a unified framework to translate between pairs of shape abstractions: TextPoint CloudProgram. We propose Neural Shape Compiler to model the abstraction transformation as a conditional generation process. It converts 3D shapes of three abstract types into unified discrete shape code, transforms each shape code into code of other abstract types through the proposed ShapeCode Transformer, and decodes them to output the target shape abstraction. Point Cloud code is obtained in a class-agnostic way by the proposed PointVQVAE. On Text2Shape, ShapeGlot, ABO, Genre, and Program Synthetic datasets, Neural Shape Compiler shows strengths in TextPoint Cloud, Point CloudText, Point CloudProgram, and Point Cloud Completion tasks. Additionally, Neural Shape Compiler benefits from jointly training on all heterogeneous data and tasks.

Figure 1: Overview of Neural Shape Compiler, and three transformation examples.

Figure 2: Neural Shape Compiler helps obtain hierarchical information for the reconstruction result of a single image, including its structural descriptions, regularities, and how to assemble it.

=Results

Figure 3: Text2PointCloud. Neural Shape Compiler can generate corresponding point clouds to the text prompt with structural details, while the baselines generate inaccurate structures or fewer alignments with the text prompt.

Figure 4: Text2PointCloud more results.

Figure 6: PointCloud2Text. One description per sentence. The shown shapes are from test sets, and Neural Shape Compiler tell their structures well.

Figure 6: PointCloud2Program. Neural Shape Compiler can well infer programs for the shown shapes of unseen categories and reconstruct shapes in a creative manner (e.g., represent the semicircle leg with bars).

=Resources

  • Data can be found in [Github].
  • Our code and pre-trained models are released in [Github].

=Related Publication

LLVM: A Compilation Framework for Lifelong Program Analysis & Transformation

Chris Lattner, Vikram Adve,