Neural Shape Compiler

3D shapes have complementary abstractions from low-level geometry to part-based hierarchies to languages, which convey different levels of information. This paper presents a unified framework to translate between pairs of shape abstractions: Text ⬌ Point Cloud ⬌ Program. We propose Neural Shape Compiler to model the abstraction transformation as a conditional generation process. It converts 3D shapes of three abstract types into unified discrete shape code, transforms each shape code into code of other abstract types through the proposed ShapeCode Transformer, and decodes them to output the target shape abstraction. Point Cloud code is obtained in a class-agnostic way by the proposed PointVQVAE. On Text2Shape, ShapeGlot, ABO, Genre, and Program Synthetic datasets, Neural Shape Compiler shows strengths in Text → Point Cloud, Point Cloud → Text, Point Cloud → Program, and Point Cloud Completion tasks. Additionally, Neural Shape Compiler benefits from jointly training on all heterogeneous data and tasks.

Figure 1: Overview of Neural Shape Compiler, and three transformation examples.

Figure 2: Neural Shape Compiler helps obtain hierarchical information for the reconstruction result of a single image, including its structural descriptions, regularities, and how to assemble it.

Figure 3: Text2PointCloud. Neural Shape Compiler can generate corresponding point clouds to the text prompt with structural details, while the baselines generate inaccurate structures or fewer alignments with the text prompt.

Figure 4: Text2PointCloud more results.

Figure 6: PointCloud2Text. One description per sentence. The shown shapes are from test sets, and Neural Shape Compiler tell their structures well.

Figure 6: PointCloud2Program. Neural Shape Compiler can well infer programs for the shown shapes of unseen categories and reconstruct shapes in a creative manner (e.g., represent the semicircle leg with bars).

• Data can be found in [Github].
• Our code and pre-trained models are released in [Github].