$\dagger$: "B. Dec." is a baseline method considered in our experiments. This baseline excludes color and material properties from the generation process, i.e., it generates a 3D shape, then predicts color conditioned on the shape through a decoder, and then the material through another decoder. The choice of this baseline attempts to answer the question of whether there is any benefit of incorporating the physical materials in the generation process. Please refer to our paper for more details.

$\ddagger$: "B. Perc." is a baseline method considered in our experiments. This baseline uses perceptual models to estimate material properties based on an off-the-shelf 3D generation model. Specifically, we adopt TRELLIS, a state-of-the-art 3D generation model, to generate textured 3D shapes given image or text conditions. To obtain the material properties of each generated shape, we then leverage an open-vocabulary 3D part segmentation model, Find3D, to get the part labels for the sampled surface points. Note that the query part names we provide to Find3D are derived from the set of part labels in our dataset, which comes from 3DCoMPaT200. Finally, we leverage ChatGPT-4o by providing it with two renderings of the generated 3D object and asking it to estimate the material properties for each part of the object retrieved by Find3D. Please refer to our paper appendix for more details.