For designing a wide range of everyday objects, the design process should be aware of both the human body and the underlying semantics of the design specification. However, these two objectives present significant challenges to the current AI-based designing tools. In this work, we present a method to synthesize body-aware 3D objects from a base mesh given an input body geometry and either text or image as guidance. The generated objects can be simulated on virtual characters, or fabricated for real-world use. We propose to use a mesh deformation procedure that optimizes for both semantic alignment as well as contact and penetration losses. Using our method, users can generate both virtual or real-world objects from text, image, or sketch, without the need for manual artist intervention. We present both qualitative and quantitative results on various object categories, demonstrating the effectiveness of our approach.
Our method generates a variety of semantics and body-aware objects from input text prompts.
Real-world fabrication. We fabricated the objects in the real world with 3D printing. The objects can be worn as accessories on real people and characters.
Our goal is to design rigid objects that satisfy diverse contact constraints for different body shapes and semantics. We synthesize body-aware 3D objects from a base mesh given an input body geometry and either text or image as guidance. We propose to use a mesh deformation procedure that optimizes for both semantic alignment as well as contact and penetration losses. The user has the option to specify the semantic goals with a text prompt or an input image. Depending on the input modality, our system uses different losses to guide the optimization.
We introduce two losses that enable generated objects to integrate seamless onto an user's body. We introduce $\mathcal{L}_c$ for contact loss, and $\mathcal{L}_p$ for penetration loss, and we define the overall body-aware loss to be $\mathcal{L}_b = \mathcal{L}_c + \mathcal{L}_p$.
Given contact vertices $\mathcal{V}_c$, the contact loss $\mathcal{L}_c$ is defined as
\begin{equation}
\mathcal{L}_c(\mathcal{V}, \mathcal{V}_c) = \lambda_c\frac{1}{|\mathcal{V}_c|} \sum_{v_c \in \mathcal{V}_c} \min_{v \in \mathcal{V}} || v_c - v||^2_2,
\label{eq:contact_loss}
\end{equation}
where $\lambda_{contact}$ is a tunable weight. This encourages the object to be in contact with the body vertices specified by the input contact vertices. To reduce penetration between the object and the body mesh $\mathcal{M}_b$, we include an additional loss $\mathcal{L}_p$:
\begin{equation}
\mathcal{L}_p(\mathcal{M}, \mathcal{M}_b) = \sum_{d_i < D} {d_i}^2, \label{eq:pen_loss} \end{equation} where $d_i$ are signed distances between the object and the body mesh, and $D$ is the penetration distance threshold. In total, the body-aware optimization loss is defined as: \begin{equation} \mathcal{L}_b(\mathcal{V}, \mathcal{V}_c, \mathcal{M}, \mathcal{M}_b)=\lambda_c \mathcal{L}_c(\mathcal{V}, \mathcal{V}_c) + \lambda_p \mathcal{L}_p(\mathcal{M}, \mathcal{M}_b). \end{equation}
Shape optimization through Jacobians. The design parameterization plays a significant role in the difficult design optimization problem. Naive optimization of the mesh deformation through vertex displacement can result in significant artifacts and is prone to convergence to local minima. Inspired by Neural Jacobian Fields, we indirectly optimize the deformation map by optimizing a set of per-triangle Jacobian matrices $J_i\in\mathbb{R}^{3\times 3}$ for every face $f_i\in\mathcal{F}$. The deformation map $\Phi^\ast$ is computed as the mapping with Jacobian matrices that are closest to $\{J_i\}$, solved via the following Poisson optimization problem: % By solving the following Poisson optimization problem, we compute a deformation map $\Phi^\ast$ as the mapping with Jacobian matrices for each face that are closest to $\{J_i\}$: \begin{equation} \Phi^\ast = \min_{\Phi} \sum_{f_i\in\mathcal{F}}|f_i|\lVert \nabla_i(\Phi) - J_i\rVert_2^2, \label{eq:poisson_equation} \end{equation} where $\nabla_i(\Phi)$ denotes the Jacobian of $\Phi$ at triangle $f_i$ and $|f_i|$ is the area of that triangle.
Different body shapes. We evaluate our system on different body shapes, ranging from human adults and children to virtual characters such as dinosaurs and cartoon-looking cows.
Sketch to body-fitting 3D object. We show a sketch application with ShapeCraft. We ask a user to draw a sketch of an object, and we use ControlNet to convert the sketch into a 2D image. The image is used as an input to our image-guided mesh deformation method.
This work is in part supported by the Stanford Institute for Human-Centered AI (HAI), the Stanford Center for Integrated Facility Engineering (CIFE), Nissan, and NSF RI #2338203. MG is supported by NSF GRFP. Ruohan Zhang is partially supported by Wu Tsai Human Performance Alliance Fellowship.
@misc{guo2024shapecraft,
title={ShapeCraft: Body-Aware and Semantics-Aware 3D Object Design},
author={Michelle Guo and Mia Tang and Hannah Cha and Ruohan Zhang and C. Karen Liu and Jiajun Wu},
year={2024},
eprint={2412.03889},
archivePrefix={arXiv},
primaryClass={cs.CV}
}