TapMo: Shape-aware Motion Generation of Skeleton-free Characters

Jiaxu Zhang1†, Shaoli Huang2†, Zhigang Tu1*, Xin Chen3, Xiaohang Zhan2, Gang Yu3, Ying Shan2,
1Wuhan University, 2Tencent AI Lab, 2Tencent PCG
Paper Code arXiv

TapMo is a text-based animation pipeline for generating motion in a wide variety of skeleton-free characters.

Abstract

Previous motion generation methods are limited to the pre-rigged 3D human model, hindering their applications in the animation of various non-rigged characters. In this work, we present TapMo, a Text-driven Animation Pipeline for synthesizing Motion in a broad spectrum of skeleton-free 3D characters. The pivotal innovation in TapMo is its use of shape deformation-aware features as a condition to guide the diffusion model, thereby enabling the generation of mesh-specific motions for various characters. Specifically, TapMo comprises two main components - Mesh Handle Predictor and Shape-aware Diffusion Module. Mesh Handle Predictor predicts the skinning weights and clusters mesh vertices into adaptive handles for deformation control, which eliminates the need for traditional skeletal rigging. Shape-aware Motion Diffusion synthesizes motion with mesh-specific adaptations. This module employs text-guided motions and mesh features extracted during the first stage, preserving the geometric integrity of the animations by accounting for the character's shape and deformation. Trained in a weakly-supervised manner, TapMo can accommodate a multitude of non-human meshes, both with and without associated text motions. We demonstrate the effectiveness and generalizability of TapMo through rigorous qualitative and quantitative experiments. Our results reveal that TapMo consistently outperforms existing auto-animation methods, delivering superior-quality animations for both seen or unseen heterogeneous 3D characters.

Method Overview
An overview of the TapMo pipeline. Given a non-rigged mesh and a motion description input by the user, the Mesh Handle Predictor $\lambda (\cdot)$ predicts mesh handles and skinning weights to control the mesh. The Shape-aware Motion Diffusion $\mu (\cdot)$ generates a text-guided and mesh-specific motion for the character using the motion description and the mesh deformation feature ${f}_{\phi}$ extracted by the Mesh Handle Predictor.

Results


Technical Paper


Jiaxu Zhang, Shaoli Huang, Zhigang Tu, Xin Chen, Xiaohang Zhan, Gang Yu, Ying Shan
TapMo: Shape-aware Motion Generation of Skeleton-free Characters
arXiv:2310.12678, 2023.

BibTeX

@misc{zhang2023tapmo,
      title={TapMo: Shape-aware Motion Generation of Skeleton-free Characters}, 
      author={Jiaxu Zhang and Shaoli Huang and Zhigang Tu and Xin Chen and Xiaohang Zhan and Gang Yu and Ying Shan},
      year={2023},
      eprint={2310.12678},
      archivePrefix={arXiv},
      primaryClass={cs.GR}
}