Skinned Motion Retargeting with Residual Perception of Motion Semantics & Geometry

1Wuhan University, 2Tencent AI Lab
CVPR 2023

R2ET is a neural motion retargeting model that can preserve source motion semantics and avoid interpenetration in target motion.


A good motion retargeting cannot be reached without reasonable consideration of source-target differences on both the skeleton and shape geometry levels. In this work, we propose a novel Residual RETargeting network (R2ET) structure, which relies on two neural modification modules, to adjust the source motions to fit the target skeletons and shapes progressively. In particular, a skeleton-aware module is introduced to preserve the source motion semantics. A shape-aware module is designed to perceive the geometries of target characters to reduce interpenetration and contact-missing. Driven by our explored distance-based losses that explicitly model the motion semantics and geometry, these two modules can learn residual motion modifications on the source motion to generate plausible retargeted motion in a single inference without post-processing. To balance these two modifications, we further present a balancing gate to conduct linear interpolation between them. Extensive experiments on the public dataset Mixamo demonstrate that our R2ET achieves the state-of-the-art performance, and provides a good balance between the preservation of motion semantics as well as the attenuation of interpenetration and contact-missing.

Method Overview

Overview of the proposed network R2ET, which has three decoupled modules, i.e., the skeleton-aware module $\Delta \mathcal{F}_{s}$, the shape-aware module $\Delta \mathcal{F}_{g}$, and the balancing gate $\mathcal{F}_{w}$. The Distance Matrix (DM) and the Distance Field (DF) are two types of distance measurements that guide the network to learn the information of semantics and geometry.

Technical Paper

Jiaxu Zhang, Junwu Weng, Di Kang, Fang Zhao, Shaoli Huang, Xuefei Zhe, Linchao Bao, Ying Shan, Jue Wang and Zhigang Tu*
Skinned Motion Retargeting with Residual Perception of Motion Semantics & Geometry
In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.


  title={Skinned Motion Retargeting with Residual Perception of Motion Semantics \& Geometry},
  author={Zhang, Jiaxu and Weng, Junwu and Kang, Di and Zhao, Fang and Huang, Shaoli and Zhe, Xuefei and Bao, Linchao and Shan, Ying and Wang, Jue and Tu, Zhigang},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},