Unsupervised Domain Adaptation for 3D Hand Mesh Reconstruction via Mask-mesh Consistency

Abstract

Recent studies have explored the use of hand-object interaction videos to develop human-like skills for robots. Estimating hand pose and reconstructing the hand mesh is crucial for transferring human motion to robot motion in hand-object interaction videos. However, acquiring hand pose and hand mesh annotations is challenging, which often limits data collection to virtual or controlled laboratory environments. In this study, we propose a source-free domain adaptation method based on pseudo-labeling for hand mesh reconstruction using mask-mesh consistency. First, we construct a hand mesh reconstruction model with two separate heads: a segmentation head and a mesh head. Additionally, we assess the consistency between predictions from the two heads (segmentation masks and reconstructed meshes) to create reliable pseudo labels during adaptation to the target domain. These pseudo labels are then utilized to fine-tune the model for adaptation to the target domain. After applying our approach to the target domain, the model’s performance improves by 5% across all metrics for hand mesh reconstruction. © ICROS 2025.