Automatic Sign Dance Generation with Diffusion Based Inpainting

Abstract

Our research presents a novel method for creating sign language for music. Our approach involves automatically generating sign language that incorporates both the musical and lyrical aspects of songs, resulting in a full-body choreography of sign dance. We have created the first dataset for sign dance, which is unique in its emphasis on lyrically-driven sign language and its inclusion of full-body dance movements. The dataset focuses on the expressiveness of terminal joints, such as fingers, and rhythmic body movements in response to music. It is a valuable contribution to sign language research. Our methodology is inspired by the dance generating diffusion model, which integrates spatio-temporal motion inpainting in the sampling process. This approach ensures natural body movement synchronization with music while preserving key joint movements essential for sign language. To validate our approach, we conducted quantitative and qualitative evaluations, including a user study with certified sign language translators. The results demonstrate that our method effectively aligns with the music beat and has potential applications in enhancing sign language translation workflows.