Sign Dance Maker: A Generative AI-Assisted Framework for Inclusive Music Performance Support for Sign Language Interpreters

Abstract

Inclusive musical performances have been crucial in bridging the gap between hearing and Deaf communities, with sign language interpreters playing a vital role. However, these interpreters face significant challenges regarding musical expression and translation efficiency. We introduce a generative AI-based Sign Dance Maker (SDM) tool to address these issues. Informed by focus group interviews with four experienced music sign language interpreters, we designed SDM to integrate a fine-tuned dance generation model with a Large Language Model. This system enables text-based 3D motion editing, specifically supporting visual feedback and a streamlined translation process. A user study with 10 interpreters confirmed that SDM enhances creative expression while reducing cognitive burden in both qualitative and quantitative measures. Beyond performance applications, our findings suggest potential extensions to sign language education and improving accessibility in essential services, contributing to more inclusive cultural experiences through AI-assisted sign dance creation.