Beyond Sound: AI-based Multimodal Music Systems for d/Deaf and Hard-of-Hearing Individuals from Appreciation to Creation and Therapy

Abstract

From an inclusive art perspective, artistic expression should not be confined to specific senses or restricted to particular groups. Emerging technologies, such as Generative AI, expand creative possibilities by engaging multiple senses. This advancement is particularly significant for individuals with disabilities, as it challenges the notion of “limited senses” and instead highlights their enhanced abilities in other sensory domains, offering new opportunities for artistic expression. As a multimodal art form, music integrates images, video, and text, making it inherently accessible beyond auditory perception. However, deaf and hard-of-hearing (DHH) individuals often encounter barriers to musical engagement due to limited access to sound. To address this, research has primarily focused on music-sensory substitution systems that translate musical elements into visual and tactile feedback. While these approaches help develop musical skills such as sound perception, instrument playing, and singing, they often fail to capture music's deeper artistic and emotional value. This study aims to support active and independent musical engagement for DHH individuals by integrating music-sensory substitution systems with Generative AI. This research highlights the intrinsic value of music for DHH individuals and aims to develop a technology-based inclusive music accessibility system. Rather than focusing solely on skill acquisition, it seeks to foster deeper experiences such as pleasure, self-awareness, and self-reflection. To achieve this, four key research objectives will be pursued: (1) conceptualizing rich emotional experiences in music through alternative sensory modalities, (2) developing an interactive music-making system based on song signing, (3) implementing an AI-driven autonomous music composition tool, and (4) designing and validating an LLM-based therapeutic music system for self-reflection. This study holds significance in three key aspects. First, it promotes inclusivity by recognizing DHH individuals as active participants in music culture. Second, it introduces a new multimodal paradigm for musical engagement that extends beyond the DHH community to benefit children, the elderly, and neurodivergent individuals. Third, it fosters empowerment by enhancing creative expression and self-confidence through technology, overcoming traditional limitations associated with hearing impairments. By emphasizing the need for continued attention to marginalized groups in technological advancements, this research contributes to more inclusive and equitable musical experiences. Ultimately, it envisions a society where technological innovations bridge gaps and enrich artistic expression for all, regardless of sensory abilities.