AI-Based User State Modeling for Interactive Systems: Creative Authoring, Skill Adaptation, and Affective Reconstruction

Abstract

Interactive systems increasingly use AI-based user state modeling to provide adaptive, personalized, or context-sensitive support. However, modeling a user state is not sufficient by itself. A system may estimate affect, performance, preference, or task context, but the value of that model depends on how the modeled state is integrated into the user’s ongoing interaction. This dissertation investigates AI-based user state modeling for interactive systems, focusing on tasks where users’ practices, behaviors, or experiences are difficult to directly observe, specify, or reconstruct. The dissertation examines this problem through three interactive systems. Sign Dance Maker supports creative authoring for inclusive music performance. Rather than dynamically modeling each user’s style during interaction, it uses design-time findings about sign language interpreters’ translation habits and authoring practices to create an editable visual authoring workspace with AI-generated translation drafts, dance references, and previews. AAA-CPR supports skill adaptation in CPR training by developing a wearable tactile-sensing platform, collecting force-labeled compression data, training a real-time force-estimation model, and connecting modeled behavioral states to adaptive audio- haptic feedback. PREFAB （PREFerence-Based Affective Modeling for Low-Budget Self-Annotation） is a cognition-informed approach to retrospective affective self-annotation. It identifies affective inflection regions, focuses user annotation on selected moments, and reconstructs the remaining trace through interpolation. Across these systems, technical evaluations and user studies show how user-informed design and AI-based state modeling can support interaction when design findings or model outputs are translated into concrete user-facing mechanisms. Sign Dance Maker showed how users’ authoring practices can inform interactive system design and motivated the need for more adaptive interactive system by integrating user state modeling. AAA-CPR then demonstrated how behavioral state modeling can support adaptive skill training, and PREFAB showed how cognition-informed affective state modeling can reduce annotation burden while preserving user judgment. This dissertation contributes three AI-mediated interactive systems, empirical findings across creative, embodied, and affective tasks, and design implications for integrating AI-based user state modeling into interaction.