Seeing and Sensing Intention: Resolving Midas Touch with Vision-Motion Fusion in Mixed Reality

Abstract

The Midas Touch problem in XR (Extended Reality) environments is a significant factor that can degrade user experience. The vision-based interaction often leads to false alarms, where unintended user gestures are recognized. Particularly in MR environments, users can interact with virtual and real objects alternately, posing a risk of misrecognizing gestures intended for real objects as intentions towards virtual objects. We collected two kinds of data during interactions across multiple MR environments to analyze and address the Midas Touch problem: (1) false alarm data from interactions with real objects and (2) true pinch data from interactions with virtual objects. We present a multimodal model and analysis to understand the Midas Touch problem in MR environments. © 2025 Elsevier B.V., All rights reserved.