A mixed reality-based remote collaboration framework using improved pose estimation

Abstract

Mixed Reality (MR) technology integrates digital content with the real world to enable a cohesive user experience. Accurate pose estimation is crucial for aligning virtual content with physical surroundings, ensuring the virtual elements appear naturally in the user’s environment. This paper proposes a learning-based approach for accurate pose estimation using a monocular RGB (Red-Green-Blue) image, eliminating the need for markers and depth sensors. The method leverages YOLO6D (You Only Look Once Six-Dimensional) and a RoI (Region of Interest)-based color augmentation technique combined with Principal Component Analysis to enhance the accuracy of 6-DoF (Degrees of Freedom) pose estimation, while mitigating the effects of background variations and lighting changes. The proposed pose estimation method is incorporated into an MR-based remote collaboration framework, ensuring consistent and robust information rendering onto target objects across various devices. This integration enhances the reliability and effectiveness of MR-based remote collaboration. Experimental results demonstrate the superior performance of the proposed method, establishing it as a strong foundation for future MR-based remote collaboration frameworks. © © 2025. Published by Elsevier B.V.