Robust Scene Perception for Mobile Robots via 2D Vision Foundation Models

Abstract

Mobile robots operating in unstructured human environments must recognize, track, and segment a wide variety of objects, and Vision Foundation Models （VFMs） have recently made such perception broadly generalizable. Transferring this capability to an embodied platform, however, is far from straightforward: a robot observes the world through a moving camera, in environments whose appearance and object distributions differ from the training corpus, and under tight on-board computational budgets. This thesis studies two facets of the robustness these deployment challenges expose. The first is temporal robustness under ego-motion, and the second is distributional robustness under domain shift. In both cases, it shows that a powerful 2D VFM can be better leveraged for real robotic perception.

The first part addresses the loss of object identity caused by the frequent occlusion and frame-out events of egocentric viewing. We propose R3DVOT, a 3D-aware video object tracking framework that augments 2D VFMs with geometric reasoning. Its two core components, Position Modeling and Position-Aware Memory Selection （PAMS）, lift mask candidates into a canonical 3D world coordinate system using camera pose and depth, track the target's world-frame state with a Kalman filter, and control memory updates by 3D spatial consistency, so that identity is anchored to physical position rather than to a fragile 2D projection. This maintains long-term identity consistency through the occlusion and frame-out events typical of egocentric indoor perception, without additional training or dense 3D reconstruction.

The second part addresses adaptation to a new deployment domain without labels, under the constraint that VFMs are far too large to fine-tune on-device. In practice these two obstacles occur together: the model that most needs to adapt to an unfamiliar indoor scene is exactly the one too heavy to adapt on a mobile platform. We resolve them jointly with a cloud–edge test-time adaptation framework that separates where adaptation happens from where perception is used.

Heavy adaptation runs in the cloud, while the edge device performs forward inference only and adapts by loading updated weights, without ever computing a gradient. This separation is what makes the design effective. Because adaptation no longer has to fit on the device, the cloud can host a large VFM as a frozen teacher that the edge could never run, and its predictions do not depend on the student's own changing outputs, so they provide a stable supervisory signal that avoids the collapse seen in self-supervised adaptation over long horizons.

At the same time, the on-device loop stays lightweight, so the robot can keep uploading frames and downloading refreshed weights as it moves, and its perception continues to improve while it operates in the target environment. Since prior test-time adaptation has been limited mostly to classification and semantic segmentation, extending it to instance-level and open-vocabulary segmentation is itself new. A single framework, using task-specific heads and bipartite matching for instance-level queries, covers semantic segmentation, closed-set instance segmentation, and open-vocabulary instance segmentation.

Although the two parts target different perception tasks, they follow the same idea: each takes a powerful 2D VFM and makes it more usable for real robotic perception. R3DVOT restores 3D spatial consistency to keep object identity stable under ego-motion, and the cloud–edge framework enables label-free, on-device domain adaptation of large foundation models. Together, they move toward scene-perception systems that remain robust as both the viewpoint and the environment change, moving these systems closer to reliable deployment in the unstructured settings where mobile robots are increasingly expected to operate.