A Simulation Benchmark for Dexterous Peg-in-Hole Assembly With Force-Tactile-Based Pose Estimation Across Multiple Embodiments

Abstract

In recent years, advances in robotic hand hardware and AI technologies have spurred increased research into dexterous embodiments. However, the application of most robotic hands remains confined to pick-and-place or in-hand reorientation, with limited capabilities in contact-rich and precise tasks such as assembly. For dexterous hands in particular, occlusions caused by the fingers and the hand body hinder precise visual perception, and in-hand object slippage or rotation introduces uncertainty into control. To facilitate research on this problem, we introduce a simulation benchmark in Isaac Gym for the alignment and insertion phase of dexterous peg-in-hole assembly, spanning multiple robotic arms, dexterous hands, and peg geometries. Building on this benchmark, we develop DexPiHNet, a Transformer-based pose-estimation network that fuses arm wrist force and finger tactile signals to infer both the in-hand peg pose and the peg-to-hole pose, and we use it within a unified closed-loop insertion strategy. In simulation, the proposed method achieves an average insertion success rate of 49.68% across the embodiments in the benchmark, outperforming learning-based temporal baselines and classical force-based search controllers at matched model size. A comprehensive pose-error analysis further characterizes the failure modes of the estimator and shows that closed-loop failures are primarily driven by error accumulation toward the late phase of insertion. We position this work as a simulation benchmark and a first learned baseline for dexterous peg-in-hole assembly rather than a deployment-ready solution; bridging the sim-to-real gap and validation on physical hardware are explicit directions for future work.