Paper accepted at 2024 Conference on Robot Learning (CoRL)

6-9 November, 2024, Munich, Germany

Juan Del Aguila Ferrandis, Joao Moura and Sethu Vijayakumar, Learning Visuotactile Estimation and Control for Non-prehensile Manipulation under Occlusions, Proc. The Conference on Robot Learning (CoRL 2024), Munich, Germany (2024). [pdf] [video] [citation]

Abstract

Manipulation without grasping, known as non-prehensile manipulation, is essential for dexterous robots in contact-rich environments, but presents many challenges relating with underactuation, hybrid-dynamics, and frictional uncertainty. Additionally, object occlusions in a scenario of contact uncertainty and where the motion of the object evolves independently from the robot becomes a critical problem, which previous literature fails to address. We present a method for learning visuotactile state estimators and uncertainty-aware control policies for non-prehensile manipulation under occlusions, by leveraging diverse interaction data from privileged policies trained in simulation. We formulate the estimator within a Bayesian deep learning framework, to model its uncertainty, and then train uncertainty-aware control policies by incorporating the pre-learned estimator into the reinforcement learning (RL) loop, both of which lead to significantly improved estimator and policy performance. Therefore, unlike prior non-prehensile research that relies on complex external perception set-ups, our method successfully handles occlusions after sim-to-real transfer to robotic hardware with a simple onboard camera.