IPAB Workshop - 07/07/2022

Speaker: Zhaole Sun

Title: Deformable Linear Object Perception in 3D

Abstract: Deformable linear object (DLO such as ropes, wires, tubes, and pipes) perception in 3D space is crucial for DLO manipulation. However, compared to rigid body perception, it is difficult to perceive DLOs in 3D space from one RGB or RGBD image due to their thin and deformable structures. Previous DLO segmentation methods do not segment DLOs well because of occlusions, image blur, extreme lighting, and transparent material, and depth sensors usually fail to capture full depth information of DLOs. To address these problems and provide a suitable DLO state estimate for downstream tasks like DLO tracking, we propose a DLO perception pipeline that consists of shape recovery and depth completion—after coarse segmentation of DLOs, which often contain gaps, holes, and occlusions, the proposed method can recover the DLO shape by pixel-wise filling gaps and holes and then perform depth completion to infer complete depth maps of the segmented DLOs. The pipeline is trained using only synthetic data without needing any Sim-to-Real fine-tuning and, through experiments, we demonstrate that the proposed shape recovery and depth completion methods lead to 22% and 14% reductions of 2D errors of DLO masks and 3D errors of DLO projected depth masks measured by Chamfer distance compared to directly applying coarse 2D masks on depth maps. Besides the perception pipeline, we present a dataset containing DLOs with different configurations from simulation environments for training the shape recovery process and real-world DLO datasets for valuation.