Undergraduate Research Project 1 - Robot Tactile Sensing Deep Learning Application

Project report

UROP1100_QIN ZHENG YAN.pdf

Project Video 

YouTube

哔哩哔哩

GitHub

github

Abstract

This research aims to automate the process of inserting data wires into circuit board slots using deep learning and tactile feedback. By addressing the challenges in industrial cable assembly, particularly with soft body objects, the goal is to enable robots to perform complex tasks that require tactile feedback. The study explores developing a deep learning model for classifying plugging actions for robotic control of plugging cables. The results indicate successful control and classification within the training data, but challenges arise when generalizing to new insertion positions. Future work includes expanding the dateset, improving tactile sensing capabilities, and investigating hybrid tactile sensing methods. The research contributes to the development of more efficient and flexible robotic systems for manipulating soft body objects in industrial applications.

Introduction

Tactile feedback remains a challenge in robotics, despite advancements in vision and hearing capabilities. While current tactile-based robotic arm systems excel at specific tasks like object manipulation and handling changing weights, they struggle with more complex tasks that require tactile feedback. This limitation is particularly evident when manipulating soft body objects such as cables and wires. The predefined movements of robotic arms hinder their ability to adapt to the messy and chaotic nature of these objects, leading to poor performance. Dynamic adjustments are necessary to account for misalignment and errors, but teaching robots to perform such tasks without predefined movements poses a significant challenge, especially in the industrial assembly of circuit connection wires and cables, where human labor still dominates.

The goal of our research was to develop a system capable of inserting data wires into circuit board slots. This system aimed to address the challenges in industrial cable assembly, enabling robots to automate cable plugging. In this paper, we will share our exploration of building a deep learning model as the first step to controlling the robotic arm, the process of collecting training data, and the challenges encountered during the development of such a model

Methodology

Our research focused on a specific scenario involving a flat data cable for the Raspberry Pi camera and the corresponding camera cable slot on the Raspberry Pi. We secured the Raspberry Pi on a 3-axis CNC platform, while a gripper held the cable in place. To capture tactile feedback, we mounted a 3-axis tactile sensor on one side of the gripper. The cable was secured using a flexible gel, with a clip providing additional security. The tactile sensor produced three values, representing the displacement of the gel surface along each axis, with the normal range from -2000 to 2000. 

Assumptions

To simplify our research, we made several assumptions. First, we assumed that the cable would be brought to a roughly accurate position relative to the slot using cameras. Tactile feedback would then be used to dynamically control the robot arm in case of misalignment and to determine the success of the plug. We reduced the plugging motion to a single axis, specifically the y-axis, which is perpendicular to the wide side of the cable. We also assumed that the plugging speed would remain constant throughout the experiment.

Dynamic motion control

To achieve dynamic robot motion control, we adopted a trial and error approach similar to binary search. Using the tactile sensor and the moving platform setup described above, we trained a deep learning model to classify the state of the plugging action into three categories: high, success, and low. Our system employed trial and error until the cable was successfully plugged in, following the control logic outlined below: 

 1. Plug in the cable (z-axis) until the sensor exceeds the force threshold. 

 2. Extract recorded data from step (1) and obtain the plug state classification from the model. 

 3. Increment the y position if the model predicts "too low," or decrease the y position if the model predicts "too high." Repeat step (1) if the model's output is not "success." 

Plug status classification model

For the plug status classification, we employed an LSTM network with a dense layer as the classification model. We classified three insertion conditions: too high, too low, and success. We used separate models for the classification of the success or not and the too high or too low classes. The outputs of these models were combined for the trial and error control algorithm

Data collection

To collect data for training and testing, we developed a control interface for the tactile sensor and CNC platform. The CNC platform allowed us to move the Raspberry Pi and enable sensor recording. We simulated different insertion positions (too high, too low, success) by moving the Pi board towards the data cable. The insertion distance was set to 2cm with a period of 3 seconds. We collected data in batches, with each batch containing around 10 to 50 insertion data of the same position. We collected approximately 10 batches of data for each class type, and each batch typically had a unique insertion position. The original dataset for the three classes had the following lengths: success (626 records), high (243 records), low (246 records)

We have defined the conditions for what class should the data be in. The ‘success’ class that means the y position is in the range for the cable to be perfectly plugged in. The ‘high’ or ‘low’ class means the y position is not correct, and the cable is not plugged in entirely into the slot.

Results

The performance of the classification models varied significantly depending on the training and testing data. When the training data included data from the same batches (same positions) as the testing data, the models achieved high accuracy of approximately 95% on unseen data. However, when the training and testing data came from different batches (different positions), the accuracy dropped to around 60%. This result indicates that the models struggled to generalize to new insertion positions.

Conclusion

Our research explored the use of deep learning and tactile feedback to automate cable plugging tasks for soft body objects. While we achieved some success in controlling the robotic arm using trial and error and classification models, there are still challenges to overcome. Collecting more diverse training data, improving tactile sensing capabilities, and exploring hybrid control approaches or reinforcement learning can further enhance the automation process. Advancements in these areas could pave the way for more efficient and flexible robotic systems capable of handling a wide range of soft body object manipulation tasks in various industrial applications. I would like to express my appreciation for Prof. Shen for providing me this opportunity to do research in his lab with his students. I found this experience fruitful and rewarding。