2) AR application for Pick and Place operations

The aim of this work was to learn how to utilize a Virtual Reality (VR) Head Mounted Display (HMD) for robotic applications. As a starting point, I decided to utilize one of the unity’s robotic hub tutorials. Specifically, the pick and place tutorial.

The niryo one robotic arm was simulated in Unity. The robot communicated with moveit running in an external PC using ROS to increase the level of autonomy. The communication was stablished by a ROS-TCP-Connector. Through the VR HMD the user could command the autonomous stack to initiate a pick and place operation. To ease the way the commands were transmitted to the autonomous stack a custom graphical interface was created using the Mixed Reality Toolkit.

To increase and ease interaction, the following design conisderations were taken:

1. Spatial information was incorporated by providing a label describing each simulated object within the application.
2. The user could position the object anywhere within the vicinity of the environment to do the pick and place operation. Additionally, the position of the object could be moved back into its original position by selecting a button within the custom graphical interface.
3. Visual feedback was given to the user after communicating with the autonomous stack. For example, when the user commanded to start the pick and place operation through the custom graphical interface the phrase “Pick and place has started” appeared to let the user know that the command was sent.