Jeff Hebert
Vice President of Engineering

Automation AR and Jobs: A Demonstration

Editor’s Note: On November 16, 2017, Synapse and Cambridge Consultants showcased brand new, groundbreaking technology, product design, and innovative development at our annual Innovation Day event.

The Day included presentations highlighting the progress and future of technological innovation and over two dozen technology demonstrations with application in consumer, medical, industrial, telecommunications, and other sectors.

One of the technology demonstrations Synapse presented was “Enhancing Human Capabilities through Augmented Reality”. In this blog, Jeff Hebert introduces the demo.

Demonstrating Our Worker Augmentation Concept

In a recent blog post, I said that augmented reality and machine vision technologies are unlocking the ability to train workers to perform skilled tasks and address some fluid and varied scenarios more economically than by investing in automation.  With this concept in mind, we recently embarked on an internal investment project at Synapse to build such a system, picking the use case of an industrial worker assembling high-mix, low-volume circuits.  Many other applications for this ‘human augmentation’ technology also exist, including industrial equipment maintenance, worker training, and even medical procedures.

Our demonstration has a workspace with a pegboard and a number of unassembled, colorful circuit parts to be placed onto the board, each with its own special location and orientation.  Above the workspace is a basic video camera and in front of the user is a monitor.  There are no instructions or other visual clues as to what to do beyond the screen highlighting parts and instructing the user where to place them.  The system also checks each step performed for accuracy and helps the user fix issues as they occur.

Our demo setup - an augmented workstation

We built our demo with a basic video camera and screen as opposed to a head-mounted display (HMD) to focus attention on the software portion of the system as well as show how such an augmented workspace can be a relatively inexpensive investment with readily-available hardware.  On the periphery of the screen in our demo, we show what the vision algorithm sees and how it works with picture-in-picture visualizations.  For applications with a dedicated workspace like the one we’re showing, we’ve noticed that an HMD can pose challenges in terms of comfort for all-day use as well as battery life.

Our Custom Machine Vision Technology

Augmented reality is a hot subject and has lots of promise, but out-of-the-box functionality is focused on projecting images into the user’s field of view with little understanding of the environment.  Apple’s AR Kit and software running on AR glasses like Hololens can understand the basic surfaces and space around the user—the walls, floor, ceiling, and basic objects like tables—and project imagery accordingly.  This functionality is great for entertainment, gaming, and displaying reference material, but it doesn’t truly enhance the user’s own capabilities and senses.  We wanted to show how AR can be used to intelligently guide a user in real time to see things they can’t see and do things they don’t already know how to do.

AR Application.png
A typical AR application showing projection of objects into a scene Image licensed under the Creative Commons Attribution-Share Alike 4.0 International license

To get a better understanding of the scene and objects of interest within it, either custom hardware and sensors or much-more-sophisticated machine vision systems are required.  For this demo, we wanted to see what we could achieve with only a basic video camera as the input, but other applications may benefit from or require custom sensors and hardware.

To successfully identify the circuit parts for our demo in all orientations, distances from the camera, lighting conditions, velocities, and amounts of occlusion by the user’s hands, we needed to create a robust machine vision system and train it with a lot of data.  The classic way to train a vision system how to identify an object is to feed it a huge set of images of the object from all angles, in many environments (think using every result from a Google image search of a cat to identify a cat in a novel photo).  Collecting such a data set is incredibly time consuming for a novel object and the process needs to start all over again if the object of interest changes.  

To do this in a much more efficient way and set up a platform to speed the training of new objects, our team created a synthetic data set of our circuit parts by creating 3D solid models of them and using a gaming engine to render the objects in all orientations and lighting conditions with thousands of random background images.

Animation showing images from our synthetic training data generation system

On top of this machine vision system and synthetic dataset creation tool, our team built a lightweight application and graphical interface to show the user what part should be assembled next, where it should go, and it what orientation.  As the user performs a step, the system checks it for accuracy and then moves on to the next step or suggests corrective action.  We tested the system with different users and made improvements and adjustments for usability, accuracy, and latency along the way.

Looking Forward

We’re excited about this demonstration not only because it shows how we can use AR and machine vision technology to train workers to do something economically useful, but also because we’ve built reusable components in the process that we can leverage and reference for future client projects to speed them up and reduce risk.  We’re seeing an explosion of AR, machine vision, and machine learning opportunities and are looking forward to helping more clients in this space as it develops.


See what else is new...

August 8, 2018

One Step Closer to Instant Package Delivery

The holy grail of last mile logistics is cost-efficient instant package delivery, to wherever the recipient happens to be. The DelivAir system we’ve prototyped takes us a step closer to that by using drones, and a novel precision location and authentication technology to deliver packages directly to your hands.

May 9, 2018

Alexa, what more could you do?

We're exploring ways to take the voice enabled user interface to the next level by augmenting it with other technologies or optimizing it with custom engineering.

See what else is new...

May 21, 2018

Unlocking Lower-Cost Robotic Automation

The critical decision regarding investment in automation comes down to economics, and for many applications, automation still doesn’t pencil out. As Elon Musk recently acknowledged regarding automation at Tesla, humans are underrated—we’re flexible and capable. The trend toward automation will continue though, and we see a way to leverage low-latency, high-accuracy, outside-in tracking technology from virtual reality to significantly reduce the cost and expand the opportunities for robotic automation systems. Most of today’s robotic automation systems are currently burdened by expensive inside-out encoding and tracking systems as well as higher costs and weights to handle accuracy despite worst-case loads and degrees of freedom.

August 8, 2018

One Step Closer to Instant Package Delivery

The holy grail of last mile logistics is cost-efficient instant package delivery, to wherever the recipient happens to be. The DelivAir system we’ve prototyped takes us a step closer to that by using drones, and a novel precision location and authentication technology to deliver packages directly to your hands.

June 28, 2018

Making Best Practices the Default with GitLab

GitLab was recently in the house for DevOps Stories Seattle, "a single day symposium dedicated to real-life DevOps transformation stories." Hosting the event here in our Seattle office was a great opportunity for us to cross-pollinate with other people and organizations that are using GitLab to solve their own challenges. Software Engineering Program Lead, Jason Haensly, joined in on the knowledge-sharing with a presentation describing how we use GitLab throughout the product development process, and why we think it's such a win.