Editor’s Note: On November 16, 2017, Synapse and Cambridge Consultants showcased brand new, groundbreaking technology, product design, and innovative development at our annual Innovation Day event.
The Day included presentations highlighting the progress and future of technological innovation and over two dozen technology demonstrations with application in consumer, medical, industrial, telecommunications, and other sectors.
One of the technology demonstrations Synapse presented was “Enhancing Human Capabilities through Augmented Reality”. In this blog, Jeff Hebert introduces the demo.
Demonstrating Our Worker Augmentation Concept
In a recent blog post, I said that augmented reality and machine vision technologies are unlocking the ability to train workers to perform skilled tasks and address some fluid and varied scenarios more economically than by investing in automation. With this concept in mind, we recently embarked on an internal investment project at Synapse to build such a system, picking the use case of an industrial worker assembling high-mix, low-volume circuits. Many other applications for this ‘human augmentation’ technology also exist, including industrial equipment maintenance, worker training, and even medical procedures.
Our demonstration has a workspace with a pegboard and a number of unassembled, colorful circuit parts to be placed onto the board, each with its own special location and orientation. Above the workspace is a basic video camera and in front of the user is a monitor. There are no instructions or other visual clues as to what to do beyond the screen highlighting parts and instructing the user where to place them. The system also checks each step performed for accuracy and helps the user fix issues as they occur.
Our demo setup - an augmented workstation
We built our demo with a basic video camera and screen as opposed to a head-mounted display (HMD) to focus attention on the software portion of the system as well as show how such an augmented workspace can be a relatively inexpensive investment with readily-available hardware. On the periphery of the screen in our demo, we show what the vision algorithm sees and how it works with picture-in-picture visualizations. For applications with a dedicated workspace like the one we’re showing, we’ve noticed that an HMD can pose challenges in terms of comfort for all-day use as well as battery life.
Our Custom Machine Vision Technology
Augmented reality is a hot subject and has lots of promise, but out-of-the-box functionality is focused on projecting images into the user’s field of view with little understanding of the environment. Apple’s AR Kit and software running on AR glasses like Hololens can understand the basic surfaces and space around the user—the walls, floor, ceiling, and basic objects like tables—and project imagery accordingly. This functionality is great for entertainment, gaming, and displaying reference material, but it doesn’t truly enhance the user’s own capabilities and senses. We wanted to show how AR can be used to intelligently guide a user in real time to see things they can’t see and do things they don’t already know how to do.
To get a better understanding of the scene and objects of interest within it, either custom hardware and sensors or much-more-sophisticated machine vision systems are required. For this demo, we wanted to see what we could achieve with only a basic video camera as the input, but other applications may benefit from or require custom sensors and hardware.
To successfully identify the circuit parts for our demo in all orientations, distances from the camera, lighting conditions, velocities, and amounts of occlusion by the user’s hands, we needed to create a robust machine vision system and train it with a lot of data. The classic way to train a vision system how to identify an object is to feed it a huge set of images of the object from all angles, in many environments (think using every result from a Google image search of a cat to identify a cat in a novel photo). Collecting such a data set is incredibly time consuming for a novel object and the process needs to start all over again if the object of interest changes.
To do this in a much more efficient way and set up a platform to speed the training of new objects, our team created a synthetic data set of our circuit parts by creating 3D solid models of them and using a gaming engine to render the objects in all orientations and lighting conditions with thousands of random background images.
On top of this machine vision system and synthetic dataset creation tool, our team built a lightweight application and graphical interface to show the user what part should be assembled next, where it should go, and it what orientation. As the user performs a step, the system checks it for accuracy and then moves on to the next step or suggests corrective action. We tested the system with different users and made improvements and adjustments for usability, accuracy, and latency along the way.
We’re excited about this demonstration not only because it shows how we can use AR and machine vision technology to train workers to do something economically useful, but also because we’ve built reusable components in the process that we can leverage and reference for future client projects to speed them up and reduce risk. We’re seeing an explosion of AR, machine vision, and machine learning opportunities and are looking forward to helping more clients in this space as it develops.
As the only form of public transit that is readily capable of supporting social distancing, shared bikes and scooters will be an essential component of an effective urban economic recovery strategy in the coming months. Prior to COVID-19, the city of San Francisco announced that future shared scooter platforms will be required to have deterrents to riding on sidewalks. Naturally, being both SF residents who will be affected by this decision, and curious engineers with micromobility industry experience, we decided to leverage our expertise in machine learning to explore the specifics involved in implementing a system which can determine if a scooter is riding on a street or a sidewalk.
Architecture development for innovative products is exciting because options abound and we can exercise our creativity. At the same time, uncertainty lurks in the user experience, business case, and technical realms. Learn how we navigate this challenging terrain to help our clients deliver world-class products to market.
The recent success of smart speakers has been a great leap in human-digital interaction, but there’s still a lot of space for developers and companies to cover to create smart devices and environments with truly engaging and intuitive interfaces. While voice command technology can handle simple tasks, the interaction can fall short because it has modest understanding of human intent.
Synapse is a product development firm. We work with the best companies in the world to drive innovation and introduce cutting-edge devices that positively impact our lives. Fueled by a desire to solve complex engineering challenges, we develop products that transform brands and accelerate advances in technology.