« The summer intern
I was a summer intern...and it was a game changer. From facial recognition to tinkering with toy dart guns to counting coffee consumption -- I DID IT ALL!
my facial recognition + coffee consumption project
Background
At the University of Colorado (and probably most other schools) there are two sides to summers. On one side, students get to take a break from classes, exams, and papers to enjoy the summer. In Boulder, that means tubing down Boulder Creek, hikes through Chautauqua Park, and trekking to your favorite bar every weekend. On the other side, it means finding an internship. We’re told as students that an internship during our time in college is crucial to finding a career after graduation. My summer internship brought me to nio.
It was a job posting from a previous nio intern (now full-time nio software engineer, Kylie Dale) that caught my eye. I did some more research on the company and the team and it really started to pique my interest so I sent in an application. I met Kylie for coffee and one conversation confirmed what I was already thinking, the technology was exciting, the people were smart, and the internship opportunity was unique. After meeting some more team members on the office visit, I was welcomed to the nio team as a software engineering intern.
Startup culture
I was quickly thrown into the startup culture as a member of the team. I wasn’t tasked with getting coffee or running mind-numbing errands. I was immediately given the opportunity to create. It was explained to me that this summer program was less about what I could provide to the development of nio and more about what I could make using the platform. I was asked to come up with a project that would both excite me and benefit from the use of the nio platform.
Why the nio platform is great!
My favorite part of the nio software platform is how much it can do for users with very little information; it listens for data and can do something smart based on that. But what if it could see? Instead of listening for the user to interact with it, what if it could see and recognize them? This lead me to facial recognition.
nio provides real-time processing for any type of signal, so facial recognition seemed to make sense. The nio platform also allows for a variety of nodes and processing blocks to connect painlessly. Combine that with a little machine learning, and facial recognition became increasingly less complicated to implement.
My face in nio
Visual display of facial recognition output
How I did it
I started with a trained neural network (provided by the Dlib library with a face_recognition python api created by Adam Geitgey) and used both face detection and encoding algorithms to get facial measurements. Those measurements were then compared against a list of known encodings to generate differences between them. Normalizing the differences between two encoding matrices provided a confidence value that was used to accurately identify a face from a video frame.
Wrapping all of this functionality into a nio block meant that anyone could be accurately identified in real time. nio would provide all of the reference data to the trained network and would use the resulting data in multiple ways.
The real challenge was implementing real-time logic. I needed a project to test my work. With the help of some nio nerds, we connected the facial recognition instance to a dart gun so I could shoot the people that were kind enough to donate reference encodings (i.e. their face).
"Wrapping all of this functionality into a nio block meant that anyone could be accurately identified in real time."
The dart gun project
The dart gun project was both fun and complicated. The gun would rotate on a homemade turret and fire at a user-set target. It needed to quickly identify faces from a camera and use both name and location data to control actuation and triggering of the simple turret. The name and location of the face would then be sent to a raspberry pi attached to the gun. The name of the person in front of the gun would be compared against the name of the set target. When those names matched, the face location data would be used to control the motors to both center the face in the frame and aim the gun below the face as an added “safety” feature. Once the target was found, centered, and aimed, a signal would be sent to the power relay attached to the trigger and fire the gun. Control engineering concepts were applied to create a dart gun that could both see and think.
For a kid that grew up shooting Nerf darts at homemade targets, and mostly my sister, this project was a dream come true. But I had some time left in my internship and wanted to explore facial recognition even more. I ended up finding a way to still be the coffee intern, but in a nio-fied way. I used facial recognition to count the beverage intake of everyone in the office.
The coffee counter
The system for counting beverages had to live in the background and work without extended interaction from the employees. The facial recognition instance on a server would grab broadcast video frames from a raspberry pi camera attached to the front of the coffee machine in the break room. It would identify anyone standing in front of it. This data was then combined with measured amperage values that the coffee machine pulled when making a beverage. Several blocks with simple tasks came together to create the logic that would count how many cups of any beverage an individual had made throughout the day. Employees never had to open an app to log their coffee consumption, and nobody ever had to yell to their voice-controlled speaker, “Tell the food tracker to log 8 oz of hot chocolate for Tyler!” The user simply had to stand in front of the machine and make their drink selection, allowing nio to do all of the work behind the scenes.
What I gained from working at niolabs
My internship at nio kept me creative and always on my toes. I had to learn fast and prototype quickly, which helped me to contribute to the vision shared by everyone who works there. The nio software platform is an amazing innovation tool. It allowed me to spend less time on setup, and spend more time on engineering the intelligence behind the system.
As my summer comes to a close and I prepare for my final year of college, I know exactly what I am working towards. I couldn’t have done it without the help of nearly everyone at nio. I truly enjoyed being a part of the team and I’m thankful that I was treated like more than just the summer intern.