Machine learning can be used for more than violating your privacy for a social media challenge. For example, one fascinating application has been developed by Instrumental AI, which uses machine learning to detect defects and anomalies in photographs of parts during various stages of assembly, primarily in the electronics manufacturing industry.
Instrumental was founded by Anna Shedletsky, a former Apple engineer with two degrees from Stanford. A mechanical engineer by training, Shedletsky led system products design for the Apple Watch for six years. “I led the team that designed the physical product itself, as well as being responsible for the first production line. We had to prove mass production yields at mass production speeds on the first line,” said Shedletsky.
In this role at Apple, she noticed how very small defects on the line could cause huge delays and problems in the product lifecycle and profitability. So, she set out to build technology tools that could address these defects. “I had this kind of naïve, but correct insight that there aren't great tools for this. that we struggle by just throwing people and money at these problems. And if Apple didn't have it, it didn't exist.”
Anna Shedletsky, CEO, Instrumental AI
Engineering.com sat down with Shedletsky to ask questions about the technology and applications.
Would a system like this be typically used to trace issues in components prior to assembly, or during final assembly?
All of the above. Process issues, part quality issues, workmanship issues, and design issues. We primarily do our first deployment in the new product introduction (NPI) stage. We’ll primarily be looking for a combination of part quality and design issues in the design phase. Some of our customers, like Motorola, have actually pushed us up in their supply chain. So they're actually using this technology with some of their suppliers to prevent poor quality parts from leaking into their final assembly stage. We had a customer where we're in three phases of their supply chain. Those parts start in Thailand, they go to Korea, and then they end up in China.
We work with those factories, as well as the main China assembly facility. So, we can look at both quality issues created by suppliers, as well as quality issues that are created in on-site final assembly.
How is all this data collected? Does it require human input, such as a SKU barcode that's scanned?
The unique insight that we had is that often when a problem occurs, there's not the right kind of high resolution data available as soon as the problem is discovered. So, then an engineer has to go collect additional data before you can even get started on solving the problem. What instrumental does is we actually collect data proactively. Specifically, we take images of every single unit at key stages of assembly. And those images, as you might have heard, are worth a thousand words. You don't need to know what you're looking for when you take the image, but that image could be very valuable for identifying these types of issues that we were discussing.
The way that we get these images is with drop-in camera boxes, or we can take images off of pre-existing automation equipment, if our customers have cameras already. We read bar codes out of the images to make those traceable, or you can manually scan a barcode if there's no barcode on a part.
You mentioned to me that the technology uses machine learning to identify defects that conventional vision systems cannot. So, what are you detecting that conventional systems can't?
Conventional systems are usually used for very specific applications. For example, measuring a gap, or detecting if certain screws are present or not. For these systems, there's a preconceived notion of what the value of that system is to justify buying it and putting it on the line. That's how vision has been used in the past. These vision scenarios are very rules-based: the screw is either there or not. But that detection won't find if the screw is stripped, for example, unless it's been pre-programmed to do that as part of that specific deployment.
Instrumental is a generalized tool. It can actually discover issues that it’s never seen before and that our customers have never seen before. So that means that we ultimately go in places that typically industrial vision or human inspectors haven't been able to provide value in the past. For example, we’ll go after major sub-assemblies have been built and do inspection across those entire sub-assemblies. And we can identify a wide variety of different types of issues that would be difficult to wrap specific specifications around.
For example, in a system which measures a gap, you could say point four plus or minus point one, like there is a very specific specification. But what about glue volume and glue dispense? What about bubbles in that glue? What about if you have solder? And is the solder, does the solder look right? Is it cold solder or is it going to make a good connection? These things could use conventional vision to inspect if it was pre-planned. However, often it's too expensive to actually deploy vision systems to do these, because each individual algorithm you have to set up on a machine vision system will cost you many hours of a consultant’s time, whereas Instrumental programs itself.
So, you drop our system in, we look at thirty units of data, which usually we get in the first day of being on the line, and our algorithms start to program themselves to be able to find other types of defects that they know about and ones they don't know about yet.
Can those initial thirty units be normal?
Yes. We don't need red rabbit defects to be able to set up these algorithms. We can certainly take and use red rabbit data, but we don't need it to be effective. Key point: those thirty units, you said normal units, they don't have to be golden either. That's often another constraint or problem in development, is you can't even build one golden unit, let alone 30. And that's actually been a problem. And so just normal input is good. There can be defects in it too. That's fine, no problem. It doesn't have to be special in any way.
Is the system retraining over time and getting beyond those initial 30 data points?
Yes. Thirty is when we allow the algorithm to start working because there are some use cases in which that's enough and ready for the production line. And then as we get more data, then we can tune in and prove these algorithms. They effectively tune in and prove themselves. What we do need to know is what is defective, because what we find is difference, but our customers needed to tell us which differences actually correspond to defects.
Can I generate, for example, Pareto charts of what defects are appearing?
Yes. We have Paretos. We have trends, because we know what these defect rates are over time. So, we can show run charts that show different types of defects and when they're occurring.
A lot of times, our customers view that when they roll into production, things shouldn't change. Like, “it's production! Everything's stable. Everything's fixed.”
But actually, at least in the consumer electronics industry, things are not fixed, because operators are turning over at very high rates, so things are constantly changing. For example, one of our customers found that every Monday morning at 8:00 a.m., they would have a spike of defects such as a missing screw or something silly like that. And no matter what they did on training, they've always had this spike. And so, being able to kind of monitor that and understand that you're going to be able to stop those units. That is a key value that this technology provides.
I used to do this as an engineer, where you get a problem, figure out what's going on, do failure analysis, create corrective actions, validate it's good, and then make the change and do that many times over during development. So, our technology is augmenting and supporting that by identifying these issues more quickly, identifying trends to help understand if your issues are being fixed or whether they're increasing, and being able to essentially take multiple data sets together and be able to identify correlations.
Is it possible to close the loop and have the system identify a defect and then change the machine control to prevent the defect without human intervention?
That's the vision of what we're trying to do. With the software that we've built today, vision is just our first application. There's this whole data pipeline which manufacturers are pushing additional data into, so the idea is that someone else will build the automation. These automated lines are going to be filled with different vendors of automation. So, you're going to need one common place for this data to go into, and you want this common place to be some place that you can access from all of the sites in the supply chain.
We're thinking bigger than just one factory floor. We're thinking about all the upstream suppliers and also the downstream customer experience with the products, like for quality escapes and things like that. So, that's the vision of Instrumental is to create self-optimization feedback loops across this entire supply chain. We're a manufacturing data company that doesn't sell to manufacturers. We actually sell to brands, like Motorola. The reason is because they own the supply chain, the factories don't.
Is it challenging sometimes to get a manufacturer to allow a network connected, cloud connected box onto their production floor?
As I mentioned, we work with brands. The brand is our paying customer, and then they often have a factory partner, companies like Flex, Foxconn, or Pegatron where they actually build the stuff. So, we come into the factory on two well worn ruts. The first rut is that we're just the test station on the line, and we happen to require a data line. So, our customer specifies, "Oh yeah, we want an internal station here. They need two outlets and a data line." And a factory says, "Okay. It's a test station." And it’s just kind of part of their normal flow. The other well worn rut is third party QA.
So, these are services such as Underwriters Laboratories (UL). Factories are constantly having people come and essentially look over the data and what they're doing. So, we come in in these ways. We haven't ever been in a situation where our customer cannot get internet and have prevented us from being able to run. We designed this to work in China, after all.
What kind of benefits has Motorola seen using this technology?
They're piloting the system in production, and their expectations are to see more dollars and cents, because we're essentially intercepting defects that they don't have good ways to catch.
We work on every phone they've made since 2018. It's about six or seven products at this point, and key takeaway values for them include enabling them to accelerate product maturity during development, meaning they find issues faster and they fix them. They find them in the first build instead of the third build, and there are so many stories. Each particular program there seem to be ten or twenty different situations in which each of those problems would have cost them $150k or more, such as finding a tooling issue on the first day of the build instead of all the way at the end after you've already replicated tools.
That product maturity acceleration and the production of experiments and mistakes involving tooling specifically and DOEs that have to be run to fix things have been significantly reduced with Instrumental. They also saw an acceleration in the ramp. Their time to stability was much shorter over the seven programs that use Instrumental versus the ones that didn't. So, they saw a significant delta there. It means their yields are higher faster, so they're essentially saving a lot of money they would normally spend on rework and potentially shipping bad units to customers.
Motorola is interesting in that they kind of lost some of their gleam on their brand name in recent times, but Motorola from an engineering and manufacturing process standpoint is the ‘OG’. Apple's process was replicated off of Motorola's. We work now with a bunch of Fortune 500 companies, and they all use the Motorola process. Six Sigma was invented at Motorola. They really are leaders and innovators in the product development process and manufacturing process.
How important is cyber security when you are storing high-res images of the product during various stages of assembly?
Super important. We have pre-production images of products in highly competitive spaces, and so from the beginning we built this system with enterprise-grade security. In many ways, cloud security can be more secure than on-premise systems, because on-prem systems don't necessarily get patches and updates. Cloud systems that are fully encrypted are getting updated constantly with any latest security patches. That's part of working with a new customer. We go through extensive security reviews to make sure our system is up to par. Again, as I mentioned, we work with these Fortune 500 brands that like to be protected. The key point is making sure the system is secure for their data.
I guess lastly because you're in the machine learning space, this is just kind of a side question: what are your thoughts on the term "AI" being used? I think there's a lot of hype surrounding it. What do you think about using the term "AI"?
I hate AI! I hate industry 4.0 and industrial IoT too. I think the key point is that AI is a collection of technologies, of which machine learning is one. Computer vision is another. Then, frankly, more than 50% of it is just math and statistics and Pearson coefficients. But those sound a lot less sexy.
I'm pretty skeptical on the buzz words. We do use them because that's how people find us, because they're looking for these types of solutions, but personally I'd rather just be 100% clear and concise about what it is we do. We find defects. That's where we try to position ourselves, but that's my personal vendetta against buzz words. Especially industry 4.0 and industrial IoT. I don't know. What do you think about it? You probably use it all the time?
I don't know. I agree about ‘industrial IoT’, because I think that you can usually be a lot more specific. But we already have ‘industry 5.0’, so the buzzwords clearly aren’t slowing down!
Yeah, well. I think that people at various larger brands are trying to co-opt these terms as well. The term ‘Industry 4.0’ was invented in Germany, but [a certain American conglomerate] kind of took it over and spent all the marketing dollars on it. But then they never delivered anything, so I have a skepticism around something that was a big marketing thing, but then there was no delivery on the promise. I think that industry 4.0 is frankly a little short-sighted around the capabilities, and that's probably what 5.0 means, but I've seen several different definitions of 5.0, so it's clearly not clear.
The 5th one, as far as I understand it, is about human and machine communication, cooperation, collaborative technologies.
In my opinion, the next thing after automation is full autonomy. It's not ‘humans in the loop.’ That's just people who are trying to make themselves feel better that they're going to get replaced. I think it's about autonomy and intelligence, and essentially creating systems that can do this stuff. Sure, a human can be in the loop to push the button to make the change, to make sure that the machines don't go awry. But I think that the collaborative stuff, I don't think that's the future. That’s just part of 4.0.
That's an interesting take!
How are the parts usually presented to the camera? For example, on a conveyor, like you might have with a conventional vision system?
It can depend. Usually, as I mentioned, most of our customers don't have vision in the places that would be most valuable for the way Instrumental works. So, we'll drop in place a station that will have a fixture. The part will be manually loaded and we'll capture the image.
Consumer electronics is still a 98% manual assembly. There are a lot of people on these lines. So, in a production line, we sit between two operators, so that it’s neutral. A person loads the part. We scan the bar code, take the picture and do the analysis and then pass the results to the downstream operator.
If our customers have automation equipment or cameras that are already taking images from conveyor lines, for example, we can ingest those images and process these images and then give realtime results back to the MES for that line, so they actually intercept those parts.
Are there other industry applications besides electronics manufacturing?
Essentially any discrete manufacturing serialized product. We think in general for modest volumes $50 or more in value of the product itself. We primarily today work on electronics applications, but we have medical device customers that have products that don't have any electronics in them, such as pump systems and things like that where it's medical applications. Some of those parts are as cheap as $3.00 in terms of how much the part costs, but the reason there is return on investment is because the cost of a defective unit getting through is very, very high in the medical industry.
We also have customers in the automotive industry, and we believe that there'll be some really interesting applications in defense and government. And we're just kind of starting that now.
For more details on vision systems, check out Eye Spy: The Basics of Robot Vision Systems.