For many manufacturing processes, automation is an ideal way to streamline throughput, improve quality and repeatability, and solve labor challenges. If you’re beginning your journey in robotic automation, check out this article on the benefits of industrial robotics, and this article on pitching robotics projects to management.
When looking to automate the dull and repetitive tasks of their most bored and underutilized workers, many production managers take for granted the inherent human skill involved in even the most mundane tasks. Handling objects requires motor skills and visual processing. To replicate these with an automated system can be complex.
According to Keith Vozel, Product Manager for Software at Yaskawa Motoman, most customers begin their search for an automation solution with an understanding of the problem, and not much else. They know they want to get parts from A to B, but often don’t know what vision solutions are out there.
In this article, we explain the basics of what’s out there, and match common problems to solutions. For expertise, we reached out to robotic systems integration professionals from leading companies, including Yaskawa Motoman, Nachi Robotic Systems, Inc., and Universal Robots.
The 1, 2 and 3 Dimensions of Robot Vision
Most engineering projects are a quest for the simplest solution. In robotic vision, this raises the question: How many dimensions do you need? The answer to this question is usually found within another question: how many dimensions can you constrain?
If your parts are in a bin, they are completely unconstrained in all three dimensions. Your system needs to determine the part’s position and orientation in all six axes. If your parts are presented on a surface, such as a tabletop or a conveyor, the z-axis height is fixed, and a three-dimensional camera is probably overkill. If your parts are constrained in another axis, such as in a chute or narrow surface, the camera only needs to detect the part’s location along one axis.
For one-dimensional detection, a laser rangefinder or optical distance sensor could be used.
According to Keith Vozel, product manager for software at Yaskawa Motoman, 2D vision systems are much more common than their 3D counterparts, partly due to the increased complexity and computational requirements of 3D vision.
2D robot vision is more than just a camera. Systems also include lighting, interfaces, and software, including, in many cases, plugin packages for the robot controller. The Yaskawa Motosight 2D system is an example of a 2D vision system. It includes a camera and configuration software.
3D vision can include multiple cameras and laser displacement systems. Laser displacement systems work essentially like LIDAR, measuring the reflected laser light returning from the scanned surface. Multiple camera systems use triangulation algorithms to interpret information gathered from multiple perspectives on a part. The Nachi Robotics NX-AV Vision system is an example of an image processing system that can be configured for this type of stereoscopic 3D vision, as well as 2D vision. Bluewrist ScanXtream is an example of a 3D point cloud processing system that uses points of light to interpret the shape of objects in images.
Lighting Issues with Robot Vision Systems
“Everybody in the world has a different issue with lighting. Robot person says it’s the light, and the lighting person says it’s the robot,” said Vozel. “Many cameras now have integrated lighting and use different spectrums to avoid being affected by changes in ambient light. Our system uses a structured blue light as part of a single solution to reduce the effect of external light.” Robin Schmidt, vice president of research and development at Nachi Robotics, agreed: “A common problem in any vision application is proper and consistent lighting. Changes in lighting, depending on shiny parts, surface features, lots of camera setup in order to have the sensors and applications stay consistent. For example, morning and evening may be fine, but when light increases in the middle of the day, a system could have issues because External light is coming in and changing the vision,” said Schmidt. He suggested red-spectrum light, with filters surrounding the work cell to restrict ambient light.
Configuration: What to Expect
Many robotic systems in factories today are not intended to perform the same task for their entire life cycle. It’s normal for a robot to be reassigned to a different task as contracts and demands change. So, if you require a robotic vision system now, it’s important to consider whether you have the in-house knowledge to reconfigure your vision system for a new application. According to Vozel, more complex systems may require more programming know-how, but many systems are designed with user-friendly configuration software. “You might have tools on the robot side or the camera to configure what it’s looking for, how the object is shaped, where it’s looking. In the 3D vision world, it’s a little different depending on the implementation. Some older implementations require that you actually use programming and call system-provided subroutines to interpret the image. You have to provide the image, but you have to use programming techniques to find the item and figure out its position. With other systems, including one Yaskawa Motoman offers, all you need is a CAD model of the part and the system does the rest.”
Schmidt, of Nachi Robotics, explained it this way: “Well, there’s always been in the early days, it was an engineer’s environment and an engineer’s tool. You had to have advanced skills to get in and make changes to parameters. But in trying to make things easier for non-engineers, manufacturers are coming up with ways to make changes in a more automated way, with a user interface. It’s similar to an ATM: the system is still highly secure, relatively complex; but the menus are there to limit the user’s choices. You can’t open a mortgage or perform advanced banking functions at an ATM, but anyone can easily perform basic functions with no expertise.” In other words, the configuration software should be a consideration when choosing a vision system, and if you don’t have the in-house expertise, making advanced configuration changes after the initial installation could be costly.
Is Robot Vision Necessary? Scenario: Robot Barista
Considering the advanced force and torque sensors available for industrial robots, do robots always need to ‘see’ what they’re doing?
Universal Robots recently announced the e-Series, a new line of their collaborative robots. The most significant upgrade was the inclusion of a six-axis force and torque sensor at the flange of the robot. I asked Esben Ostergaard, CTO of Universal Robots, why the company chose force sensing instead of vision for this upgrade. Here’s what Ostergaard had to say: “A vision system was considered, and it still is. A lot of our robots are using vision to do different things. If you consider adding a camera onto all robots, we did consider that, but it’s very hard to know what camera to add. There is a reason why you can buy so many different cameras—it’s because all applications require different solutions, in terms of light, resolution, and more. We also noticed that our competitors who did launch a robot with a built-in camera have recently released versions without.”
While robot vision systems are essential to some applications, they are also highly specialized, with certain systems best suited to certain applications. In addition, some applications don’t require vision at all, with a little ingenuity in program and process design.
For example, consider this demonstration of the KUKA iiwa collaborative robot. At timestamp 2:07 of this video, the robot uses a tactile search to confirm the presence of a paper cup on its gripper. This is done using the torque sensors equipped on the axes of the robot. After this check, the robot performs a task.
So, what is it about this demonstration that makes vision unnecessary? Well, as you can see at timestamp 1:57, the cup is placed in the gripper by a human operator. In a factory environment, it would probably be part of the robot’s program to locate and pick up the object by itself.
Robots without vision capabilities can locate objects if they are in a predictable location, usually fixtured in place. This demonstration of the Kawasaki DuAro, a dual-arm collaborative SCARA robot, shows the fixturing used for tools and parts. However, custom fixturing and tooling can get expensive.
Let’s revisit paper coffee cups. Cups may likely be located in a dispenser like this one. In this case, the cup is fixtured in place. A robot without vision could reliably grab a cup from this dispenser. The system could be made even more reliable by adding a sensor to the dispenser to alert the robot when a sleeve is empty.
Then again, a dispenser like this would require a human operator to refill the dispenser. It would be better if the robot could pick cups out of a case, so that human input would be limited to delivering and preparing cases.
Could a robot do this task?
Now, the task of picking a cup is similar to what’s known as bin-picking, in which the robot locates and picks objects from a bin. In these tasks, neither the location in X, Y or Z is controlled, nor the orientation in A, B or C. Our cups look something like this:
The cups seem structured, as they are arranged in a uniform orientation and placed in neat stacks. However, their placement is probably not fixed enough for a robot to accurately pick them. As cups are picked, the heights of each stack will reduce, and stacks may shift or topple over. The interaction of the gripper may cause the cup stacks to sway or wobble. If the robot misses a cup, a collision may crush or damage the cups.
For these reasons, a 3D vision system, such as the Bluewrist system demonstrated in this video, would work.
While Bluewrist’s software is hardware agnostic, the setup in the demonstration uses a Microsoft Kinect sensor to create a 3D point cloud of the target area. Next, the software determines which object to pick without colliding with other objects or with the bin. This solution could be ideal for our case of cups, as it could correctly identify the height of the stack of cups and position the gripper accurately.
However, 3D vision has much higher computational demands than 2D vision processing. Vozel explained: “In 3D, there is much more image processing involved in comparing a 3D image of the part to point cloud or 3D image data, and that’s very compute-intensive. It’s not as time sensitive as 2D vision. If you’re really looking for fast cycle time, 2D is still the king of that.”
In the scenario of the robot barista, cycle time is not really a factor. The robot needs a cup for every new order, leaving plenty of time for the “If we have a vision process that works but still takes a long time, in an application where the robot is trying to capture the image and do the processing, maybe after the robot picks part number 1 and gets out of the way, the vision system is starting to try to get the next image while the robot is delivering part number 1. So, you can try to bury some of that processing time within other tasks, rather than waiting for it sequentially,” suggested Schmidt. With this sort of efficient programming, our barista robot would likely not suffer any delay before picking a new cup, as all the image processing would be complete by the time it finished making the previous order.
However, 2D vision may be a more cost-effective option. Schmidt explained how a single-camera, two-dimensional system can use algorithms to compare the part profile to an ideal model and use that information to interpret the position. For example, if a cup’s bottom surface is parallel to the floor, it should appear in the image as a circle. If it appears as an oval, the software can determine the angle of the cup. To determine the height of the cup, sensors on the end effector or a second camera at a different perspective could be used.
Besides showing the differences between 2D and 3D vision, this scenario also illustrates one very important factor: that the design of your process is critically important to the design of your vision system. In fact, design changes to your process, or even to your part, could save you money, as well as installation lead time for the robotic system integration, and even cycle time when it’s all up and running.
Bottom line: whatever your problem, a robotic system integrator can probably get you set up with a vision system that gets the job done. But without examining the bigger picture, you won’t get the most efficient solution.
Redesigning your Part and Process for Better Automation
If your factory uses industrial robotics, it’s unlikely that they were just plopped into place one day without a full study of the costs, benefits and workflow changes. There’s no reason to assume a vision system would be any different. While you may think the problem is getting your parts from A to B, the problem could be the ‘A’ and the ‘B’.
Schmidt gave this example: “If you are trying to pick up a pencil, and you need to orient the eraser end a certain way, if you can have the pencils coming down so that your eraser is always within 90 degrees and pointing in a certain direction, that’s much easier than having to pick it up from within a full 360 degrees of rotation. Anything you can do to limit those variables lends to less complexity or greater success.”
Beyond the process, even the part itself could be redesigned or changed to improve success of the automation. If a highly reflective part is causing problems for the camera, could the surface finish be changed? If a part is challenging for a robot to grip, could a feature be added? While it may seem ridiculous to change your part designs to suit the automation, doing so could save you big time in costs and headaches down the road.
For more coverage of automation and robotics, follow me on Twitter at @engcom_automate!