How to Implement Bin Picking in your Manufacturing Operation
by Adil Shafi
, President, ADVENOVATION, Inc.
ADVENOVATION, Inc. Posted 04/04/2007
This article is targeted towards the End – User manufacturing community. It is intended to provide a brief overview of Bin Picking’s progress towards reliable and widespread use, with Vision and/or Light Guided Robotic techniques, and then to provide a methodology to consider, carefully test, and implement reliable Bin Picking.
Turning the Completely Impossible into the Obviously Possible
When Thomas Alva Edison began to work on inventing a light bulb, it was generally considered an unreliable and impossible task. When with self belief and perseverance he succeeded, he looked back and said that he had to succeed since he ran out of methods that could not succeed. Today, satellite images show impressive images of lights in industrialized regions on earth at night.
Learning from failures and the experience of others before him, Sir Edmund Hillary defied the conventional reservations of his time and summitted Mount Everest. Today, so many people summit Mount Everest each year that it is commonly joked that soon we will have a weather insulated escalator to go up to the top.
In our manufacturing community we have similar parallels. A generation ago, most welding was done by people, often with inspectors after welding stations. Today, manual welding is questioned and rare. Just six years ago, 3D Vision Guided Robotics performing AutoRacking (or pick or place stamped metal parts from or onto racks) was virtually unprecedented. Presently, we have hundreds of cells running AutoRacking reliably in our industry and some companies implement AutoRacking on every new manufacturing program.
I believe that the same is true of Bin Picking. A few solutions have been running in production for more than three years and more are being implemented each year. Within a decade or so, all Bin Picking will be automated. Our next generation will wonder why people would want to pick parts manually, more slowly and more expensively than a fast robot from a bin. Manual bin picking will then become questioned and rare.
Bin Picking, in the past three years, has quietly but steadily made advances in commercial production lines. A good review of successful solutions in our manufacturing industry was published in Automation World’s February 2006 issue, www.automationworld.com. The article was entitled ‘‘Vision Guided Robotics: In Search of the Holy Grail’‘.
Ease of Bin Picking is driven and prioritized by two factors: 1) The geometry of the part, and 2) The degree or severity of randomness of parts in bins. The first, easiest, and financially most justifiable solutions have been in the automotive powertrain area; most notably engine blocks. These parts are well machined, are rich in geometric features, skewed slightly in x, y, z, yaw, pitch, roll directions and are heavy (thereby slow and hence expensive to manually handle). This has been a perfect first storm to enable Bin Picking.
There are many enablers currently driving more solutions into the fold of reliable Bin Picking. These include: Advances in computational processing power, vision recognition tools, mathematical algorithms, flexible lighting, a continuous reduction in commercial pricing, and a growing collection of techniques in handling, gripping and staging an overall problem into more easily handled steps.
A rough analogy is that 16 = 4 x 4, but 16 is also 4 + 4 + 4 + 4. Addition is easier to do than multiplication. The same problem can be reduced into several smaller equivalent problems.
A tough bin picking challenge can be simplified by breaking the problem into individual retrieval only first, which may be imprecise in finding a part centroid, but then using a simpler 2D pick and place stage for precise final placement. Such two-stage operations can reliably run entire bins and meet a six second part-to-part, bin acquisition to precision pins placement cycle time. Fast, fixed mount camera solutions are now running in production at four second part-to-part cycle times.
Good Applications That Are Ready for Reliable Bin Picking in Production Now
The following applications have now become feasible for reliable Bin Picking:
- PowerTrain (Engines, Cylinder Heads, Axle Shafts, Differential Carriers, Pinions, Round Parts with Stems, Connector Rods, Piston Heads, Brake Rotors and Stacks of Gears).
- Stamping (Flat or bent metal plates with multiple holes, roughly stacked stampings with a progressive skew).
- Final Assembly Products in Boxes in T/C/F (Trim Chassis Final) pick operations for placement into cars on moving lines; see related discussion about Vision Servoing at the Robotic Industries Association website.
- Strips of medical tablets, flat but randomly arranged in boxes.
- Bags of products e.g., chips, salsa, cheese, cement, etc.
- Lateral or upright layers of tubes (copper, plastic, PVC).
- Layers of products e.g., wooden planks, plastic sheets.
How to Implement Bin Picking in your Manufacturing Operation
The following steps are recommended to evaluate, justify and implement Bin Picking.
The instructions below are a bit precise but not difficult to follow.
- Take pictures of your parts with a cell phone or a digital camera from an electronics store. You will need two cameras for your part and bin image analysis.
Individual Part Pictures (IPP)
- Consider each part that you manufacture. Place each of your parts on a flat surface. Review the multiple stable resting positions in which each part can be placed on a flat surface (for example, a soft drink can has two stable resting positions: One ‘‘standing up’‘ with its circular footprint on the flat surface, and one ‘‘lying on its side’‘ with its circular planar ends perpendicular to the flat surface (the resting position in which it can roll on a flat surface).
- Then for each of your parts, take a picture of each Stable Resting Position (SRP). The camera should be aimed at about a 45 degree angle to the flat surface, looking down towards the part. Collect this as your library of Individual Part Pictures (IPP). This is essentially a two-dimensional array of pictures, where the first index is your part number, and the second index is the part’s SRP.
Bin Randomness Pictures (BRP)
- The next step is to take each of your part types, and review how randomly they are found in actual bins in your manufacturing operation.
- Using a tripod or a temporary structure, setup two fixed-mount cameras above each bin. Depending on the size of your bin, adjust the size of the view so that the Field of View (FOV) of your image is indeed the entire bin. Place the first camera directly above the bin pointing straight down or perpendicular to the flat horizontal plane of the bin below. Let’s call this Camera 1 or C1. Place the second camera at a 45 degree angle above the bin, looking downward, so that it sees the C1 scene from an angle from any side (select one fixed side) of the bin. Again setup the FOV so that it has as much part content in it as possible as what C1 can also see. Let’s call this camera at 45 degrees Camera 2 or C2. When looking at a bin, the planar 2D views of C1 and C2 will not be in the same direction nor scale and the C2 images will be skewed and that is fine.
- Then for each of your parts, place a bin of parts below C1 and C2 (as they normally occur in production to the level or randomness that you typically find them). Take multiple pictures of each bin and several examples of randomness of parts that you will see. Organize and maintain a pair of C1 and C2 image pairs for every scene.
Take This Pictorial Information to the Experts: Evaluate and Believe by Seeing Demos
- Take this pictorial information to experts in the field of Bin Picking. You can use an Internet search engine (enter ‘‘Bin Picking’‘). Request examples of their past work as well.
You can also attend and meet speakers at the Machine Vision & Robotics Track Session 6: Advances in Vision Guided Robots at the International Robots, Vision and Motion Control Show in Rosemont, Illinois (Chicago) on June 8 – 11, 2009 http://www.robots-vision-show.info/robots_vision_show_info.html. There will be several Bin Picking demonstrations running at the show.
- Request an evaluation of your parts from the pictorial information collected above. It is then possible to obtain a budgetary estimate to automate your Bin Picking operation. If the payback on investment is justifiable, then proceed with the following steps.
- The first key to success is to insist on a pre – sale demonstration with exactly your parts. This is a critical step to not misunderstand and to not create failures. It is very important to ask for a completely reliable, uninterrupted retrieval of all parts, or negotiated manual intervention for certain cases of part randomness. It is the only way to adequately protect the risk in these projects for five parties : End – User, Systems Integrator, robot company, vision company, and software enabling company.
Sometimes these roles are provided by the same company, however Bin Picking experience and a single line of project responsibility from a Systems Integrator is critical to your success in this area.
Seeing is Believing
It is highly recommended that your factory floor personnel visit and review vendor demonstrations, since they often know of rare and exceptional cases that can stop production. It is critical to gain a comfort level by seeing several, continuous, uninterrupted and realistic demos running from completely full to completely empty bins before issuing a purchase order.
Part Variation Management
- The second key to success is Part Variation Management (PVM) in your operations. It is very important to separately study, log, plan and manage manual–to–automatic retrofits versus new part programs. In a retrofit situation, it is possible and recommended to take hundreds of unobtrusive pictures (see C1 and C2 image gathering methods above), and to be able to run simulated pickups of those images offline.
This process protects being caught off guard after good laboratory demos and runoffs at the vendor site, while remaining unaware of true variation in a plant. Sadly, this is often realized late in a project when the vendor arrives at the End – User plant for final implementation, only to discover that a number of variation cases were unexpected, misunderstood and unplanned for in advance.
These types of mistakes create disillusionment and delay in future confidence, and ultimately delay the time advantage in financial benefit to End – Users. It often takes a year or two for a typical End – User to recover, reinvestigate and reinvest. In the meantime, other global End – Users gain competitive advantage by avoiding these mistakes.
- Thirdly, it is recommended that you review and benchmark, through actual test, ease of use for non – technical operators, training at Operator, Technician and Engineering levels, a FMEA (Failure Mode Engineering Analysis), and rigorous procedures for backups, version control, and access to 24/7 vendor support.
Bin Picking is a manufacturing solution whose time has now come. There are many examples of Bin Picking that are ripe for success and financial benefit to End – Users. The content above provides a methodology for analysis and evaluation. It also provides project management guidelines critical to protect End – User success.
The article’s author, Adil Shafi, President, ADVENOVATION, Inc., welcomes questions and comments at 734-516-6761. For more Bin Picking related information and content, visit Robotics Online, Tech Papers.