Congratulations! You have decided to automate your manufacturing processes and streamline the production of your product(s). To arrive at this point, you spent critical time and effort on justifying this investment decision. Maybe you needed a solution that would support additional production capacity to satisfy customer demand, or to launch a new product, or as a result of an improvement initiative to lower your manufacturing costs. Whatever the reason, the investment you are about to make in the pursuit of your automation journey also requires due diligence in selecting the right automation system provider. Vetting automation suppliers need not be an art – it can be a science if the best practice methodology is applied.

Setting yourself up for success 

For the purpose of this article, we will focus on custom automated systems, meaning solutions that have not been referenced, designed, built, or implemented before. Setting yourself up for success starts with defining what success looks like. A significant performance indicator for measuring the success of an automated system over the asset’s life is OEE (Overall Equipment Effectiveness). A successful automated system has a favorable OEE at the time of commissioning but also throughout the warranty period and through its useful life. After all, the investment in automation isn’t trivial and you should expect dependability, reliability, and quality throughout the system’s lifespan. Traditionally, the agreed-upon contractual OEE between the automation supplier and the manufacturer at acceptance is 85% or better. Achieving and sustaining this means the quality of the product being produced, and the system’s performance and reliability are aligned with and meeting the specifications set forth by you.

Furthermore, tolerances linked to production output and machine uptime are supplementary measures that ensure the product is produced at a sustainable rate over the asset’s life.

Two additional indicators to evaluate the success of your investment are performance to project budget and on-time delivery. In the context of a custom automated system and supplier selection, the budget should be considered from two perspectives; the contracted purchase price as determined by the supplier’s initial proposal, and the on-going cost of maintaining the system’s performance. When the solution is delivered, it should align to the anticipated budget with limited number of unplanned variances (change orders) that catch you off guard. In addition, you do not want exposure to supplemental investment in future. As years pass, the automated system should guarantee salable product being produced without defect and at the volumes expected, avoiding latent process failures. Finally, your automated system needs to arrive on time, again per your purchase contract. A delayed delivery may impact your ability to start realizing the planned savings, or far more critically impact your ability to produce and sell your product which delays revenue.

A last critical element to setting yourself up for success is the user requirements specification (URS). You use the URS to communicate to all potential automation suppliers what your requirements are for the anticipated automation. You will use the URS to verify that all the requirements were achieved prior to accepting the solution. There are numerous resources that can provide guidance on how best to structure a URS. This article will focus on how to use a URS and not go into specifics that differentiate a solid URS from a poorly written one. The URS assists in vetting the potential suppliers as it becomes a method to fairly compare the tendering of each supplier against a standard reference of requirements.

Besides employing metrics to set yourself up for success, there are different analytical tools that can be used to demonstrate and validate that the automated system you desire to purchase will function precisely as you projected in terms of:

  • How operators interact with the automation 
  • Process workflow 
  • Inspections and inspection processes 
  • Recovery methods for fault conditions

Leveling the playing field for your vetting process

So now that you have set yourself up for success, how do you select the right supplier? Not all suppliers are created equal. Not all suppliers will be appropriate for your project. And not everyone within your own organization will have the same set of criteria for determining the best candidate. Let’s consider the available toolset and the evaluation matrix in particular.

The evaluation matrix is a qualitative tool for comparing the various supplier candidates against criteria that are important to you, your project, and your organization. It is also intended to identify your confidence level in the ability of the various supplier candidates to deliver on their claims, i.e. establish whether they “can do what they say they will do”. After all, any supplier can develop an elegant proposal and concept noting they can meet every aspect of your URS.

This tool helps to minimize unpleasant surprises that can result in a delayed delivery and a system that is not compliant with the initial URS. Surprises mean additional costs and worse, the failure of the anticipated outcomes of the automation implementation.

The power of an evaluation matrix is threefold; its ability to predict successful project outcomes, its ability to align and manage definitions and expectations of your key stakeholder groups, and its ability to flag risk areas thus allowing you to proactively implement mitigations. The best evaluation matrix includes evidence to substantiate your evaluation and/or the supplier’s claims.

Recognizing that there are many different stakeholders, here are some examples of criteria that can be included in the evaluation matrix and that should be backed up with evidence.

Business-Related Criteria
  • Financial Strength – What are the revenue, debt, cash flow, credit rating, etc. position for the candidate? 
  • Does the candidate have the financial depth to not only manage the project in terms of cash flow but also invest in a service team to ensure the automation OEE is sustainable over its lifespan? 
  • Business Continuity – Can the candidate manage the business through disruption or catastrophic events, like the COVID pandemic? Does the candidate have the appropriate plans in place and bench strength to continue executing projects? 
  • Experience – Does the candidate have relevant experience? Are there testimonials or references provided? Does the candidate have a history of dialoguing on and challenging the customer’s URS for the sake of optimizing the desired outcomes or pointing out a potential pitfall(s)? 
  • Footprint – Does the candidate have a global or multi-national footprint that provides the capability to build in different locations to support scalability and/or pivot as a risk mitigation strategy.
Execution-Related Criteria
  • On-Time Delivery – Does the candidate measure its On-Time Delivery in a manner that is repeatable, and that the customer comprehends? What has been the candidate’s track record over the last 3 jobs, 6 jobs, 12 jobs? 
  • OEE Performance – How does the candidate collect, analyze, and present detailed machine performance data for maintenance, operator, and engineering planning?  Can the candidate provide historical OEE data representative of previously installed systems? Does the candidate have a repeatable process that demonstrates how it measures OEE performance? 
  • Change Order Process – Does the candidate have a well-defined and documented change order process? Under what circumstances does the candidate employ the COR process? What do previous customers say about the candidate’s use of CORs? 
  • Project Management Methodology – Does the candidate have a defined and comprehensive project management process and discipline that controls the communication cadence, report formats, and information the customer receives concerning performance to plan during the build and the appropriate escalation paths if required? 
  • Staffing – Does the candidate have the appropriate staff, expertise, and capability to address the specific nuances of your particular project?
Services-Related Criteria
  • Service Strategy – Does the candidate have a post-purchase service strategy available to ensure sustainable OEE is achieved post-warranty period of its installed equipment? 
  • Ongoing Operator/engineer Training Tools – What processes and tools does the candidate employ or offer to ensure a customer’s employees’ understanding of the equipment is current, and to address any turnover in the customer’s employee base? 
  • Service Locations – Where is the closest services site? What are the candidate’s commitments to service levels and response times? 
  • Spare Parts – What is the candidate’s approach to the supply of spare parts? What is the commitment to delivery time?
Quality-Related Criteria
  • Quality System – Does the candidate have a quality system? Can the candidate provide evidence of quality audits, findings, and corrective actions? Can the candidate provide and demonstrate execution of its internal protocols that ensure equipment performance prior to customer acceptance? 
  • Quality Certification – Does the candidate have any quality certifications? Are they relevant to your project?
Project-Related Criteria
  • Proposal Basics – Are price, delivery schedule, system rate, and size clearly identified? 
  • Compliance with the URS – When you perform your own comparison of the candidate’s proposal and your URS, is the offering compliant with the URS? Did the candidate provide a detailed response to your URS that demonstrates understanding and compliance with the stated requirements? If the offering is not compliant, how does the candidate identify this?
  • Options – What is the candidate suggesting as options or alternatives to your URS? Are they legitimate options, or are they just addressing exclusions or exceptions to the URS? 
  • Exclusions & Exceptions – Does the candidate identify any exclusions or exceptions? What is the nature of these? Do these raise any concerns about capabilities and/or future change orders? Are they acceptable exclusions or exceptions? 

The scoreboard and total cost of ownership

At the end of the process, you need some kind of scorecard that provides the final justification for the selection of one supplier over all the others. Your decision should be based on the criteria that are absolutely critical to your business success and should take into account the evaluation matrix, either as a single line item (overall performance on the evaluation matrix) or as a subset of the scorecard. A weighting factor can be introduced if there is an agreement within the key stakeholder group that some criteria are more important than others. What it shouldn’t be is a decision based on price alone.

Instead, consider Total Cost of Ownership. Total cost of ownership is a management accounting concept that can get lost in the evaluation, but it is too important to be ignored. What is included in the total cost of ownership? Any direct or indirect costs that, over the lifetime of the asset, contribute to the cost of implementing that particular solution, for example:

  • Investment at the time of purchase
  • Costs associated with risk management and maintenance of the equipment once installed onsite
  • Costs associated with sustaining OEE including headcount and overtime costs to maintain performance
  • Investment in additional assets because of poor performance 

Lessons learned-some final do’s and don’ts

When it comes to automation, the saying “the devil is in the details” is very true. Getting into the details in terms of vetting your automation supplier selection is not a recommendation but your obligation. Do take the time upfront to plan out and define the requirements for your desired automation and you will significantly increase the probability of a successful implementation and ensure the automation solution has a long and useful life. Do focus on separating the ‘needs’ from the ‘wants’ as you could be paying for something that provides little value to what the end goals are for the automation solution.

A good rule of thumb when considering automation is ensure you have a reliable process first and then automate redundant tasks. Don’t automate an unreliable process. A poorly understood process and high defect rate passed on to an automation supplier will result in a poor return on investment more often than not.

Do ensure that your URS includes your quality requirements in addition to any functional and process requirements. Neglecting to define what quality is and what the desired level of quality is leaves the definition in the hands of your supplier and this may not be satisfactory to your organization. Specifying OEE alone will not suffice as OEE is a combination of performance, availability, and quality. At a minimum, DO call out your desired yield target.

Don’t ignore red flags when considering which supplier and proposal to select. The inability of a supplier to provide a compliance matrix in response to the issued URS, or to demonstrate a sustainable OEE should be probed, and if the decision is to proceed regardless then DO make sure to identify and implement mitigations. You need to move beyond selection based on the proposal scope, price, and schedule. You need to request evidence that can quantifiably demonstrate a supplier’s performance.

Do be receptive and responsive to questions from your potential suppliers regarding your URS. Strong automation suppliers should be comfortable challenging a customer’s URS based on their lessons learned in terms of what has worked and not worked when automating a process. To simply just comply to the URS is not always in the best interest of the supplier or the customer. 

At the end of the day, no one enjoys surprises. The philosophies, approaches, and tools described within this article are to help mitigate those unwelcome surprises. Project risk is real and failure to achieve the long-term objectives of your automation’s performance can’t be ignored. When the automation supplier can achieve the customer’s expected performance, it’s a beautiful thing.

What comes easy, won’t last. What lasts, won’t come easy.

Ntsiki KaCaleni