Everyone is excited about the new technology options that pop up nearly every day in talent management and recruiting. Unfortunately, along with all of the excitement and the new capabilities, there are also many perils.
One of the most common problems is when talent leaders or ad-hoc tech purchasing committees buy technology impulsively, without an in-depth assessment. They often rush into the purchase simply to be able to say that “Yes, of course, we are using technology.” As a result, they end up spending hundreds of thousands of dollars on technology that never actually improves either their talent or business results. That lack of a business impact from tech purchases was highlighted in a recent survey of executives. According to Deloitte, human resources technology spending has “grown by 50 percent … in the last five years.” But the spending is not resulting in improved HR performance because “72 percent of non-HR business leaders rate HR performance as adequate, getting by or underperforming.
I’ve been involved in the hands-on implementation of technology since 1966. And as a result of that all-to-often painful experience, I estimate that because of the use of inadequate and even shallow assessment criteria, up to 80 percent of all corporate talent management technology purchases don’t produce a measurable ROI for the corporation within two years.
There are two primary reasons for the high failure rate of new talent-tech implementations. The first is a weak foundation within HR to support any new technology purchases (Note: I covered this topic in the 10/31/16 ERE.net article entitled “Talent-Tech Is Hot … But Without the Right Foundation, It Will Likely Disappoint”).
The second reason for failure is that purchases of talent tech are made without a comprehensive objective assessment checklist.
A Checklist Covering the Best Criteria for Assessing Talent-Tech Solutions and Vendors
Anyone with more than a few years of experience in talent management probably already realizes that most of what startup talent vendors say or promise is not 100 percent reliable (to put it kindly). In order to ensure success and a positive ROI from purchasing technology, develop a detailed checklist for assessing all talent vendors and their products. Incidentally, don’t be surprised that when talking to vendors that they automatically reject the need for an assessment checklist. And, when you show them an assessment checklist, don’t be surprised when they automatically discount each of the individual items on this or any similar checklist. My recommendation is to instantly reject any vendor that badmouths the need for a tech purchase checklist with objective criteria. This is because any tech purchase ties a firm to a particular way of acting years to come, so it is essential that you make a perfect buying decision every time.
Every firm, should, of course, develop their own final technology purchase screening criteria that best fits their situation. However, the 15 criteria that I have found to be effective in assessing talent-tech vendors is listed below. First, the “Top 10” followed by additional considerations.
The Top 10 Most Critical Vendor/Product Assessment Criteria
The most critical assessment factors are listed first.
- Demand proof that a vendor solution has actually improved business results — don’t ever operate on the false assumption that technology automatically produces superior results over traditional human-based. Instead, in each case, demand proof of tech’s superior capabilities and its proven results for at least one firm. If the vendor doesn’t have proof that whatever they are selling directly and measurably improves one of four key performance factors, pass on buying it. Those key performance factors are 1) an improvement in the on-the-job performance of new hires, 2) an improvement in the productivity of current employees, 3) an improvement in the rate of employee innovation, or 4) an increase in revenue generation. In fact, in my experience, the more excuses that a vendor provides for not having or needing this kind of business impact data /proof, the quicker that the vendor should be disqualified from consideration. If the vendor has some limited business impact data or if they are currently undergoing a Beta test with a company to provide this proof, that is an acceptable short-term answer. If they’re willing to waive a portion of their fees if their technology doesn’t produce the targeted results at your company, that’s also a positive indicator.
- Split sample proof is the most convincing way to prove that a product works — be wary of vendors that in lieu of producing actual product performance data, instead only provide cherry picked quotes from users or the names of firms that have used the product. In business, the best way to prove that any new technology works is to conduct a split sample side-by-side test. This is where you apply your new talent technology approach to half of a group and do nothing different in a similar control group. Then you compare the performance data from the two equal groups to see how much the solution improves performance. This approach is the way that product and drug managers prove the effectiveness of what they produce. If the vendor doesn’t have split sample proof, ask them to consider it. Where it is feasible, consider doing it at your firm in order to unambiguously prove how well the new technology actually improves results.
- Assess the effectiveness of each individual product feature — even if the overall product works, you must still check each of the individual features to ensure that each one performs to the quality and volume specifications.
- Assess the quality of service and vendor responsiveness — even a product that has the latest technology embedded in it won’t produce great results at your firm if the vendor doesn’t provide great service to support their product. Unless your startup is willing to offer a service level agreement, conduct a due diligence “service check” with a sample of their existing customers (that you pick out). Assess their response time, their satisfaction rates, and their service quality. Also, try to avoid “consultant switch,” which is where the initial highly knowledgeable consultant that sells your product is replaced with a less-capable junior associate immediately after the purchase is completed.
- Vendor reliability and longevity must be assessed — in the volatile world of business, the stability of even established firms needs to be continually questioned. And the volatility among talent management firms is no different. Do you remember those many vendors that pushed referral programs during the last few years? Where are they now? Failure is so common among startup vendors that you cannot afford to be naïve about longevity, because as many as 9 out of 10 startup firms across all fields fail. Assume that the startup that you’re dealing with may have a limited future. It may be bought out by an industry giant (and their product will disappear) or they may simply fade away because they are not well managed, well funded, or their product will be surpassed by a better technology. If you have a choice among vendors in a particular product area, look for these seven factors. They are larger firms, they have a positive cash flow (or even profitability), their top managers have been in the industry for several years, they have a rising stock price (if they are a public firm), and they have sufficient capital funding to last them for more than 18 months. In addition, because the continued growth of a vendor is essential for their product to continually improve, consider looking at their own talent capabilities to determine if, in this tight job market, they are able to continually recruit new talent and retain their best employees. Finally, firms that use 6 Sigma measures or that have ISO type certifications should get extra consideration. It also might be a good idea to assume that there will be an economic downturn next year, so be extra cynical about assessing vendor longevity during tough economic times.
- The best vendors have an impressive array of performance metrics — almost without exception, the very best vendors are true believers in kaizen or continuous performance improvement. These firms will have extensive metrics built into their product or process that cover important performance areas including volume, quality, time, satisfaction, costs, and quantified business impacts. These metrics are critical because all new technology products have flaws, and the firm won’t be able to identify and fix them unless they have extensive performance metrics. The best will also have comparison benchmark numbers so that you can tell how your firm is doing when compared to others that are also using the product.
- Determine if your secrets, data, and privacy will be well protected — in many cases, vendors will have access to your firm’s internal data. Obviously, you need to have a data security expert look closely at the quality of the protections that they provide. If the vendor doesn’t seem sufficiently concerned about your protecting company secrets or data breaches, you need to reject that vendor. You also need to protect your data and make sure that it will still be available to you if the firm goes away.
- Assess whether the vendor is aware of all potential problems — after 20 years of assessing vendors, I have found that the very best are not naïve about potential problems. So when asked, they can quickly provide you with a long list of the potential problems that a purchaser might likely encounter during implementation and operation. If your vendor doesn’t have such a list of problems, I found that, it is unlikely that they have prepared a solution to counter each one if it arises. If necessary, create your own list of potential problems and especially look for the costly “unintended consequences” that might occur in other departments or functions.
- The very best vendors offer pay-for-product performance — you should start with a healthy skepticism for vendors. However, you can reduce some of that skepticism if the vendor is willing to put their fee at least partially at risk based on the performance of the product. I have found that successful vendors that have a great product are at least willing to talk about the possibility of including a performance clause in the contract. Those vendors that automatically discount the need for pay for performance clause should be automatically rejected.
- Be wary of initial cost and time estimates — don’t be surprised when vendors provide you with an initial lowball purchase price. But select a sample of current users and survey them to see if there was a significant variation between their initial cost estimate and their final actual costs. Of course, when you are calculating the total cost of the technology, you must include in your own internal implementation cost estimates the cost of manager and employee time during implementation and operation. If time to implementation or time to operational status is important, compare the vendor’s time estimates to the experience at of some sample customers.
Additional Vendor/product Assessment Criteria to Consider
In addition to the top 10 criteria provided above, you should also consider including the following items in your assessment checklist.
- Assess global capabilities and the ability to service large corporations — most startup vendors will assert that they have global capabilities, I have found that few actually do. If your firm has major operations around the globe, be cautious about U.S.-based vendors that claim that they currently have, or they will soon have, a “global reach.” Have one or more of your international country HR leaders assess the product closely to see if it fits their unique needs. Where relevant, work with your legal staff to make sure that the technology meets the laws and regulations in each of the countries where you have employees. Also be wary of startup vendors with mostly small clients that claim that their technology will also work at large international corporations, because dramatic scaling up is usually highly problematic.
- In many cases, external data must supplement internal data — the very best algorithms require “big data” in order to be accurate. So in order to maximize effectiveness, you will often have to supplement internal corporate data with a large amount of external data. That external data might include LinkedIn, Department of Labor data, criminal databases, economic growth data, and Census data. And, especially in cases where you’re trying to accurately predict the future, you need to be extra sure that external data will also be used.
- Run your own employees through the process — after a split sample test, one of the best ways to determine if a technology-driven assessment process is accurate is to run your own employees through it. For example, in the case of an electronic assessment process, run a sample of your best and worst employees through it. In the case of a recruiting sourcing tool that uses external databases, your own best employee should come out near the top of the list of top prospects.
- Candidate assessment factors are validated — in the cases where you are assessing applicants, candidates, or employees, use only assessment criteria that have been formally validated (i.e. statistically proven to predict on-the-job success). Be careful of any vendor supplied validation data that doesn’t precisely fit your jobs or your company.
- Are you provided with a competitive advantage? — vendors that provide technology to your firm can also provide the same tools and services to every one of your competitors. If all of your talent competitors simultaneously have access to the same product, you are merely keeping up, and you cannot be providing your firm with a competitive advantage. If the vendor is willing to give you exclusive use for a period of time among your talent competitors, that is a positive factor.
Final Thoughts
I am frequently approached by startup talent vendors. But not surprisingly to me, they never call back after I provide them with a list of criteria that a buyer should use to assess their product. I don’t even know of a single talent-tech vendor that recommends using a checklist or that provides one to potential customers. These vendors don’t even see the need for formal assessment. And it’s not because there is any dishonesty issue (there isn’t one) but instead, I find that it is because 1) they are in love with all technology, whether it works or not, and 2) they don’t really understand talent management or business because they’re really just technologists. As a result, they don’t understand the need for product performance metrics and they wouldn’t know what metrics to use, even if they did understand the need for them.
I encourage readers to pursue talent-tech because it has many extraordinary capabilities. However, at the same time, take a moment to remember the many historical failures in the tech-talent field: for example, ATSs, job matching software, referral software, and automated sourcing. Because the failures far outweigh the number of tech vendor firms that have lasted even five years, it pays to use a scientific approach to selecting vendors in this important but risky area. My final heads-up warning is don’t let your enthusiasm for new technology outweigh your business sense, so conduct a thorough due diligence and be cynical about anything that you hear.