Before you read this article, please take a moment to help us learn more about your organization’s online hiring process. Rocket-Hire and ERE are currently conducting our second annual online screening and assessment usage survey. The data we collect from this survey will help to provide members of the ERE community with important data about trends in the usage on online screening and assessment tools. We plan on reporting our findings in an upcoming ERE article. Visit www.rocket-hire.com/survey to take our short, 10 minute survey and register for a chance to win a free copy of the Rocket-Hire Buyer’s Guide. As always, all results will be kept completely confidential. The use of online pre-screening tools has grown tremendously over the last few years. The basic idea behind pre-screening is to ask candidates direct questions about their skills, experiences, and interests. These questions are asked early in the hiring process to screen out unqualified applicants from the candidate pool. Applicants must pass pre-screening before they can be considered for interviews or other more in-depth staffing assessments. Most pre-screening tools use relatively simple questions to assess basic skills and qualifications. Common questions include, “What is your level of skill using Excel?” or “How many years of experience do you have as a manager?” Rocket-hire’s review of the online staffing assessment market uncovered over 20 vendors who offer pre-screening tools. Although they differ in sophistication and functionality, every pre-screening system we have seen allows clients to write and score their own questions. Some systems also provide clients with recommended pre-screening questions for different types of jobs or skills. Popular publications such as Business 2.0 and the Wall Street Journal have noted the growing use of pre-screening and called attention to potential problems with these tools. However, little objective systematic research has actually been done to evaluate whether using pre-screening tools truly leads to hiring better candidates. This prompted us to conduct our own investigation into the effectiveness of pre-screening. To do this we looked at information from several sources including:
-
- Statistical analysis of pre-screening data collected from several thousand candidates applying for a variety of professional and hourly jobs at a Fortune 500 company
- Discussions with recruiters and other staffing professionals who use pre-screening tools to staff exempt and non-exempt positions
- Relevant research reported in sources such as the Journal of Applied Psychology, Personnel Psychology, and the Society of Industrial & Organizational Psychology.
Having spent considerable time poring over a variety of data on pre-screening, we feel we can now confidently answer the question, does pre-screening lead to more effective hiring? The answer is a resounding “sometimes.” The purpose of this two-part article series is to share the results of our study and provide guidance on how to effectively use pre-screening as an assessment method. In this installment, we critique eight widely promoted “half-truths” about pre-screening. These represent common, but only partially substantiated, claims that are often made by pre-screening vendors. Next month’s installment will provide some suggested best practices for effectively using pre-screening tools. We call these statements half-truths because they represent things about pre-screening that, while not completely false, are not totally true either. They are perhaps best thought of as risks that need to be acknowledged and managed when implementing pre-screening systems Half-Truth #1: Pre-screening is simple and easy to use. First of all, it can be very difficult to write good pre-screen questions. Hiring managers are notoriously poor at identifying job requirements, let alone writing effective questions for assessing these requirements. Secondly, it is very difficult to write pre-screen questions that validly assess less tangible “soft” skills, such as ambition, work ethic, customer service, or innovation. As a result, pre-screening tends to have very limited utility for jobs where performance depends primarily on soft skills as opposed to specialized skills, knowledge, or qualifications. This is an important consideration, because the effectiveness of any pre-screening initiative is entirely dependent on the quality of the questions used. As they say in the computer industry: garbage in, garbage out. Our work with companies implementing pre-screening suggests that the learning curve for these systems can often exceed six months. One leading vendor indicated that learning to use their pre-screening system requires “a full day of training followed by regular, one-on-one coaching sessions for a month.” In other words, these are not simply “plug and play” tools. Half-Truth #2: Pre-screening identifies the best candidates. Pre-screening can effectively eliminate candidates who clearly lack key job qualifications. While pre-screening is effective for screening out the “bottom half” of the candidate pools, pre-screening tools lack the accuracy needed to differentiate between “great” candidates and those that are merely acceptable. It is not uncommon for the majority of qualified candidates to receive nearly identical pre-screening scores. Often the differences in pre-screening scores between candidates scoring at the 90th percentile and those scoring at the 60th percentile will depend on responses to one or two questions. These questions end up having a major impact on a candidate’s overall pre-screen ranking, even though the questions may be relatively weak indicators of actual candidate potential. For example, we found that the main difference between top scoring candidates and candidates scoring at the 75th percentile for one pre-screening questionnaire depended primarily on how candidates rated their knowledge of a single software system. While this question was job relevant, it was not considered to be a “make or break” question. The bottom line is that while pre-screening can help eliminate those applicants who are unqualified, it does not do a good job at identifying which of the remaining candidates are the most qualified. Half-Truth #3: Pre-screening is not affected by candidates “faking” their responses. It should come as little surprise that candidates often respond in an overly favorable manner when asked to rate their knowledge, skills, and experience as part of a job application. Many candidates would rather be eliminated during an interview with an actual person than to be automatically screened out by a machine. If this means stretching the truth and indicating that they had “extensive management experience” even if though it was informally as a team leader, so be it. After all, what constitutes extensive experience and expertise is largely a matter of perspective. One of the main problems with faking is that those candidates who do respond in an open and candid manner are likely to receive much lower scores than candidates who stretch the truth. Half-Truth #4: Pre-screening allows you to increase the quality of candidates by “raising the bar.” One of the supposed benefits of pre-screening is that it lets you effectively manage large numbers of candidates by allowing you to “raise the bar” so that you only look at the top candidates. This benefit might be true if it were not for the problems raised by half-truths #2 and #3. While pre-screening tools can always be used to eliminate candidates, there is often little guarantee they are eliminating the right candidates when they are used to screen out candidates at the upper end of the candidate pool. Several recruiters we spoke to even indicated that the best candidates are often not those with the highest scores, but those falling roughly between the 50th and 85th percentiles. These candidates tend to be the most candid and realistic about their skills and experiences. In contrast, candidates with the highest scores often appear to be “faking” their responses and to lack accurate insight into their strengths and weaknesses, or else they tend to be overqualified for the position. Half-Truth #5: Pre-screening questions are more effective than resume reviews. Another supposed benefit of pre-screening is that it can save recruiters from having to spend long hours reading through resumes. This is true insofar as pre-screening questions can automatically screen out resumes from candidates who lack certain specific job requirements. However, recruiters still need to read through the resumes of the 50% or so of the remaining candidates who meet the minimum job requirements. We would argue that including some level of resume review is actually a good thing. Unlike pre-screen questions that ask candidates to respond to questions about their skills, resumes reflect how candidates proactively describe themselves. This is an important difference, since how we describe ourselves on our own is different from how we describe ourselves in response to questions from others. Resumes allow candidates to provide more enriched and unique descriptions of their skills and experiences as they see themselves. Half-Truth #6: Pre-screening is not a form of selection. We once heard a pre-screening vendor make the following statement: “Our system is not a selection system and does not need to be held to the same standards as tools used to make selection decisions. We simply use pre-screening to place applicants into two groups, those you want to look at and those you don’t.” What this vendor failed to acknowledge (or perhaps did not want to acknowledge) is that removing applicants from the candidate pool and placing them in a group that will not be considered for potential hire is equivalent to making a selection decision for those candidates. Although there is still legal discussion around what constitutes an “applicant,” we would argue that any pre-screening system that places applicants into different groups based on whether they are considered to be high or low potential candidates is functioning as a selection system, and should be held to the same legal standards and guidelines as any other selection instrument. It may be possible to implement pre-screening tools so that they do not need to be treated as a formal selection tools. This requires using pre-screening purely as a way to prioritize recruiting efforts, without specifically labeling candidates as “qualified” or “unqualified.” The easiest way to do this is to rank order candidates based on the pre-screening reports and simply work down the list reviewing candidates until you identify those you wish to pursue further. Note that this approach has the same basic effect as eliminating low scoring applicants since it is unlikely you will get that far down the list before finding some potentially suitable candidates to hire. However, it never specifically labels candidates as ineligible for hire, and thus the system never actually makes a selection decision. Half-Truth #7: Candidates do not mind answering pre-screening questions. Research clearly indicates that candidates are willing to answer fairly lengthy sets of pre-screening questions. However, this does not mean that candidates necessarily like these questions. The candidates we have talked to recognize why pre-screening tools are used and are willing to enter data into these systems. In general, we expect that candidate reactions to pre-screening systems are likely to depend on the same things that influence candidate reactions to any selection tool: ease of use, questions that are clearly job relevant and easy to answer, and timely and meaningful feedback on the status of their application. Half-Truth #8: Pre-screening leads to better hiring decisions. Pre-screening will almost always reduce the time needed to process candidates. However, there are many reasons why pre-screening may not necessarily lead to better hires. In order for pre-screening to increase the chances of hiring the right candidates, careful attention must be made to ensuring the system is appropriately designed and deployed. This means attending to the various risks highlighted earlier in this article. Unfortunately, many pre-screening vendors focus far more on staffing efficiency as opposed to staffing effectiveness. As one staffing manager told us, many pre-screening systems were designed by people who have a background in supply-chain automation. These vendors focus more on volume of product moved than actual quality. It’s as though they believe that the value of a staffing system should be measured in terms of kilos of candidates placed, with little attention given to the actual performance of candidates after they have been hired. Although this article admittedly has a very critical tone toward pre-screening, we are not suggesting that companies should abandon the use of pre-screening tools. To the contrary, we are strong advocates of pre-screening as an effective method for identifying talent provided the tools are rigorously designed and effectively used. Pre-screening will almost always increase the ability of recruiters to efficiently process large quantities of candidates. However, it may not lead to more effective identification of high-quality candidates. In fact, poorly implemented pre-screening tools can systematically screening out the best candidates, leaving a “pre-screened” pool of candidates who are largely inexperienced, insincere, or both. This risk is compounded by the variety of pre-screening vendors who have proven themselves ready and willing to sell their systems to clients without actually testing whether their tools actually lead to better hiring decisions. In the next installment, we will provide some specific guidelines and best practices for effective use of pre-screening.