IT-related jobs are the poster child for the “talent crunch” that defines our current labor market.
With almost a million unfilled IT jobs, 3.7 million tech job postings, and almost 300,000 more jobs to be added in the coming year, the demand for skilled IT professionals is as real as it gets. But when you look at the supply side of the equation, there is a serious imbalance that presents a clear and present danger to the economy.
At the same time, pre-hire assessments for IT professionals are pervasive. By my current count, there are at least 100 companies that provide some form of testing for IT skills. Even more common are homegrown coding assessments in the form of “interview homework” and on-site “whiteboard challenges.” These hands-on exercises created by hiring managers require applicants to apply their knowledge of coding to solve replicas of real problems faced on the job.
When comparing these two trends, something clearly does not compute. By putting up barriers that constrict applicant flow, employers could be adding fuel to the labor-scarcity fire.
A core concept in testing is the impact of the ratio of applicants to job openings, or the “selection ratio.” When there are many applicants for few positions, screening tools have tremendous value because they allow employers to be very choosy. With a poor selection ratio, the value of testing goes down significantly. But empty seats can be very costly, possibly even more so than the expense of filling them with less qualified hires.
Willingly reducing the applicant pool in a drought seems to indicate that the juice from adding an employment test to the hiring process must be worth the squeeze. Possible reasons for this phenomenon include:
- IT jobs are desirable and pay well, which leads to over-exaggeration of skill level that can be detected using tests.
- Many developers are self-taught and have not taken a traditional route, meaning their resumes often lack supporting evidence of their skill level. Tests can help shed light on a dark resume.
- There is a wide range of performance among coders — some are true artists, while others are drones. The value of hiring an artist can often exceed that of hiring multiple coders with average skills. A good test can help differentiate the artists from the rest of the herd.
- The sense of ownership of the hiring process and the comfort associated with quality control provide confidence.
- Testing is a tried-and-true value-add. We know testing works for almost any job, so why not take advantage of it?
4 Categories of Coding Assessments
The value in a coding assessment is in using it correctly, especially when the supply of candidates is low. Getting it right starts with understanding the different flavors of coding assessments and making sure you choose the right type for your needs. Coding assessments fall into four basic categories:
- Cognitive tests. These can be used for almost any job and provide an excellent way to look at bigger-picture abilities required for coding (i.e., fault-finding, logical reasoning, etc.). They’re also valuable when hiring for potential (vs. experience). While this is an increasingly popular strategy in a tight labor market, the error rate of hiring for potential alone can be high, making testing a big value-add.
- Multiple-choice skills tests. These are basic tests with questions that gauge knowledge of a particular technical skill or coding language. They are great for determining if applicants actually have their claimed level of expertise, but they lack the ability to identify true artists or allow applicants to show their stuff.
- Vendor-provided coding exercises/interviews. These are typically hands-on coding problems that are simulated using an online platform. They present a highly realistic exercise that can provide insight into a candidate’s true ability level. They also engage applicants by giving them a chance to show you their abilities. There is a huge number of these types of tests available — indeed, they are the fastest-growing segment of the IT testing market.
- Homemade whiteboard coding challenges. It is very common for hiring managers to create their own coding challenges. These often take the form of take-home exercises or live in-person challenges using an actual whiteboard. These assessments can be great because they typically map very closely to the types of problems faced on the job. But be careful — these tests require oversight and due diligence to ensure they are effective and legally compliant.
Ensuring Validity
It is important to understand that all types of coding assessments are subject to the same set of rules pertaining to any employment screen, test, or interview. This is a good thing because it helps ensure that tests deliver on their potential. Making sure you are choosing the right test and using it correctly really boils down to one core thing — making sure the test you are using is validated. (Test validity is a complex subject in its own right, and beyond the scope of this article, but you can learn more about it here.) In a nutshell, key aspects of validating an IT assessment include:
- A quality construction process. Questions must be constructed by subject matter experts (SMEs) who can verify measurements. In the case of cognitive and skills assessments, psychometric analyses must be conducted to ensure that tests are reliable and measure what they are supposed to.
- Documenting job relevance. Users must be able to document that what is measured is actually required to perform the job. This can be accomplished in many ways, all of which require a documentation process in which SMEs verify the relevance of the test.
- Structure and standardization. It is critical that the same assessment is given to all applicants for a given req and that people are evaluated and scored using a structured evaluation process to support decision-making.
In general, coding assessments present a lower level of risk than typical assessments. They are extremely “face valid,” meaning it is easy to see their job relevance. If the job requires expertise in Java and the test measures Java skill level, there is not much grey area. Coding simulations are essentially work samples, miniature replicas of the job. And work samples have some of the highest levels of predictive accuracy.
Managing Risk
Still, there are two scenarios where coding assessments present a higher level of risk, and require more oversight:
- Cognitive tests. Because these usually measure traits that are not job-specific, it may not be as easy to show how they relate to job performance, thus eroding confidence of both hiring professionals and applicants alike. While cognitive tests can be really predictive, they create more risk because they consistently show bias. If using a cognitive test for any situation, IT jobs included, it is a good idea to ensure that the vendor has documentation of its validation for similar jobs or that you do a local validation study if possible.
- Whiteboard challenges. In this realm, renegade, one-off exercises created by individual hiring managers are the norm. While they can be highly job-relevant and very effective, ad hoc challenges often violate some basic rules of testing and thus present risk. Risk factors include that applicants don’t always get the same test, there is often no oversight or quality control to ensure challenges are constructed properly, and that tests often lack a structured evaluation process, leading to potential for bias. Getting coding challenges right requires standardization and consistency — related to both the creation and evaluation processes.
When considering using a coding assessment, it is also critical to account for the candidate’s point of view. Coding assessments are a polarizing subject among developers. While many candidates love the opportunity to showcase their abilities, coding-challenge assessments can be a time-suck for applicants, especially when they get no feedback from an employer. Consequently, as with any assessment, it is important for employers to be on their best behavior when using a coding assessment. (Check out “How to Keep Pre-Hire Assessments From Wrecking the Candidate Experience.”)
Ultimately, while reducing the applicant pool for IT job openings may be scary for employers, wasting time and money on dud hires is an even less attractive alternative that can wreak havoc for years to come.
