Seemingly out of nowhere, 2023 has begun with an explosion of hype about the conversational AI tool called ChatGPT. Most of this press has been negative, focusing on the tool’s ability to act as a credible imposter, facilitate cheating, and take our jobs. It seems almost no industry is immune from dystopian thoughts about the changes this technology will bring.
Certainly, all this buzz is an overreaction. Or is it?
It’s hard to know for sure because we are just getting started on a long long journey with conversational AI. In his recent column for Forbes, Lance Elliot does an excellent job of explaining what ChatGPT is and exploring its potential impact. While Elliot does cover a range of potential long-term problems that may arise, he provides no definite answers, concluding that only “time will tell” if it is good or bad for mankind.
The Freakout Over ChatGPT
For anyone who has interacted with ChatGPT or has read some of the press about it, there is a tangible feeling that this one is different. It is certainly not one of the dumb chatbots with which we are used to interacting.
In some ways ChatGPT is like the groundbreaking ‘90s search engine Ask Jeeves, whose ability to provide relevant search results based on questions stated in plain English seemed like magic at the time. But instead of just returning links, ChatGPT returns curated information that can be used to immediately accomplish specific tasks such as writing essays, taking exams, applying for a job, and even answering job-interview questions.
ChatGPT’s wow factor and its fear factor arise from results that are well-written, accurate, and thorough enough to provide instant perceptions of credibility and usefulness. In particular, when it comes to hiring, ChatGPT has already been making news by passing certification exams, getting cover letters through the gauntlet, and acing job interviews.
These stories have the talent acquisition community more concerned than excited. But are these concerns justified? Is there a positive side to ChatGPT3 when it comes to hiring now and in the future?
To answer these questions, we must understand that hiring is a two-way dialogue between applicants and employers, so any thought about ChatGPT’s use in hiring must consider both sides of the fence.
From the perspective of talent acquisition professionals, ChatGPT is a problem that is likely to get worse as the tool gains increased powers. The idea that important parts of the job-application process (e.g., cover letters, interview responses, etc.) are being done by machines, not actual job applicants, is definitely concerning.
But candidates are likely to see things differently. Applying for jobs is a hard and mostly thankless task. Humans cannot effectively process the volume of applications, and while machines can, their hit rate is far from perfect. Job seekers must do whatever they can to get noticed.
(Of course, this is not new. Remember the days when job seekers would use the watermark feature to fill their resumes with invisible key words to try and get past digital gate keepers?)
In one impromptu experiment, when used to write fake cover letters, ChatGPT’s were seen as enough to move fictional candidates forward despite the letters’ generic wording, dry writing style, and lack of personality.
So, sure, ChatGPT can handle the first steps of the hiring process, but let’s be honest. How important are cover letters really?
Neil Taylor, founder of the communications consultancy Schwa, tested his recruitment team and hiring software by using ChatGPT to answer a written screening question. ChatGPT’s initial answer to the question, while accurate, was very simplistic and not sufficient to allow it to move forward in the process. Only after Taylor continued to work with ChatGPT to refine its answer was it able to create one that was credible enough to pass. Taylor’s take on his experience provides a very good summary of ChatGPT’s abilities as a job applicant:
“It’s dramatizing the gap between something that’s really good and something just pretty good,” said Taylor.
“Lots of people can produce pretty good— and so can ChatGPT.”
So, ChatGPT can rise slightly above mediocrity, which can often be just enough to get noticed. But when it comes to these more complex stages of the process where trained humans are evaluating applicants’ characteristics and qualifications more directly, ChatGPT’s flair for regurgitating information that lacks complexity, personality, and individuality provide it little to no advantage.
It is therefore too soon to fear ChatGPT as a ubiquitous imposter that wreaks havoc on hiring processes across the globe. ChatGPT is a great research tool. It can be a good companion for job seekers, allowing them to easily access important information about a company and its available jobs. And while it may get candidates noticed, it won’t be enough to get them hired.
But what about the other side of the fence? Does the technology provide any advantage for those who build and support the hiring process itself?
While they do not do the exact same work as recruiters and talent acquisition professionals, IO psychologists play an important role in designing accurate and compliant hiring systems. My team of IO psychologists have been obsessively playing with ChatGPT to see how it performs on the tasks that are our bread and butter. These include:
- Job analysis. Breaking down a job into the traits, skills, and competencies required for effective performance in a specific job.
- Building test plans. Taking the results of the job analysis and selecting the specific assessments and scales that are ideal for assessing candidates for the job.
- Content creation. Developing custom content to fulfill the test plan (e.g., structured interview questions, situational judgment scenarios, etc).
What Our Own ChatGPT Experiment Revealed
When we asked it to do some of the above tasks, we were impressed by its ability to return very organized information that is pretty darn accurate. But these responses were relatively generic and do not do a good job of finding subtleties and nuances. When it comes to building assessments that are optimally predictive for a specific job and environment, such nuances make all the difference.
For instance, we asked ChatGPT to do a job analysis for a contact center agent position for which we have created a validated assessment. Our question was: “Can you please provide job analysis results for the Expert position at Company X?”
Here was the ChatGPT response:
I’m sorry, as a language model AI, I don’t have the capability to conduct a job analysis or provide specific job analysis results. However, based on my general knowledge, a job analysis for the Company X Expert position might include the following results:
- Job Duties and Responsibilities: As an Expert at Company X, the main responsibilities might include providing technical support, resolving customer inquiries, and troubleshooting issues related to products and services.
- Essential Skills and Knowledge: The Expert position at Company X might require technical knowledge in a specific product or technology, excellent customer service skills, problem-solving abilities, and effective communication skills, both verbal and written.
- Physical Demands: The Expert position at Company X may require prolonged periods of sitting or standing and the ability to use a computer for extended periods of time.
- Working Conditions: The Expert position at Company X might involve a fast-paced work environment, with a focus on providing excellent customer service and meeting performance goals.
- Qualifications and Experience: The Expert position at Company X may require a high school diploma or equivalent, prior customer service or technical support experience, and certifications in relevant products or technologies.
Note: These are general results and may vary based on the specific company, location, and role requirements.
When we asked ChatGPT what assessments should be used to evaluate applicants for this position, it replied as follows:
For the Expert position at Company X, the following assessments can be used:
- Technical Aptitude Test: to assess the candidate’s technical knowledge and skills relevant to the role.
- Customer Service Skills Assessment: to evaluate the candidate’s ability to handle customer inquiries, resolve issues and provide excellent service.
- Communication Skills Test: to assess the candidate’s written and verbal communication abilities.
- Problem-Solving Assessment: to evaluate the candidate’s ability to analyze information and make decisions.
- Time Management Assessment: to assess the candidate’s ability to prioritize tasks and meet deadlines.
- Online Simulation: to evaluate the candidate’s ability to work under pressure, handle multiple tasks and provide efficient service.
Note: It’s advisable to consult with local employment laws and regulations and also with Company X’s HR department to ensure the assessments comply with legal requirements and align with company policies.
So how close did ChatGPT come to our professional results? Its suggestions are credible and are accurate in a generic sense. But compared to the reality of this job and the ideal assessments to be used, it whiffed.
The two biggest determinants of success for the job in question are the ability to influence customers calling in for tech support to purchase extended warranties, and a true passion for mobile devices and related technology (we call this “tech curiosity”). While ChatGPT’s suggestions are indeed relevant, an assessment that does not account for influence/sales ability and tech curiosity would fail at predicting success in Company X’s Expert role.
We found similar results when we asked the tool to create situational judgment scenarios and structured interview questions, credible but generic responses that are a good starting point and can likely be found with an in-depth web search.
Will ChatGPT Take Away Recruiting Jobs?
Will ChatGPT make IO psychologists and talent acquisition professionals irrelevant in the next few years?
Not so much. The tool can provide a good way to help understand the basic qualifications needed for a job and even generate job descriptions, interview questions, and suggested ways to screen for it. But relying on these results alone will not be sufficient for optimizing the hiring process for a given role in a specific company, leaving a good bit of money on the table.
In the end, ChatGPT is extremely impressive. While it returns responses that are extremely well-written and articulated, it is just an assistant, not a replacement. Like most AI, it can automate tasks such as web research to save time, but it lacks the detail and nuance to do the job on its own.
C. Thi Nguyen offers this excellent summary:
“GPT-3 is incredibly powerful, but it’s only as good as its training data. And GPT-3 achieves its power through the vastness of its training data-set. Such data-sets cannot be hand-picked for some sensitive, subtle value. They are most likely to be built around simple, easy-to-capture targets. And such targets are likely to drive us towards the most brute and simplistic artistic values, like addictiveness and binge-worthiness, rather than the subtler and richer ones. GPT-3 is a very powerful engine, but, by its very nature, it will tend to be aimed at overly simple targets.”
For now, our jobs are safe! But once this thing has a soul, then we need to worry. How long until that happens is anyone’s guess.