You Won’t Always Hit the Home Run in a Technical Interview
Technical interviews are full of surprises, but that doesn’t mean they are random.
Technical interviews are the bane of every novice programmer and the reason for countless hours practicing the herculean task of writing code unaided by Stack Overflow. In spite of all our efforts, we will inevitably be hit with an unexpected question in an interview. If the “regret to inform” email reaches our mailbox some days later, it’s hard not to curse at whoever thought a silly question said anything about our abilities as a programmer. This might have been the frustration that inspired Aline Lerner to write the blog post “Technical interview performance is kind of arbitrary. Here’s the data.” While the data used in the blog post is solid, the post’s point is weakened by how it does not consider the effect that interview question types have on candidate scores.
The blog post explores data from 299 interviews performed by 67 users of interviewing.io, a platform founded by Lerner. Users of interviewing.io practice interview problems and are rated in a scale from 1 to 4. The data describes the scores from users who practiced at least twice. Lerner backs a claim that interview performance is arbitrary by pointing to the high volatility in scores. The idea makes sense, and memories of failed interviews hits the reader close to home, but the fact is that things are more complicated than that.
The main flaw of Lerner’s argument is that it does not consider how the interview question types can affect scores, yet be an indicator of candidate performance in a role. Leetcode.com, the leading coding question practice platform, separates its questions into 40 different categories, each testing a different skill or family of algorithm. Interviewers will often select questions of a categories that most closely fit with the job requirements. This category will, with no doubt, correlate with a person’s score when answering a question. Without taking that into consideration, the blog post’s argument looses significant strength.
The visualizations provided in the blog post are consistent with this theory. The charts presented in the blog post consistently tell us that a) scores are volatile and b) people with higher mean technical scores have more consistent probabilities of success. This could be explained by how people with lower scores have a large disparity between question categories. The blog post even mentions that “many people who scored at least one 4 also scored at least one 2”. While the blog post frames this as evidence that this person should be seen as a 3 by employers, it might as well be evidence they should be 2s for one kind of job and 4s for another. I’d like to make it clear that this is a theory I cannot prove with the data I am given, though neither can the content of the blog disprove it.
To illustrate this problem, imagine you are preparing for an back-end software developer interview with Two-Sigma, a quantitative investment firm. Due to the nature of quantitative finance, Two-Sigma mostly asks coding problems related to statistics. You, on the other hand, are mostly comfortable with bit manipulation questions. During the interview, the interviewer asks you a statistics coding problem, you don’t go well, and don’t get the job. Lerner’s blog suggests your performance was arbitrary, when in fact it was highly correlated with the question category and job requirements.
It’s easy to blame a process for our missed opportunities, but it’s harder to realize things are more complicated than what they seem. Aline Lerner makes a valid point calling technical interviews arbitrary, though she does not consider how interviews will likely not ask totally random questions, as the users in the dataset were. While more data might prove my doubts are unsubstantiated, it seems like the best remedy for reducing the chances of a curve ball is doing your homework on the company and position. Still, we can always dream of scoring a home run.
Sources
- Lerner, A. (2018, December 23). Technical interview performance is kind of arbitrary. Here’s the data. Retrieved from http://blog.interviewing.io/technical-interview-performance-is-kind-of-arbitrary-heres-the-data/
- Problems. (n.d.). Retrieved November 17, 2020, from https://leetcode.com/problemset/all/
