The main thing about reference checks
Do not outsource your hiring decision
Fair warning: this is written in a moment of exasperation at yet another person claiming how they turned down a candidates because the backchannel said they were “in the top 10% but not in the top 1%”.
That is not how reference checks ought to be done. After years of hiring some incredible, incredible people, here are my rules for reference checks. They are not going to be in a neat MECE because I’m quite literally banging away at the Cherry MX Brown keys in front of me out of aforementioned exasperation.
Umang’s Five Rules of Reference Checks
Do not outsource your hiring decision
Calibrate the reference
Find the superfans
Everyone has haters
Performance changes, behavior rarely does
Let’s dive into each.
1. Do not outsource your hiring decision
By letting reference checks swing your decision, or hinging your decision on the reference check (“we’re on the fence but let’s decide after references” is not an uncommon conclusion of hiring debriefs) you are essentially outsourcing your hiring decision. Don’t do this. Reference checks should be just one input into your decision, and not even the most important one.
Questions like “would you rate this candidate as top 10% of all PMs you’ve worked with or the top 1%?” are essentially outsourcing your decision — you are asking for a judgement call. Ask for relevant information, preferably facts, that will feed into your decision.
2. Calibrate the reference
The most important part of doing a reference check is figuring out how much weight you can accord to the input coming from a third party. You’re not just evaluating the candidate, you are also evaluating the references, whether provided by the candidate or a backchannel.
Someone you have worked with in the past is obviously the best because you understand their judgement and you have shared context (i.e. you are calibrated). That is not common, however, so spend some time researching the reference and asking them how they worked with your candidate and use your judgement to calibrate their input. See #1.
3. Find the superfans
I know everyone says that candidate-provided references are low utility because they are obviously going to be positive. To which I say: yes, and that is exactly why these are important. Does the candidate have 3-5 people from their professional lives who will not only take 15 precious minutes out of their workday at short notice, but also sing high praises about them without a twinge of embarrassment?
If someone has even a few superfans, they must have done something right. Find superfans from different points in their career.
4. Everyone has haters
It is nearly impossible to do anything of import without encountering resistance. It is nearly impossible to surmount all resistance quickly without ruffling feathers, going around people, challenging authority, breaking process, ignoring conventional wisdom, etc. While ultimately in business the (shared) wins result in forgiveness of past trespasses, some grudges remain. If you get a negative reference, don’t let that be a detractor. Instead, dig in and try to understand it more. See #2 and #1.
5. Performance changes, behavior rarely does
Most reference check input on skills and performance is not high value: that’s your job to evaluate, and you should have already gotten signal on those in the interviews. See #1.
Questions like “what did this person not do well” about a role 5 years ago has no utility in your evaluation. Skills change all the time and, besides, sometimes people get put in the wrong role, or the role morphs a little to be ill-suited for them in just one particular dimension.
The more important insight is what happened when they didn’t “do well”. Behavior changes infrequently, perhaps only on long timescales and career arcs. Look for patterns in behavior, especially if you notice potential pitfalls during the interview process.
That’s it. There is no need to overcomplicate this process. I typically like to ask them what it was like working with the candidate with a lot of examples, and let them talk for 15 minutes. I find I learn a lot more this way than preparing a list of questions. Sometimes I will have specific hypotheses to evaluate.