My better half reports from assignment at the business school at INSEAD that Dr. Jack Soll made a clever analogy likening most interviewing processes to the Pepsi Challenge "Sip Test". I should be somewhat careful here to note that I’m not sure how far the analogy went, and I want to be careful here because academics live based on being rigorous in approach and analysis. Suffice it to say that my wife and I had the initial impression that Jack’s analogy was very clever (and pretty darn good too).
First I’ll bring readers up-to-date on the "Sip Test", which has been highlighted recently in Malcolm Gladwell’s book, "Blink". Scott Liben gets at it pretty well in a recent article:
I’m more than old enough to remember the commercial campaign a couple
of decades ago called the Pepsi Challenge. It capitalized on what
marketers call a "sip test," in which even committed Coke drinkers
chose Pepsi over their usual cola when they weren’t told which was
which. Even when the makers of Coke conducted their own such tests, a
majority preferred Pepsi. Rattled by the results, they launched New
Coke, a direct product of market research — and one of the most
notorious product flops of all time.
The problem, in short:
The "sip test" or CLT (for Central Location Test) in which Pepsi always
prevailed turned out to have very little in common with the way buyers
of soft drinks actually consume such products on their own. Gladwell
quotes Carol Dollard, of Pepsi’s new-product development department,
who says "Sometimes a sip tastes good and a whole bottle
doesn’t. That’s why home-use tests give you the best information. The
user isn’t in an artificial setting. They are at home, sitting in front
of the TV, and the way they feel in that situation is the most
reflective of how they will behave when the product hits the market."
Now research on how well interviews correlate with actual job performance once a person has been hired has shown that there is "very little" to "some" correlation. A person that did well interviewing may either do well or poorly on the job. The probability of getting it right, is measureably better than a flip of a coin.
So is interviewing for a job basically like a sip test where a 2-hour interview process has little bearing on how well a person will do in the long-run as a hire? Very disheartening. To top that off with the fact that most people are not aware of interviewing biases that may happen between first round and second round interviews with respect to interviews and the use of screening versus selection processes (that is, you need to be part of the crowd and on par in one case but stand out in the other case), the story just gets worse.
The second article I cited above hints that having more structured interviewing processes works better.
In light of that, some tactics that I have used that seem to work well (where each may also have some issues in different contexts [too detailed to go into here though]) are:
- establishing an arm-length relationship (a 1099 relationship) first with contract-to-hire terms if all goes well after a period of time
- establishing a trial period in an employment agreement (with somewhat higher salary terms in the trial period) to compensate the contractor
- having some joint problem solving tasks with the prospective hire, prior to an employment agreement being established
So on the surface, hiring good people matters a lot. Are we certain that our hiring process is not a sip test though?