Robin points out that people dislike track records across a range of contexts, and poses a puzzle: why?
Naively, track records seem pretty helpful for deciding who to trust and hire. Yet Robin points to a lack of track records or interest in track records for doctors, lawyers, pundits, teachers and academics.
He has also answered his puzzle in a later post, but I’m not going to read his answer until I’ve tried to guess myself (though I did glance at it enough to learn it is about elites and status).
Here are some theories I can think of:
- The tracked don’t like it. Most of the effort to avoid track records is from the people who would be tracked. They don’t like it because they are risk averse, and influence is not particularly well correlated with skill. When a large class of people spread throughout society dislikes something and most other people don’t automatically have strong views, most people’s views will end up being negative.
- Track records show mixed allegiance. People habitually try to publicize information which supports their own allies. Publishing track records in general is agreeing to publicize a giant pile of information, randomly supporting friends and foes alike. For instance if all records of doctor success become public, this would somewhat endanger any doctors you care about: doctors in your political party and friendship circles and minority doctors and nice doctors so on. That is, unless you had been careful to only side with competent doctors, which you probably haven’t, since you haven’t seen their track records. Similarly, if you read about the track records of pundits, half of them will be pundits you like looking bad or pundits you don’t like looking good, so you will have to surmise that the thing is bunk.
- People aren’t quantitative. Customers are not interested in track records because they are not interested in anything quantitative.
- Track records conflict with perceived quality. People perceive traits like charisma directly as goodness, and so when explicit indicators don’t align with it, they doubt the explicit indicators.
- Track records are hard to integrate. Customers have an overall sense of the worth of particular people, and the explicit measure only tells them one thing. They don’t want to trust the explicit measure on its own because they know it misses important things. But they don’t know what makes up their overall sense, so it’s hard to integrate the explicit track record into it. Similarly, suppose I have a general sense that I want to live in Berkeley instead of Boston, knowing lots of facts. Then I find that people in Boston have 10% better romantic success. I know that I don’t want to move on that basis, and I’m not really sure how to integrate that info into my overall view, except by crossing my fingers and hoping my subconscious took care of it. In this case, I might be disinterested in the romantic success figures, and generally wish people would stop talking about them.
- Track records undermine hypocrisy. Customers are looking for characteristics other than those they say they are looking for, and sellers are selling the characteristics that customers want. Both sides have to be vague about the measure of success, because if they explicitly and carefully measured the thing they say they want, it would make it harder to get the thing they really want. For instance, if people really want teachers to be good at indoctrinating their children into some particular culture, they might prefer to use their judgment rather than test scores, since they can’t ask for a cultural indoctrination test, directly.
- Publishing track records makes enemies. People who might publish track records know that some substantial fraction of the people being tracked will hate them for it, and on net this is bad for them. When they say that people aren’t interested, they mean ‘not interested enough to justify the backlash’.
- Track records bring unwanted responsibility. If customers have to decide which doctor to use, they enter a world where there are good and bad doctors and it is their own choice that determines whether their treatment is good. They prefer to live in the world where they can hand their problems to doctors and not be responsible, because that’s the whole point. So they just stubbornly live in that world. The same goes for other professionals.
- Track records for professionals make decisions harder. Perhaps people would like track records if the track record told them exactly what to do, but they just some more information to take into account, and the people already have trouble making decisions about things like who to hire, and so more information just makes them feel worse about not using it.
- Track records are socially awkward. Track records lead to embarrassing situations where you have to say ‘I really want to see Dr Phillips, not any of your other doctors’, which is pretty painful compared to some tiny chance of death, so it seems better not to know about them.
- People don’t like careful analysis that much. People basically don’t collect information and make informed decisions about almost anything, except sometimes when it is required professionally.
Ok, Robin’s answer is that the elites currently have opinions on who is good, or on heuristics that should be used to judge people, and they see other sources of information on this question as a threat to the influence of their opinions. They respond by trying to undermine the alternative sources, and punish the people responsible for them.
This doesn’t ring particularly true to me, but perhaps I’m misunderstanding. For instance, when I choose a doctor, I feel fairly ignorant about what elite opinion is on which doctor I should choose. I know that a few schools are considered good, though if I choose a doctor who didn’t go to the good schools over one who did, I find it hard to imagine any threat of elite retribution that I might fear. Nobody except my doctor knows which doctor I go to, and even if they did I assume it would usually mean nothing to them—it’s not as if doctors are public figures. And indeed, I don’t feel very worried.
I’ll leave the criticisms of my own theories to the reader. Do you have a preferred explanation?
Meteuphoric 
For instance, when I choose a doctor, I feel fairly ignorant about what elite opinion is on which doctor I should choose.
I don’t know if I am not stretching the meaning too far, but seems to me that the elite opinion in this case is something like: “Shut up, and take the first doctor available in the hospital.”
That means, the elite does not work on the level of “which doctor should Katja choose”, but on the level of “which doctors should be allowed to work at hospitals, and what specifically should they be allowed to do there”. All doctors in the hospitals are already approved by the elite, and trying to choose the best among them feels like second-guessing the elite. (If some doctor is not good enough, it is the elite’s job to notice them and remove them from the hospital; it is not your job to personally research them online and avoid them.)
And what’s the track record of tracking records?
People who support track records show strange reluctance to any kind of tracking records of tracking records, when one might think they’d be most enthusiastic about it.
Robin keeps talking about it, but never has anything more than a bunch of anecdotes to support it.
We use various kinds of quality rankings for a lot of things from ebay sellers to schools – most incorporate measurable outcomes (but rarely limit themselves to that) – it turns out this is not terribly useful in practice – they basically amount to saying that a brain surgeon has much higher death rate than a dentist so they must definitely suck. They will occasionally pick a really awful dentist, but you’ll need very complex models to get that signal out of raw data.
I would say it’s likely a combination of people not being quantitative, not liking careful analysis, an track records being hard to integrate. I would add though that doing track records well is a much harder problem then Robin assumes it is in his post. The real world is complex and many dimensional, so controling for difficultly so that the record is useful is often going to be hard.
As a minor tangent. Point two supposed that pundit track records would be 50-50 between people you like and don’t, but people have actualy tracked pundits and that is not the pattern that emerges. Domain experts tend to be consistently correct about their domain and non experts tend to be random. Since experts in a given area tend to agree with each other on the big things, if you look at pundit records you will find all the ones you like doing well or all of them doing poorly.
One big problem with track records is that track records are likely to have disproportionate effect on someone who got unlucky. Track records are not absolute indicators of quality, but are only associated with quality on a statistical basis. And if you measure quality on a statistical basis, you’re going to find that one person who statistically is only slightly less quality gets *drastically* affected by the fact that nobody is willing to patronise him. (And perhaps he just got an unlucky track record and is not actually low quality at all, but is indistinguishable statistically from someone low quality.)
In order to avoid the effect of someone who is 1% worse statistically losing 100% of the business, you need to use a measure that is noisy, *and* which has different noise every time. Track records are noisy, but the noise doesn’t vary every time someone looks at the track record. It’s the same problem with hiring people based on IQ scores or based on the statistical propensity of people with the same skin color to commit crimes.
I don’t want a world where a random half of all the doctors around go broke because they happened to have gotten a bad track record by chance and the patients are doing the statistically logical thing by never patronizing them.
Jiro’s concerns mirror my concerns. I was surprised to not see “people do not like to be pigeonholed for past mistakes” on the list of possible solutions to the puzzle. There are valid and invalid interpretations to track records, but given that most valid interpretations lie in the statistical realm, I am cynical at the possibility that a typical person would interpret the track record in a normative way or would take a charitable enough stance going in.
“Unlucky” extends to genetic luck, which contributes to my caution.
My point is that *even if the interpretation is valid*, track records are a bad idea.
The problem is that people pick the doctors (or employees, etc.) from the highest score down. At some point, they stop picking. If they stop picking before they run out of doctor slots, there will be doctors below some cutoff who never get picked (even if their score is only slightly below the score of doctors above the cutoff).
Which means that some doctors will end up permanently unemployed, because of people’s *completely rational* interpretation of the track records. Some of them may actually be less competent, but they need not be a *lot* less competent. Some of them may be unlucky. Regardless of which, the prospect of some doctors becoming permanently unemployed is really bad for society, both because we don’t want people to be permanently unemployed unless their incompetence is really severe, and because it would make entering the medical profession very risky.
You can fix this either by using a measure with variable noise (i.e. not track records) or by having the lower scoring doctors charge less (which fails because of patient risk-aversiveness).
And the same applies to hiring people based on race or IQ.
Another problem with track records is Goodhart’s law. Track records typically record one narrow metric, which is highly gameable. Instead of competing over quality, businesses now compete on gaming the relevant metric.
For example, there are 3 common outcomes to a potential lawsuit: winning, losing, or settling out of court. It is easy to track whether a lawyer wins or loses and hard to track whether they settle. If you look at track records of lawyers, then they will become incentivized to take many easily won cases to court in order to artificially boost their win percentage.