[PREV - QUESTIONING_DEBIASING] [TOP]
SCATTER_THE_NOISE
August 7, 2023
October 24, 2023
December 11, 2023
"Noise" (2021) by Kahneman, Sibony and Sunstein
The subject at hand is improving
organizational decision-making. The
claim is that while assessing the
accuracy of decisions can be difficult,
it's easier to observe when multiple
judgments are scattered widely. The
authors argue that simply trying to
reduce this inconsistency without Identifying bias in a series
worrying quite so much about correctness of judgments would require
is a quick way to improve the process. some knowledge of the right
answer in order to determine
how far the average judgment
deviates from a correct one.
But if, for example, you have a
number of experts on staff
trying to make a prediction, you
can look at the variations
between predictions even if you
don't yet know who's right.
Now really, they don't mean "noise" they mean
"scatter": the way your hits spread out around The familiar distinction
an average when you've been aiming at a target. is between "precision"
and "accuracy"--
None of your associations with the word
"noise" suggest what they actually want Though that bit of
to talk about-- this is not, for technical jargon has
example, a book about signal-to-noise it's own issues:
ratios in a communications channel. an average English
speaker would presume
those are synonyms.
PRECISELY_ACCURATE
Their primary
example is legal This is what they
decisions: began with and tried
to generalize from.
You would hope that questions of guilt and
sentencing would be determined by the quality
of the evidence and the circumstances of the
case. In practice, it's well understood that
the individual characteristics of the judge
matter tremendously. We want a system that
approximates justice but it often seems that
what we have is a crap shoot about which judge
you get assigned.
The authors consistently treat reducing
scatter (which they insist on calling "noise")
as the golden road to reducing overall error,
with only an occasional admission that in the
process you might sometimes inadvertanly
increase bias.
But consider that one way you could fix the
problem of scatter in legal judgements is to
make every judge a hanging judge. If every
judge invariably went "thumbs down", then
scatter would be reduced to zero-- a great
improvement, right?
They may very well be correct that large
amounts of "noise" are often symptomatic
of a problem, but it doesn't necessarily
follow that reducing that "noise" is the
way to fix the problem.
Really, there's no particular reason to assume
scatter and bias are independent of each other.
Myself, I would expect that if an
organization put emphasis on reducing
scatter, then reduced scatter is what For any general
they would get even though accuracy principle you can
would quite likely go to hell. formulate, human beings
will find the wrong way
E.g. the independent experts you've to apply it.
been examining might start consulting
each other to make sure they're
clustered closer together.
Now the authors are no doubt correct that
variation is often easier to identify than
correctness, and large amounts of variation
can be a strong indication of a systemic
problem...
But the actual goal is *hitting
the target*, so you should keep
your eye on the target.
Somewhere in this book there may
be some verbiage that addresses
this point, but even if so it's
lost in the roar of the headline
message.
On nearly every page they
regard it as a given that any
reduction in noise is an
improvement in decision making,
and any improvement in decision
making can be regarded as a way
of reducing noise.
It's competely shot-through (and
for me, completely shot down)
with this confusion of symptom
and disease.
Maybe I shouldn't have been
surprised: this confusion is
evident right there on the cover,
in the book's subtitle:
NOISE
A Flaw in Human Judgement
--------
[NEXT - PRECISELY_ACCURATE]