A few weeks ago, I wrote a piece about bias and noise in decision making and focussed more on bias than on noise. In this commentary, I explain in greater detail what noise is, why it is important, and what we might be able to do about it.

To recap, bias is the tendency to consistently skew a decision in a certain direction. A judge may tend to sentence people found guilty to longer sentences than other judges. Some appraisers may regularly overestimate the value of properties. That is bias.

So then, what is noise? Danial Kahneman, a Nobel Prize-winning behavioural psychologist, recently wrote a book with Olivier Sibony and Cass Sunstein. Noise: A Flaw in Human Judgement discusses noise in decision making, its impact, and what can be done about it.

Their explanation of the phenomenon of noise is this: Presented with the same facts, people will come to radically different conclusions. And surprisingly, when asked to estimate how big the differences might be between their judgements, they vastly underestimate the magnitude of them.

For example, different pathologists rating the severity of a disease from the same set of biopsy results correlated only 61% of the time. When software engineers presented with the same problem were asked to estimate the time it would take to complete the task, the estimated time varied by 71% on average.

These variations, given the same information, happen in many different applications, such as insurance-loss adjustments, stock-portfolio valuations, real-estate appraisals, employee-performance evaluations, loan approvals, financial audits, and the like.

So that there is no danger of noise, senior management have designed systems in which most lower-level employees in their day-to-day work do not have the ability to make a decision that another employee in the same circumstances would make differently. They have to apply rigid rules — if A do B, if C do D, etc.

Dealers at a blackjack table have specific rules about their sequence of actions, such as dealing the cards from left to right, who wins and who loses, with which hand they pay the winning bet, and in what sequence they pay the winners. That is not to say they do not make mistakes, but the ability to make decisions without rigid rules has been all but eliminated.

On the other hand, senior management are allowed leeway. They are expected to use their “skill and judgement” when making decisions. This is what they get paid for and it’s sometimes vast amounts of money. The thinking is that a good manager, given the circumstances, will make the better or best decision.

However, Kahneman and his colleagues found that there were big disparities in what decisions were actually made. One would have thought that highly experienced managers would make similar decisions based on the same set of circumstances. Far from it.

Noise is rife throughout industries. So what? Well, noise costs money. When bad decisions are made, companies not only lose money as a consequence, but customers are also lost, employees feel undervalued and demotivated, and in some circumstances, lives are lost.

Humans like consistency; they want things to be fair and reliable. They do not expect big variations in outcomes, depending on the time of day a decision is made, the weather that morning, or anything else. They expect uniformity.  Apparent random differences reduce confidence and lead to a lack of trust in the decision makers.

With regards to the gambling industry, in some countries, regulators are handing out large fines based on the actions (or lack thereof) of some people. No doubt, some of the violations were egregious and worthy of large fines. But I am willing to bet that some were simple judgement calls and perhaps, with hindsight, they may have been done differently. Or perhaps not.

There will always be noise in decisions about people’s gambling behaviour and what should be done about it. Regulators, usually with hindsight, will undoubtedly think that given the evidence available, they themselves would have come to a different conclusion and therefore the operator should have done so too. Regulators need to appreciate the impact of noise in decision making and that hard-and-fast rules cannot always apply.

Kahneman proposes that organisations conduct “noise audits,” not to point out bad decisions, but to use them to make better decisions. He and his co-authors conducted a number of experiments with financial-services organisations, asking skilled professionals to evaluate some cases, as they would in the course of their usual job, and decide on a monetary amount for each case.

Prior to the experiment, the managers expected a variation of between 5% and 10% and considered these to be within acceptable limits for “matters of judgement”. What they got were differences between 34% and 64%! Long service, i.e. experience, was no indicator of more consistent decisions.

Kahneman, Sibony, and Sunstein suggest that this is brought about because a) experienced professionals overestimate the accuracy of their own judgements and b) they have a great deal of respect for their co-workers’ intelligence. These two together lead to an overestimation of agreement.

So what can be done about noise? In some circumstances, replacing human decisions with simple algorithms has been shown to lead to better (and always consistent) decisions. For example, hospital managers are now insisting that doctors follow flow charts when making diagnoses. They know that this can have a big positive impact on health outcomes.

But it is not always practical or even desirable to replace human judgements with an algorithmic process. In these circumstances, what can be done?

Consistency is the answer. Managers should be trained how to approach decision making, using similar methods to gather and analyse information. These methods can be integrated in a view of the case, then that view can be translated into a decision. Groups of managers should get together on a regular basis to review their decisions, not to point out the bad ones, but to see what can be done to improve their decisions.

None of this will weed out noise entirely, but an appreciation of noise and how to deal with it should lead to better decision making.