“Noise: A Flaw in Human Judgement” by Daniel Kahneman explains that, “Wherever there is judgement, there is noise – and more of it than you think.”

In this article, you’ll learn how inconsistency in decision-making can cost your business and what to do to reduce the cost of your business decisions.

What is Noise

You know how insurance companies are supposed to make consistent decisions when it comes to claims and premiums? Well, believe it or not, sometimes they cannot seem to agree on the same cases. A noise audit revealed that the average difference was 55% for risk assessment and 43% for claims adjusters. This means that one risk assessor may set a premium at $9,500 while the other may quote $16,700. 

This unwanted variation in judgments, which often produces costly errors for companies, is known as noise.

Noise vs. Bias

In decision-making, variable judgments happen due to various factors, such as random fluctuations or errors in data, cognitive biases, as well as external factors such as weather or even your mood.

We have learnt that the variability beyond your control is an instance of noise. However, cognitive and social biases are predominantly associated with distortions and errors in decision-making, causing us to veer off course. You can see the difference between both in the illustration down below, which shows the results of a shooting arcade. 

Team A’s shots are accurate and close to a perfect pattern, while the other three teams are inaccurate in different ways. For Team B is biased, the shots are systematically off target, and Team C is noisy, meaning the shots are widely scattered. And, finally, team D is both noisy and biased; the shots are widely scattered and systematically off target. 

Bias and noise are therefore independent components of error. One refers to a systematic deviation (bias), while the other to random scatter (noise). 

Performing Noise Audits 

The key feature of noise is that you can recognise and measure it.

Measuring noisy judgements can help you understand the extent and source of the variability in your data and identify ways to reduce it, such as improving employee training and de-biasing approaches, or incorporating decision hygiene.

So, by performing noise audits, you can increase the accuracy and reliability of your decisions, leading to better outcomes and improved performance. This can be especially valuable in complex decision-making environments, where noise and biases can have significant impacts on the quality of decisions.

Here’s a ten-step guide on how to run a noise audit.

1. Recognise That Noise Is Real

To fight noise, you first have to admit that it exists – and it’s an issue that deserves attention, especially to detail. The noise audit, however, is only worthwhile if people in the company accept that there is inconsistency in the department’ or unit’s decisions.

2. Build a Relevant Cast

Assemble a project team that should include consultants (internal or external), subject-matter experts, judges, and a high-level manager as a project manager.

Consultants examine the quality of employees’ professional judgements as well as administer the collection of data, analyse the results, and prepare a final report.

Subject-matter experts develop the cases that the judges will assess.

Judges, or professionals, make similar judgements and decisions on behalf of the company.

Finally, a project manager provides administrative support to facilitate all phases of the project, including the preparation of the final report and the communication of its conclusions to the leadership of the company. 

3. Identify One or More Units to Be Audited and Select Judges

The focus is to find and measure noise as well as stimulate changes in the unit’s operations. This includes revising the guidelines that shape professional judgements, improving training, tools, and regularly supervising their work.

4. Call It a “Decision-Making Study”

Inform audited unit managers early on that their unit was picked for a special study. Avoid terms like “noise audit” and words like “noise” or “noisy” when referring to people. Use a neutral term like “decision-making study” instead.

For instance, explain the purpose of the exercise in general terms, such as, “The organisation is curious about the decision-making process of [decision makers].” The unit managers should be in charge of data collection and brief participants on the task, with the participation of the project manager and members of the project team.

5. Choose a Judgement Task

It’s best to choose a judgement task that meets two criteria. First, it can be completed using only written information. Secondly, it can be expressed numerically (e.g., in dollars, probabilities, or ratings).

6. Develop Case Materials

Task subject-matter experts with creating the cases for the audit. They can use either brief schematic lists or detailed and realistic summaries of complex cases. They need to also prepare a questionnaire for each case to understand the reasoning behind each judgement. It should be administered only after the completion of all cases and include:

• Open questions about the key factors that led the participant to their response;
• A list of the facts of the case, allowing the participant to rate their importance;
• Questions that call for an “outside view”. For example, using past similar scenarios to forecast outcomes instead of relying on your understanding of the process.

7. Set a Deadline

Define a time frame for case materials development. Let subject-matter experts schedule a meeting and present the design of the noise audit to the executives. 

8. Anonymise the Data

It is essential to reassure the participants that their individual answers will be kept confidential and not disclosed to anyone in the organisation, including the project team.

9. Separate the Participants

Coordinate the exercise, so everyone completes it at the same time, but with physical separation and no talking while the study is in progress.

10. Let the Project Team Do the Math

The project team will perform statistical analyses, measure noise, and identify biases if possible. They will also study participants’ reasoning to understand inconsistency in decisions and look for patterns in extreme responses related to training, organisational procedures, and information provided to employees.

During a noise audit, individuals judge the same problems. Variability may result from incompetence; some judges know what they are talking about, others don’t. If there is a skill gap, it’s important to prioritise training to improve those skills.

On the other hand, even competent and well-trained professionals can still be noisy in their judgements, highlighting the importance of remaining vigilant and continuously refining your skills.

That’s a Wrap on Noise Audits

Cutting out the noise and overcoming your biases can be a huge drain. However, by incorporating regular noise audits, businesses can move to data-driven business decisions, thereby reduce hidden costs of inconsistent judgements.