machine learning bias (AI bias)

Machine learning bias, also sometimes called algorithm bias or AI bias, is a phenomenon that occurs when an algorithm produces results that are systemically prejudiced due to erroneous assumptions in the machine learning process.

Machine learning, a subset of artificial intelligence (AI), depends on the quality, objectivity and size of training data used to teach it. Faulty, poor or incomplete data will result in inaccurate predictions, reflecting the "garbage in, garbage out" admonishment used in computer science to convey the concept that the quality of the output is determined by the quality of the input.

Machine learning bias generally stems from problems introduced by the individuals who design and/or train the machine learning systems. These individuals could either create algorithms that reflect unintended cognitive biases or real-life prejudices. Or the individuals could introduce biases because they use incomplete, faulty or prejudicial data sets to train and/or validate the machine learning systems.

Types of cognitive bias that can inadvertently affect algorithms are stereotyping, bandwagon effect, priming, selective perception and confirmation bias.

Although these biases are often unintentional, the consequences of their presence in machine learning systems can be significant. Depending on how the machine learning systems are used, such biases could result in lower customer service experiences, reduced sales and revenue, unfair or possibly illegal actions, and potentially dangerous conditions.

To prevent such scenarios, organizations should check the data being used to train machine learning models for lack of comprehensiveness and cognitive bias. The data should be representative of different races, genders, backgrounds and cultures that could be adversely affected. Data scientists developing the algorithms should shape data samples in a way that minimizes algorithmic and other types of machine learning bias, and decision-makers should evaluate when it is appropriate, or inappropriate, to apply machine learning technology.

How AI amplifies data bias
Gender bias in a machine learning role labeling system

Types of machine learning bias

There are various ways that bias can be brought into a machine learning system. Common scenarios, or types of bias, include the following:

  • Algorithm bias. This occurs when there's a problem within the algorithm that performs the calculations that power the machine learning computations.
  • Sample bias. This happens when there's a problem with the data used to train the machine learning model. In this type of bias, the data used is either not large enough or representative enough to teach the system. For example, using training data that features only female teachers will train the system to conclude that all teachers are female.
  • Prejudice bias. In this case, the data used to train the system reflects existing prejudices, stereotypes and/or faulty societal assumptions, thereby introducing those same real-world biases into the machine learning itself. For example, using data about medical professionals that includes only female nurses and male doctors would thereby perpetuate a real-world gender stereotype about healthcare workers in the computer system.
  • Measurement bias. As the name suggests, this bias arises due to underlying problems with the accuracy of the data and how it was measured or assessed. Using pictures of happy workers to train a system meant to assess a workplace environment could be biased if the workers in the pictures knew they were being measured for happiness; a system being trained to precisely assess weight will be biased if the weights contained in the training data were consistently rounded up.
  • Exclusion bias. This happens when an important data point is left out of the data being used --something that can happen if the modelers don't recognize the data point as consequential.

Bias vs. variance

Data scientists and others involved in building, training and using machine learning models must consider not just bias, but also variance when seeking to create systems that can deliver consistently accurate results.

Like bias, variance is an error that results when the machine learning produces the wrong assumptions based on the training data. Unlike bias, variance is a reaction to real and legitimate fluctuations in the data sets. These fluctuations, or noise, however, should not have an impact on the intended model, yet the system is using that noise for modeling. In other words, variance is a problematic sensitivity to small fluctuations in the training set, which, like bias, can produce inaccurate results.

Although bias and variance are different, they are interrelated in that a level of variance can help reduce bias. If the data population has enough variety in it, biases should be drowned out by the variance.

As such, the objective in machine learning is to have a tradeoff, or balance, between the two in order to develop a system that produces a minimal amount of errors.

How to prevent bias

Awareness and good governance can help prevent machine learning bias; an organization that recognizes the potential for bias can then implement and institute best practices to combat it that include the following steps:

  1. Select training data that is appropriately representative and large enough to counteract common types of machine learning bias, such as sample bias and prejudice bias.
  2. Test and validate to ensure the results of machine learning systems don't reflect bias due to algorithms or the data sets.
  3. Monitor machine learning systems as they perform their tasks to ensure biases don't creep in over time as the systems continue to learn as they work.
  4. Use additional resources, such as Google's What-if Tool or IBM's AI Fairness 360 Open Source Toolkit, to examine and inspect models.

History of machine learning bias

Machine learning bias has been a known risk for decades, yet it remains a complex problem that has been difficult to counteract.

In fact, machine learning bias has already been implicated in real-world cases, with some bias having significant and even life-altering consequences.

COMPAS is one such example. COMPAS, short for the Correctional Offender Management Profiling for Alternative Sanctions, used machine learning to predict the potential for recidivism among criminal defendants. Multiple states had rolled out the software in the early part of the 21st century before its bias against people of color was exposed and subsequently publicized in news articles.

Amazon, a hiring powerhouse whose recruiting policies shape those at other companies, in 2018, scrapped its recruiting algorithm after it found that it was identifying word patterns, rather than relevant skill sets, inadvertently penalizing resumes containing certain words, including women's -- a bias favored male candidates over women candidates by discounting women's resumes.

Meanwhile, that same year, academic researchers announced findings that commercial facial recognition AI systems contained gender and skin-type biases.

This was last updated in July 2020

Continue Reading About machine learning bias (AI bias)

Dig Deeper on Pattern recognition and machine learning