AI Bias Examples: 3 Real Cases and How to Test for Bias

The clearest AI bias examples share one pattern: a model learned from data that did not look like the people it was later used on, and it made more mistakes for some groups than for others. Nobody had to intend it. The skew was in the data, the labels or the goal, and the model copied it faithfully. If you build an AI product, knowing these cases helps you spot the same pattern in your own feature before your users do.

The How Modern AI Actually Works course on Age of AI Entrepreneur ends with a lesson on what AI gets wrong, with its sources listed; sign in with your email and an emailed code to open it. This post walks through three well-documented cases, the debate over what "fair" even means, and a test you can run this week.

What AI bias means

AI bias is a pattern of errors or outcomes that falls unevenly on a group of people. A model can be accurate on average and still be far worse for one group. That is why a single accuracy number can hide the problem.

Bias usually enters in one of four places:

• The data. Some groups are rare or missing in the training set, so the model has little to learn from.
• The labels. Past human decisions become the "right answers", and any unfairness in those decisions comes along.
• The proxy. The model predicts something easy to measure that stands in for what you care about, and the two differ by group.
• The feedback loop. The model's own output shapes the next round of data, and a small skew grows.

AI bias examples from real systems

Here are three cases that researchers and journalists documented in detail. Each one shows a different way bias gets in.

1. Face analysis that worked best on lighter-skinned men

In the 2018 Gender Shades study, Joy Buolamwini and Timnit Gebru looked at two face benchmarks used to test these systems. Lighter-skinned people made up 79.6% of one and 86.2% of the other. They then built a dataset balanced by gender and skin type and tested three commercial gender classification systems on it. Darker-skinned women were the most misclassified group, with error rates of up to 34.7%. The highest error rate for lighter-skinned men was 0.8%.

The Gender Shades numbers, from the study's abstract.

The lesson for builders: the test set had the same skew as the training data, so the problem did not show up until someone tested on a balanced set.

2. A hiring tool that learned to penalise "women's"

Reuters reported in 2018 that Amazon had dropped an experimental tool for screening job applications after finding it was biased against women. It had learned from past hiring, which was mostly men, and it marked down resumes that included the word "women's" and applicants from all-women's colleges. The labels, not the code, carried the bias.

3. Risk scores and predictive policing

COMPAS is a commercial program used by US courts to estimate how likely a defendant is to reoffend. ProPublica's analysis claimed that black defendants were twice as likely as white defendants to be wrongly labelled high-risk. Whether that makes the scores unfair depends on which definition of fair you use, which leads straight into the debate below. In a separate case, a simulation of predictive policing software showed a feedback loop: more patrols led to more reported crime in the same places, which led to even more patrols.

Why fairness is hard to define

You might expect a simple fix: make the model "fair". The trouble is that there are several reasonable definitions, and they can conflict.

One view says a score should mean the same thing for every group: a risk score of 7 should carry the same real risk whoever it is given to. Another view says the mistakes should be balanced: each group should face the same rate of being wrongly flagged. A 2016 paper by Jon Kleinberg, Sendhil Mullainathan and Manish Raghavan proved that, except in narrow special cases, no method can meet three such conditions at once.

People who work on this disagree about what follows. Some argue you should pick the definition that protects the people most harmed by a wrong decision. Others argue the choice belongs to the people affected, or to law and policy, not to the engineer. Still others say some decisions should not be automated at all. A 2019 survey of the field lists many fairness definitions side by side. The course gives each side; this post does too, with no verdict.

Where bias hides in an AI product you build

Most founders today build on top of large language models rather than training face or risk models. The same four sources still apply:

• Your prompt examples may all use one kind of name, accent or writing style.
• Your eval set may only include the users you already have.
• A score your product shows (lead quality, resume fit, credit to extend) may stand in for something that differs by group.
• Your product's outputs may feed the next version's data, as in the policing loop.

Bias is close kin to the other failures the course covers, such as AI hallucination examples: in both cases the model gives a confident answer that does not hold for the case in front of it.

Worked example: test a resume screener for bias in an afternoon

Say you are building a tool that reads resumes and scores fit for a sales role from 1 to 10. Here is a paired test you can run before launch.

1. Write 20 base resumes. Mix strong, average and weak ones, all for the same role.
2. Make pairs. For each resume, make a copy that changes one thing that should not matter: the first name, a women's sports club, a college, a gap year for caregiving. Keep everything else identical.
3. Score all 40. Run each one three times, since model outputs can vary from run to run.
4. Compare each pair. Average the three scores and subtract. If the gap is more than the spread you see when you run the same resume twice, flag it.
5. Fix and rerun. Remove the field that drives the gap, change the instructions, or keep a person in the loop for that decision. Then run the same 40 again and keep them in your test set.

This does not prove your tool is fair; no single test can. It catches the obvious cases cheaply, and it gives you a record of what you checked. Add the pairs to the checks in testing LLM prompts before release, and if you grade outputs with a model, read LLM as a judge first, since a judge model can carry bias too.

Frequently asked questions

What is the most famous example of AI bias?

The Gender Shades study and the Amazon hiring tool are among the most cited. One showed face analysis erring far more on darker-skinned women; the other showed a screener learning to mark down resumes that mentioned "women's".

Can AI bias be removed completely?

Not in general. Research shows some fairness definitions cannot all be met at once, so teams choose which errors matter most for their use and test for them.

Do large language models have bias?

They learn from large amounts of human text, so they can repeat patterns in it. Test your own use with paired inputs that differ only in a detail that should not matter.

Is AI bias the same as hallucination?

No. A hallucination is a confident answer that is false. Bias is a pattern of errors or outcomes that falls unevenly on a group, even when each answer looks reasonable.

Get started

Open Age of AI Entrepreneur, sign in with your email and an emailed code, and take the How Modern AI Actually Works course: next-token prediction, how a raw model becomes a helpful assistant, and what AI gets wrong. Each lesson takes about nine minutes and lists its sources.