Bias and Discrimination

This is the most likely way an AI system damages your organisation, and it does not require anyone to intend harm.

How bias gets in

Through the training data. A model learns the patterns in its examples. If historical decisions were skewed, it learns the skew — faithfully, and at scale.

Through who is represented. If a group is thin in the training data, the model performs worse for that group. Not through malice; there was simply less to learn from.

Through the choice of label. What you decide to predict encodes a judgement. Predicting "who will be a good employee" using past promotions as the label bakes in every bias in how promotions were awarded.

Through deployment context. A model built for one population applied to another will misfire, even if it was fair where it was built.

Two documented cases

Amazon's recruiting tool. Reuters reported in October 2018 that Amazon had built an experimental system, from 2014, to rank job candidates from one to five stars. It was trained on résumés submitted over the preceding decade — a period in which the applicants for technical roles were predominantly men. The model learned that pattern. According to the reporting, it penalised résumés containing the word "women's" — as in "women's chess club captain" — and downgraded graduates of two women-only colleges. Evidence of the bias appeared as early as 2015. Amazon eventually abandoned the project, having concluded it could not guarantee the system would be neutral.

Two things are worth drawing out. First, nobody set out to build a discriminatory system. The bias came entirely from the historical data. Second — and this is the part that should worry you most — the model was never told an applicant's gender. It inferred proxies for it from language, and penalised those.

The Gender Shades study. Joy Buolamwini and Timnit Gebru, in work published in 2018, evaluated three commercial gender-classification systems across a dataset balanced by skin tone and gender. The results were stark: error rates reached up to 34.7% for darker-skinned women, against a maximum of 0.8% for lighter-skinned men.

The likely reason lies in the data. The researchers found that the standard benchmark datasets the industry had been measuring itself against were overwhelmingly composed of lighter-skinned subjects — 79.6% in one and 86.2% in the other. Systems were being optimised and reported as accurate against benchmarks that barely contained the people they failed on.

The transferable lesson is uncomfortable and general: an aggregate accuracy number can conceal near-total failure for a subgroup. A system reported as "93% accurate" may be 99% accurate for the majority and far worse for a minority, and the headline figure will never reveal it.

Why deleting the sensitive field does not work

The instinctive fix — remove gender, ethnicity, age from the data — is insufficient, and Amazon's case shows why. Models find proxies.

Postcode correlates with ethnicity and income. First names correlate with gender and ethnic background. The particular sports, schools and societies on a CV correlate with class and gender. Gaps in employment history correlate with parenthood and disability. A model optimising for accuracy will use whatever helps it predict, including these, without ever representing the protected attribute explicitly.

This is why fairness cannot be handled by hiding a column. It has to be measured.

What a non-technical manager can actually do

Five things, none requiring technical skill, all of which materially reduce risk:

Ask for accuracy broken down by group, not just overall. This single question would have caught the Gender Shades failures.
Ask what the label actually encodes. If the model predicts past human decisions, whose decisions were they, and were they fair?
Ask who is thin in the training data. Which groups have too few examples for the model to have learned them well?
Insist on a route to challenge a decision. For any consequential automated decision, a person must be able to ask why and have it reviewed. This is increasingly a legal requirement as well as good practice.
Include the affected people when defining the problem. Most discriminatory outcomes are visible immediately to someone in the affected group and invisible to everyone else.

There is a compounding version of this problem that catches organisations that thought they had handled it.

Suppose a lending model is slightly less likely to approve applicants from a particular area, because historical data from that area contains more defaults — itself possibly the result of earlier discriminatory lending.

Fewer loans are made there. Fewer loans mean fewer repayment records from that area. When the model is retrained a year later, that population is even thinner in the data, so the model is even less confident about them, so it approves even fewer.

The bias deepens with each retraining cycle, and it looks like the model getting better, because it is measured on the population it is now serving — on which its accuracy has genuinely improved.

This dynamic appears wherever a model's decisions determine what data it later learns from: hiring, lending, policing, content moderation, fraud review. Whenever the system's own actions shape the next dataset, small initial asymmetries become large ones.

The mitigations are specific. Keep measuring on a fixed, representative evaluation set rather than only on incoming live data. Deliberately retain some decisions made outside the model's recommendation, so you keep learning about the cases it would have refused. And check subgroup performance across retraining cycles, not just at launch — because a system that was fair when it shipped can drift into unfairness with nobody making a decision to allow it.

❓ Knowledge Check

An HR team proposes removing gender, age and ethnicity from a CV-screening model's inputs so that it cannot discriminate. Why is this insufficient?

📚 Flashcards1 / 6

Term

Four routes for bias

Click to flip

Definition

Skewed historical decisions in the training data; groups thinly represented; the choice of label encoding a judgement; and deployment on a population different from the one the model was built for.

Click to flip back

💡Key Takeaway

Bias enters through data, representation, the choice of label and the deployment context — no intent required, as Amazon's abandoned recruiting tool shows. Deleting protected fields does not help, because models find proxies. What helps is measurement: demand accuracy broken down by group, ask what the label really encodes, find out who is thin in the data, guarantee a route to challenge a decision, and involve affected people when the problem is defined.