AI Hiring Bias: Where It Comes From and How to Audit for It

By Kevin Kim · Updated Oct 11, 2026

AI hiring bias is a systematic difference in how an automated tool treats candidates from different groups, caused by the data it learned from, the signals it uses, or the way it's deployed, rather than by anything the candidates did. It rarely looks like prejudice. It looks like a ranking, a score or a knockout rule that happens to pass men more often than women, or younger applicants more often than older ones, and nobody notices because nobody counted.

This page is for HR directors, talent acquisition leads and in-house counsel who already use, or are about to buy, screening or assessment tools. It explains where bias enters, how adverse impact is measured, and how to run an audit you can defend. For the general concept, see bias in AI. For screening specifically, see AI resume screening. This is general information, not legal advice.

Key Facts

• Amazon's experimental recruiting engine rated candidates from one to five stars, learned from resumes submitted over a 10-year period (most from men), and penalized resumes containing the word "women's"; the team was disbanded (Reuters, October 2018).
• Under the federal Uniform Guidelines, a selection rate below four-fifths (80%) of the highest group's rate is generally regarded as evidence of adverse impact, though smaller differences can still count (29 CFR 1607.4).
• iTutorGroup agreed to pay $365,000 to settle an EEOC suit alleging its application software automatically rejected women aged 55 or older and men aged 60 or older, affecting more than 200 applicants (Duane Morris, August 2023).
• In a University of Washington test of three language models, white-associated names were preferred 85% of the time versus 9% for Black-associated names, and Black male-associated names were never preferred over white male-associated names (University of Washington, October 2024).
• New York City's bias-audit rule requires an impact ratio for each sex, race/ethnicity and intersectional category, and bars use of a tool more than a year after its last audit (NYC DCWP adopted rule).

Where Bias in Hiring AI Comes From

Bias has more than one entrance. Fixing one doesn't close the others, which is the main lesson of the documented cases below.

Two points trip up many teams. First, removing the protected attribute doesn't remove the bias, because the model can rebuild it from correlated signals. Amazon edited its programs to ignore specific gendered terms, but people familiar with the effort told Reuters that gave no guarantee the system wouldn't find other discriminatory ways to sort candidates (Reuters). Second, a rules-based tool can be just as biased as a learned one. The iTutorGroup software wasn't a mystery model; the EEOC alleged it was programmed with age cutoffs (Duane Morris).

Documented Cases Worth Knowing

Amazon's scrapped recruiting model (2018)

Reuters reported that Amazon's team had been building resume-review programs since 2014 and found by 2015 that the system wasn't rating candidates for software developer and other technical roles in a gender-neutral way. The models were trained on resumes submitted over 10 years, most from men. They penalized resumes containing the word "women's," and Amazon later disbanded the team (Reuters). Recruiters looked at the tool's recommendations but, per the report, never relied solely on its rankings. The case is the standard example of label and training-data bias.

EEOC and iTutorGroup (2023)

The EEOC alleged that iTutorGroup programmed its tutor-application software to reject female applicants over 55 and male applicants over 60, failing to hire more than 200 qualified applicants aged 55 and older in early 2020. The consent decree, filed on 9 August 2023, provided $365,000 to affected applicants, split evenly between compensatory damages and backpay, and ran for five years with training, anti-discrimination policies and EEOC monitoring (Duane Morris). This one was intentional disparate treatment coded into software, which is a different failure from learned bias, and a reminder that audits should also read the rules.

Mobley v. Workday (pending)

Derek Mobley alleges that he applied to more than 100 jobs through employers using Workday's platform and was rejected every time, and that Workday's AI tools disadvantage older applicants. On 16 May 2025, Judge Rita Lin of the Northern District of California granted preliminary certification of a collective under the Age Discrimination in Employment Act, covering applicants aged 40 and older denied employment recommendations through the platform since 24 September 2020; the court's common question was whether the recommendation system has a disparate impact on applicants over forty (Holland & Knight, Fisher Phillips). Workday says the suit lacks merit and that certification is only preliminary. As of a mid-2026 employment-law update, the case was still ongoing, with claims extended to race and disability, some state-law claims surviving a motion to dismiss in late June 2026, and Workday seeking an interlocutory appeal of the certification ruling (employment law update, Spring 2026). Nothing has been decided on the merits, so treat it as a signal about legal theory, not a finding of bias. The theory itself matters to buyers: it targets the software vendor as well as the employers who use it.

Name bias in language models (2024)

University of Washington researchers varied only the first names (120 names associated with white and Black men and women) on over 550 real resumes ranked against more than 500 job listings across nine occupations, using three language models and producing over three million comparisons. White-associated names were preferred 85% of the time and Black-associated names 9%; male-associated names 52% versus 11% for female-associated names; and Black male-associated names were never preferred over white male-associated names (University of Washington). This was a controlled research test of general-purpose models, not an audit of any commercial hiring product. It shows why you test your own configuration, and why the researchers' point that bias "wasn't necessarily visible from just looking at race or gender in isolation" argues for intersectional checks.

Adverse Impact and the Four-Fifths Rule

Adverse impact means a selection procedure advances one group at a meaningfully lower rate than another, whether or not anyone intended it. The federal Uniform Guidelines treat a selection rate for any group that is less than four-fifths (80%) of the rate for the highest-rate group as generally evidence of adverse impact, and a rate above that as generally not. They also say that smaller differences may still be adverse impact when statistically and practically significant, and that larger gaps resting on small numbers may not count. Employers are expected to keep records showing the impact of their procedures by sex and by race and ethnic group (29 CFR 1607.4).

A worked illustration with made-up numbers:

Group C's ratio of 0.70 falls below 0.80, which is your cue to investigate. Group B sits exactly on the line, which proves why the threshold is a screening signal rather than a verdict. With small groups, a handful of hires moves the ratio a lot, so pair the ratio with a significance test and look at several months of data.

How to Audit a Hiring Tool

New York City's rule gives the clearest published template, and it's useful even if you're not covered by it. It requires the audit to calculate the selection rate and the impact ratio for each category, defined as a category's selection rate divided by that of the most selected category. For tools that produce scores, it uses a scoring rate (the share scoring above the sample's median) in the same way. Calculations must be run separately for sex categories, race/ethnicity categories, and intersectional categories of sex, ethnicity and race, and the audit must report how many people fell into an unknown category. An independent auditor may leave out a category under 2% of the data if the summary explains why and gives that category's numbers (NYC DCWP adopted rule).

A step-by-step routine

1. List every automated step. Parsing, ranking, knockout questions, assessments, chat screeners, interview scoring. Bias can sit in any of them. See AI pre-employment assessments and AI in interviews for the stages beyond resumes.
2. Define the decision at each step. "Advance," "reject," "score above median." You can only measure what you've defined.
3. Collect outcome data by group. You need applicant counts and advancement counts by sex, race and ethnicity, and age band, with an "unknown" bucket. Clean inputs matter here, so start with HR data foundations for AI.
4. Compute impact ratios at each stage, not just at hire. A fair final offer rate can hide a skewed screen early in the funnel.
5. Test intersections. Men versus women and Black versus white candidates can both look acceptable while Black women, or older women, don't.
6. Test with matched inputs. Submit identical resumes that differ only in name, pronouns, graduation year or address, and see whether the output changes. This catches proxy effects that aggregate numbers can miss.
7. Check each input for relevance. For every feature, document why it's job-related and necessary, and drop the ones that are really proxies.
8. Review rejections, not just selections. Sample the low-ranked pile every week, with a person who can overrule the tool (human in the loop).
9. Decide who owns the result. HR runs the numbers, legal interprets them, and a named leader signs off on any remediation.
10. Repeat on a schedule and after any change. New criteria, new job families or a vendor model update all reset the clock.

What to do when a ratio fails

Don't just recalibrate the output to hit 80%. Find the cause first. Trace the gap to a specific input, label or step, then consider less discriminatory alternatives that serve the same business need, such as dropping a proxy feature, replacing a degree requirement with a skills test, or turning a knockout into a human review. Document what you found and what you changed. For the broader operating routine, see human oversight in HR AI.

The Legal Context in Brief

Rules differ by place and are changing, so confirm current requirements with counsel. For a fuller survey, see AI hiring laws.

• New York City. Local Law 144 bars use of an automated employment decision tool without a bias audit within the prior year and a public summary, and the Department of Consumer and Worker Protection began enforcing it on 5 July 2023 (NYC DCWP). The rule sets no pass/fail threshold; it requires the ratios to be calculated and published.
• Illinois. HB 3773, signed on 9 August 2024 and effective 1 January 2026, amends the Illinois Human Rights Act to make it unlawful to use AI that has the effect of discriminating against protected classes or to use zip codes as a proxy for them, and requires notice when AI is used for employment decisions. The Illinois Department of Human Rights is to set the notice rules (DLA Piper).
• Federal enforcement posture. Executive Order 14281, signed on 23 April 2025, states a policy of eliminating disparate-impact liability "to the maximum degree possible" and tells agencies to deprioritize enforcement of related statutes and regulations (Compilation of Presidential Documents, GovInfo). An order doesn't change the statutes or the case law, and private plaintiffs and state law remain available, which is exactly the route Mobley takes.

Common Pitfalls

• Auditing only the final hire. The skew often sits at the first filter.
• Treating the four-fifths ratio as a safe harbor. Passing 0.80 on small numbers proves little; failing it is a reason to look, not proof of illegality.
• Relying on the vendor's audit alone. It covers the vendor's data, not your job families or applicant pool.
• Removing one proxy and calling it fixed. Correlated signals fill the gap.
• Collecting no demographic data. Without it, you can't measure anything, and the Uniform Guidelines expect records by group. Handle it carefully and separate it from the hiring decision.
• Letting rankings become decisions. Reviewers read the top of a list and stop.
• Skipping re-tests. A tool that passed at launch can drift as criteria and applicant pools change.