Why "Human in the Loop" So Often Becomes a Rubber Stamp

By Victor Hoang · Updated Oct 11, 2026

Most AI policies end the same way: a human will review it before it goes out. Boards like that sentence. Regulators like it. And it's often the weakest control in the building, because the question that matters isn't whether a person is in the loop. It's whether that person is still looking.

Our view is that oversight decays by default. When AI is right most of the time, the reviewer's job becomes confirming something that is almost always fine, and attention drifts away from a task that almost never rewards it. That's not laziness. It's how people behave around reliable automation, and a decades-old research literature says so. The fix isn't a stricter policy or a second checkbox. It's designing review the way you'd design any scarce resource: spend it where the risk is, sample the rest, and measure whether it works.

Key Facts

• A systematic review of 74 studies, mostly in healthcare, defined automation bias as the tendency to over-rely on automation. Across four studies in the meta-analysis, erroneous advice was more likely to be followed with decision support than without it (risk ratio 1.26).
• Parasuraman and Manzey's 2010 review in Human Factors found that complacency appears in novices and experts alike and isn't eliminated by practice, and that automation bias persists despite training or instructions.
• In a Harvard Business School field experiment with 181 professional recruiters screening 44 resumes each, those given a more accurate AI (about 85%) did worse than those given a weaker one (about 75%). The weaker tool kept them checking.
• Article 14 of the EU AI Act requires high-risk systems to be designed so people can remain aware of the possible tendency of automatically relying or over-relying on the output, which the Act itself names automation bias.
• In a 199-person experiment, interface designs that forced people to think before seeing the AI's answer reduced overreliance, but were rated least favorably.

Why a Good AI Makes a Worse Reviewer

The uncomfortable pattern in the research is that reviewer quality falls as system quality rises. That sounds backwards until you think about what a reviewer is rewarded for.

Dell'Acqua's recruiter experiment is the cleanest illustration. The recruiters were paid to be accurate, and they knew what kind of AI they'd get. The ones given the 85%-accurate tool still did worse than the ones given the 75% tool. The paper's account is that recruiters with the better AI moved faster, delegated more, and risked "falling asleep at the wheel", while those with the weaker tool stayed vigilant, spent more time on each application, and verified what the AI told them. One caution: this is a 2022 working paper, and it studied one hiring-style task, so read it as a mechanism rather than a benchmark.

Healthcare has the biggest evidence base. In the Goddard, Roudsari and Wyatt review of 74 studies, clinicians overrode their own correct decisions in favor of wrong computer advice in about 6% of cases in one study, and negative consultation rates ran from 6% to 11% across four. Those are clinical populations making clinical calls, not office workers approving invoices, and the figures are small on purpose. A 6% rate of a reviewer being talked out of a correct answer sounds survivable until you multiply it by volume. What transfers to the workplace is the direction and the moderators. Less experienced users showed more bias, though experienced users showed it too. Higher workload, time pressure and task complexity increased reliance. Higher self-confidence reduced it.

Parasuraman and Manzey add the attention piece. Their review separates two failure modes. Complacency is the attention problem: it shows up when manual tasks compete with the automated one for attention, which describes most knowledge workers' days. Automation bias is the decision problem, where imperfect aids cause both omission errors (missing what the system missed) and commission errors (following a wrong recommendation). The authors argue both rest on attention, and that neither goes away with practice or reminders.

Vigilance Is a Limited Resource

There's an older strand of research that explains the mechanics. Psychologists have long studied the vigilance decrement: people watching for rare events get worse at spotting them as time passes. One lab study of 12 young adults describes the decline as typically appearing after 20 to 30 minutes of continuous work, and sometimes within five. That was a laboratory monitoring task, so don't convert it into a rule for your finance team. But the shape matters: sustained, low-event-rate checking is exactly the job a "human in the loop" is handed.

Put the two together and you can predict where oversight fails in a company. It fails in queues. A reviewer who sees 200 AI-drafted items a day, 198 of which are fine, has been trained by the system to expect fine. The 199th gets the same half-second as the rest.

Why Policy Alone Doesn't Fix It

Ben Green's analysis of 41 policies requiring human oversight of government algorithms makes the structural point. His finding is that the policies assume people can perform the oversight function, while the evidence suggests they often can't. A required human can also make a questionable system look acceptable, which creates a false sense of security and moves accountability onto the reviewer. Green studied public-sector algorithms, so the setting differs from yours, but the failure mode is familiar to any company that has added an approver after an incident.

For a CEO or COO the implication is awkward. A rubber stamp isn't a neutral control. It's worse than no control in one respect: it lets the organization believe a risk is managed. When something slips through, the reviewer who clicked approve is blamed, and the design that made clicking approve the rational move stays in place.

Regulators have noticed. Article 14 of the EU AI Act requires that high-risk systems be built so people can effectively oversee them while in use, and it asks that those people stay aware of the over-reliance tendency. The Act is aimed at high-risk systems, so it won't apply to most internal workflows. But the drafting shows where the thinking has gone: oversight is a design requirement on the system and the interface, not just a duty on a person. If your approach to regulation changes is to keep controls that survive the next rule, designing for attention is a safer bet than adding sign-offs.

Design Review for Attention

If attention is the scarce resource, treat it like one. Four design moves follow from the research, and none of them is "tell people to be more careful."

Spend review by risk, not by volume. Reviewing everything equally guarantees that you review nothing well. Decide which outputs are high stakes and hard to reverse, and put real reviewers there. Our earlier piece on decisions you shouldn't delegate to AI covers how to draw that line. The governance gap article sets out the stakes and reversibility test. This piece is about what happens after you've drawn it.

Sample the low-risk middle instead of approving all of it. For routine, reversible output, a random sample reviewed properly beats a full queue reviewed by reflex. Sampling also gives you something an approval queue never does: an error rate you can state.

Seed the queue and measure catches. Plant known-bad items at a low, steady rate, such as a test invoice with a wrong total or a draft with a fabricated figure. If reviewers catch them, attention is alive. If they don't, you've learned the loop is decorative before a customer teaches you. This is the single most useful metric in the whole design, and almost nobody tracks it. Tell reviewers that seeding exists. The point is accountability, not entrapment, and the accountability literature points the same way: in the Goddard review, perceived accountability reduced errors.

Make the AI harder to defer to. The same review found that less prominent advice, per-item confidence information, and showing information rather than a recommendation were associated with less bias. Buçinca and colleagues tested what they call cognitive forcing functions: for example, making people commit to their own answer before seeing the AI's. In a 199-person experiment these reduced overreliance compared with simple explanations. The catch is in the same abstract: the designs that worked best were rated least favorably by participants. Expect pushback, and decide that some friction is the point.

A Review-Design Table by Risk Tier

Use this as a starting point. The sample rates are judgment calls to adjust against your own measured catch rates, not benchmarks from the studies above.

Two columns carry the weight. "Who reviews" is about authority: a reviewer who can't reject without a three-system detour will approve. "What you measure" is about evidence: an override rate near zero and a seeded-error catch rate near zero are the same warning from two directions.

A Worked Example (Illustrative)

Take a hypothetical 300-person services firm whose AI drafts quote documents for client approval. Everything below is invented to show the logic, not to benchmark anyone.

Today every quote gets a manager approval click, 120 a week. Managers approve 100% of them, usually in under a minute. Leadership counts that as full oversight.

The redesign has three parts. Quotes under a set value, using standard terms, get a random 15% sample reviewed in depth. Quotes above the value, or with any non-standard clause, go to a named reviewer who writes their own estimate of price and terms before opening the AI draft. And one quote in about forty each week is a seeded test with a deliberate error, which the reviewers know happens.

After a month, the firm learns something it couldn't see before: the seeded-error catch rate. Suppose it's 55%. That number is the real size of the oversight program, and it's well short of the 100% the old process implied. Now the firm can act on it: shorter queues, a better interface, a different reviewer. Whether the number goes up is the test of the design.

What to Do Next

For an executive who has promised a regulator, a board or a customer that "a human reviews everything," here is a sequence that fits a quarter.

1. List every place a human approval is required on AI output. Include the informal ones. Most companies find far more than they expected, and most were added after an incident or in a policy draft.
2. Pull the approval data. For each, find the approval rate and the time spent per item. A near-100% approval rate with seconds per item is a rubber stamp, whatever the policy says.
3. Tier each one with the table above. Move low-risk items to sampling. Be willing to remove approvals you can't defend. Fewer, real reviews beat many nominal ones.
4. Seed one queue. Start with a medium-risk workflow, introduce known errors at a low rate, and track the catch rate for a month. It is the cheapest honest measurement of oversight you can run.
5. Give reviewers authority and a way out. Make rejecting the easy path, make clear who owns the outcome, and cap queue length so vigilance isn't asked to last all day. If review time is bigger than expected, that's the verification tax, and it belongs in your ROI numbers.
6. Treat agents as a separate case. As systems act rather than recommend, review shifts from a decision to a gate. The distinction between copilots and agents matters here, as do the security questions leaders should ask before an agent can spend or send anything.
7. Report catch rates to the board, not just "reviewed." If you can say how often your reviewers catch a planted error, you can say something true about oversight. If you can only say that reviews happen, you're reporting activity.

None of this means humans should leave the loop. Where stakes are high, they shouldn't. The point is that a person in the loop is a design decision with a measurable result, and that result decays unless someone looks after it.

Learn More

• The Governance Gap: What Leaders Get Wrong About AI at Work
• The Verification Tax: Why AI Time Savings Shrink Once Someone Checks the Work
• AI Copilots vs. AI Agents: Understanding the Difference Matters
• Measuring AI ROI Beyond 'Time Saved'
• Which Business Decisions Should Never Be Delegated to AI?