How AI Is Changing Work: What the Evidence Actually Shows

By Victor Hoang · Updated Sep 22, 2026

AI is measurably changing specific tasks inside specific jobs, not entire jobs wholesale. Controlled studies show real productivity gains in narrow settings (customer support, coding, writing) that vary by skill level and sometimes contradict each other. Enterprise adoption is high; measured financial impact is not. Longer-range claims about job loss or task automation are forecasts from named researchers using stated methods, not observed facts, and should be read that way.

Most writing about AI and work falls into one of two failure modes: treating a vendor demo as proof of a labor-market shift, or treating a consulting firm's forecast as if it already happened. This article separates what has actually been measured, in controlled studies with published methods, from what people are predicting, and names who is doing the predicting and how.

Key Facts

• A Stanford-led study of 5,179 customer support agents found generative AI assistance raised productivity by 14% on average, with a 34% gain for novice agents and almost no effect for experienced ones. NBER working paper 31161
• A randomized controlled trial found GitHub Copilot users completed a coding task 55.8% faster than a control group. arXiv 2302.06590
• A 2025 METR randomized trial of experienced open-source developers found AI coding tools made them 19% slower on real tasks, even though the developers believed they were 20% faster. METR, July 2025
• A Science study of 453 professionals found ChatGPT cut writing task time by 40% and raised quality by 18%, with the largest gains for lower-performing writers. Science, DOI 10.1126/science.adh2586
• McKinsey's 2025 State of AI survey found 88% of organizations use AI in at least one function, but only about 6% report using it in ways that generate significant enterprise-wide value. McKinsey, State of AI
• MIT's Project NANDA found 95% of generative AI pilots at the companies it studied showed no measurable profit-and-loss impact, despite tens of billions in enterprise spending. Fortune, August 2025
• The World Economic Forum's Future of Jobs Report 2025 forecasts 170 million jobs created and 92 million displaced by 2030, a net gain of 78 million, based on employer surveys. WEF, January 2025
• Gartner forecasts that more than 40% of agentic AI projects will be canceled by the end of 2027 over cost, unclear value, and weak risk controls. Gartner via RCR Wireless

What "AI Changing Work" Actually Means

Strip away the branding and the claim underneath most AI-and-work content is simple: generative AI that can draft, summarize, code, and answer questions is now cheap enough and good enough that people use it inside real jobs, not just in demos. That has three separate, testable layers, and conflating them is where most bad analysis starts.

The first layer is task-level effect: does a specific task, done with AI help, get faster, better, or cheaper? This layer has the most rigorous evidence, because it is small enough to run a randomized trial on. The second layer is job-level effect: does a whole role change shape, shrink, or disappear? This is harder to measure directly and mostly gets inferred from hiring data and job postings. The third layer is economy-level effect: aggregate employment, wages, productivity. This is the layer forecasts operate on, and it has the least direct evidence, because the shift is still early and the data lags by years.

Keeping these layers separate matters because a true statement at one (AI made this coding task 55% faster in a trial) gets stretched into a false-sounding claim at another (AI is replacing software engineers) without anyone showing how a task-level gain adds up to a job-level one. Economist David Autor's task-based labor models are the clearest guide here: jobs are bundles of tasks, and automating some tasks usually changes a job rather than eliminating it, at least in the short run.

The Measured Evidence: What Controlled Studies Found

Customer Support: The Stanford Call-Center Study

The most cited empirical study on generative AI and frontline work comes from Erik Brynjolfsson, Danielle Li, and Lindsey Raymond, published as an NBER working paper and associated with Stanford's Digital Economy Lab. The researchers tracked the staggered rollout of a generative AI assistant to 5,179 customer support agents at a software firm, giving them real-time suggested responses during live chats.

The headline number is a 14% average increase in issues resolved per hour after agents got access to the tool. But the average hides the more interesting finding: novice and lower-skilled agents saw a 34% productivity gain, while the most experienced agents saw almost no measurable benefit. Agents followed the tool's suggestions only about 38% of the time on average, a human-in-the-loop pattern where the tool worked less like an autopilot and more like an always-available coach the agent could choose to trust or override. The study also found improved customer sentiment and higher agent retention, and the authors argue the mechanism is that the AI model distilled the conversational patterns of the company's best agents and made them available to everyone else in real time, a version of the trust calibration every human-plus-AI workflow eventually has to solve.

That point is worth sitting with. This was not a study of AI replacing agents. It was a study of AI compressing the gap between a company's best and worst performers by transferring tacit knowledge that used to require years on the job or a good mentor.

Coding: Two Trials, Two Different Years, Two Different Answers

The coding evidence is where "AI changing work" gets genuinely contested: two credible RCTs point in opposite directions, and the difference is informative rather than a contradiction to wave away.

Sida Peng and coauthors ran an RCT with developers asked to build an HTTP server in JavaScript as fast as possible. The treatment group, using GitHub Copilot, finished 55.8% faster than the control group, with a 95% confidence interval between 21% and 89%. Less experienced developers and those coding more hours per day benefited most, the same skew toward less-experienced workers seen in the customer support study.

Then in mid-2025, the nonprofit research group METR ran a different kind of RCT: not a fresh task built to isolate one variable, but real work by experienced open-source developers on their own codebases, using whichever large language model-based tools they chose themselves. The result: developers were 19% slower with AI assistance, not faster, despite predicting beforehand that AI would speed them up by 24% and believing afterward that it had sped them up by 20%. The gap between belief and measured reality was the study's most striking finding.

These are not the same experiment, and that is the point. Peng's trial measured a scoped, unfamiliar task where AI compensates for missing background knowledge. METR's trial measured experienced developers doing familiar, high-context work, where reviewing and correcting AI suggestions can cost more time than writing the code directly. Read together, the studies suggest AI coding help works best when it substitutes for knowledge the developer lacks, and can slow experts down when it substitutes for knowledge they already hold. That is a more useful, falsifiable claim than either "AI makes developers faster" or "AI makes developers slower."

Writing and Knowledge Work: The Noy and Zhang Experiment

Shakked Noy and Whitney Zhang ran a randomized experiment with 453 college-educated professionals doing occupation-specific writing tasks, half with access to ChatGPT, published in Science in 2023. Time spent on tasks fell by about 40%, and independent grader ratings of output quality rose by roughly 18%. The productivity gains were sharply skewed toward workers who scored lowest on the task without AI help, meaning the tool narrowed the performance gap between strong and weak writers rather than widening it.

The study also documented something forecasts rarely mention: what ChatGPT actually replaced was worker effort on rough drafting, not worker judgment. Task time shifted toward idea generation, structuring, and editing, and away from the blank-page first draft. Workers also reported higher job satisfaction alongside more concern about automation, which is a useful reminder that productivity gains and worker anxiety are not mutually exclusive reactions to the same tool.

Measured Findings at a Glance

Adoption Is High. Measured Impact Is Uneven. That Gap Has a Name.

If task-level studies show something real but narrow, enterprise surveys show something broad but murky, and the two don't match up cleanly yet.

McKinsey: Adoption Without Transformation

McKinsey's 2025 State of AI research found 88% of organizations now use AI regularly in at least one business function, up from 78% the year before. But the same report found only about a third of companies have moved past scattered pilots into scaled use, and roughly 6% qualify as genuine high performers capturing significant enterprise value from AI. Separately, 47% of organizations reported at least one negative consequence from generative AI use, whether an accuracy failure, a compliance issue, or a workflow breakdown, the kind of gap that AI governance programs exist to close. Adoption, in other words, has outrun both oversight and measurable payoff for most companies, even as it becomes close to universal at the "we use this somewhere" level.

MIT NANDA and the GenAI Divide

The starkest version of this gap comes from MIT's Project NANDA, whose July 2025 report reviewed more than 300 public AI deployments, 52 structured interviews, and 153 leader surveys. The headline finding: despite an estimated $30 to $40 billion in enterprise generative AI spending, 95% of pilots produced no measurable profit-and-loss impact. The more useful finding sits underneath that headline. Buying AI tools from specialized vendors succeeded roughly twice as often as building custom AI internally, and the biggest realized returns showed up in unglamorous back-office automation, not the sales and marketing use cases that got most of the budget. The researchers frame the core obstacle as a "learning gap": tools that don't adapt to a specific workflow's context rarely stick, regardless of the underlying model's capability.

Read together, McKinsey and MIT NANDA describe the same phenomenon from different angles: broad, shallow adoption that hasn't yet become deep, measured transformation. That is not evidence AI doesn't work; the task-level studies above show it clearly can. It is evidence most companies haven't done the integration work to convert a capable tool into a measured result, the shortcut-taking that AI cultural debt describes, and it tracks with how AI-native culture work frames owning a tool versus running it well.

What Forecasts Claim, Who's Making Them, and How

Everything above is measurement: something happened, researchers recorded it, and the method is published and checkable. Forecasts are a different category entirely, and the two most-cited ones in future-of-work writing deserve to be named as claims with authors and methods rather than repeated as settled fact.

WEF Future of Jobs: A Survey of Employer Expectations

The World Economic Forum's Future of Jobs Report 2025 projects 170 million new jobs and 92 million displaced roles by 2030, a net gain of 78 million, with job "disruption" (creation plus displacement combined) touching roughly 22% of today's jobs. This is built from a large employer survey, not observed employment data, because 2030 hasn't happened yet. It is a genuinely useful signal of what employers expect and are already planning around. It is not a measurement of what AI has done to employment, and treating a forward-looking survey as a historical result is the single most common error in future-of-work coverage.

Gartner: Skepticism About Agentic AI at Scale

Gartner's own analysts, meanwhile, are the source of one of the more skeptical widely cited forecasts: that more than 40% of agentic AI projects will be canceled before the end of 2027, driven by rising costs, unclear business value, and weak risk controls. Gartner's Anushree Verma has pointed to "agent washing," where existing chatbots and automation tools get relabeled as autonomous agents without new capability behind the label, as a driver of inflated expectations that later collapse on contact with production requirements. A forecast predicting failure deserves the same scrutiny as one predicting success: it is an informed estimate from analysts who track vendor claims closely, not an outcome anyone has observed yet.

What the Evidence Does Not Yet Show

Being precise about what we don't know matters as much as citing what we do. The research above does not establish:

• Whether task-level productivity gains translate into net job losses, net job gains, or mostly a reshuffling of what existing workers spend their time on. The WEF numbers are survey-based projections, not observed outcomes.
• Whether the productivity gains in narrow controlled studies hold up over a full workweek, a full year, or across less-scripted work than customer support chats or timed coding tasks.
• Why AI helps novices more than experts in some settings (support, writing) but appears to slow down experts in others (the METR coding study), beyond the plausible but unproven theory that review overhead matters more when a person already has the expertise the AI is trying to supply.
• Whether the current gap between high AI adoption and low measured enterprise value is a temporary integration lag, as MIT NANDA's framing suggests, or a more durable ceiling on what these tools can add without deeper process redesign.

None of that uncertainty is a reason to wait and see. It's a reason to treat this like any unsettled empirical question: watch the studies, not the vendor decks, and update your plan as better evidence arrives rather than locking in on this month's headline number.

Putting the Evidence to Work

The practical takeaway for anyone planning headcount, tooling, or reskilling around AI is to sort every claim you hear into one of two buckets before acting on it. Bucket one: a measured result, with a named study, a stated sample, and a method you could go read. Bucket two: a forecast, with a named forecaster and a stated method for how they got there. Confusing the two is where plans go wrong, whether that means over-investing based on a survey of intentions or under-investing based on one contested coding study. The organizations doing well right now, per both McKinsey and MIT NANDA, treat AI adoption as an ongoing measurement problem rather than a one-time rollout, the same discipline behind building an AI-ready culture and human-agent teams that function day to day rather than just look good in a slide.