How to Implement AI in Operations: From Pilot to Scaled Process

By Tara Minh · Updated Oct 11, 2026

Implementing AI in operations means taking one repeatable piece of work, measuring how it performs today, putting an AI step into it with a person still in the loop, proving the result against that measurement, and only then copying it to other teams and sites. The order matters more than the technology. Most failed rollouts skip the measuring, or copy a pilot to a second site before anyone has written down what made the first one work.

This guide is for operations managers, site leads and process owners who've been handed an AI initiative and need the practical sequence. It stays at the level of steps and handoffs. If you're looking for who should own AI across the whole operation and who decides what, that's in the COO's guide to AI. For the wider picture, start with AI in operations management.

Key Facts

• NIST's AI Risk Management Framework asks that the behavior of an AI system be monitored when it is in production (MEASURE 2.4), not only tested before launch (NIST AI 100-1).
• The same framework asks organizations to have mechanisms, with assigned responsibilities, to supersede, disengage or deactivate AI systems whose performance is inconsistent with their intended use (MANAGE 2.4) (NIST AI 100-1).
• It also asks that processes for human oversight be defined, assessed and documented (MAP 3.5) (NIST AI 100-1).
• NIST describes the framework as voluntary and use-case agnostic, so it works as a checklist for any size of operation (NIST AI RMF 1.0).
• The sequence that holds up in practice: one process, one baseline, one pilot site, one owner, one stopping rule. Then scale.

The Seven Steps at a Glance

Step 1: Choose the Process

The most common mistake is choosing the tool first. Start with the work.

Screen each candidate process on four questions.

• Volume. Does it happen often enough, in similar enough form, that a small gain adds up? Weekly or daily work qualifies. Annual work doesn't.
• Cost of a wrong answer. If the AI errs, is the damage small and reversible (a mis-tagged work order) or large and hard to undo (a wrong payment)? Start where errors are cheap.
• Data. Is the input captured, in one place, and trusted by the people who use it? If not, the first project is the data. See operational data for AI.
• Process clarity. Can two people on the team describe the process the same way? If not, AI will automate the disagreement.

Good first candidates are tasks where the AI drafts, sorts, flags or predicts, and a person confirms. Illustrative examples by function:

Actions that move money, change a customer's account or alter a production schedule on their own come later, after the AI has earned trust on the advisory work. The deeper treatment of how much authority to give is in AI-assisted decisions in operations.

Write down the candidates you rejected and why. When a vendor or a peer pushes a pet project six months from now, that list saves a debate.

Step 2: Baseline It

You can't prove improvement without a "before." Capture two to four weeks of current performance on the process you picked, using measures the team already trusts.

Pick one primary measure and two guardrails. For invoice matching, the primary measure might be touch time per invoice, with error rate and rework as guardrails. For work-order triage, it might be time from request to assignment, with misrouted orders as the guardrail. Our piece on process KPIs helps with choosing measures, and measuring AI in operations covers how to track them once AI is live.

Three baseline habits save a lot of argument later.

1. Measure actuals, not the SOP. The documented process and the real one differ. Time a sample of real cases.
2. Note the seasonality. A baseline taken in the slow week will flatter the pilot when volume spikes, or the reverse.
3. Agree the numbers in writing before the pilot starts. If the shift lead and the analyst disagree about the baseline in week eight, you've lost the thread.

If the data or the written process is too thin to baseline, that's a finding, not a failure. The right project is then documentation, and AI for SOPs and process documentation is a reasonable place to begin.

Step 3: Design the Pilot

A pilot is an experiment with a charter, not a trial period for a tool. A one-page charter covers:

• Scope. One process, one team, one site. Say what's excluded.
• Roles. A business owner in the operations line who answers for results, a process owner who maintains the procedure, a data owner for the inputs and a technical contact who keeps the system running. One person per role, by name.
• Human checkpoints. Where a person reviews, edits or approves. NIST's framework asks that processes for human oversight be defined and documented, and that reviewers have what they need to do the job (NIST AI 100-1). A reviewer with no authority to stop the process is decoration.
• Success criteria. The primary measure and the size of improvement that would justify scaling, set before you see results.
• Stopping rule. A threshold that triggers a pause, such as "if the error rate is worse than the manual baseline for two consecutive weeks." The framework calls for mechanisms and assigned responsibilities to supersede, disengage or deactivate a system whose performance is inconsistent with its intended use (NIST AI 100-1). Decide who presses the button and how work continues without the AI.
• Duration. Long enough to see normal and busy periods, short enough to keep attention. Four to eight weeks is common, though your cycle may need more.

Step 4: Run It in Parallel

For the first stretch, run the AI-assisted method alongside the current one on the same cases. The comparison is the evidence you'll need to convince skeptics and to find out where the tool really falls down.

Keep a simple log. For every miss, record what happened, what the AI said, what the right answer was, and why the gap occurred. Patterns usually appear within two weeks: a product family the model misclassifies, a customer with unusual rules, a data field that's blank on weekends.

Fix causes, not symptoms. A model that fails on blank fields needs a data fix upstream, not a bigger pile of exceptions for the reviewer. Then move from parallel running to AI-assisted work with review, still with the baseline measure running, so you can see drift.

NIST's framework is clear that testing before launch isn't enough: the behavior of the system and its components should be monitored when in production (NIST AI 100-1). The pilot is the right time to build that monitoring habit, with a weekly check on the same measures.

Step 5: Earn Frontline Adoption

The people who do the work decide whether an AI step survives. If they feel measured by it, replaced by it or ignored in its design, they'll route around it, and the metrics will look fine until the day they don't.

What helps:

• Involve a few frontline people in the design, not just the demo. The picker who knows the aisle is blocked every Tuesday knows something the data doesn't. Give them a say in what the tool shows and when.
• Say plainly what changes in their role. Vague reassurance backfires. If a task goes away, say so, and say what replaces it. If headcount decisions are not on the table for this pilot, say that too, but only if it is true.
• Teach the review skill, not just the interface. Reviewers need to know how the tool tends to be wrong, what a good correction looks like and when to escalate. Build that into a short training session and a one-page cheat sheet.
• Make overrides easy and valued. Every override is data about a constraint the system doesn't know. Track the override rate and the reasons, and show the team what changed because of their input. A visible fix after a complaint builds more trust than any town hall.
• Name a local champion on each shift. Someone peers trust, who gets questions first and passes problems up.

Step 6: Decide and Standardize

At the end of the pilot, hold a decision meeting with the business owner, the process owner and the frontline champion. Compare results to the baseline and the success criteria you wrote in step 3. Read a sample of actual outputs, including the bad ones, not just the dashboard.

Then choose one of three outcomes, in writing.

• Scale. Results met the criteria, reviewers trust the tool and the failure modes are understood.
• Fix and repeat. The idea is sound but a cause is known: data, a missing constraint, a training gap. Fix it and run a shorter second pilot.
• Stop. The result didn't beat the baseline, or the cost of review ate the gain. Stopping is a legitimate result, and it's cheaper now than after a wider rollout.

If you scale, turn the pilot into a procedure. Update the SOP so a new hire could follow it: the steps, who reviews what, which cases escalate, what the stopping rule is and what happens when the system is down. Keep a record of the pilot's settings and the constraints you added. That record is what a second site will need.

Step 7: Scale Across Sites

Scaling is not copying. A second site has different people, slightly different data, local rules and its own habits. Treat each new site as a small version of the pilot.

Roll out in waves. Add one or two sites at a time. Pick the first wave to be similar to the pilot site, then move to harder ones.

Use a rollout kit. Bundle the charter, the SOP, the training session, the checklist of data fields to verify, the baseline method and the list of known failure modes. A site lead should be able to start from the kit without a call.

Baseline every site before switching on. The same measures, taken the same way. Gains at site two are only visible against site two's starting point.

Allow local constraints, with a record. Let each site add its own rules and log them. If five sites ask for the same exception, it belongs in the standard.

Keep one owner per site and one for the process. Site owners answer for local results. The process owner decides what's standard and what's a local variation.

Keep the stopping rule and the fallback. Every site needs to know what to do when the system is off.

If you want help on the organizational side of moving from pilot to scale, the transformation series has a stage guide at from pilot to scaled.

An Illustrative Example: Returns Triage at a Distribution Network

This scenario is hypothetical.

A distributor with four warehouses handles returns manually. Clerks open each return, read the customer's note, choose a disposition (restock, refurbish, scrap, return to vendor) and route it. Dispositions vary by clerk, and the backlog grows after promotions.

The returns manager at the largest site chooses this process because it's high volume, mistakes are fixable and the notes are already typed into the system. She baselines three weeks: minutes per return, share of returns re-routed after a first decision, and the age of the oldest unprocessed return. She finds the re-route rate is the real pain.

The pilot charter limits scope to one site and one product family. The AI suggests a disposition and drafts the note; a clerk accepts or changes it. The stopping rule: if re-routes exceed the baseline for two straight weeks, the pilot pauses. Two clerks help choose which fields the screen shows, and a shift lead is the local champion.

In parallel running, the log shows the tool misreads returns with notes in two languages and mishandles items with a manufacturer recall. The team adds a recall flag from the product master and a rule sending mixed-language notes to a person. After six weeks, the manager compares results with the baseline and decides to scale.

She writes the SOP and a rollout kit. Site two joins next, picked because its product mix is close to site one's. Site three has different disposition codes, so its wave waits for a mapping fix.

Common Pitfalls

Skipping the baseline. Without it, every result is an anecdote and every argument is about opinions.

A pilot with no stopping rule. Pilots without an end condition become permanent, unexamined tools.

Overloading reviewers. Review that's squeezed into the same time as before becomes rubber-stamping.

Scaling before standardizing. Roll out a procedure, not a person's memory of how it worked.

Forgetting upkeep. Data, products and customers change. Someone needs to own monitoring, retraining and, when needed, retirement. The framework's call for ongoing monitoring and clear responsibility to disengage a system applies here (NIST AI 100-1).

What to Measure

Track four groups together, because gain in one can hide harm in another.

• Outcome: the primary measure from your baseline, such as cycle time, touch time or cost per transaction.
• Quality: error rate, rework and re-routes, compared with the baseline.
• Adoption: share of cases where staff used the AI step, override rate and reasons, training completion.
• Control: incidents traced to the AI step, time to detect a problem and whether the stopping rule has ever fired.

Where to Go Next

If you do one thing this month, pick a single repeatable process, time twenty real cases through it, and write the numbers on one page. That page is your baseline, and it's the thing most failed pilots never had. Then read operational data for AI to check the inputs, and measuring AI in operations to plan the scorecard.