Measuring AI in Operations: OEE, Fill Rate, Forecast Accuracy and Unit Cost

By Tara Minh · Updated Oct 11, 2026

Measuring AI in operations means tying each AI tool to one operating KPI, fixing a baseline before it goes live, and comparing results against what would have happened without it. The KPIs are mostly old and familiar: overall equipment effectiveness (OEE) for equipment, fill rate for service, forecast accuracy for planning and unit cost for money. What's new is the discipline. A model that "feels" faster proves nothing until a number moves by more than normal variation.

This article is a companion to AI in operations management. It gives the formulas, the baseline window to use for each KPI, the AI lever that should move it, and a method for attributing the change to the AI rather than to the season, the product mix or a new supervisor. For the wider rollout, see implementing AI in operations.

Key Facts

• OEE is calculated as Availability x Performance x Quality, where availability is run time over planned production time, performance is net run time over run time, and quality accounts for parts that miss standards (OEE.com).
• A world-class OEE of 85 percent is often cited, and OEE.com links that figure to Seiichi Nakajima's work. The same site states that there is no one ideal OEE score or industry standard that fits every company (OEE.com).
• Fill rate has no single standard definition. It can be counted by order, by line or by unit, so two reports using the same name can show different numbers (DCL Logistics).
• MAPE becomes infinite or undefined when any actual value is zero, and it penalizes negative errors more heavily than positive ones (Hyndman and Athanasopoulos).
• A before-and-after comparison alone can't support a causal claim, because factors other than the change influence outcomes over time (World Bank DIME Wiki).
• The rule that follows: pick one KPI per AI use case, define it in writing, freeze a baseline, and compare against a control.

Why AI Needs a Different Measurement Habit

Most operations teams already track OEE, fill rate and cost. The trouble is that AI projects tend to be judged on activity (models deployed, alerts sent, hours saved on a task) rather than on the operating result. Those are inputs. A planner who spends less time on the forecast hasn't improved the forecast.

Three things make AI harder to measure than a new machine or a new shift pattern.

The effect is spread thin. An AI tool for maintenance might save 40 minutes of unplanned downtime a week on each line. That's invisible in a monthly OEE number unless you've measured the baseline carefully.

The context moves. Demand, product mix, staffing and supplier reliability all change during a pilot. If fill rate rises the quarter you launch an AI inventory tool, was it the tool or a slow season?

People change behavior when they know they're watched. Operators and planners often work harder or more carefully during a pilot, which inflates early results.

The fix isn't a fancier dashboard. It's agreeing in advance what will be measured, how, and against what comparison.

The Four Core KPIs, With Formulas

OEE: equipment and line performance

OEE multiplies three ratios: Availability x Performance x Quality (OEE.com). Availability asks how much of the planned time the machine actually ran. Performance asks whether it ran at the intended speed while running. Quality asks how many of the parts were good the first time.

Because the three multiply, small losses compound. Three factors at 90 percent each produce an OEE of about 73 percent, not 90 (OEE.com). That's why a small AI-driven gain in one factor, such as predictive maintenance reducing stoppages, shows up less than people expect in the top-line number. Track the three components separately and use OEE as the summary.

AI levers: predictive maintenance and anomaly detection act on availability, scheduling and parameter tuning act on performance, and vision inspection acts on quality.

About the benchmark: you'll often hear that 85 percent is world class. OEE.com describes that as a commonly held view tied to Nakajima's work, and says plainly that no single score suits every company (OEE.com). A high-mix job shop and a continuous packaging line have very different realistic ceilings. Compare your line with its own history first, and use the 85 percent figure only as a loose reference.

Fill rate: service to the customer

Fill rate is the share of customer demand you met from stock. The common variants are:

• Order fill rate: orders shipped complete divided by total orders placed, times 100.
• Line fill rate: order lines completely filled divided by total order lines.
• Unit (or case) fill rate: units shipped on the first shipment divided by units ordered.

Those definitions follow the usual industry treatment, and the same source notes there's no standard fill rate measure across businesses (DCL Logistics). A 10-line order with one line short counts as a miss for order fill rate but 90 percent for line fill rate. Choose one definition, write it down, and never compare it with a number built on another.

AI levers: better demand forecasts and safety stock settings (see AI inventory optimization). If you buy from suppliers, the supply-side counterpart is on-time, in-full delivery, covered in AI supplier performance scorecards.

Forecast accuracy: planning quality

There's no single accuracy number. The standard textbook measures are MAE (average absolute error), MAPE (average absolute percentage error) and MASE (error scaled by a simple benchmark) (Hyndman and Athanasopoulos). MAPE is easy to explain but breaks down when actuals are zero or near zero. MASE compares your forecast with a naive one, so a value below 1 means you beat the naive method. WAPE (total absolute error divided by total actual volume) is a common practical choice because large items count for more, and bias (total forecast minus total actual, over total actual) shows whether you're consistently high or low.

The full treatment, with a worked example, is in AI demand forecasting. Two rules matter for measurement. Judge the forecast at the lag where you commit money, such as eight weeks ahead for an eight-week lead time. And always compare against a naive baseline, like last week or the same week last year. If the AI can't beat it, there's no gain to report.

Unit cost: what it costs to make or serve one

Unit cost is total cost divided by units produced or served: fixed plus variable cost over output. It's the KPI finance cares about most, and the hardest to attribute, because it moves with volume, input prices, wages and overhead allocation.

Hold the definition steady. Decide which costs are in (labor, materials, scrap, freight, overhead share), which are out, and at what volume you measure. A rising volume lowers fixed cost per unit with no help from AI. For service work, AI cost-to-serve analysis shows how to break cost down by customer and channel. A practical habit is to track cost per unit at constant volume, or to report variable cost per unit separately.

KPI Measurement Table

The baseline windows here are working rules of thumb for setting a stable comparison, not published standards. Lengthen them when your process has strong seasons, and shorten them when the process changed recently and old data no longer represents it.

How to Measure AI Impact, Step by Step

1. Name one KPI and one use case. "Reduce unplanned downtime on line 3 with predictive maintenance" is measurable. "Improve operations with AI" isn't.
2. Write the definition. Formula, data source, which events count, who owns the number. Put it in the pilot charter before launch.
3. Freeze the baseline. Take the baseline window from the table, and record its average and its spread (the week-to-week range). The spread matters: a 2 point gain means little if the KPI normally swings 4 points.
4. Pick the comparison. The best option is a control group: a similar line, site, product family or customer segment that doesn't get the AI. If you can't split groups, use a before-and-after comparison and treat it as weaker evidence.
5. Run the pilot long enough. Cover at least one full cycle of whatever drives your variation: a production campaign, a month-end, a demand season.
6. Compute the difference in differences. The change in the AI group minus the change in the control group. That's the estimate of the AI effect, and it removes anything that hit both groups equally. The method rests on an equal-trends assumption, which can't be proved but can be assessed by checking that the two groups moved together before the pilot (World Bank DIME Wiki).
7. Track the whole cost. Include licenses, integration, data cleanup and the time of the people who review outputs. A KPI gain that costs more than it returns isn't a win.
8. Report the range, not just the point. "OEE rose 2 to 4 points against a normal weekly swing of 3" is honest. "OEE rose 3 points" is not.

For the data side of steps 2 and 3, see operational data for AI.

Illustrative Example: AI Forecasting and Fill Rate

This example is hypothetical, with invented numbers, to show the mechanics.

A distributor has two regional warehouses with similar product ranges. The team applies an AI forecasting tool to the north warehouse and leaves the south warehouse on the old method as the control. Both have a 13-week baseline.

• North warehouse order fill rate: 91.0 percent before, 93.5 percent after. Change: +2.5 points.
• South warehouse order fill rate: 90.5 percent before, 92.0 percent after. Change: +1.5 points (a good season lifted everyone).
• Difference in differences: 2.5 - 1.5 = 1.0 point attributed to the AI.

A naive before-and-after reading would have claimed 2.5 points and overstated the result by more than double. The team also notes the weekly swing in fill rate was about 1.5 points, so the 1.0 point gain is within noise and needs a longer run before they call it a win. That's the useful answer: promising, not proven.

Pitfalls to Avoid

Measuring activity instead of outcome. Tickets closed by an AI assistant and models in production are not KPIs.

Changing the definition mid-pilot. Recoding downtime reasons, switching fill rate from order to line level, or changing which costs count in unit cost will all produce a "gain" from nothing.

Cherry-picking the pilot site. If you put the AI on your most organized line, it'll look good for reasons unrelated to AI. Document how sites were chosen and compare their pre-pilot trends.

Ignoring spillovers. If staff or customers are shared between the AI group and the control, the control gets contaminated and the measured effect shrinks.

Using MAPE alone. It misbehaves on low-volume and zero-demand items. Pair it with WAPE and bias, or use MASE (Hyndman and Athanasopoulos).

Chasing one KPI at the expense of another. Higher fill rate bought with excess inventory raises cost. Faster line speed can lower quality. Always read KPIs in pairs: service with cost, performance with quality, accuracy with bias.

Declaring victory at launch. Novelty effects fade. Keep the measurement running for several cycles after go-live, and watch for model drift as products and demand change.

What to Do Next

If you do one thing this month, pick a single AI use case already in flight, write the KPI definition on one page, and freeze 8 to 13 weeks of baseline data. Then find a comparison group, even an imperfect one. For help choosing which operating numbers belong on a shared view, see process KPIs.