AI Quality Assurance for Support: Scoring Every Conversation

By Linh Ngo · Updated Oct 11, 2026

AI quality assurance (AI QA) for support is the practice of using a language model to score every customer conversation a human agent handles against a written rubric, instead of having a reviewer grade a small hand-picked sample. The model reads the transcript, answers the rubric's questions (Did the agent identify the real issue? Was the solution correct? Was the tone appropriate?) and produces a score you can slice by agent, team, topic or week.

The appeal is coverage. A manual QA program can only read as many tickets as its reviewers have hours for, so most conversations, including some of the worst ones, never get looked at. AI can read all of them. But reading everything isn't the same as reading it correctly, and a score that agents don't trust is worse than no score. So this guide covers three things: how AI QA works, how to calibrate it against human reviewers, and how to keep it fair. It's part of the AI for Customer Service collection, so the overview of AI in customer service is a good starting point if you want the wider context.

Key Facts

• Zendesk's QA product advertises "100% coverage with AutoQA" and invites teams to stop manually selecting tickets and analyze every interaction (Zendesk). That's a vendor claim about coverage, not about accuracy.
• Zendesk's own settings documentation says its automatic evaluations are for guidance only, shouldn't be used for performance decisions, and should supplement, not replace, human performance reviews (Zendesk).
• Cohen's kappa measures agreement beyond chance. In McHugh's widely cited interpretation, 0.60 to 0.79 is moderate agreement, 0.80 to 0.90 is strong, and above 0.90 is almost perfect (McHugh, Biochemia Medica, 2012).
• Research on LLM judges found strong models can reach over 80% agreement with human preferences, the same rate at which humans agree with each other, but also documented position, verbosity and self-enhancement biases (Zheng et al., 2023).
• Under the EU AI Act, AI systems used to monitor and evaluate the performance and behaviour of workers are listed as high-risk in Annex III, point 4(b) (EU AI Act, Annex III).

How AI QA Works

Traditional QA is a loop with a bottleneck in the middle. A QA analyst picks tickets, opens each one, scores it against a scorecard, writes comments, and sends feedback to the agent. Because the analyst's time is fixed, the sample stays small, and sampling is rarely random. People tend to review the tickets that were escalated, got bad CSAT, or happen to be short.

AI QA keeps the scorecard and replaces the analyst's reading step. The pipeline looks like this:

1. Ingest. Conversations from chat, email, voice transcripts and tickets flow in with metadata: channel, topic, agent, customer tier, outcome.
2. Analyze. A model reads each conversation and answers each rubric question, usually with a short justification quoted from the transcript.
3. Score. Answers roll up into a conversation score and then into agent, team and topic views.
4. Route. Low-scoring or high-risk conversations go to a human reviewer. Exceptional ones go to a coaching library.

That last step is why this belongs to the scoring-and-routing pattern in the ACE Framework: the model doesn't just grade, it decides which conversations a person should look at. Done well, your human reviewers spend their hours on the 5% that need judgment instead of the 95% that don't.

Zendesk QA's autoscoring ships with default categories such as Greeting, Empathy, Spelling and grammar, Closing, Solution offered, Tone, Readability and Comprehension. Some are simple yes/no checks, Tone is a weighted score, and Readability is a fixed three-point scale (Zendesk). "Did the agent greet the customer?" is almost mechanical. "Did the agent understand the customer's issue?" requires judgment. That difference drives the rest of this article.

Manual QA vs. AI QA

The honest takeaway: neither replaces the other. AI gives you breadth, humans give you ground truth, and the calibration process described below is what connects them.

Real Examples by Function

The rubric changes by where the conversation happens. A few concrete cases:

Tier 1 customer support. A customer writes in about a double charge. The rubric checks that the agent verified identity, found the duplicate, explained the refund timeline, and didn't promise something policy forbids. AI is good at the last check: it can flag every conversation where an agent used a phrase like "guaranteed" or quoted a refund window that doesn't match the current policy.

Billing and collections. Compliance matters most. The rubric might require a disclosure at the start and ban certain threats, and AI QA can check every conversation for both where a manual sample would catch them by luck.

Technical support. Diagnosis is harder to grade automatically. It often works better as "flag for review if the customer came back within a week about the same issue" than as an absolute grade.

Internal IT help desk. The "customer" is an employee, and rubrics lean on resolution quality and whether the agent pointed to the right article, which also exposes gaps in your support data.

Designing the Rubric

The model is only as good as the question you ask it. Most failed AI QA programs fail at the rubric, not the model.

A workable scorecard has four to eight criteria, each written as an observable, answerable question rather than a vague trait. Compare "Was the agent empathetic?" with "Did the agent acknowledge the customer's stated problem before moving to a solution?" The second one can be checked from the transcript, and two reviewers are far more likely to agree on it.

Two design choices pay off. First, always allow "can't tell" or "not applicable." A model forced to pick yes or no will guess, and guesses on missing context are where unfair scores come from. Second, weight criteria by what customers actually feel. Correct resolution should outweigh a spelling slip.

Calibration: Making the Score Believable

Calibration is the step that separates a dashboard people argue with from one they use. The idea is simple: have humans and the AI score the same conversations, then measure how often they agree and fix the gaps.

Step 1: Build a gold set

Pick 100 to 200 conversations that cover your channels, topics and difficulty levels. Include some easy ones, some ambiguous ones, and some you already know went badly. Have at least two experienced reviewers score each one independently, then meet to resolve disagreements. The result is a "gold set" with agreed answers, and the disagreements you had along the way are valuable by themselves because they show where your rubric is vague.

Step 2: Run the AI on the same set

Score the gold set with the model. Compare criterion by criterion, not just the total score. A total can look fine while one criterion, say "Accuracy", is quietly wrong half the time.

Step 3: Measure agreement properly

Raw percent agreement is misleading, because two raters who both say "pass" most of the time will agree often by chance. Use Cohen's kappa, which discounts chance agreement. Here's a hypothetical to show why it matters:

The two agree on 80 of 100 conversations, so raw agreement is 80%. But if both raters pass about 80% of conversations, they'd agree about 68% of the time by luck alone. Kappa is (0.80 minus 0.68) divided by (1 minus 0.68), which is about 0.38. On McHugh's scale that's weak agreement (McHugh, 2012), even though the headline says 80%. Most of the AI's "agreement" was just both of them saying pass.

Compute kappa for each criterion, and also for the human reviewers against each other. That second number is your ceiling. If your two best reviewers only reach 0.65 with each other, expecting the AI to hit 0.9 is unrealistic, and the real fix is a clearer rubric. The LLM-judge research points the same way: strong models can match the rate at which humans agree with each other, which is a useful target, not a magic one (Zheng et al., 2023).

Step 4: Fix the gaps, then re-test

Where agreement is low, you have three levers: rewrite the criterion to be more concrete, add worked examples (a "good" and a "bad" transcript) to the prompt, or take the criterion out of automatic scoring and keep it human-only. Some criteria, like real empathy, may never reach acceptable agreement. Score what the AI scores reliably and leave the rest to people.

Step 5: Keep calibrating

Keep a standing weekly or monthly session: a reviewer grades a fresh random sample blind to the AI's score, and you track kappa over time. A falling number is an early warning that something changed, such as a new product line, a policy update, or a new language. If you run multilingual support, calibrate per language, since agreement in English tells you nothing about Portuguese.

Fairness: Scoring People Is Different from Scoring Tickets

A QA score isn't just data. It feeds coaching, bonuses and sometimes performance reviews. That raises the stakes for errors and for bias, and it's the part most teams under-plan.

Know the known biases. The LLM-as-judge literature documents position bias, verbosity bias (longer answers favored) and self-enhancement bias (Zheng et al., 2023). In support, verbosity bias has a direct translation: an agent who writes long, polished replies can outscore an agent who solves the problem in two crisp lines. Test for it by checking whether score correlates with message length after you control for outcome.

Check for group-level differences. Compare scores by tenure, shift, language, region and contact type. Some gaps are real. Others show the rubric penalizing non-native phrasing or a harder queue: if billing-dispute agents score lower than password-reset agents, the model may be grading difficulty, not quality. Normalize by topic before comparing people.

Be transparent with agents. Publish the rubric. Let agents see the exact criteria, their own scores, and the quoted passage that triggered each judgment. People accept a low score they can see the reason for. They resist a number that appears with no explanation.

Build an appeal path. Any agent should be able to dispute a score and get a human to look at it within a set time. Track the overturn rate. A high overturn rate on one criterion means that criterion isn't ready for automation. Feed overturned cases back into your gold set.

Keep a human in the decision. Even the vendors say so. Zendesk's documentation states that its automatic evaluations are for guidance only and should supplement, not replace, human performance reviews (Zendesk). Use the AI score to decide which conversations a person reads, and use the person's judgment to decide anything that affects pay or employment.

Know the regulatory direction. In the EU, AI systems intended to monitor and evaluate the performance and behaviour of workers appear in Annex III of the AI Act's high-risk list (EU AI Act, Annex III), and Article 14 requires that people overseeing such systems stay aware of the tendency to over-rely on their output, known as automation bias (EU AI Act, Article 14). Check with legal counsel for your jurisdiction, and keep in mind that works councils and employee-monitoring laws in some countries add their own requirements.

Implementing AI QA, Step by Step

1. Define the purpose. Coaching, compliance and customer experience need different rubrics. Start with one.
2. Write the rubric with agents, not just managers: four to eight observable criteria, with "can't tell" allowed.
3. Calibrate before launch. Build the gold set and check kappa per criterion, as above.
4. Run in shadow mode for a few weeks alongside your current QA, without showing scores to agents, or label them "beta, not used in reviews".
5. Write down how scores may be used. Coaching topics and which tickets to review, yes. Bonuses and discipline, not until trust is earned.
6. Route, don't just report. Low scores, compliance flags and disputes go to a human queue. Standout conversations go to the coaching workflow as examples.
7. Close the loop with agents. Show results with quoted evidence and an appeal button from day one.
8. Connect it to the stack. QA findings show where real-time suggestions or knowledge articles fail. For measuring a customer-facing bot instead, see AI resolution rate.
9. Review quarterly: re-run the gold set, check kappa and score gaps across groups, retire criteria that don't hold up.

If you're still deciding what stays human, automate or assist frames that call, and how AI changes support team roles covers what happens to QA analysts: they usually move from scoring tickets to owning the rubric, the gold set and calibration.

Common Pitfalls

Trusting coverage as accuracy. "Every conversation scored" says nothing about whether the scores are right. Ask for agreement numbers against human reviewers on your own conversations.

Scoring what's easy, not what matters. A team scoring 98% on greetings and 60% on correct resolution has a measurement problem, not a quality success.

Skipping "can't tell". Without it, the model invents answers when the transcript lacks context, such as account state it never saw.

Ranking people on raw scores. Agents on hard queues get punished. Compare within topic and treat gaps as questions to investigate.

One-time calibration. Agreement decays as products and policies change. Schedule the recalibration.

Teaching agents the score, not the skill. If an empathy phrase raises the score, you'll get empathy phrases. Keep human review in the mix.

Blaming agents for knowledge gaps. Many low accuracy scores trace back to outdated help articles. Fix the source before coaching the person.