Explainability and Auditability of AI in Finance

By Jane Ha · Updated Oct 11, 2026

Explainability and auditability in finance AI mean two separate promises: you can say why the system produced a specific result (explainability), and you can prove afterward what the system did, with which inputs, under which version, and who reviewed it (auditability). Regulators, auditors and customers each lean on a different half. A credit applicant wants the reason for a denial. An auditor wants the evidence trail. A model risk team wants both, plus proof that someone independent challenged the model.

This guide is for CFOs, controllers, heads of risk and finance managers who are putting AI into credit, fraud, forecasting, close or reporting work. If you want the plain-language definition of the technique itself, read What is explainable AI?. This article covers the finance-specific part: what has to be explained, to whom, and what you need on file.

Key Facts

• Regulation B requires that a statement of reasons for an adverse credit action "must be specific and indicate the principal reason(s)", and statements based on internal standards or a failed credit scoring threshold are not enough (12 CFR 1002.9).
• The April 2026 interagency model risk guidance, SR 26-2, supersedes and replaces SR 11-7 and is expected to be most relevant to banking organizations with over $30 billion in total assets (Federal Reserve).
• That guidance puts generative and agentic AI models outside its scope, while its principles apply to traditional statistical models and non-generative, non-agentic AI (SR 26-2 attachment).
• The EU AI Act classes AI systems intended to evaluate the creditworthiness of natural persons or set their credit score as high-risk, and requires such systems to technically allow automatic recording of events (logs) over their lifetime (Annex III, Article 12).
• NIST separates the two ideas: explainability is a representation of the mechanisms underlying a system's operation, while interpretability is the meaning of its output in the context of its designed purpose (NIST AI RMF 1.0).

Why Finance Holds AI to a Higher Standard

Most business functions can live with a model that works but can't say why. Finance mostly can't, for three reasons.

Decisions carry legal duties. When a lender turns down an applicant, US rules require a specific reason. The reason has to reflect what actually drove the decision, not a generic label.

Numbers feed statements people rely on. If a model sets a reserve, drafts a flux explanation or matches ledger items, someone has to be able to show the auditor how that number came to exist. The controls side of that is covered in AI and internal controls. This article covers the layer underneath it: the documentation and records that make a control testable.

Errors are expensive and slow to find. A drifting forecast or a biased scoring model can run for months before anyone notices, and the only way to reconstruct what happened is to have kept the records.

What Regulators and Auditors Actually Expect

The expectations differ by jurisdiction and by use case, so it helps to separate them.

Credit decisions (US). Regulation B, which implements the Equal Credit Opportunity Act, requires a creditor to tell a declined applicant the principal reasons in specific terms. The rule gives a creditor 30 days after receiving a completed application to notify the applicant of the action taken (12 CFR 1002.9). The practical consequence for AI: if your model can't produce reasons you can stand behind, you have a compliance problem, however accurate the model is. Post-hoc explanation tools approximate a model rather than reproduce it, so the team has to be able to check that the approximation is faithful.

Model risk management (US banks). The 2011 guidance, SR 11-7, has been replaced by SR 26-2, which takes a risk-based approach and says effective challenge is critical analysis by objective experts, performed by people with the right expertise, enough independence and the standing to get changes made. It also says an effective model inventory holds enough information to understand model risk at the individual and aggregate level, and that adequate documentation supports effective model risk management (SR 26-2). The guidance excludes generative and agentic AI from its scope, but it also says a bank's own governance practices should guide the controls for tools it doesn't cover. If you're a smaller institution or a corporate finance team, treat it as a well-tested template rather than a rulebook.

Vendor models. The same guidance says sound practice includes understanding a vendor model's conceptual soundness, design, development data and performance, and monitoring its outcomes, even when the vendor withholds code or data (SR 26-2). "The vendor says it's accurate" is not documentation.

EU. Credit scoring of natural persons is a high-risk use under Annex III of the AI Act, and high-risk systems must technically allow automatic logging over their lifetime (Annex III, Article 12). The Act's timeline has been amended, and the current consolidated text lists 2 December 2027 as the application date for Annex III systems (Article 12). Legal teams should confirm the status for their own situation, but the engineering lesson is the same either way: build the logging now, because retrofitting it is harder.

Financial reporting auditors. External auditors aren't asking for model internals in the abstract. They want to know the AI step is a known, owned part of a documented process, with evidence that someone checked the output.

Explainability by Function: What "Explained" Means in Practice

"Explainable" isn't one thing. The right explanation depends on who is asking and what's at stake.

The pattern across the table is the same. Explanation is about drivers and basis, and auditability is about the record. A model can have one without the other: a simple rule-based scorer is easy to explain but still useless to an auditor if it logs nothing.

The Explanation Toolkit, and Where It Falls Short

Two techniques come up constantly for complex models. SHAP "assigns each feature an importance value for a particular prediction" (Lundberg and Lee). LIME explains an individual prediction by fitting a simple, interpretable model around the neighborhood of that prediction, treating the underlying model as a black box (Ribeiro et al.).

Both are useful for the "why this result" question. Neither is a substitute for governance, for three reasons:

• They explain a prediction, not the model's fitness. A clean explanation of a biased score is still a biased score.
• They're approximations. Treat the output as evidence to be validated, not as the model's actual reasoning.
• They don't produce a customer-ready reason on their own. Someone has to map factor contributions to the principal reasons a notice can state accurately.

For high-stakes decisions, it's worth testing whether a simpler model gives up little accuracy, because a model you can inspect directly is easier to defend than one you must approximate.

Documentation That Holds Up: Model Cards and Inventories

A model card is a short document that travels with a model. The original proposal describes cards as documentation of benchmarked performance across different conditions, the intended use context, and how the evaluation was run, so that users know where the model works and where it doesn't (Mitchell et al.). The authors focus on vision and language, and say the idea applies to any trained model. For a finance team, the idea ports directly. See AI model cards for the general format.

A finance version might include:

1. Purpose and owner. What decision or process it supports, and the named business owner.
2. Intended and out-of-scope use. For example, "scores consumer unsecured applications; not for small-business lending."
3. Data. Training sources, time window, known gaps, and the exclusions you applied.
4. Method in one paragraph. Model type, key features, and how outputs are turned into decisions.
5. Performance. Accuracy measures, tolerance, results by segment, and the backtest against actuals.
6. Explanation method. How reasons or drivers are produced, and how faithful they were shown to be.
7. Limits and known failure modes. Where it is weak, and what triggers manual handling.
8. Controls. Reviewer, thresholds, fallback if it's down, and change history.
9. Validation. Who challenged it, when, and what they found.

Pair the cards with an inventory: one list of every model and AI tool in use, with owner, purpose, tier and card location. SR 26-2 itself notes that an effective inventory has enough information to understand model risk, which is the test to apply to yours. For vendor tools, the card will be partly empty. Write that down rather than leaving it blank, and decide whether the gaps are acceptable for the risk.

Audit Trails: The Record of What Actually Happened

Documentation says what a system is meant to do. An audit trail shows what it did on a given day. The logging principle in the EU rule is a good general standard: record events over the lifetime of the system, in a way that supports monitoring and investigation (Article 12). For the deeper treatment of trails when AI takes actions inside systems, see audit trails for AI actions.

A usable finance audit trail captures, for each material AI-influenced item:

• Input: the data or document the system received, or a pointer to it
• Version: model, prompt or rule set, and configuration in force at the time
• Output: the score, entry, flag or text produced
• Explanation: the drivers or reasons generated with it
• Human action: who reviewed it, what they changed or overrode, and when
• Downstream result: the decision, posting or notice that followed

Retention should match the record it supports. Ask your records and legal teams how long credit files, journal support and workpapers must be kept, then make sure the AI logs aren't deleted sooner. Data handling for these logs is its own topic, covered in AI finance data security.

How to Build Explainability and Auditability, Step by Step

1. Inventory every model and AI tool in finance. Include vendor features switched on inside existing systems. You can't document what you haven't found.
2. Tier by consequence. A model that sets a customer's credit outcome or a reserve belongs in a stricter tier than a tool that drafts an internal summary.
3. Set the explanation requirement per tier. Name who must be able to get an explanation, how fast, and in what form (customer notice, analyst view, auditor workpaper).
4. Choose models with the requirement in mind. For top-tier uses, test whether a more transparent model is nearly as accurate before accepting a black box.
5. Write the model card before go-live. Make launch approval conditional on a completed card and a named owner.
6. Get independent challenge. Someone with the right expertise and independence reviews the model and its explanation method, and can require changes.
7. Turn on logging from day one. Capture inputs, versions, outputs, explanations and human actions. Test that you can reconstruct a single decision end to end from the logs.
8. Run a reconstruction drill. Pick ten past decisions at random and have someone outside the build team explain each from the records alone. Gaps show up fast.
9. Monitor and re-document. Track accuracy, drift and override rates. Update the card and re-challenge when the model, data or use changes. See AI accuracy controls in finance for monitoring ideas.

Common Pitfalls

• Explaining the model but not the decision. Global feature importance doesn't answer why this applicant was declined or why this entry was flagged.
• Generic reasons. Sending the closest checklist reason when the real driver is something else fails the specificity test in Regulation B.
• Trusting the explanation tool blindly. Post-hoc methods approximate behavior. Test them against known cases.
• Model card as a one-time PDF. A card that doesn't change when the model changes becomes a false record.
• Logs without versions. Knowing what the output was is useless if you can't tell which model or prompt produced it.
• Vendor opacity accepted by default. "Proprietary" is a reason to monitor outcomes harder, not to skip documentation.
• Review without evidence. A reviewer who approves but leaves no record of what they checked gives the auditor nothing to test.
• Explainability built after launch. Retrofitting logs and documentation costs far more than designing them in.

What to Measure

• Coverage: share of models and AI tools with an owner, tier and current card
• Reconstruction rate: share of sampled decisions that can be fully explained from logs alone
• Explanation quality: how often reviewers or validators reject the generated reasons as inaccurate or too vague
• Override rate: how often humans change the output, and why
• Freshness: time since each card and validation was last updated, against the change log
• Issue closure: time to resolve validation and audit findings

Where to Go Next

Start with the inventory and tiering, then write cards for your top tier and run one reconstruction drill. If you're still mapping where AI fits, AI in finance and accounting gives the wider view.