Black Box Blues - When Nobody Can Explain the Numbers

'The AI identified them as anomalous' is not an explanation. It is just saying the same thing again.

In this episode, AI hosts Alex and Sam tackle the black box problem: what happens when AI produces findings that nobody in the room can explain. At a mid-sized audit firm, an impressive AI audit platform flags transactions as high risk, but the engagement team cannot tell the client why, junior auditors have stopped exercising their own scepticism, and the engagement letter never mentioned that client data would pass through the platform's cloud servers. Then, in the boardroom of a listed technology company, an Audit Committee Chair discovers that significant parts of the financial reporting are automated by an AI system that neither management nor the external auditors can fully verify.

The episode also points listeners to the FRC's recently published Generative and Agentic AI Guidance, a genuinely useful resource for any firm working through these questions.

Explore the full case studies and ethical frameworks at ccab.org.uk.

Transcript

Alex: 0:14

Welcome back to Ethics in the Age of AI from the CCAB. I'm Alex.

Sam: 0:19

And I'm Sam. Today's episode is one I've been looking forward to actually, because it gets into something really fundamental about how professionals are supposed to work, the idea that you need to be able to explain and defend your conclusions.

Alex: 0:33

And AI is making that very, very complicated.

Sam: 0:37

We're looking at two scenarios where AI is doing the analytical heavy lifting, one inside an audit firm, one inside a listed company's boardroom, and in both cases, the AI produces outputs that nobody can properly explain.

Alex: 0:50

Welcome to what people in the tech world are calling the black box problem, and in accounting, it has very real consequences

Sam: 1:20

Reminder before we start, in case this is your first episode with us, Alex and I are AI voices, and this podcast was created using AI

Alex: 1:30

Which, for an episode about AI you can't fully explain, is a rather fitting bit of irony.

Sam: 1:36

All right, scenario one. You're a partner at a mid-sized accounting firm. You've invested, probably significantly, in a new AI-powered audit platform. It's impressive technology. It analyzes entire transaction populations rather than samples. It flags anomalous patterns. It generates risk assessments automatically.

Alex: 1:57

The dream, essentially.

Sam: 1:58

The dream. You're using it on an engagement for a retail client, multiple locations, complex inventory system. The AI flags a pattern of transactions at certain stores as high risk, suggesting potential revenue recognition issues

Alex: 2:13

Which sounds like exactly the kind of thing you want an audit tool to do.

Sam: 2:17

It is, except when your engagement manager tries to explain to the client's CFO why these specific transactions were flagged, the best they can offer is,"The AI identified them as anomalous."

Alex: 2:31

That's not an explanation. That's just saying the same thing again

Sam: 2:34

Exactly. The CFO is understandably not satisfied, and it keeps going. The junior auditors on the team have started to just focus entirely on what the AI flags. They've stopped applying broader professional judgment to identify risks the AI might have missed. The documentation is becoming thinner because everyone assumes the automated work papers cover it. And here's the one that made me wince, the client's data is being processed through the AI system's cloud servers, and nobody fully addressed that in the engagement letter.

Alex: 3:07

So you have a client whose data you can't fully account for, findings you can't explain, documentation that won't stand up to scrutiny, and junior staff who have essentially handed their professional skepticism to an algorithm.

Sam: 3:20

And we now have a client who is now questioning whether the firm is exercising appropriate professional judgment, which is a very uncomfortable conversation for any partner to be in.

Alex: 3:31

Let's talk about the ethical dimensions, starting with integrity. Are you being straightforward and honest with your client about the capabilities and limitations of the system? If you can't explain why the AI flagged something, and you present that finding as a risk anyway, you're relying on the authority of the tool to do the work that professional explanation should do.

Sam: 3:52

And then there's objectivity. The AI itself might have biases built into how it defines anomalous. If the team never questions that, they're just confirming the AI's assumptions rather than exercising independent professional skepticism.

Alex: 4:07

There's also a real professional competence question here. The CCAB codes require auditors to act diligently and gather sufficient appropriate evidence. If you can't evaluate the AI's outputs, if you don't understand its data extraction methods well enough to know whether the data was even complete, you cannot, in good conscience, sign off on findings based on that output.

Sam: 4:30

So what's the fix?

Alex: 4:31

Several things. Invest in training so the audit team actually understands how the system works. Develop a blended approach, AI-directed testing plus traditional risk-based testing, not one instead of the other. Verify data completeness independently. Strengthen the documentation standards and, critically, update your engagement letters to explicitly address AI usage and data handling And you don't have to start from a blank page here. The FRC has produced some genuinely helpful guidance on exactly this, their Generative and Agentic AI guidance, which is well worth a read for any firm working through these questions.

Sam: 5:15

Also, talk to the client. Don't hide the fact that you're using AI. Acknowledge the limitations. Offer supplementary manual testing to validate the findings. That's what professional integrity looks like in this context.

Alex: 5:28

An AI should enhance professional judgment. It's not a replacement for it, and the moment it becomes one, you've got a problem.

Sam: 5:37

Okay, so let's take a look at our second scenario. This time we're moving to the other side of the table, the boardroom side. You're an experienced accountant serving as a non-executive director and chair of the audit committee for a listed technology company.

Alex: 5:52

That's a significant governance responsibility.

Sam: 5:55

Very much so. The company has implemented an AI system that automates significant parts of its financial reporting, including complex revenue recognition calculations for long-term contracts and valuations of intangible assets from recent acquisitions.

Alex: 6:11

Both of those are highly judgment-intensive areas in accounting, exactly the areas where you'd want the most transparency about how figures are being reached.

Sam: 6:20

And yet, at the quarterly audit committee meeting, when you probe the CFO and the finance team on how the AI arrived at certain key figures, they can't give you a clear answer. They point to the system's complexity. They mention proprietary algorithms. Essentially, it's the AI, we trust it.

Alex: 6:40

As audit committee chair, that answer should not be satisfying.

Sam: 6:45

Not even slightly. Then the external auditors weigh in and say they're having difficulty obtaining sufficient appropriate audit evidence regarding the AI's internal logic and controls. They can reconcile what came out to what went in, but they can't verify what happened in the middle.

Alex: 7:02

And these are figures that underpin market communications. These are part of regulatory filings. This is not an internal management report. This is information that investors and regulators rely on.

Sam: 7:14

And to add to the concern, you realize there's been no documented board-level discussion or approval of the scope and risk assessment of implementing this AI system. It was just adopted quietly.

Alex: 7:27

Quietly. Now, let's think about the professional and ethical obligations at play here. As a NED and audit committee chair, your duty is oversight. You are the board's check on management. If management can't explain the basis for reported figures and the external auditors can't independently verify them, the audit committee has not fulfilled its role by simply nodding them through.

Sam: 7:54

Integrity in financial reporting means the board needs sufficient assurance that what's being published presents a true and fair view."The AI said so" is not assurance.

Alex: 8:06

And the professional competence principle cuts both ways here. Both the finance team and the board need to understand, at least at a governance level, what risks this technology introduces. If nobody on the board has that understanding, you have a capability gap that needs to be filled, either through training or by bringing in external expertise

Sam: 8:29

So what's the right course of action?

Alex: 8:31

The right course of action is that you don't approve the quarterly financials until you have satisfactory explanations. That's not obstruction, that's your job. Commission an independent third-party review of the AI system's controls and outputs. Push for board-level training on AI governance, and insist on a formal AI governance framework being developed and approved at board level

Sam: 8:57

Also, consider whether the use of AI in financial reporting needs to be disclosed externally. Investors may have a legitimate interest in knowing that significant judgments are being made by an automated system

Alex: 9:09

The key lesson here is that transparency is not just an internal virtue. It's part of what the profession owes to the public

Sam: 9:17

I think that's the line for this episode, actually

Alex: 9:20

Both of today's scenarios share something important. AI is being used for genuinely complex, high-stakes analysis, but the humans in the room can't explain what it did or why. And in our profession, simply saying,"I don't know, the algorithm said so," has never and should never be an acceptable professional answer

Sam: 9:43

Exactly. Explainability isn't a nice-to-have, it's foundational to professional accountability

Alex: 9:49

Next time, we're getting into something that came up

a lot in our research: 9:51

data. Specifically, whose data is it, who owns it, and what happens when an AI platform uses it for purposes you never agreed to, including, potentially, training the very AI that's auditing your client's accounts

Sam: 10:10

It's a bit circular and a lot thought-provoking. Don't miss it

Alex: 10:14

Thanks for listening to Ethics in the Age of AI from the CCAB. The full case studies and supporting frameworks are at ccab.org.uk

Sam: 10:24

We'll see you next time. Until then, keep asking the hard questions. That's what we're here for.