CCAB Ethical Leadership Podcast

Whose Data Is It Anyway?

• CCAB

Share episode

Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.

0:00 | 12:04

Client data does not belong to the firm. Student data does not belong to the university. So what happens when AI starts using that data in ways nobody agreed to?

In the final episode of the series, AI hosts Alex and Sam look at data, consent and trust. A big global firm wants to train its own large language model on 'synthetic' data generated from years of confidential client information, and the technology team insists confidentiality is maintained. The IESBA Code says otherwise. Meanwhile, a university accounting department pilots an AI teaching platform whose plagiarism checker starts falsely accusing international students, while vague vendor terms leave open the question of whether student work is quietly training a commercial AI.

The series closes with the line at the heart of it all: the accounting profession requires an inquiring mind, the exercise of professional judgement and ethical decision making with clear accountability. Those things cannot be automated.

Explore the full case studies and ethical frameworks at ccab.org.uk.


 

Speaker:

Welcome to Ethics in the Age of AI from the CCAB. I'm Sam.

Speaker 2:

And I'm Alex. In today's episode, we're talking about data, specifically the data of clients and students, and what happens when AI enters the picture and that data starts being used in ways that weren't originally agreed to.

Speaker:

We've got two very different settings today. One is a big global accounting firm building its own AI model. The other is a university accounting department trying to modernize its teaching. Both run into some uncomfortable questions about consent, privacy, and where the ethical lines are.

Speaker 2:

And in both cases, there's a real temptation to say, "It's fine. We've sort of addressed it." That temptation is exactly what we're here to challenge.

Speaker:

But before we go on, we need to make the same disclosure we've made all series. The two of us are AI voices, and this podcast was created using AI. Worth holding in mind for an episode about who owns data and how it gets used.

Speaker 2:

Right, let's set the scene. You're a partner at a big global firm responsible for innovation and technology. The firm is developing its own large language model, its own AI, to enhance audit, tax, and advisory services. Genuinely exciting, potentially transformative.

Speaker:

And to train a large language model effectively, you need large amounts of high-quality, relevant data.

Speaker 2:

Which a big global firm has in extraordinary abundance. Years of client financial data, transaction records, contracts. The technology team comes to you with a proposal: use this client's data to create what they're calling synthetic data. Essentially, a statistically equivalent version of the data that doesn't contain actual client information, and use that synthetic data to train the model.

Speaker:

And on the surface, that sounds like a reasonable way to protect client confidentiality while still benefiting from the richness of real-world data.

Speaker 2:

It does, and the technology team is confident. Their team leader says, "We're not using actual client data. We're using synthetic data. Confidentiality is maintained. Let's move."

Speaker:

But there are people in the room who aren't so sure.

Speaker 2:

Several concerns surface. First, synthetic data is created from actual client data. Even if the end product doesn't directly identify clients, the first step of the process, processing the original confidential information, is itself a use of that data, and clients have never consented to that.

Speaker:

And this isn't a theoretical concern. The IESBA Code, the International Ethics Standards for Accountants, is explicit on this point. It requires client authorization before confidential information is used in the development of technology, full stop. The Financial Reporting Council has also clarified this

Speaker 2:

So the technical argument, we're only using synthetic data, doesn't actually hold up against the ethical and regulatory standard. The processing step counts.

Speaker:

What about the risk of the synthetic data still containing traces of actual client information?

Speaker 2:

The technology team themselves acknowledge this. Synthetic data generation isn't perfect. There's a small but real risk that actual data occasionally leaks through. And for clients in regulated industries, financial services, healthcare, even that small risk may cross a line.

Speaker:

And then there's competitive pressure. Competitors are doing this. The firm risks falling behind if it doesn't move forward.

Speaker 2:

Which is a classic objectivity threat. When commercial urgency starts to feel more pressing than your professional obligations, that's exactly when you need to slow down and apply the conceptual framework.

Speaker:

So what should the firm do?

Speaker 2:

Start by acknowledging the legal and ethical position clearly. Explicit client authorization is required, not assumed, not inferred from vague engagement letter language, actually obtained.

Speaker:

And that means going to clients and having an honest conversation about what you'd like to do with their data, why, and what safeguards are in place.

Speaker 2:

The FRC has actually noted that clients may be quite open to this if it's explained properly. The issue is doing it without asking What you need to do is develop a consent framework. Give clients the option to opt in or out. Begin with a pilot using only data from clients who have genuinely agreed, and explore whether you could create useful synthetic data based on patterns observed through human expertise over time, which, under the IA-SBA provisions, can be done without specific client authorization.

Speaker:

There's also something to be said here about the integrity signal this sends. If a firm is willing to use client data in ways clients haven't agreed to, even for ostensibly good reasons, what does that say about the trust relationship?

Speaker 2:

Exactly. Clients entrust firms with some of their most sensitive financial information in the world. That trust is foundational to the entire profession. You don't erode it quietly for a competitive advantage.

Speaker:

Now let's zoom out from big global firms to a university accounting department because the students who'll be practicing the profession in five years are already navigating these questions.

Speaker 2:

Let's imagine that you're a professor of accounting at a well-regarded university. The department has licensed a third-party AI platform designed to help students with accounting exercises, give instant feedback on assignments, and check for plagiarism.

Speaker:

And the rationale is actually quite sound. Students are already using AI tools anyway. A bespoke, free, vetted tool should at least give the department some oversight of what's happening.

Speaker 2:

That's the logic. But during the pilot, things get complicated. The AI's feedback on complex case study assignments is often generic and occasionally just wrong. It can't grasp nuanced arguments or context-specific applications of accounting standards

Speaker:

Which matters enormously in a subject where the whole point is developing professional judgment

Speaker 2:

Then there's the plagiarism checker. It starts flagging submissions from international students, students whose sentence structures and writing patterns differ from typical native English conventions. These students suddenly find themselves under investigation for plagiarism they didn't commit

Speaker:

That's a genuinely distressing situation for a student, and it's a textbook example of algorithmic bias. The system wasn't trained on diverse enough writing patterns, so it treats linguistic difference as suspicious

Speaker 2:

And meanwhile, students are beginning to over-rely on the AI to generate answers rather than developing their own analytical thinking, which is precisely the opposite of what an accounting education is supposed to do

Speaker:

Then comes the data question. The terms of service from the AI vendor are vague, particularly about whether student assignment data is being used to train and improve the AI model

Speaker 2:

So potentially, student work, which may contain original analysis, personal reflections, even sensitive contextual information, is being fed back into a commercial AI system without meaningful consent

Speaker:

Under GDPR, student data is personal data. Processing it for a purpose students haven't consented to, like improving a commercial AI, is a compliance problem, not just an ethical one

Speaker 2:

And then there's the fundamental question: Is this tool actually preparing students for the ethical demands of the accounting profession, or is it training them to short-circuit the very critical thinking the profession depends on?

Speaker:

What should the professor do?

Speaker 2:

First, evaluate the tool's accuracy rigorously and systematically. Compare its feedback against expert human assessment before relying on it for anything consequential. Address the bias in the plagiarism checker directly with the vendor. And critically, use this tool for formative purposes only, practice and feedback, not for summative assessment, not for final grades, until accuracy and fairness can be properly validated.

Speaker:

Ensure human oversight of all AI-generated feedback. Faculty should always review before a student sees a result, especially for anything that carries stakes for the student.

Speaker 2:

Review the data privacy terms with the university's legal and data protection teams, not just on your own. Get explicit consent from students if their data is going anywhere beyond the immediate marking function. And create a clear, accessible appeals process so students have recourse if the AI makes an unfair call.

Speaker:

There's also something worth saying here about what we're modeling for students. If we teach future accountants that it's acceptable to delegate judgment to an AI and not look too closely at how it works, we've given them a very dangerous professional habit.

Speaker 2:

The accounting profession requires an inquiring mind, the exercise of professional judgment, and ethical decision-making with clear accountability. Those things can't be automated. They have to be taught, practiced, and earned.

Speaker:

Both of today's scenarios are really about consent and respect. Client data doesn't belong to the firm. Student data doesn't belong to the university. Trust is extended to you for a specific purpose. And when AI starts using that data for purposes that go beyond what was agreed, you've crossed a line that professional ethics doesn't let you quietly uncross.

Speaker 2:

The hard truth is that it's technically permissible and it's ethically sound are not always the same thing The CCAB codes ask you to apply both the letter and the spirit of professional standards, and sometimes the spirit asks more of you

Speaker:

That's it for this series. We've covered AI in strategic decisions, AI in audit and financial reporting, and AI and data privacy. Three episodes, six case studies, 15 very uncomfortable questions

Speaker 2:

and hopefully some useful frameworks for thinking through the ones you'll encounter in your own practice

Speaker:

The full CCAB case studies with all the ethical considerations and recommended courses of action are available at ccab.org.uk. We really do recommend reading them in full

Speaker 2:

Thank you for listening to Ethics in the Age of AI from the CCAB. Until next time, keep asking, not just can we do this, but should we?

Contributor

Tom Parker

Host
Contributor

Natalie Chisholm

Producer