Who Is Allowed to Know That?
Permissions are not something you add to an AI project. They are a property of the data underneath, and a chat window cannot acquire them.
A customer drops into your office to talk about next quarter. Your account manager pulls up their account to check something, and without thinking about it, angles the monitor a few degrees away.
Nobody asked her to. Nobody would call it a security control. But the screen shows what this customer pays, what it costs you to serve them, and the name of the competitor down the road who gets a better rate. She knows, so she turns it.
That half-turn is most of the permission model in a lot of businesses. It works because a person is doing it, and a person knows who is sitting in the other chair.
Now put an AI assistant in front of the same records and let the customer type the question. The assistant does not know who is in the other chair. It does not turn the screen.
Why does this get filed under compliance?
Because it looks like a policy question. Who should see margin, who should see other customers' prices, who should see the whole region rather than their own patch. It goes to IT or to legal, somebody writes it down, and the AI project carries on.
It is not a policy question. It is a question about the data underneath. Either the system already knows, record by record, who each piece of information belongs to and who may see it, or it does not. If it does not, no amount of policy sitting next to it will be enforced by a machine that answers in under a second.
We set out the wider version of this in your AI is not guessing, it is reading bad data very confidently, where permissions are one of four conditions that have to be true before an AI answer can be trusted. This piece is about that one condition, because it is the first thing a buyer asks about and the last thing most vendors can show.
What happens when the data underneath has no rules?
Companies that have tried it already know. When Computerworld's Matthew Finnegan reported a Gartner survey in September 2024, 40% of respondents rolling out Microsoft 365 Copilot had delayed it by three months or more over data "oversharing", and 64% said information governance and security had taken significant time and resources to deal with. The assistant was not the problem. It simply surfaced documents that had always been reachable and that nobody had ever labelled properly.
Researchers have measured how far that goes. In a 2024 study published at ACL, Shenglai Zeng and colleagues gave an assistant a private store of real company emails to answer from, then attacked it. With GPT-3.5 doing the answering, 116 of 250 crafted prompts returned passages copied word for word from the private emails. That was a deliberate attack on a test dataset, not an ordinary user, but the mechanism is the point: whatever an assistant can retrieve, a determined question can get back out.
And deleting something is not always the end of it. A team from the University of Texas at Austin, publishing as ConfusedPilot in 2024, found that content deleted from a document kept appearing in Copilot's answers for between roughly forty seconds and three minutes, because the assistant was answering from a cached copy.
None of these are stories about a careless model. In every case the model behaved as designed. It answered from whatever it could reach.
Why can't you just tell the AI what not to say?
Because an instruction is a request, and a permission is an absence.
"Do not reveal other customers' pricing" is a sentence in a prompt. The model will usually follow it, and the whole of the research above is about the times it does not. A permission works differently. The pricing that a customer may not see is simply never handed to the assistant answering that customer. There is nothing to leak, because it was never in the room.
An AI cannot repeat what it was never given. That is the only version of AI safety on business data that does not depend on the model behaving well.
It also means the work has to be done before anybody asks a question. The assistant has to be answering as the person who asked, using only what that person could already see. Which assumes the system knows what that is.
What does a permission model actually have to know?
Two things, and most systems only know the first.
The first is which views a person can open. Finance can open the cost reports. The warehouse cannot. That is permission by module, and nearly every business system has it.
The second is which rows inside those views belong to them. A regional manager sees utilisation, but only for their own region's vehicles. A distributor and a shop buy the same product from the same catalogue, and neither may know what the other pays. That is permission by record, and it is where most stacks quietly give up. The rule lives in a spreadsheet, in a separate portal running on a copy of the data, or in a sales team's habit of turning the screen.
An assistant will flatten anything that is only enforced by habit.
How can you tell if your systems are ready?
Run this test before anybody demonstrates an assistant to you.
- Log in as a customer, a salesperson and a regional manager, one after another.
- Open the same screen as each of them.
- Check whether they see different numbers, and whether that difference is enforced by the system rather than by being asked not to look.
If the answer is yes, the hard part is already done and an assistant inherits it. If the answer is that customers are protected because they use a separate portal fed by a nightly copy, that copy is what an assistant will read, and its rules are whatever survived the export.
How this runs on one platform
In Illuminate, reports and dashboards are scoped to roles and permissions at both levels: which widgets a role can see, and which data within those widgets is visible to them. Finance sees cost. Operations sees utilisation. A regional manager sees only their region's assets. These views are configured by Illuminate for each client as part of the relationship, not assembled by the client from a blank screen.
The same rule faces outwards through Kart eCom. Every portal user is a Ryse CDP account, and the account type decides at login which catalogue they see, which price list applies and what they are allowed to do. A distributor and a direct shop log in through the same address. The shop does not know what the distributor pays, and the distributor does not know what the shop pays. That isolation is enforced at the account, not by running separate storefronts.
This was built because customers and staff needed it, before any conversational layer was on the table. Which is the point. An assistant that answers as the person asking, on top of permissions like these, can only ever surface what that person could already see.
The useful question about an AI assistant is not how clever it is. It is whether, before anybody typed anything, your systems already knew who was allowed to know that.
Frequently asked questions
Can an AI assistant leak confidential company data?
Yes, if it can retrieve information the person asking is not entitled to see. Research published at ACL in 2024 showed that a determined set of prompts can pull passages word for word out of the private documents an assistant answers from. The reliable defence is to make sure confidential records are never retrieved for that user in the first place, rather than relying on the model to keep them to itself.
Why can't a prompt instruction stop an AI revealing data?
Because an instruction is something the model usually follows, not something it is unable to break. Carefully worded or persistent questions can get round it. A permission removes the data from what the assistant can see for that user, so there is nothing to reveal regardless of how the question is phrased.
What is the difference between role-based access and record-level access?
Role-based access decides which screens, reports or modules a person can open. Record-level access decides which rows inside them belong to that person, such as one region's vehicles or one customer's prices. An AI assistant answering business questions needs both, because most confidential information sits at the record level.
What should a business check before connecting an AI assistant to its data?
Log in as several different users, open the same screen as each of them, and confirm the system itself shows each one a different, correct slice of the data. If confidentiality depends on separate portals, exported copies or staff discretion, those rules will not carry over to an assistant.
References
Finnegan, M. (2024). Microsoft 365 Copilot rollouts slowed by data security, ROI concerns. Computerworld, 27 September 2024, reporting a Gartner survey.
Zeng, S., Zhang, J., He, P. et al. (2024). The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG). Findings of the Association for Computational Linguistics: ACL 2024. Open access.
RoyChowdhury, A., Luo, M., Sahu, P., Banerjee, S. and Tiwari, M. (2024). ConfusedPilot: Confused Deputy Risks in RAG-based LLMs. arXiv:2408.04870. Open access.
More in this series
This piece takes one of four conditions from a larger argument. The overview, with the research on why AI answers go wrong on operational data, is your AI is not guessing, it is reading bad data very confidently.
The other spoke so far is what your dispatcher knows that your system does not, on the correction that makes a broken plan work and why an AI never sees it.
Talk to us about who should see what in your operation
For media inquiries, please contact:
contact@illuminate.ae
