AI Is Bad at Reading Databases. So Stop Asking It To.

The best-known test of AI on business data gave every question a written hint. Take the hints away and people and AI both drop about 20 points. Your business has never written its hints down.

On her first morning, a new hire is asked a simple question: how many orders are late?

She doesn't answer. She asks what you mean by late.

Late against the date we promised, or the date we planned? Does a split delivery count? What about the customer who moved his own delivery and is now ringing to ask where it is?

You'll want to promote her by lunchtime. That question is the most useful thing anyone will say all week.

Ask an AI the same thing and it won't ask anything. It will just give you a number. A very tidy one.

In your AI is not guessing, it is reading bad data very confidently we quoted one result from a well-known research test of AI on business data. This piece goes back to that test, because the most important part of it is the part nobody quotes.

The short version. AI tools that write their own database queries get business questions wrong far more often than the demos suggest, mostly because nobody has told them what the data means. The safer approach is a semantic layer: one agreed list of what each business number means, which the AI chooses from instead of making up its own. And that list should know who is allowed to see what.

What did the BIRD text-to-SQL test actually measure?

The test is called BIRD. It was published in 2023 by Jinyang Li and colleagues at the University of Hong Kong and Alibaba, and presented at NeurIPS, one of the biggest AI research conferences.

It is not a toy. It uses 95 real databases, 33.4 GB of data, from 37 different fields. Exactly the kind of mess a business builds up over the years without ever meaning to.

The idea is simple. Someone asks a question in plain English. The AI turns it into a database query, written in SQL, the language databases speak, and runs it. Researchers call this text-to-SQL. The answer is either right or wrong.

People who know databases took the same test. They scored 92.96%.

Now the bit that rarely makes the headlines. Every question came with a short note attached. The researchers called it evidence. It explained what the data meant, because the database never says. Women are stored as gender = 'F'. A basketball centre is recorded as pos = 'C'. A rate is one column divided by another.

In other words, every question came with a cheat sheet.

What happens when you take the hints away?

The researchers, to their credit, ran the test both ways.

With the notes, people scored 92.96%. Without them, 72.37%.

GPT-4, the strongest model in the final paper, scored 54.89% with the notes. Without them, 34.88%.

Both lost about 20 points. Human or machine, it made no difference. Take away the meaning and everyone gets worse.

The paper's own error analysis says the same thing another way. 40.8% of the AI's mistakes came from misunderstanding what the values in the database meant. Another 41.6% came from looking in the wrong tables or columns altogether.

Writing the query was never the hard part. Knowing what the business means is.

In the test, somebody kindly wrote the notes down. In your business, nobody has.

Your notes sound like this. Status 4 means cancelled, but still billable. "Available" means on the shelf and not already promised to someone else. Late means late against what we told the customer, not what the plan said. These rules live in a few people's heads and an email thread from 2019. No AI has ever read them. They have never left the building.

Won't a better model fix it?

It is getting better, and fast. The BIRD leaderboard now has entries above 82%, up from the mid-fifties in the 2023 paper. That is real progress, and worth saying.

Every one of those scores still came with the cheat sheet.

And tidy tests are kind to everyone. When researchers built Spider 2.0, presented at ICLR 2025, they used real company databases instead, many with more than 1,000 columns. The same AI setup that solved 91.2% of an older, cleaner test, and 73% of BIRD, solved 21.3% of the real ones. The more the data looked like an actual business, the lower the score went. Which will surprise nobody who has ever opened an actual business's database.

So a better model helps where someone has already written down what everything means. It cannot help with the meaning nobody wrote down. And the risk keeps the same shape. Every question gets a brand-new query, which is a brand-new chance to be wrong. A wrong answer sounds exactly like a right one. Same tone. Same confidence. Same lovely formatting.

A better model is wrong less often. It is not wrong in a way you can see.

So what is the alternative?

Stop asking the AI to write the query at all.

Instead, the business decides what its important numbers mean, once, properly, and gives each one a name. On-time delivery rate. Available stock. Free space on a route this week. Each one is calculated the same way every time, by the system, and checked by the people who own it.

Then the AI gets a much smaller job, and one it is genuinely good at: working out which of those numbers you are asking about, and for which branch, week or customer. It chooses. It does not compose. Every number it hands you was agreed by a person before anyone asked.

Less creative. Far more useful. Rather like a good accountant.

This is called a semantic layer. Think of it as a dictionary for your business numbers: one word, one meaning, agreed by the people who own it. We didn't invent it. Analytics companies have been making the case for years. dbt Labs has written up research in which GPT-4 answered insurance questions correctly 16.7% of the time from raw tables, and 54.2% of the time once the business meaning was set out first. Cube published a comparison in 2026 in which three current models went from between 45% and 51% to around 68% when they were given a written description of the business's measures. Both companies sell semantic layers, so they are hardly neutral, and the Cube study used just 99 questions on one dataset. Treat the exact numbers with care. The direction is hard to argue with.

What do analytics semantic layers leave out?

Most semantic layers were built for analysts. They describe the past. Revenue last quarter. Sales by channel. Customers lost by month.

Operations doesn't live in the past. It lives in this afternoon. Can this order go out today? Can we squeeze one more job onto Thursday's route? The honest answer depends on things that change by the hour. Is that truck actually on the road, or booked into the workshop? Is the driver on shift? Is there already a problem on that route? Is the stock really free, or quietly promised to someone else?

If those things are not defined numbers too, one of two things happens. Either the AI writes its own query to find them, and you are back in the test without the cheat sheet. Or it never sees them at all, and answers from the plan. And the plan, as anyone in operations will tell you, is mostly a hopeful document.

So for an operation, the list of numbers has to include what is happening now, not just what happened. And a question about right now has to read live data, not last night's export.

How this runs on one platform

In Illuminate, the numbers a business runs on are defined in its reporting layer. They are mapped to that business's own data and set up by Illuminate with the client, so nobody starts from an empty page.

The layer reads from two places, on purpose. Questions about right now read live operational data: shipments on the road, current asset status, open work orders, where the drivers are. Questions about trends read from a separate data warehouse, a copy kept just for reporting, so a heavy quarterly report never slows down the people trying to get trucks out of the gate.

What is happening now is already part of those numbers. Cargo TMS counts exceptions on the dispatch board, next to ready, dispatched and en route, so trouble is a number rather than a sticky note. Tagz AMS knows whether each vehicle or machine is available or in maintenance. Depot WMS shows which orders are already holding each unit of stock, so "available" really means available.

A semantic layer that knows who is asking

Illuminate takes this one step further. Its semantic layer is secured by the same permissions control as the rest of the platform, the access control list, or ACL. That is simply the record of who is allowed to see what.

So the dictionary does not only say what a number means. It says who may see it. The finance team can see what things cost. The operations team can see how busy each truck is. A regional manager sees their own region, and only their own region. If a number is not yours to see, it is not hidden after the answer is written. It was never on your list in the first place.

That matters even more once AI arrives. An AI that chooses from the list can only choose from numbers the person asking is already allowed to see. It cannot cheerfully tell the warehouse team your margin on your biggest customer, because that number was never on the warehouse team's list. Why that has to be built into the data, rather than requested politely in a prompt, is the subject of who is allowed to know that?

None of this needs an AI. It is the list any AI would have to choose from, and it has to exist before you can trust one.

So when someone offers to let you "ask your data anything", don't ask how clever their model is. Ask who wrote the cheat sheet. Then ask who is allowed to read it.

Frequently asked questions

Can AI accurately answer questions from a company database?

Only partly. In the BIRD research test, the strongest model in the 2023 paper answered 54.89% of questions correctly when given written hints about what the data meant, against 92.96% for people, and leaderboard entries have since passed 82%. Without the hints, both people and AI scored about 20 points lower, and in a real business those hints are rarely written down.

What is a semantic layer?

A semantic layer is a set of business figures defined once in advance, such as on-time delivery rate or available stock, each calculated the same way every time. An AI working on top of one chooses which defined figure answers a question instead of writing its own database query, which removes a whole class of errors.

Should a semantic layer respect user permissions?

Yes. A semantic layer decides what each business number means, so it should also decide who is allowed to see it. Otherwise an AI choosing from the layer can hand any user any number. In Illuminate, the semantic layer is secured by the platform's access control list (ACL), so finance, operations and regional managers each see only the figures their role allows, and an AI working on top could only choose from those.

Why do AI models get database queries wrong?

Mostly because databases do not record what their contents mean. In the BIRD study, about 41% of errors came from misunderstanding database values and about 42% from choosing the wrong tables or columns. Codes, statuses and local definitions such as what counts as late are knowledge a model cannot learn from outside the business.

How accurate is text-to-SQL on real enterprise databases?

Much less accurate than on research benchmarks. In the Spider 2.0 study, presented at ICLR 2025, an AI system based on OpenAI's o1-preview solved 91.2% of tasks on an older benchmark and 73% on BIRD, but only 21.3% of tasks drawn from real enterprise databases, many with more than 1,000 columns.

Do operational questions need anything a standard semantic layer lacks?

Yes. Analytics semantic layers mostly define historical measures such as revenue or churn. Operational questions such as whether an order can go out today depend on live state: vehicle availability, driver shifts, open exceptions and reserved stock. Those need to be defined figures too, read from live data rather than an overnight export.

References

Li, J., Hui, B., Qu, G. et al. (2023). Can LLM Already Serve as a Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs. Advances in Neural Information Processing Systems 36. Open access.

BIRD leaderboard, test set results, checked 15 October 2026.

Lei, F., Chen, J., Ye, Y. et al. (2025). Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows. International Conference on Learning Representations (ICLR 2025). Open access.

Ganz, J. (2024). Semantic Layer as the Data Interface for LLMs. dbt Labs.

Rumiantsau, M. and Fokeev, I. (2026). Semantic Layers for Reliable LLM-Powered Data Analytics. Published by Cube. arXiv:2604.25149.

More in this series

This piece takes one argument out of a larger one. The overview, with the research on why AI answers go wrong on operational data, is your AI is not guessing, it is reading bad data very confidently.

The other pieces so far: what your dispatcher knows that your system does not, and who is allowed to know that?

Talk to us about the figures your business actually runs on

For media inquiries, please contact:
contact@illuminate.ae