How does text-to-SQL work, and how reliable is it?

The promise is old and simple: anyone in a company types a question in plain words, and gets an answer from the data without waiting for an analyst. Large language models made it workable, because they write SQL well. What they lack is not SQL. It is knowing what your tables mean.

How it works

A text-to-SQL system gives a model 3 things: the question, a description of the database (table names, column names, types, and ideally a sentence on what each one means), and any instructions about the business. The model writes a query. The system runs it and shows the result, and a good one shows the query too.

The model never sees the data itself, only its description. So everything depends on that description being enough to choose the right table and column.

How good is it?

Benchmarks give a sharp answer. On BIRD, a test of over 12,000 questions across 95 real databases, the people it was tested with (data engineers and database students) answered about 93% correctly, measured by whether their query returned the right result. The systems on its leaderboard score below that.

Spider 2.0, published in 2024, tried something closer to company work: 632 tasks drawn from enterprise databases, often with more than 1,000 columns, across systems such as BigQuery and Snowflake. An agent built on OpenAI's o1-preview, which solved 91.2% of an older, simpler benchmark, solved 21.3% of these. The gap shows where the difficulty lives: not in writing SQL, but in finding the right data among thousands of columns with unclear names.

What makes it work in a company

The fixes are mostly data work, not model work.

• Describe the tables. A column called amt_2 tells a model nothing. A description saying it is the order total in euros after discounts tells it everything. Databricks' Genie, which turns questions into SQL over Unity Catalog tables, relies on annotated tables and columns for exactly this.
• Define the metrics once. A semantic layerSemantic LayerOne shared definition of business terms and metrics, such as revenue or active customer, that reports, dashboards and AI assistants all read instead of writing their own. means the model asks for "revenue" rather than inventing a formula for it.
• Give it verified answers. Genie lets an author add example queries and trusted assets, queries whose logic a person has checked. An answer that uses one rests on logic a person checked.
• Keep it read-only. Genie's generated queries are always read-only, so a misunderstood question cannot change data.

How to use one well

Treat the answer as a draft from a fast analyst who joined last week. Read the query, not only the number. Check which table it used and whether the filters match what you meant. When the question is one people ask every week, have someone write and verify the query once, so the next answer is looked up rather than guessed.

The risk is not an obviously broken answer. It is a plausible number from the wrong table, stated with the same confidence as a right one. That is the HallucinationHallucinationWhen a model states something false as if it were true, such as a wrong date, a made-up quote or a source that does not exist. problem in a database's clothes.