I spent a Saturday last month trying to delete myself from my own analytics workflow. The plan was lazy and a little smug: instead of writing the same GROUP BY for the hundredth time, I’d point a local model at my Postgres database and just ask it questions in plain English. No cloud, no API bill, no shipping my schema off to someone else’s server. Ollama on my laptop, a 7B model, done before lunch.
It mostly worked. That’s the honest headline. It also handed me a query that quietly returned the wrong number, and I didn’t catch it for an hour. So this isn’t a “local LLMs replaced my SQL” post. It’s the more useful version: here’s where a small local model writing SQL actually earns its keep, and here’s exactly where it’ll bite you if you trust it.
Why run a text to sql LLM locally at all
The obvious answer is cost, and it’s real. A local model doesn’t bill you per token, so you can let people hammer it with sloppy questions all day and the only thing you spend is electricity.
The less obvious answer matters more if you work anywhere regulated. A recent benchmark study on biopharmaceutical manufacturing data made this concrete: organizations under FDA guidance, EU GMP, and the EU AI Act often can’t send production data to a cloud API at all, which makes a locally hosted model the only option that’s even on the table (arXiv:2606.01338). That framing stuck with me. The interesting question isn’t “is the local model as good as GPT-5.” It’s “is the local model good enough to be useful when the cloud model isn’t allowed in the building.”
I’ve written before about which local models I keep on disk and which I deleted, so I won’t relitigate the hardware here. For SQL specifically you don’t need a giant model, which is the part that surprised me.
Setting it up took ten minutes, not ten hours
I expected a weekend of glue code. It was closer to a coffee break. Ollama handles the model serving, and a coding-tuned 7B like Qwen2.5-Coder is small enough to run on a laptop with 16GB of RAM.
# pull a coding-tuned model and start chatting
ollama pull qwen2.5-coder:7b
ollama run qwen2.5-coder:7b
The actual trick isn’t the model, it’s the prompt. A bare “write me SQL for X” gets you confident nonsense. You have to hand the model your real schema every single time. Here’s the minimal version I landed on:
import ollama
SCHEMA = """
Tables:
orders(id, customer_id, total_cents, status, created_at)
customers(id, name, country, created_at)
"""
def ask(question: str) -> str:
prompt = f"""You are a Postgres expert. Given this schema:
{SCHEMA}
Write a single read-only SQL query answering: {question}
Return only SQL, no explanation. Use only columns that exist above.
"""
resp = ollama.chat(model="qwen2.5-coder:7b",
messages=[{"role": "user", "content": prompt}])
return resp["message"]["content"]
print(ask("revenue by country for completed orders last month"))
That “use only columns that exist above” line is doing more work than it looks like. Drop it and the model starts inventing fields that sound plausible. More on that in a second.
Where the small models actually held up
For the boring 80% of questions, a local 7B is genuinely fine. Single-table filters, counts, grouped sums, date ranges, basic joins on obvious foreign keys. The benchmark I mentioned tested four open models (Qwen2.5-Coder 7B, Llama 3.1 8B, Mistral 7B, and a domain model) on 60 questions against a real manufacturing database, and the general-purpose coders produced runnable SQL for essentially every prompt. The structure was almost always right.
That tracks with my own poking. “How many customers signed up in March,” “total revenue by country,” “orders stuck in pending for more than a week” all came back clean and correct on the first try. If your day job involves answering the same shape of question for non-technical teammates, a local text to sql LLM is a real time-saver. It turns a Slack ping into a query you skim and run instead of one you write from scratch.
The classic academic yardstick here is the Spider benchmark, and the gap between “scores well on Spider” and “works on my weird production schema” is exactly the gap you’re about to fall into.
And where they quietly lied
Here’s the query that cost me an hour. I asked for “average order value for repeat customers,” and the model gave me something that ran without error, returned a number, and looked completely reasonable. The problem: it defined “repeat customer” using a column that didn’t exist, so it had silently invented a customers.is_repeat boolean and the planner happily ignored my real data.
The query didn’t crash. It just answered a different question than the one I asked.
This is the failure mode nobody warns you about. A syntax error is a gift, because you see it immediately. A semantically wrong query that runs clean is the dangerous one. The benchmark study measured this directly and reported non-trivial hallucination rates even on models that produced valid SQL every time. Valid and correct are not the same property, and small models blur that line more than big ones do.
I’ve ranted about this pattern in a separate post on catching LLM hallucinations, and SQL is the worst case of it. A made-up sentence is easy to spot. A made-up JOIN condition that returns 4,212 rows looks exactly as authoritative as the correct one.
The guardrails I won’t skip again
After the wrong-number incident, I stopped treating the model’s output as an answer and started treating it as a draft that has to clear a few gates before anyone sees a number. Three things changed.
First, the database connection is read-only. Not “I’ll be careful,” actually read-only at the role level, so a hallucinated DELETE is impossible by construction.
CREATE ROLE llm_reader LOGIN PASSWORD 'redacted';
GRANT CONNECT ON DATABASE analytics TO llm_reader;
GRANT USAGE ON SCHEMA public TO llm_reader;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO llm_reader;
-- no INSERT, UPDATE, DELETE. ever.
Second, every generated query gets an EXPLAIN before it runs for real. If the planner references a relation or column that isn’t in my schema, it errors there, which catches the invented-column case before it can return a confident wrong number.
def validate(query: str, conn) -> bool:
try:
conn.execute(f"EXPLAIN {query}")
return True
except Exception as e:
print(f"rejected: {e}")
return False
Third, and least technical: the SQL is always shown to the human, not hidden behind a chat bubble. The point of the tool is to save typing, not to save reading. If you find yourself wanting to hide the query, you’ve quietly turned a helpful drafting tool into an unaudited oracle, which is the opposite of what you want against real data. This is the kind of internal tooling I build for clients, and you can see some of it in my work.
If you’re choosing where to even point this thing, my notes on SQLite versus Postgres cover which engine I reach for, and a model’s schema-awareness is one more reason I lean Postgres for anything an LLM will touch.
So would I actually ship this?
Yes, with a clear boundary. As a drafting assistant for read-only analytics, with a human reading the SQL before trusting the number, a local 7B is a solid, cheap, private win. I use it most days now for first-draft queries.
As an autonomous agent that runs its own SQL and reports numbers to a dashboard with nobody checking? Not yet, and the hallucination data is why. The models are good enough to help a person and not yet good enough to replace one against data that matters.
If you want to try it this week, here’s the smallest useful experiment: install Ollama, pull qwen2.5-coder:7b, create a read-only role on a copy of your database, and wire up the schema-in-the-prompt pattern from above. Then ask it ten questions you already know the answer to. The ones it gets wrong will teach you more about your own schema than the ones it gets right.
