Confession: the first AI automation I sold to a client was a mistake, and I didn’t know it for four months. It was a small e-commerce shop, three people, and they wanted the “AI reads the support inbox and replies” thing everyone was selling in 2024. I built it. It worked in the demo. Then it spent a quarter politely telling customers their refund was “being processed” when nobody had processed anything, because the model was excellent at sounding like a refund had been processed and nobody had wired it to the actual refund system. The owner found out from a one-star review.
I’ve been thinking about that project again this week, for an odd reason. OpenAI published a case study on Gilbert + Tobin, a large Australian law firm that has rolled ChatGPT Enterprise and Codex across most of its operations. It’s a vendor story, so take the adoption numbers with the usual salt. But the interesting part isn’t the numbers. It’s what they explicitly did not automate, and how much governance a firm with real money and real lawyers decided it needed before letting AI touch anything. If a law firm with a CIO and a business transformation team draws the line that carefully, a three person shop with me as the entire IT department should probably draw it more carefully, not less.
So this is a post about when not to use AI automation in a small business, written by someone who makes part of his living building AI automation for small businesses. That should tell you I’m not against it. I’m against the version of it that ends in a one-star review.
What a law firm chose not to automate
Read the Gilbert + Tobin piece with an eye for the negatives and a pattern shows up. The firm introduced ChatGPT to operations teams first, then marketing, recruitment, finance and technology. The legal work itself, the advice clients pay for, stayed on separate approved platforms with their own controls. The CEO’s line was “AI is not cheating”, but the framing throughout is that people stay responsible for constraining the task, checking outputs and approving the final work product.
Their Codex examples are the same shape. Audit reports across 300 entities, prepared by Codex and reviewed by a person. Eleven hundred files checked and renamed for upload, reviewed by a person. A KYC and anti-money-laundering check workflow that does the research and processing steps and then produces a report for human sign-off. In every example the model does the long boring middle and a human owns the start and the end.
There’s a detail I keep coming back to. They built a custom GPT trained on the CEO’s writing and priorities so executives could pressure test ideas before taking up his time. And then the case study goes out of its way to say it does not make decisions on his behalf or speak for him. Someone thought hard about where that line was and wrote it down.
That’s a firm with hundreds of staff, an information security function, and clients who will sue if data leaks. My clients are usually a founder, two employees and a Shopify store. They don’t have a governance team. So the question for me is: which of those lines transfer down to a business that size, and which ones are big company overhead?
The line that transfers: automate the middle, not the ends
Almost every AI automation I’ve seen fail in a small business failed at one of the ends. Either the model decided what the task was, or the model decided the task was done. The middle, the part where it reads forty PDFs and pulls the invoice numbers into a spreadsheet, almost never fails in a way that matters. If it gets one wrong, a person catches it at the end, because a person is at the end.
My refund bot broke both ends at once. It decided a support email was a refund request (start), and it decided to tell the customer the refund was handled (end). The bit in the middle, drafting a polite reply, was flawless. That was the problem. A flawless middle with no one at the ends is the most dangerous configuration there is, because it looks like it’s working right up until it very much isn’t.
So the first rule I now apply, before a single line of code, is this. Draw a box around the automation. Write down what enters the box and who decided it should. Write down what leaves the box and who checks it before it has an effect in the real world. If either of those “who” answers is “the model”, stop and ask whether the business can survive the model being confidently wrong there a hundred times before anyone notices. Usually it can’t.
I wrote about the positive version of this, the jobs that actually are worth handing to a model, in my post on AI automation examples and which skills LLMs replace. This post is the mirror image.
When not to use AI automation at all
Some things aren’t a middle. I’ve built up a short list of situations where I’ll now tell a client no, or at least “not like that”, and most of them come from getting it wrong once.
Anything that sends money or makes a commitment. Refunds, discounts, delivery promises, quotes. A model that promises a customer a discount has made a contract on your behalf, and “the AI said it” is not a defence anyone will accept. Gilbert + Tobin let Codex prepare audit reports, not sign them. The same applies to a Shopify store’s refund policy.
Anything where being wrong is invisible. A summarizer that drops one line from a contract, a classifier that files a complaint under “general enquiry”. Nobody sees the mistake at the moment it happens. The cost turns up weeks later as a churned customer or a missed deadline, and you can’t trace it back. If you can’t measure the error rate, you can’t decide whether it’s acceptable, and “we’ll find out” is not a plan.
Anything you don’t do often enough to review. This one surprised me. The Gilbert + Tobin examples are irregular jobs: an audit sweep, a bulk rename. Those work because a person is sitting right there when it runs. The danger zone is the low volume, unattended automation. It runs twice a week, no one is watching, and by the time someone looks at the output there are two months of confidently wrong records to unpick. High volume tasks get caught fast because someone is always looking. Low volume, unattended tasks rot.
Anything where the business hasn’t written down the rule. If the founder can’t tell me in plain English how they decide whether a lead is worth calling, I can’t encode it, and the model will happily invent a rule that sounds plausible. Half the “AI automation” projects I’ve turned down were really “please make a decision for us that we haven’t made ourselves” projects. That’s a consulting job, and the model is not the consultant.
The permission problem nobody budgets for
The part of this that doesn’t get talked about in small business AI automation pitches is what the automation is allowed to touch. Docker published a piece this week, Below the Harness, that opens with Norm Hardy’s 1988 confused deputy problem: a program that takes an action using its own permissions rather than yours. Their argument is that every AI agent is that deputy now, because it inherits your credentials, your repository access, your ability to call whatever API you can call, and that organizations will need a runtime layer underneath all their agents where rules get enforced in one place.
That’s aimed at enterprises running fleets of agents. But the confused deputy shows up in a three person business just as hard, and with fewer safeguards. The typical small business automation I’m asked to build runs on the owner’s own accounts. The owner’s Gmail. The owner’s Stripe key. The owner’s admin login to the store. Because setting up a scoped service account is friction and the demo is due Friday. So the automation can do anything the owner can do, which includes issuing refunds, deleting products and emailing the entire customer list. It probably won’t. But “probably won’t” is doing a lot of work when the thing deciding what to do is a model reading customer emails, some of which will eventually contain instructions.
I covered the technical shape of that last risk in the tool schema problem in AI agent security, so I won’t repeat it. The small business version is simpler and more boring. Before any automation goes live, it gets its own account, with the minimum permissions it needs, and the owner’s credentials come out of the config. If the client says that’s too much hassle, that’s useful information about whether the automation should exist.
What the small version of governance actually looks like
I don’t think a three person company needs a governance team. I do think it needs about a page of writing, and I now refuse to build the automation until that page exists. It has four things on it.
What the automation is allowed to do, in plain sentences. “Read incoming support emails. Draft a reply. Put the draft in a folder. Never send.” Not a system prompt, a sentence a non-technical owner can read and agree to.
What it’s not allowed to do, which is usually the previous list with the word “never” in front of the scary verbs. Never send. Never refund. Never change a price.
How you’ll know it’s wrong. Somebody’s name, and how often they’ll look. “Sara checks the drafts folder every morning before sending.” If nobody’s name goes here, the automation is unattended, and see above.
How to turn it off. A single switch the owner can flip without calling me. This sounds trivial. It is the most important line on the page. The refund bot took a week to fully stop because it was wired into three places and only I knew where.
That page is the small business translation of everything in the Gilbert + Tobin story: approved tasks, what you can enter, how outputs get reviewed, and who is accountable. It takes an hour to write and it has saved me from at least two projects that would have gone the way of the first one.
Where AI automation does earn its keep
I want to end on the other side, because “don’t” is a cheap thing to write and I’d be a hypocrite if I stopped there.
The automations I’ve built that clients still pay for a year later have all had the same profile. High volume, so mistakes surface fast. A human at both ends. Read-mostly, meaning the model’s job is to look at things and produce a draft or a classification, and the effect on the world happens when a person clicks. Invoice data extraction into a review queue. First draft replies that a person sends. Lead notes summarized before a call. Tagging and sorting a backlog so a person can triage it in ten minutes instead of an hour.
None of these are exciting. All of them save real hours, and all of them have a fail state that is “a person spends an extra minute” rather than “a customer gets a promise nobody will keep”. The cost side matters too; a review queue that a person clears is far cheaper to run than an agent loop that retries until it’s sure, and I’ve written before about what happened to LLM costs after the free lunch ended.
If you’re a small business owner reading this and someone is pitching you an automation, ask them where the human at each end is. If they say there doesn’t need to be one, that’s your answer. If you’re a developer building these things, the way I’ve come to think about scoping them is written up in more detail on my site, along with the projects where I learned it the hard way.
What to do this week
Pick one automation you’re already running, or the one you’re about to build, and write the one page. Four sections: allowed, never, who checks and how often, how to turn it off. If you can’t fill in “who checks”, don’t ship it. If you can’t fill in “how to turn it off” in under a sentence, you’ve got a bigger problem than AI, and this is a good week to fix it.
