Okay, this is going to sound like a niche stats story, but stay with me, because it’s the most concrete thing that’s happened to search traffic all year. On August 8, ChatGPT quietly changed how it searches the web. Not the interface. The queries themselves. One day its background searches were almost never scoped to a specific website, and the next day nearly one in five of them were. Nobody announced it properly. A tracking company caught it in their data, and the numbers are wild enough that I’ve spent a week rethinking how I write for my own blog.
I’m generally allergic to the “SEO is dead, now it’s all about AI” crowd. Every few months someone rebrands the same advice and sells it back to us with a new acronym. But this change is real, it’s measurable, and it has already vaporized most of one very large website’s AI visibility. So let’s look at what actually happened before the hype merchants get to it.
What changed on August 8
Promptwatch, a company that runs automated tracking of how chatbots answer prompts, noticed that ChatGPT Search’s use of the site: operator jumped from 0.37 percent of background queries to 16.8 percent in a single day. That’s roughly a 46x increase overnight. For weeks the share had hovered between 0.3 and 0.5 percent, dipped briefly in early August in what looks like a staged rollout experiment, then spiked on August 8 and stayed there. The timing lines up with the GPT-5.6 rollout, and with a vague OpenAI announcement on August 6 that didn’t mention any of this.
At the same time, the average number of searches ChatGPT runs per response nearly doubled. So it’s not just scoping queries to specific sites; it’s running more searches overall, and more of them are aimed at a particular domain.
Simon Willison covered the data with a caveat worth repeating: Promptwatch can only see the prompts they track, so the exact percentages are their sample, not gospel. Fair. But a 46x jump in a tracked sample on one specific day isn’t noise. Something shipped.
Fanout queries, in plain terms
If you haven’t run into the term: when you ask ChatGPT something with search enabled, it doesn’t run your question as one query. It fans out. Your one prompt becomes several background searches, each phrased differently, and the model reads the results and stitches an answer together. You ask about the best way to structure a Next.js app, and behind the curtain it might run four or five searches with different phrasings.
The site: operator changes what those background searches mean. Before August 8, ChatGPT was searching the open web and taking what came back, the way you or I would. Now it’s frequently deciding, in advance, which specific domain it wants an answer from, and going there directly. That’s not browsing anymore. That’s the model keeping a mental list of sources it trusts for particular topics, and skipping everyone else.
If your site is on that mental list for a topic, you get read and cited. If it isn’t, you don’t just rank lower. You’re not in the room.
Reddit found out what the stakes are
Here’s the part that should get your attention. Promptwatch also tracked what happened to Reddit’s citations across the same change. From mid-July to August 7, Reddit averaged 3.83 percent of all ChatGPT Search citations, which is enormous when you remember how many total citations that represents. By the week of August 14, it was 0.52 percent. An 86 percent relative drop, in about a week, from a model-side change no site owner had any say in.
Reddit didn’t change anything. Their content didn’t get worse. A model update rearranged who gets read, and one of the biggest sites on the internet lost most of its AI search presence between two Tuesdays.
I keep turning that over. My blog’s ChatGPT referrals are a rounding error next to Reddit’s, but the mechanism is identical. Whatever visibility any of us have in AI search is sitting on top of model behavior that can be reshuffled overnight, without an announcement, without a migration guide, without a deprecation window.
What llm seo actually means, minus the hype
So what do you do about it? This is where I’m supposed to sell you a GEO checklist. I’m not going to, because most of what’s sold as llm seo or generative engine optimization is regular technical SEO wearing a lanyard.
Here’s my honest read on what the site: operator shift rewards. The model is choosing domains it already associates with a topic. That association gets built from training data, from links, from being cited by other sources the model reads. Which means the boring old advice still carries: be the canonical answer for something specific, publish under stable URLs, use headings that state what a section answers, and be linked to by pages that talk about your topic. None of that is new. What’s new is the payoff structure. Search rankings degrade gracefully; position three still gets clicks. Model source lists look more binary. You’re consulted or you’re invisible.
There’s one genuinely new implication, and it cuts against classic SEO instinct: broad topical sprawl probably helps less than depth. If ChatGPT scopes queries like site:yourdomain.com when it wants an answer about one subject, being the tenth-best generalist site helps you with nothing. Being the site for one narrow thing gives the model a reason to come to you on purpose. I’d rather own three queries than rent three hundred.
And measurement has to come before tactics. I made this argument about model behavior generally in my post on LLM evaluation tools: if you aren’t tracking how models treat you over time, you’re doing vibes, and vibes fail silently. The Reddit collapse is exactly the kind of thing you only catch with a baseline.
How to tell if ChatGPT is reading your site
You don’t need a tool subscription for the first step. OpenAI’s fetchers identify themselves in your server logs. Grep for them:
# nginx access log, adjust path to taste
grep -E "OAI-SearchBot|ChatGPT-User|GPTBot" /var/log/nginx/access.log \
| awk '{print $7}' | sort | uniq -c | sort -rn | head -20
OAI-SearchBot is the search crawler, ChatGPT-User shows up when a live response fetches your page, and GPTBot is training collection. The middle one is the interesting signal: it means your page got pulled into an actual answer for an actual person. When I ran this on my own logs, the pages ChatGPT fetches were not the pages I would have guessed, and two of my highest-Google-traffic posts had never been fetched at all. Google’s picture of my site and the model’s picture of my site are different pictures.
Second step, five minutes: ask ChatGPT the three questions your best posts answer, with search enabled, and see who it cites. If it’s citing a worse article than yours, you’ve learned where you stand. I do a chunk of my consulting work around exactly this kind of unglamorous measurement, and the pattern never changes: people optimize for a year before spending the afternoon it takes to find out where they actually are.
This week’s homework
Run the grep above on your access logs and write down three numbers: how many OAI-SearchBot hits, how many ChatGPT-User hits, and which page got fetched most. Then re-run it in a month. That’s it. That’s a baseline, it costs nothing, and when the next silent model-side change reshuffles who gets read, you’ll be one of the few people who can see it happen in their own data instead of finding out from someone else’s dashboard.