We Audited 50 Sites for AI Visibility — What We Found

We re-ran our machine-readable AI visibility audit on the same fifty B2B websites on October 10, 2026, six weeks after the first pass. Still nobody blocks the AI crawlers: not one site turns away GPTBot, ClaudeBot or PerplexityBot. What did move is the boring part. Homepages with no structured data at all fell from sixteen to twelve, and the count clearing our four-point hygiene test went from fifteen to twenty.

Updated October 2026: three things are new since this first published. We re-ran the whole audit, so every figure below now comes in pairs and you can see what six weeks of quiet engineering work looks like. Cloudflare changed its default on September 15, which gives the "nobody blocks" finding an expiry date. And a much larger study of llms.txt landed, covering 137,000 domains instead of 900, pointing the same way the small one did.

What We Measured, and How

We picked fifty companies across four groups: twenty marketing and sales software firms, ten operations, HR and fintech platforms, ten developer-infrastructure and payments companies, and ten professional-services firms. All are established brands with real marketing budgets. If anyone has had the time and money to prepare for AI search, it is this cohort.

For each domain we check four things a machine can verify without judgment calls. First, the robots.txt file, and specifically what it says about thirteen named AI crawlers — among them GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and Bytespider. Second, whether the site serves a real llms.txt, the plain-text summary file some teams now publish for language models. Third, the JSON-LD structured data on the homepage: does it declare an Organization, a WebSite, a breadcrumb trail, an FAQ. Fourth, two pieces of ancient on-page hygiene that still matter — a single h1 heading and a meta description.

The first run was August 27, 2026. The second was October 10, forty-four days later, same script, same domains. Both raw data sets are published alongside this piece — the August run and the October run — so anyone can check the arithmetic or re-run it.

Finding One: Nobody Is Blocking, and the Default Is Moving Anyway

In August, forty-five of the fifty served us a readable robots.txt. Of those forty-five, zero blocked GPTBot. Zero blocked ClaudeBot. Zero blocked PerplexityBot, OAI-SearchBot, ChatGPT-User or Google-Extended. One site turned away CCBot, Bytespider and Meta's agent — all three refusals came from the same company, a sales-intelligence platform whose own product depends on scraping. That was the entire resistance.

Six weeks later the picture is identical. Forty-four sites served robots.txt, the same lone holdout blocks the same three bots, and the frontier crawlers still walk in everywhere. Only ten of the forty-four mention any AI crawler by name, exactly as before. The other thirty-four say nothing about them, so those bots inherit whatever the wildcard rule allows — and in every single case, the wildcard lets them through. Most companies got to a wide-open default by accident rather than by decision.

Here is why that may not hold. Cloudflare announced in July that from September 15, 2026, new customers, new sites from existing subscribers and free-tier accounts would default to allowing search crawling while blocking training and agent crawlers on pages carrying ads, unless the owner opts out. Crawlers that bundle search with training and agent work, giving owners no way to separate the two, get blocked on those pages as well. Engadget covered the announcement, including the rename of Pay Per Crawl to Pay Per Use, which pays publishers when their content shows up in an AI answer rather than when a bot fetches the page.

None of our fifty is a new free-tier site, which is exactly why their robots.txt files have not budged. But the floor is being raised underneath the rest of the web by a vendor rather than by a policy debate. Decide your position on AI crawlers before your CDN decides it for you.

One more wrinkle. Six sites never gave us their robots.txt: four returned a 403 to automated requests, one a 404, and one large consultancy stopped answering automated requests altogether between the runs, having previously served both robots.txt and llms.txt without complaint. A site that refuses robots.txt to a non-browser client is making a bet about which visitors matter, and that bet now includes the crawlers behind AI answers.

Finding Two: The llms.txt Files Are Getting Bigger and No More Useful

Thirty-three of the fifty serve a genuine llms.txt, down one from August. The median file grew from about 15KB to about 17KB and the median link count from roughly 85 to 93. The tails grew far harder than the median: one developer-tools company took its file from 3.7KB to 244KB in six weeks, a consultancy went from 5KB to 156KB, and a sales platform grew its index twentyfold. The largest in the sample is a 2.3MB file listing over nine thousand links. One site still returns an HTML page at /llms.txt, which looks like a file to a casual check and is worthless to a parser.

The one genuine improvement: every file we found in October contains links. The prose-only files from August, which defeated the whole purpose, are gone. If you are deciding whether to build one, our primer on what llms.txt is and whether your site needs one walks through the format itself.

Set that against the wider web. SE Ranking crawled roughly 300,000 domains and found llms.txt on 10.13 percent of them, work covered by Search Engine Journal in November 2025. Our cohort adopts the file at more than six times the rate of the general web.

Here is the awkward part, and the evidence got much heavier since we first wrote this. Ahrefs looked at all 137,210 domains in its analytics product that received traffic in May 2026. Twenty-eight percent published an llms.txt, about 38,000 files. Ninety-seven percent of those files were requested zero times that month. Among the requests that did arrive, AI retrieval bots accounted for 1.1 percent — 233 requests in total — while SEO audit tools were the largest single category at 21.7 percent. Ahrefs notes its own customers skew technical, so treat the adoption figure as an upper bound.

That supersedes the smaller server-log study we cited before, which covered around 900 domains and found 1,227 requests for llms.txt files, not one from a frontier lab's crawler. Same finding, two orders of magnitude more data. Google has said its Search systems ignore llms.txt. No major model provider has committed to reading it in production answers. The file is being adopted enthusiastically and read by almost nothing.

Finding Three: The Structured Data Gap Is Closing

We fetched forty-nine homepages in both runs; the same consultancy refuses automated requests to its homepage.

In August, sixteen of those forty-nine — nearly a third — contained no JSON-LD structured data of any kind. In October it is twelve. Declared Organization blocks went from thirty-one to thirty-five, WebSite from sixteen to eighteen. Breadcrumb trails held flat at five. FAQ markup held flat at four, though the membership churned: one sales platform dropped its FAQ block while a competitor added one.

Five sites went from no structured data at all to a declared Organization, and each fixed more than one thing at once. The sales-intelligence platform added JSON-LD, an Organization block, a WebSite block, FAQ markup and its first h1, and doubled its llms.txt, all inside six weeks. That is not drift. That is somebody working through a checklist.

The regression is just as instructive. One of the Big Four accounting firms lost its structured data and its meta description between runs, failing both checks it had passed. Hygiene is not a project you finish. It is a thing a redesign can quietly take away from you.

Finding Four: The Fundamentals Improved Faster Than the Fashionable Stuff

Nine of the forty-nine homepages had no h1 at all in August. Not a wrong h1 — none. In October that is five. Homepages carrying exactly one h1 went from thirty-four to forty: six sites that shipped zero, two or four h1 tags now ship exactly one.

Six homepages carried more than one h1 in August; now four do. A well-known email platform still ships six of them on a single page, and one consultancy went the wrong way, from four to five.

Eleven had no meta description in August. Now nine do. Among the ones that fill the field, lengths still run from six characters to 201, which tells you how little thought it gets even where somebody remembered it.

Think about what an extraction system does with that. It arrives, looks for the clearest statement of what this company is, and finds a hero animation with no heading and no summary. It can still read the rendered text, but you have removed every signal you controlled and left the machine to guess.

Finding Five: Twenty of Fifty Now Clear a Low Bar

We set a deliberately modest test. Does the homepage declare an Organization, carry exactly one h1, include a meta description, and serve a working llms.txt? Four things, all of them a morning's work.

Fifteen of fifty passed in August. Twenty pass now. Drop llms.txt from the test and the count goes from twenty-seven to thirty-two. The direction is right and the absolute number is still poor: three fifths of a cohort of well-funded brands fails a check any competent developer could clear before lunch.

What Six Weeks of Data Actually Shows

The obvious read is "add schema, add llms.txt, win AI search." We do not believe that, and the evidence does not support it.

Ahrefs tracked 1,885 pages that added JSON-LD markup and reported in May 2026 that AI citations barely moved: a 2.4 percent change in AI Mode and 2.2 percent in ChatGPT, both indistinguishable from noise, alongside a small decline in AI Overviews. The llms.txt research points the same way, harder than before. If you are hoping a markup sprint will make ChatGPT recommend you, the data says no.

What the two runs together show is more useful than either alone. The signal everybody talks about did not move: llms.txt adoption dropped by one file while the files themselves bloated. The signals nobody writes conference talks about — one heading, one description, one Organization block — improved on eight separate domains in six weeks, against three that went backwards. Teams are quietly fixing the cheap, verifiable things, which is the correct order of operations even if it never makes a headline.

These signals are not levers, they are hygiene — and hygiene is how you find out whether the underlying facts exist at all. A team that cannot produce an Organization block usually cannot answer the question underneath it: what do we do, who exactly for, and what makes us the right call. The markup is downstream of clarity.

The stakes are real even if the fix is unglamorous. Ahrefs compared 300,000 keywords against aggregated Search Console data and found that where an AI Overview appears, the top organic result takes about 58 percent fewer clicks than the trend without one would predict. They are careful to call that a correlation, and it covers desktop only. Pew Research Center tracked the browsing of over 900 US adults and found that when a Google AI summary appeared, people clicked a search result 8 percent of the time, against 15 percent when no summary was present. They clicked a link inside the summary in 1 percent of visits. The traffic does not come back. Being described accurately inside the answer is what is left.

That is why our own AI visibility audit starts with the claims on the page, not the markup around them. If a buyer cannot extract your pricing model, your specialty and your proof from plain text, no structured data will rescue you. Structure a page so the answer comes first and the machine has less room to guess.

How to Run This on Your Own Site This Afternoon

Pull your robots.txt and read it properly. Find out whether your position on AI crawlers is deliberate or inherited. Both are defensible; not knowing which you have is not. Check your CDN's bot settings in the same sitting, because that is where the real enforcement now happens.

Fetch your own /llms.txt and confirm it returns plain text rather than an HTML error page. If you do not have one, that is a reasonable choice on current evidence, and lower priority than everything else here.

View source on your homepage and search for application/ld+json. Confirm there is an Organization block and that its details match your footer, your Google Business Profile and your LinkedIn page. Contradictions between those sources are a genuine problem.

Count your h1 tags. You want one, and it should say what you do rather than what you feel. Read your meta description out loud: if it does not name the buyer and the outcome, rewrite it.

Put the whole check on a calendar. Three of our fifty went backwards on at least one signal in six weeks, which is the most practical finding of the re-run — a passing score is a snapshot, not a state.

Then do the part that matters more than all of it: open your five highest-intent pages and check whether a stranger could extract your specialty, your pricing shape and one piece of verifiable proof without scrolling. Our guide to getting cited by ChatGPT, Perplexity and AI Overviews covers that process in detail, and our primer on generative engine optimization explains where it sits inside search strategy.

What This Study Does Not Show

We measured machine-readable eligibility, not outcomes. We did not ask any assistant whether it recommends these brands, and a site can score badly here and still get named constantly because its reputation is built elsewhere. Reputation beats markup, every time.

We also measure declared policy, not enforcement. A robots.txt that welcomes GPTBot tells you nothing about whether a CDN rule turns that bot away at the edge, and the 403 responses we collected show that some of these sites filter non-browser clients well before robots.txt enters the conversation. The Cloudflare change widens that gap rather than narrowing it. Reading robots.txt is the cheapest check available, and it is not the same as watching your logs.

This is also two points in time, one page per domain, and one requesting client. Six weeks is long enough to catch a release cycle and far too short to call a trend. A fuller audit would test many pages, repeat monthly, and compare against actual assistant answers.

We publish the limits because a study that only flatters its own method is marketing, not research.

Frequently Asked Questions

Should we block AI crawlers?

For most B2B companies, no. Blocking removes you from the systems buyers increasingly use to build shortlists, and both runs show your competitors are not blocking either. What changed is that your CDN may now make this call for you by default, so the real task is to make it deliberately and write it down.

Do we need an llms.txt file?

On current evidence it is optional, and the evidence hardened while we were watching. Ahrefs found that 97 percent of the llms.txt files it surveyed across 137,000 domains went unrequested for a full month. Build one if it is cheap. Do not treat it as a strategy.

Does schema markup get us cited?

Not on its own. Ahrefs found little citation movement from adding markup. Schema helps machines parse facts you have already made clear, which is worth doing for its own sake.

Which fix gives the most return?

Answer-first writing on the pages that carry buying intent. Lead each page with a direct statement of what you do and who it serves, then support it with specifics a system can quote.

Did anything actually improve between the two runs?

Yes, and only in the unglamorous places. Five homepages gained structured data and one lost it. Four gained a first h1. Three gained a meta description and one dropped one. The four-point pass rate went from fifteen of fifty to twenty, and llms.txt adoption fell by one.

How often should we re-run this check?

Quarterly for the technical signals, and after every significant site release. One site in our fifty lost its structured data in six weeks, so re-running is as much about catching regressions as measuring progress.

Where does this fit in a wider marketing audit?

It belongs alongside demand, positioning and channel economics rather than in a silo. Our marketing audit covers the technical layer and the commercial one together, because a site that reads well to a model and badly to a buyer has solved the wrong problem.