Case Study #6 - 603 Crawls From Four Operators

Week four is the quiet one. After thirteen AI operators crawled Home Crew USA in week three, four showed up between October 29 and November 3, 2026. Their combined traffic fell from 1,464 requests to 603, a drop of 58.8 percent, while every other kind of request grew enough to push the site’s total from 9,716 to 10,014.

That is the whole story in one line, but the details are more useful than the headline. The bots that left did not leave because the site broke. They left because their crawl was finished, and the ones that stayed kept reading pages without trouble.

The daily shape

Requests peaked on November 2 at 3,020 and bottomed on November 1 at 762. The AI share never exceeded 163 requests in a day, and on the quietest day it was 53.

The 3,020 spike on November 2 came mostly from non-AI clients. That day the site served 1,183 pages, sent 589 redirects, and missed 941 times. The AI crawlers contributed 92 requests to it.

Four operators, one of them loud

Amazonbot made 405 requests, or 67.2 percent of all AI crawler traffic. It got a page 402 times and missed 3 times. That is a 99.3 percent success rate, and it is the second week in a row that Amazonbot has been the largest single crawler.

Googlebot sent 127 requests. It received 88 pages, 31 redirects, and 8 misses. OpenAI’s three agents, ChatGPT-User, OAI-SearchBot, and GPTBot, made 51 requests and every one returned 200. Bingbot made 20 requests, all 200.

Served means a 200 response. The 31 Googlebot redirects were the www and http variants pointing at the canonical host, not misses.

Across the four operators, 561 of 603 requests returned 200. That is 93.0 percent, up from 36.5 percent in week three, when most of the fleet was still probing for files that were never there.

The nine that stopped

Week three had thirteen operators. Week four had four. The absentees: Anthropic, Perplexity, ByteDance, Baidu, Common Crawl, Cohere, Meta, You.com, and Apple. Together they made 608 requests in week three, and zero in week four.

A launch crawl behaves like this. A site gets linked, discovered, and swept by a wide set of crawlers in a short window, then the set narrows to the operators that keep a continuous index. Nothing in the week four data suggests the site was penalized or blocked. The robots file did not change, the pages did not move, and the crawlers that returned were served normally.

If you want the week three numbers for comparison, Case Study #5 has the full breakdown of the 1,464 requests and the 764 config-file misses.

What the misses look like now

Misses rose from 912 to 2,514. That sounds bad until you see what they are. Of the 2,433 404s, 1,987 ended in a file extension. Only 446 looked like page requests.

The wlwmanifest pattern is a WordPress client asking every possible install path whether the site is a WordPress blog. The doubled slash comes from a probe that joins a base URL with a path starting with a slash. Twelve install-path guesses account for 261 requests: //shop at 30 and eleven others at 21 each. Adding xmlrpc.php and wp-login.php brings the WordPress family to 301.

The new entry is /.env.production at 46 requests. That is a Laravel and Node convention. This site is static Astro, so the file does not exist and never will, but scanners keep asking. I confirmed all four of these paths still return 404 on the live site today.

The redirect volume is worth a note. 2,308 requests got a 301 and 331 got a 308. Both come from the site’s own canonical rules: http to https, www to apex, and a trailing slash on service pages. I checked each of those with curl, and they point where they should.

The QlyzeBot burst

One crawler did not fit the AI pattern. QlyzeBot made 924 requests on October 29 and November 2, split 468 and 456. It received 923 pages and one redirect, read 3.7 MB, and skipped the other four days entirely.

That is 9.2 percent of the site’s six-day traffic, delivered in two bursts, from a crawler that is not in the AI operator list. It looks like an analytics or audit crawler doing a scheduled sweep. If it keeps appearing, it belongs in its own bucket rather than the AI totals.

What this means if you care about AI search

Concentrated beats broad. Four operators made 603 requests and missed 11 times. The wide sweep in week three made 1,464 and missed 912, mostly because it was hunting for files. A narrow crawl of real pages is the outcome you want.

The site answered 93 percent of AI requests this week. That is the number I would track. Total AI requests will swing with launch crawls and seasonal sweeps, and a single week’s drop says little on its own.

For the sitewide picture, including the 444 crawls and 272 dead ends on nonsenseo.com, the November 3 crawl report covers the same window from the other domain.

What I am not claiming

I did not measure rankings, citations, or referrals. This is request data from one zone, filtered to eyeball traffic, and nothing more. A crawler fetching a page is not the same as a model using it.

I also did not count QlyzeBot as an AI crawler, and I did not count the 301s as failures. If you define either differently, the percentages move.

FAQ

Did AI crawlers stop visiting Home Crew USA? Not all of them. Four operators still crawled the site between October 29 and November 3, 2026, led by Amazonbot at 405 requests. What stopped was the long tail: nine of the thirteen operators seen in week three sent nothing in week four.

Why did 2,433 requests return a 404? Most of them were never pages. 1,987 of the 2,433 misses ended in a file extension, which means the client asked for something like wlwmanifest.xml or .env.production rather than a URL on the site. WordPress path guesses alone accounted for 301 requests.

What is /.env.production and why does it get 46 hits? It is a Laravel and Node convention for environment files. Home Crew USA is a static Astro site with no such file, so every request for it returns 404. Scanners ask anyway, and it was the single most requested missing path this week.

Does a 58.8 percent drop in AI crawler traffic hurt AI search visibility? Not by itself. Week three carried a launch crawl from thirteen operators. Week four kept the four that index continuously. The pages still resolve, and the crawlers that matter for answers still fetched them.

What is QlyzeBot? A crawler that identified itself as QlyzeBot made 924 requests on October 29 and November 2, all but one answered with 200. It read 3.7 MB in two bursts and did not appear on the other four days.