Case Study #4 - 1,504 Crawls, 292 Wrong Turns, Zero 5xx

Home Crew USA has now been live for nine days. Week one delivered 460 pages, 458 of them in Google, and 1,199 AI crawler requests without a single error. Week two asks the boring follow-up with a useful answer: what does the site look to crawlers once the launch stampede ends?

This is case study four. Case study #1 built nonsenseo.com, case study #2 tracked its first week, and case study #3 proved the pattern repeats on a second domain. This one stops being about launch day and starts watching the maintenance regime, mistakes included.

Short answer: quieter, deeper, and honest about its misses. Every number below comes from the same public artifacts as before.

Six quieter days

Between October 17 and 22 the site served 16,593 requests. AI crawlers sent 1,504 of them, 9.1 percent, about 4.8 MB of responses. The total stayed between 1,271 and 4,290 per day, the same band as launch week.

The AI line tells the real story: 620 requests on the 17th, a midweek floor of 59 and 60, a 510 spike on the 21st, then 94 to close. The launch sweep from case study #3 is finished. What remains is maintenance crawling, and it arrives in bursts, not steady streams.

The leaderboard shuffled

Every operator from week one came back. The order did not survive.

Meta’s external agent went from a footnote to first place with 456 requests and zero failures. Amazonbot kept its schedule. Googlebot cut its volume in half and, for the first time, logged failures, 68 of them. ClaudeBot cooled from 523 to 137 now that the initial index exists.

The also-rans tell the sharper story. Applebot, Bytespider, and CCBot turned in nothing but misses. OAI-SearchBot missed on two thirds of its 97 visits. One week of volume leadership is a snapshot, not a ranking.

Where the 292 misses went

Of the 1,504 AI responses, 292 were 4xx: 280 404s and 12 403s. Not one 5xx. The misses spread across 137 paths, and the repeat offenders read like a museum of stale assumptions: /userfiles, requested 16 times and referenced nowhere on the site, plus /fly.toml, /.env.mail, /vercel.json, /api/console/api_server, and /press/ and wp-json guesses from crawlers expecting an old WordPress install. Another 131 hits went to dot paths like /.boto and /.ssh/known_hosts.

Crawlers treat unknown domains like neglected servers with forgotten deploy configs. This one answers every probe with a clean 404 at static-page cost. That is the maintenance regime working, not breaking.

What they read once the sweep ended

robots.txt, 116 times counting a redirect, and sitemap-index.xml, 69 times, still open every crawl. Then real pages: the homepage 31 times, moving labor in Calabasas 11, lawn care in Whittier 11.

Depth is the headline. AI hits touched 602 distinct paths in six days, up from 248 in three. The fleet graduated from homepage and sitemap to the long tail of city pages.

Check it yourself

curl -s https://homecrewusa.com/sitemap-0.xml | grep -c "<loc>"
curl -s https://homecrewusa.com/sitemap-0.xml | grep -o "<lastmod>[^<]*</lastmod>" | sort | uniq -c
curl -s https://homecrewusa.com/userfiles -o /dev/null -w "%{http_code}\n"
curl -s https://homecrewusa.com/llms.txt | head -20

The sitemap still lists 460 URLs. The userfiles probe answers 404, exactly as the numbers above say. Same artifacts, same receipts, one week later.

What we won’t tell you

How the pages get generated and refreshed. Which retrieval experiments run in the background. What the domain earns and what it costs. The engagement buys the outcome; the kitchen stays closed.

The misses are in here on purpose. A case study with only wins is an advertisement. 292 documented 4xx answers on paths that do not exist is evidence that someone actually looked.

FAQ

Did anything break in week two?

No. 1,181 responses were 2xx, 31 were 3xx, and 292 were 4xx, with zero 5xx. Every repeated miss we traced landed on a path the site does not have, like /userfiles or a dotfile config probe.

Who crawled the most?

meta-externalagent, 456 requests without a failure. Amazonbot followed at 440 and Googlebot at 241, down from 550 in week one.

Is attention fading after launch?

Total AI requests rose from 1,199 in three days to 1,504 in six. The daily rate fell from about 400 to about 250 once the launch burst decayed, touching 59 and 60 midweek before a 510 spike. Nine days of data says maintenance crawling, not abandonment.

What changed since Case Study #3?

Same domain, later phase. The operator ranking shuffled completely, crawl depth more than doubled from 248 to 602 distinct paths, and errors went from zero to 292, all 4xx, all explainable.

If you want a site that keeps its receipts after launch week too, that is the service.