Is your firewall blocking the AI bots your robots.txt allows?
It can. On 5 November 2026 Abondance reported that its host had been blocking GPTBot addresses and some PerplexityBot IPs since 25 August, even though robots.txt allowed those bots. After the list was removed, both bots reappeared in the logs. A robots.txt Allow does not guarantee arrival if a firewall, CDN or WAF stops the request first.
- Abondance put
/llms.txtlive on 1 June 2026, andllms-full.txtfrom 13 September. - From 1 June to 4 November 2026 the two files received 1,657 requests; only 10 (0.6%) came from identified AI bots.
- ChatGPT-User opened them twice (8 June and 25 August). GPTBot and PerplexityBot showed 0 requests in the published table.
- Update on 5 November: the host found that since 25 August a firewall / suspicious-IP list had been blocking GPTBot addresses and some PerplexityBot IPs. The list was removed the same day, and both bots reappeared in the logs.
- robots.txt allowed those bots; the site worked normally. A blocked bot causes no error on the site side — it simply stops coming.
- These findings are from one site. A request is not proof that content was used. Absent requests do not prove bots never use llms.txt, and visits do not prove ranking or citation benefits.
What happened
In a 5 November article, Victor Lerat at Abondance shared server logs for the site’s llms.txt files. The first read looked stark: almost no identified AI bots. The same-day update (20h32) explained the gap: after 25 August, a host firewall list had been stopping GPTBot and some PerplexityBot IPs before they could reach the files — even though robots.txt allowed them. Once that list was removed, both bots showed up again. Zeros in the table after 25 August must be read with that limit.
What this means for your site or store
- Check logs for official bot IPs, not only User-Agent strings. Match known GPTBot / PerplexityBot ranges against access and error logs.
- Audit firewall, CDN and WAF lists for “suspicious IP”, bot or rate-limit rules that may block crawlers robots.txt already allows.
- Treat robots.txt Allow as permission, not arrival: a Allow rule does not force the bot onto your server if something upstream drops the connection.
- Use our how-to on checking which AI crawlers can access your site to walk robots.txt, headers and common block points together.
The technical details
robots.txt is a polite signal. Firewalls, CDNs and WAFs enforce network policy. When they disagree, the network wins and the bot never appears in your application logs — so “no visits” can mean “blocked before arrival,” not “bots ignore this file.” Do not generalise one site’s table into industry proof that bots never fetch llms.txt, and do not treat a request as evidence of ranking or citation value.
Want to know whether AI agents can reach your site? Run our free AI check.
FAQ
Sources
- Fichier llms.txt : que disent les logs d’Abondance ?, Abondance (Victor Lerat, 5 Nov 2026)
