We Examined the AI Crawl Policies of Hosting Companies. The Industry Is Mostly Silent.
The hosting industry has spent 2026 talking about an “agent-ready web.” Vendors advertise crawler-level controls, AI-aware infrastructure, and machine-readable publishing standards. The messaging suggests a mature ecosystem where companies actively manage how AI systems interact with their content. To verify that assumption, we stopped looking at marketing and inspected the actual machine-readable declarations these companies