Research checked 5 October 2026. Practical recommendations below are Divel’s editorial guidance; examples are illustrative.
A business may want its public pages discoverable in AI search while making a different decision about model training. Those goals should be discussed separately before someone copies a blanket crawler-blocking rule into the website.
Search access and training are separate controls
OpenAI’s crawler documentation distinguishes OAI-SearchBot, used for search discovery, from GPTBot, associated with potential training use. The controls are independent. Its publisher FAQ also explains that ChatGPT referral links include utm_source=chatgpt.com for analytics. Allowing access does not guarantee a citation or traffic. Crawler documentation and publisher FAQ.
Start with an access inventory
Our recommended audit records what the business wants discoverable, the current rules and who owns changes. Include public service pages, articles and pages that should remain excluded. Then ask the website maintainer to compare that intent with the live configuration.
- Review robots.txt rules by crawler rather than relying on a generic template.
- Check whether hosting, firewall or security settings interfere with intended public access.
- Inspect important pages for unexpected noindex directives or unavailable content.
- Document the configuration before changing it, with a rollback plan.
Make pages useful after discovery
Technical access is only the starting point. A discovered page still needs accurate information, clear authorship where appropriate and a useful next step. For a service business, that can mean explaining scope, locations served, project inputs and how to request a conversation.
Review referral quality
Create an analytics view for identifiable ChatGPT referrals and examine landing pages and meaningful actions. Treat it as a partial view: attribution can be affected by consent choices, browser behaviour and the measurement setup. Do not assume every AI-influenced customer journey will carry a referral label.
An illustrative audit finding is a strong article that receives visits but few enquiries. Before commissioning another batch of content, check whether its topic fits the business, its contact path is clear and the enquiry form works.
Make the policy deliberate
Have marketing and the website owner agree the discovery goal, then let the maintainer implement it carefully. Review the result after deployment. Crawler settings should reflect the business’s policy, rather than an accidental side effect of an old template.
Need help applying this to your website? Explore Divel’s relevant service or discuss your priorities.