[Verse] Overview of Open AI Crawlers Open AI uses web crawlers (“robots”) and user agents to perform actions for its products, either automatically or triggered by user request. Open AI uses the following robots dot TXT tags to enable webmasters to manage how their sites and content work with AI. Each setting is independent of the others – for example, a webmaster can allow OAI Search Bot to appear in search results while disallowing GPT bot to indicate that crawled content should not be used for training Open AI’s generative AI foundation models. For search results, please note it can take ~24 hours from a site’s robots.txt update for our systems to adjust. [Verse] OAI Search Bot OAI Search Bot is for search. OAI Search Bot is used to link to and surface websites in search results in the Search GPT prototype. It is not used to crawl content to train Open AI’s generative AI foundation models. To help ensure your site appears in search results, we recommend allowing OAI Search bot in your site’s robots.txt file and allowing requests from our published IP ranges . [Verse] Chat GPT-User Chat GPT-User is for user actions in Chat GPT and Custom GPTs. When users ask Chat GPT or a Custom GPT a question, it may visit a web page to help answer and include a link to the source in its response. Chat GPT users may also interact with external applications via GPT Actions. Chat GPT User governs which sites these user requests can be made to. It is not used for crawling the web in any automatic fashion, nor to crawl content for generative AI training. [Verse] GPT Bot GPT Bot is used to make our generative AI foundation models more useful and safe. It is used to crawl content that may be used in training our generative AI foundation models. Disallowing GPT Bot indicates a site’s content should not be used in training generative AI foundation models. [outro]
