Robots.txt
Your robots.txt file tells search engines and AI crawlers what they may crawl. These issues cover a missing, broken or overly strict file.
- AI crawlers blocked in robots.txt: Your robots.txt blocks one or more AI crawlers from the whole site, which can keep your pages out of AI answers or out of model training.
- robots.txt blocks Bingbot from the whole site: Your robots.txt stops Bingbot from crawling any page, which keeps your site out of Bing results and the Copilot answers built on Bing’s index.
- robots.txt blocks every crawler from the whole site: Your robots.txt tells all crawlers, including Googlebot and Bingbot, not to crawl any page, which can take your site out of search and AI answers.
- robots.txt blocks Googlebot from the whole site: Your robots.txt stops Googlebot from crawling any page on your site, so Google cannot read your content for Search or its AI features.
- robots.txt contains lines crawlers cannot read: Some lines in your robots.txt are not valid rules, so crawlers skip them and the file may not block or allow what you intended.
- robots.txt does not list your sitemap: Your robots.txt has no Sitemap line, so crawlers that read it first get no pointer to the list of pages you want found.
- robots.txt file is empty: Your robots.txt file exists but is blank, so it cannot point crawlers to your sitemap or give them any crawl rules.
- robots.txt file not found: Your site has no robots.txt file, so crawlers get no crawl rules and no pointer to your sitemap in the first file they request.
- robots.txt returns a server error: Requests for your robots.txt fail with a 5xx server error, which can make search engines stop crawling your whole site for a while.
Last updated on