robots.txt blocks Bingbot from the whole site

What it is

The rules Bingbot follows in your robots.txt disallow the entire site. Bingbot obeys a group that names it, such as User-agent: Bingbot, or the User-agent: * group when none does. Sites sometimes block it on purpose to reduce crawl load, and sometimes by accident through a group copied from another site.

Why it matters

Bingbot builds the Bing index. While it is blocked, Bing cannot read new or changed pages, and by Bing’s own account  that index serves its AI answers in Copilot as well as its search results. Other services that draw on Bing’s results are likely to miss your pages too.

How Asky checks it

Asky reads your robots.txt during its daily site discovery and at the start of each technical audit. It collects every group whose User-agent is Bingbot (in any letter case), or the * groups when none is, and reports the issue when those rules disallow two sample paths inside the site, /a and /z9/page. The longest matching rule wins, Allow on a tie. Groups for other Microsoft crawlers are not read, and a block on some paths only is not reported. When the * group and Googlebot’s rules block the site too, the all-crawlers issue is raised as well.

Reported as a warning with medium severity.

How to fix it

1. Open https://example.com/robots.txt for your own domain. Find the group naming Bingbot, or the User-agent: * group if there is none, and its Disallow: / or Disallow: /*.
2. Remove that rule, or narrow it to the paths Bing should skip.
3. A group naming Bingbot replaces the * group for it completely, so copy into it any * rules you still want it to follow.
4. If the block was there to reduce load, slow Bingbot down instead: Bing Webmaster Tools lets you set its crawl rate by the hour .
5. To leave it as it is: if you block Bing deliberately, mark the issue resolved.

Example

A group on https://example.com shuts out Bing only. The fix drops it so Bingbot follows the shared rules:

# Before
User-agent: Bingbot
Disallow: /

User-agent: *
Disallow: /checkout/

# After
User-agent: *
Disallow: /checkout/

Sitemap: https://example.com/sitemap.xml

← Back to Robots.txt