Crawlers and JavaScript
A browser runs your site’s JavaScript before a visitor sees the page. Most crawlers do not: they read the HTML your server sends and stop there. If a page’s headings, text or links only appear after JavaScript runs, those crawlers never see them.
This page lists the crawlers that matter for search and AI visibility and whether each one runs JavaScript. Asky uses the same split in a technical audit, described below.
The list
| Crawler | Operator | Feeds | Runs JavaScript |
|---|---|---|---|
| Googlebot | Google Search, AI Overviews, Gemini answers that use Google’s index | Yes. Googlebot renders pages with an up to date Chromium, often in a second pass after the first fetch. | |
| Bingbot | Microsoft | Bing, Copilot answers that use Bing’s index | Yes, with limits. Microsoft renders with an up to date Edge, and recommends that JavaScript-heavy sites still serve content in the HTML. |
| Applebot | Apple | Siri, Spotlight, Safari suggestions, Apple Intelligence | Yes. Apple documents that Applebot may render pages in a browser, and asks that the resources a page needs stay crawlable. |
| GPTBot, OAI-SearchBot, ChatGPT-User | OpenAI | ChatGPT training, ChatGPT search, pages ChatGPT opens for a user | No. They read the served HTML. |
| ClaudeBot, Claude-SearchBot, Claude-User | Anthropic | Claude training, Claude search, pages Claude opens for a user | No. They read the served HTML. |
| PerplexityBot, Perplexity-User | Perplexity | Perplexity answers and the pages it opens for a user | No. They read the served HTML. |
| Meta-ExternalAgent, Meta-ExternalFetcher | Meta | Meta AI | No. They read the served HTML. |
| Bytespider | ByteDance | TikTok and ByteDance models | No. It reads the served HTML. |
| CCBot | Common Crawl | The open web archive many models train on | No. It reads the served HTML. |
| AskyBot | Asky | Your monitoring in Asky | No by default. A technical audit can render pages on request, see below. |
The “No” rows come from independent measurements of what these crawlers fetch and execute, most recently Vercel’s study of AI crawler traffic from December 2024, and from the operators’ own crawler documentation, none of which claims rendering. The “Yes” rows come from the operators’ documentation: Google’s JavaScript SEO basics , Microsoft’s Bing Webmaster guidance on JavaScript and Apple’s About Applebot . Operators change their crawlers; this list was last reviewed in October 2026.
What it means for your site
If your pages are server-rendered, every crawler in the table reads the same content a visitor sees, and there is nothing to do.
If parts of a page exist only after JavaScript runs, the three crawlers that render will see them and the rest will not. For the AI assistants Asky monitors, that content is invisible: ChatGPT, Claude and Perplexity never read it. The usual fix is to serve the content in the HTML, through server-side rendering, static generation or a pre-rendering step, so every reader gets it.
What this means for your audit
A technical audit in Asky reads your pages the way most crawlers do: the HTML your server sends, without JavaScript. That is the honest view of what ChatGPT, Claude and Perplexity read.
The audit’s Configure Scan step has a switch, Prefer JavaScript rendering. With it on, Asky reads each page as a browser renders it, which is what people visiting your site and the rendering crawlers in the table see. A page that cannot be rendered, for example because the site answers the browser with a bot challenge, is read without JavaScript instead, and the audit says how many pages that happened to and why. Each rendered page counts as two pages against your audit allowance, and the crawl takes longer because every page is visited twice. The sitemap step shows the estimated time before you start.
AskyBot itself never runs JavaScript. Rendering happens through a separate browser service that fetches the page once more, so your firewall sees a browser visit in addition to AskyBot’s request. See AskyBot for how to allow it.