Free marketing tool

robots.txt Generator

Build a robots.txt in seconds — with per-bot control over every AI crawler, a Next.js app/robots.ts export, and a tester that tells you whether a given path is actually blocked. No signup.

Site rules

Add a Sitemap line

Points crawlers at /sitemap.xml on the URL above.

Include explanatory comments

Adds # comment lines so the next person understands the file.

Write explicit Allow groups for permitted AI bots

Careful: a bot with its own group ignores the * group entirely, including your Disallow lines.

Block the entire site (staging)

Emits Disallow: / for every crawler. Not a security control — the file is public.

AI crawler control panel

7 of 11 blocked

Training bots feed model updates. Retrieval bots fetch or index pages so an assistant can answer with your content and link back to you. They are worth treating differently.

GPTBotTraining

Crawls pages to build training data for OpenAI models. Blocking it does not remove you from ChatGPT search.

Blocked
OAI-SearchBotRetrieval

Builds the index behind ChatGPT search. Retrieval only — what it crawls is not used to train models.

Allowed
ChatGPT-UserRetrieval

Fetches one page live when a ChatGPT user or a GPT action follows a link. Never used for training.

Allowed
ClaudeBotTraining

Crawls pages that may be used to train Anthropic models.

Blocked
Claude-UserRetrieval

Fetches a page live when a Claude user asks about it. Retrieval, not training.

Allowed
CCBotTraining

Builds the public Common Crawl archive, which most AI labs use as raw training data.

Blocked
PerplexityBotRetrieval

Indexes pages so Perplexity can cite them in answers. Blocking it removes you from those citations.

Allowed
Google-ExtendedTraining

Not a crawler — a token that opts your pages out of Gemini training. No effect on Googlebot or rankings.

Blocked
Applebot-ExtendedTraining

Not a crawler — opts you out of Apple Intelligence training. Applebot still crawls for Siri and Spotlight.

Blocked
BytespiderTraining

Crawls for ByteDance model training. Widely reported to ignore robots.txt, so treat this as a request.

Blocked
Meta-ExternalAgentTraining + retrieval

Crawls for Meta AI training and for indexing content inside Meta products.

Blocked
# robots.txt for https://example.com
# Generated with MarketingDB — /tools/robots-txt-generator

# All other crawlers
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /cart
Disallow: /checkout
Disallow: /*?utm_

# OpenAI — model training
User-agent: GPTBot
Disallow: /

# Anthropic — model training
User-agent: ClaudeBot
Disallow: /

# Common Crawl — model training
User-agent: CCBot
Disallow: /

# Google — model training
User-agent: Google-Extended
Disallow: /

# Apple — model training
User-agent: Applebot-Extended
Disallow: /

# ByteDance — model training
User-agent: Bytespider
Disallow: /

# Meta — model training and product indexing
User-agent: Meta-ExternalAgent
Disallow: /

Sitemap: https://example.com/sitemap.xml

Save this as public/robots.txt — it is served at yoursite.com/robots.txt with no build step.

Test a URL against these rules

Applies real robots.txt precedence: the most specific user-agent group wins, then the longest matching pattern, and Allow beats Disallow on a tie.

Blocked — GPTBot may not crawl /admin/secret

  • Matched the User-agent: GPTBot group — this group overrides the wildcard group completely.
  • Winning rule: Disallow: / — the longest matching pattern in that group.

How a robots.txt file works

A robots.txt file is a plain text file at the root of a host that tells crawlers which URLs they may request. It is built from groups. Each group opens with one or more User-agent lines naming the crawlers it applies to, followed by Allow and Disallow lines holding URL path patterns. Two wildcards are supported: * matches any run of characters, and $ anchors the end of the URL, so /*.pdf$ catches every PDF. Paths are case-sensitive.

A few rules that trip people up: the file must live at the root of the host and be named exactly /robots.txt, each subdomain needs its own copy, and Sitemap lines are global — they apply to the whole file rather than to the group above them, so their position does not matter. Google ignores Crawl-delay entirely, though Bing and Yandex still honour it.

Which rule wins: precedence in practice

Crawlers do not read your file top to bottom and stop at the first hit. Two rules decide everything, and the tester above applies both:

  1. One group per crawler. A bot obeys the group whose user-agent matches its own token and ignores every other group, including User-agent: *. This is the trap behind most broken files: give GPTBot its own group with Allow: / and it stops obeying the /admin/ disallows you wrote in the wildcard group.
  2. Longest pattern wins. Inside the chosen group, the most specific match decides — the rule with the longest path pattern. If an Allow and a Disallow tie on length, Allow wins. So Disallow: /blog/ plus Allow: /blog/public/ leaves the public folder crawlable. Anything no rule matches is allowed by default.

robots.txt blocks crawling, not indexing

This is the costliest misunderstanding in SEO, and it is worth being blunt about. Disallow means do not fetch this URL. It does not mean do not list this URL. If another site links to a disallowed page, search engines can still show it in results — bare URL, no description, with a note that no information is available. Your page ends up in the index looking broken.

To keep a URL out of the index you need a noindex meta robots tag or an X-Robots-Tag HTTP header — and the crawler has to be allowed to fetch the page to see it. Blocking a URL in robots.txt and adding noindex to it is self-defeating: the crawler never reads the tag, so the page can stay indexed indefinitely. Pick one. For anything genuinely sensitive, use authentication instead of either.

AI crawlers: training versus retrieval

Lumping every AI bot into one block list costs you traffic. Training crawlers such as GPTBot, ClaudeBot, CCBot and Bytespider collect pages that may end up in a model. You get nothing back from them, which is why blocking them is a defensible default. Retrieval agents are the opposite trade: OAI-SearchBot and PerplexityBot build the indexes behind answer engines, while ChatGPT-User and Claude-User fetch a single page live because someone asked about it. Those visits produce citations and clicks. Blocking them is how sites quietly disappear from AI answers.

Google-Extended and Applebot-Extended are a third category: not crawlers at all, but opt-out tokens. They control whether pages Googlebot and Applebot already fetched can be used for Gemini or Apple Intelligence training. Setting them has no effect on crawling, indexing or rankings in Search.

Be clear-eyed about what any of this buys you. robots.txt is a request, not a fence. There is no enforcement: a crawler can read your file, ignore it, and fetch every page anyway — Bytespider has been widely reported doing exactly that, and any scraper can simply send a browser user-agent string. Directives also do nothing about content already collected or already inside a trained model. If you need a real block, enforce it at the edge with firewall or WAF rules, rate limiting, or bot management on your CDN. Treat the file above as the polite, public-facing half of that policy.

robots.txt for Next.js

Next.js gives you two ways to ship this file, and the generator emits both. Drop a static file at public/robots.txt and it is served verbatim at /robots.txt with no build step. Or add app/robots.ts in the App Router, exporting a default function typed as MetadataRoute.Robots, and Next generates the file for you — which lets you branch on an environment variable and serve a blanket disallow on preview deployments while production stays crawlable. Ship one or the other, never both: two sources for the same route is how a staging disallow ends up live.

robots.txt — frequently asked

Where does robots.txt go, and does it cover subdomains?

It must sit at the root of the host it applies to, at exactly /robots.txt in lowercase — https://example.com/robots.txt. A file in a subfolder is ignored. Every subdomain is a separate host, so blog.example.com needs its own robots.txt; the one on example.com does not apply to it. http and https on the same host are also treated separately.

Does robots.txt stop a page from appearing in Google?

No, and this is the most expensive robots.txt mistake. Disallow stops crawling, not indexing. If other sites link to a blocked URL, Google can still list it, usually with the note that no information is available for the page. To keep a page out of the index you need a noindex meta tag or an X-Robots-Tag header — and the crawler has to be allowed to fetch the page to see it. Blocking a page in robots.txt and adding noindex at the same time cancels the noindex out.

How do I stop ChatGPT and other AI tools from training on my site?

Add a Disallow: / group for each training crawler — GPTBot, ClaudeBot, CCBot, Bytespider and Meta-ExternalAgent — plus the Google-Extended and Applebot-Extended opt-out tokens. The AI crawler panel above does this in one click with the block-training preset. Keep in mind it is a request: honouring robots.txt is voluntary, and content already scraped or already in an existing model is not affected.

Will blocking AI crawlers hurt my SEO?

Blocking training bots does not. Google-Extended is a training opt-out only: it does not change how Googlebot crawls you or how you rank in Search. What does cost you is blocking retrieval bots — OAI-SearchBot, ChatGPT-User, Claude-User and PerplexityBot are how answer engines find and cite you, so blocking them removes you from that referral traffic. That is why the block-training, allow-retrieval preset is the default here.

Is it safe to list private paths in robots.txt?

No. Anyone can read your robots.txt, so a line like Disallow: /internal/2026-acquisition/ is a public signpost to the thing you are hiding. Use robots.txt for crawl budget and duplicate or low-value URLs. Anything genuinely private needs authentication, a noindex header, or both — not a Disallow line.

Next, make your pages legible to the crawlers you did allow with the schema markup generator, or browse all free tools.