robots.txt generator
One small file that tells crawlers where they may go. Easy to write, and easy to accidentally block your whole site with.
Your rules
Start from a preset, then adjust. The file builds itself on the right.
Your robots.txt
How to use this robots.txt generator
- Enter your site address and start from a preset.
- Add the paths to block or allow, and list your sitemaps.
- Choose your AI crawler rules, then copy or download the file and upload it to your root folder.
Who it is for
For site owners and SEO specialists deciding which crawlers can access a site, including those used by AI answer engines.
What robots.txt is, and is not
It is a plain text file at yoursite.com/robots.txt that asks crawlers not to visit
certain paths. Well-behaved crawlers obey it. Badly behaved ones read it as a list of interesting
places to look.
It is not security. Anything genuinely private needs a password. Putting
Disallow: /secret-admin/ in a public file is an advertisement.
The mistake that removes sites from Google
Those two lines block your entire site. They belong on a staging server, and they get copied to production more often than anyone likes to admit. It is one of the most common causes of "we relaunched and lost all our traffic".
Disallow does not mean deindex
This trips up a lot of people. If a blocked page has links pointing at it, Google can still list the URL in results — it just cannot see the content, so it shows a bare link with no description.
To actually keep a page out of search, let the crawler in and use a noindex meta tag.
Blocking it in robots.txt prevents Google from seeing the noindex tag, which does the opposite
of what you wanted.
The AI crawler question
You can block GPTBot, ClaudeBot, Google-Extended and the rest. Whether you should is a real decision, not an obvious one:
- Block if your content is your product and being trained on gives you nothing.
- Allow if you want to appear in AI answers that cite sources. That is becoming genuine referral traffic, and you cannot be cited if you are not readable.
Note these are different bots for different jobs. Google-Extended controls AI training
only — it does not affect your normal Google Search ranking at all.
Keep it boring
For most small sites the ideal robots.txt is three lines: allow everything, and point at your sitemap. Complexity here creates risk without much reward.
Common questions
How do I allow ChatGPT and Claude to read my site?
Make sure robots.txt does not block their crawlers, such as OAI-SearchBot, GPTBot and ClaudeBot. Each company's documentation explains what its crawlers do, so you can choose separately for search and training.
Where does the file go?
The root of your domain, so it loads at yoursite.com/robots.txt. It will not work in a subfolder, and each subdomain needs its own.
Will blocking a page remove it from Google?
No. It stops Google reading the page, but the URL can still appear if other pages link to it. To remove a page properly, allow crawling and add a noindex meta tag.
Does Crawl-delay work?
Google ignores it. Bing and Yandex respect it. If Google is hitting your server too hard, set the crawl rate in Search Console instead.
Should I block AI crawlers?
It depends on what you want. Blocking protects your content from training use; allowing keeps you eligible to be cited in AI answers, which sends real visitors. There is no single right answer.
Do I even need a robots.txt?
Not strictly — no file means everything is allowed. Having one with a sitemap line is still worth the two minutes.
Need more customers as well?
This tool is free and always will be. If you also want ads, search and follow-up that bring in customers, I offer a free 45-minute consultation for businesses in the USA, UK, UAE, Australia and India.
Book a free consultationBuilt and maintained by Premkumar Ragav, an independent performance marketing specialist. Updated November 2026.