Robots.txt generator and tester.
Build a robots.txt in a few clicks, decide which AI crawlers may read your site, or test an existing file to see who is allowed to fetch a given page.
How to use the Robots.txt Generator & Tester
- Choose "Generate" to build a file, or "Test" to check an existing one.
- Pick a preset, tick the AI crawlers you want to block and add any folders to keep out.
- Copy or download robots.txt, upload it to your site root, then test it here.
Good to know
robots.txt asks crawlers politely; it does not protect private pages. Anything secret needs a login. A rule that is too broad can hide your whole site from Google, which is why the tester shows exactly who may fetch a path.
Questions about this tool
It is your choice. Blocking GPTBot, ClaudeBot and similar crawlers keeps your content out of those assistants' answers and training; allowing them lets them quote and link to you.
At the root of the domain: yoursite.com/robots.txt. It does not work in a sub-folder.
Not reliably. Disallow stops crawling, not indexing. To keep a page out of search results use a noindex tag and let Google crawl it.
Allow everything, disallow /wp-admin/ (but allow admin-ajax.php), and add the address of your sitemap. The WordPress preset does exactly that.
Keep going
SEO Checker
Score any page or a whole website on more than 50 checks, with a PDF report.
Open →Title & Meta Checker
Check length in characters and pixels, and preview the Google result on desktop and mobile.
Open →Meta Description Generator
Describe your page and get three titles and descriptions written at the right length.
Open →llms.txt Generator
Build the llms.txt file that gives AI assistants a plain-text map of your site.
Open →Schema Markup Generator
Create FAQ, Article, Organization and Breadcrumb structured data to paste into your page.
Open →Heading Checker
See the H1 to H6 outline of any page and find missing, duplicate or skipped headings.
Open →Google Rank Checker
See where your website ranks on Google for any keyword, and who ranks above you.
Open →Want every page written, checked and published for you?
InfinityWriter researches, writes and scores articles against these same checks, and publishes them to your WordPress site. Ten articles a month are free.