Skip to content

Robots.txt Generator

Make a correct robots.txt file for WordPress, WooCommerce, Shopify or any site. Block AI crawlers, add your sitemap and test any page.

Live preview Block AI crawlers

1. Start from

3. Rules for one bot (optional)

How to use the Robots.txt Generator

  1. 1

    Pick a starting point

    Choose your platform, like WordPress or Shopify, or start from Allow everything.

  2. 2

    Add your details

    Add your website, pages to block and your sitemap link.

  3. 3

    Choose extras

    Block AI crawlers or SEO tool bots, and add rules for one bot if you need them.

  4. 4

    Test and download

    Check any page against your file, then copy or download robots.txt.

Where does it go?

Upload the file to the main folder of your site, so it opens at yoursite.com/robots.txt. In WordPress, plugins like Rank Math and Yoast also let you paste it.

Blocking is not hiding

robots.txt asks bots not to visit a page. It does not protect private pages. Use a password for those, and a "noindex" tag to keep a page out of Google.

Do not block your CSS and JS

Google needs your theme files to see your page like a visitor. Never block /wp-content/ or /wp-includes/ on WordPress.

Questions people ask

What is a robots.txt file?

It is a small text file at yoursite.com/robots.txt that tells search engine bots which pages they may visit. Google, Bing and most good bots follow it.

How do I stop AI tools from using my content?

Tick "Block AI crawlers". It adds rules for GPTBot, ClaudeBot, Google-Extended, CCBot and other AI bots. It does not affect your normal Google ranking.

Can robots.txt remove a page from Google?

Not on its own. A blocked page can still show in Google if other sites link to it. Use a "noindex" tag on the page instead, and do not block it in robots.txt so Google can see that tag.

Where do I put the file?

In the main (root) folder of your website, so it opens at yoursite.com/robots.txt. On WordPress, Rank Math and Yoast let you edit it from the dashboard.

What does the page test do?

It reads your file the same way Google does: the most specific rule wins, and Allow wins a tie. Type a page path and pick a bot to see if it can visit.

Want your site found on Google?

We set up technical SEO for WordPress and custom sites: robots.txt, sitemaps, speed, schema and Search Console.

Get an SEO check

More free tools

All tools
Web IDE Write HTML, CSS and JavaScript in your browser with a live preview, a console and share links. Free, no sign-up. Developer and SEO Color Converter Convert colours between HEX, RGB, HSL, HSV and CMYK. Pick colours from an image, check text contrast and get matching shades. Developer and SEO Urdu Typing Test Test your Urdu typing speed online. Works with English keys too, no Urdu keyboard needed. Typing New English Typing Test Check your typing speed and accuracy, then download a free report. Typing