Robots.txt Generator

Create a professional robots.txt file for your website's SEO

🤖 Robots.txt Generator

Create a professional robots.txt file for your website's SEO

Default Robot Access

Default access for robots:

Crawl Delay (seconds):

Time delay between page requests (optional)

Sitemap Location

Sitemap URL:

Enter the full URL to your XML sitemap

Common Directories

Block /wp-admin/ (WordPress admin)

Block /cgi-bin/

Block /tmp/

Block /private/

Custom paths to disallow (one per line):

Enter paths you want to block, one per line

Specific Bot Rules

💡 Common Use Cases

Block specific folders: /admin/, /temp/, /private/
Block file types: /*.pdf$, /*.zip$
Allow specific bots: Googlebot, Bingbot
Sitemap: Always include your sitemap URL for better indexing

Generating your robots.txt file...

Generated Robots.txt

📂 How to Install robots.txt File

🔷 Method 1: Manual Upload (Recommended)

Download the file using the "Download File" button above
Connect to your server via FTP/SFTP using:
- FileZilla (Free - Download)
- WinSCP (Windows)
- Cyberduck (Mac)
- Or use your hosting File Manager (cPanel, Hostinger, etc.)
Navigate to your WordPress root directory:
/public_html/ or /www/ or /httpdocs/

This is the same folder where you see: wp-config.php, wp-content, wp-admin
Upload the robots.txt file to this root directory
If a robots.txt file already exists, backup the old one before replacing

🔷 Method 2: Using cPanel File Manager

Login to your cPanel or hosting control panel
Open File Manager
Navigate to public_html folder
Click Upload or Create New File
If creating new: Name it robots.txt and paste the content
If uploading: Select the downloaded robots.txt file
Set file permissions to 644 (read/write for owner, read for others)

🔷 Method 3: Using WordPress Plugin (Not Recommended)

⚠️ Note: Some WordPress plugins generate dynamic robots.txt. A physical file will override plugin-generated ones. We recommend using a physical file for better control.

🔷 Standard Web Server Installation

Download the file using the "Download File" button above
Connect to your server via FTP/SFTP or use your hosting File Manager
Upload to your website's root directory:
Common paths:
• /var/www/html/ (Linux/Apache)
• /usr/share/nginx/html/ (Nginx)
• /home/username/public_html/ (Shared hosting)
• C:\inetpub\wwwroot\ (Windows/IIS)
• /Library/WebServer/Documents/ (macOS)
Place in the root - The robots.txt must be in the root directory, not in subfolders
Set permissions to 644 (Linux/Unix servers)
chmod 644 robots.txt

🔷 FTP Connection Details You'll Need

Get these from your hosting provider:

FTP Host: ftp.yourdomain.com or server IP
Username: Your FTP username
Password: Your FTP password
Port: 21 (FTP) or 22 (SFTP)

✅ How to Verify Your robots.txt is Working

Step 1: Direct Browser Check

Open your browser and visit:

https://yourdomain.com/robots.txt

Replace "yourdomain.com" with your actual domain

✅ You should see your robots.txt content displayed in the browser

Step 2: Google Search Console

Go to Google Search Console
Select your property (website)
Go to Indexing → robots.txt Tester
Test your URLs to ensure they're blocked/allowed correctly

Step 3: Online robots.txt Checker

Use online tools to validate your syntax:

⚠️ Common Issues:

404 Error: File not in root directory or not uploaded
Permission Denied: Check file permissions (should be 644)
Not updating: Clear browser cache or use incognito mode
WordPress plugin conflict: Disable robots.txt plugins if using physical file

✨ Pro Tips:

Wait 24-48 hours for search engines to recognize changes
Submit your sitemap in Google Search Console after updating robots.txt
Keep a backup of your robots.txt file
Test thoroughly before deploying to production

Share With Group:

How to use

Learn how to use the Robot.txt generator TOOL

What This Tool Does:

Optimize your website's search engine crawling with our free Robots.txt Generator. This essential SEO tool helps you create a properly formatted robots.txt file that tells search engines which pages to index and which to ignore. Block admin areas, protect private directories, and ensure search engines find your sitemap. Get your professional robots.txt file instantly with copy-paste convenience and detailed upload instructions.

Step 1: Configure Default Robot Access

Choose how you want search engines to access your website:

Allow all robots – Let all search engines index your entire site
Disallow all robots – Block all search engines from indexing
Custom rules – Create specific rules for different parts of your site

Optionally, set a crawl delay (0-60 seconds) to control how fast bots can request pages from your server.

Step 2: Add Your Sitemap URL

Enter the full URL of your XML sitemap (e.g., https://yoursite.com/sitemap.xml). This helps search engines discover and index your pages more efficiently.

Step 3: Select Common Directories to Block

Check the boxes for common directories you want to block from search engines:

/wp-admin/ – WordPress admin area (recommended for WordPress sites)
/cgi-bin/ – CGI scripts directory
/tmp/ – Temporary files directory
/private/ – Private content directory

You can also add custom paths in the text area, one per line.

Step 4: Add Specific Bot Rules (Optional)

Click “Add Bot Rule” to create rules for specific search engine bots like Googlebot, Bingbot, or others. For each rule:

Enter the bot name (e.g., Googlebot) or use * for all bots
Choose to Allow or Disallow
Specify which paths the rule applies to

Step 5: Generate Your Robots.txt File

Click the “Generate Robots.txt” button to create your file. The generated content will appear below with syntax highlighting.

Pro Tips:

Common Use Cases:

Block admin areas and sensitive directories from search engines
Prevent duplicate content by blocking parameter URLs
Control which bots can access your site
Guide search engines to your sitemap for better indexing
Set crawl rate limits to reduce server load

Tips for Best Results:

Always include your sitemap URL in the robots.txt file
Test your robots.txt before deploying to production
Don’t block CSS or JavaScript files needed for rendering
Remember that robots.txt is publicly accessible
Keep a backup of your robots.txt file before making changes

0 0 votes

TOOL Rating

0 Comments

Oldest

Newest Most Voted

Inline Feedbacks

View all comments

0 0 votes

TOOL Rating