How to Generate robots.txt: 7 Steps for AI Indexing in 2026

how to generate robots txt
TL;DR: Generating a robots.txt file in 2026 requires more than blocking bad bots. You need to manage AI crawlers (like GPTBot and Google-Extended), avoid blocking critical JavaScript, and use the Disallow: directive carefully to prevent SEO issues. This guide covers a new 5-point validation framework and a 30-day checklist.
Quick Answer: To generate a robots.txt file for your website, create a simple text file named robots.txt in your root directory. Define your User-agent: (e.g., * for all bots) and use Disallow: to hide specific paths. In 2026, you must also include directives for AI crawlers using User-agent: GPTBot and User-agent: Google-Extended.
Key Takeaways

Table of Contents

Why robots.txt Still Matters in 2026

Many SEOs assume that robots.txt is a relic of the early 2000s. That assumption can hurt your site. In 2026, AI crawlers and Google-Extended are the primary new agents you must manage. If you don’t explicitly block GPTBot or CCBot, your content can be used to train large language models (LLMs) without your permission. Conversely, blocking Google-Extended prevents Google from using your page data for Bard or Vertex AI.

However, robots.txt does not guarantee your content won't be used for AI training—some crawlers ignore it. A Disallow is a request, not a law. You may need additional terms of service or legal notices for full protection.

What is robots.txt?

robots.txt is a plain text file placed in the root of your website (e.g., example.com/robots.txt). It tells compliant web crawlers which parts of your site they are allowed or not allowed to access. It follows the Robots Exclusion Protocol (REP).

Key components:

Step-by-Step: How to Generate robots.txt

Below is a concrete workflow for creating and deploying a robots.txt file that handles both standard search engines and AI crawlers.

Step 1: Identify Paths to Block

Review your site architecture. Common paths to disallow include:

Avoid blocking paths that contain images, CSS, or JavaScript that Google needs to render the page.

Step 2: Write the Directives

Open a plain text editor (like Notepad or VS Code). Structure your file as follows:

User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /search/
Allow: /search/help/

User-agent: Googlebot-Image
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://example.com/sitemap.xml

Step 3: Upload to Root Domain

Upload the file to the root of your website so it’s accessible at yourdomain.com/robots.txt. If you use a Content Management System (CMS) like WordPress, plugins such as Yoast SEO or Rank Math can generate this file automatically.

Step 4: Validate the File

Use the Google Search Console robots.txt Tester. Paste the URL and check for syntax errors. A valid file must not exceed 500 KiB in size.

Step 5: Monitor Crawl Stats

In Google Search Console, navigate to Settings > Crawl stats. If you see a sudden drop in crawled pages, your robots.txt might be too restrictive. Adjust accordingly.

Managing AI Crawlers (GPTBot, Google-Extended, CCBot)

AI crawlers require special attention in 2026. Here is a comparison of common AI user agents:

Crawler Purpose Directive Example Respects robots.txt?
GPTBot OpenAI model training (GPT-4, GPT-5) User-agent: GPTBot
Disallow: /		 Typically yes
Google-Extended Google AI (Bard, Vertex AI) User-agent: Google-Extended
Disallow: /		 Yes
CCBot Common Crawl (training many LLMs) User-agent: CCBot
Disallow: /		 Sometimes
Claude-Web Anthropic model training User-agent: Claude-Web
Disallow: /		 Usually yes

Important limitation: Some crawlers, especially less reputable ones, ignore robots.txt entirely. For high-stakes content, combine robots.txt with contractual terms in your website’s Terms of Service.

5 Common Mistakes to Avoid

Even experienced SEOs make these errors:

  1. Blocking CSS and JS: This prevents Google from seeing the full page layout, leading to poor indexing. Example: Disallowing /wp-content/ on a WordPress site.
  2. Using / for all bots: Blocking the entire site from all crawlers will de-index your site.
  3. Not including a Sitemap directive: Google uses this to find your sitemap faster.
  4. Case sensitivity issues: Paths are case-sensitive. /Admin/ is not the same as /admin/.
  5. Forgetting to update when moving URLs: If you restructure your site, update your robots.txt accordingly.

The 5-Point robots.txt Validation Framework

This framework helps you evaluate whether your robots.txt is effective for 2026. Score your file using a simple qualitative scale: Needs Fix, Acceptable, or Optimal.

Criteria Needs Fix Acceptable Optimal
1. AI Crawlers No directives for GPTBot or Google-Extended Blocks GPTBot only Blocks all known AI crawlers (GPTBot, Google-Extended, CCBot, Claude-Web)
2. Resource Access Blocks CSS/JS/images Blocks only non-critical assets Allows all CSS, JS, and images
3. Sitemap Declaration Missing Present but outdated Present and matches actual sitemap URL
4. Intention Clarity Uses / without clear reason Blocks some admin paths Blocks only low-value or sensitive paths
5. Validation Check Never tested Tested once Tested after every major site change

How to use: Review each row. If any criterion is in the Needs Fix column, prioritize fixing it. Aim for all Optimal scores.

How This Applies in Practice

Different websites need slightly different approaches. Here’s how the advice changes:

Frequently Asked Questions

What is the difference between Disallow and Allow?

Disallow tells a crawler which path it should not access. Allow is used to override a Disallow for a specific sub-path. For example, if you disallow /blog/ but want to allow /blog/seo-guide/, you include both rules.

Can I use robots.txt to block all AI crawlers?

You can list user agents like GPTBot, Google-Extended, and CCBot with Disallow: /. However, not all AI crawlers respect robots.txt. For full protection, combine this with a Terms of Service clause prohibiting scraping for AI training.

How does Google find my robots.txt file?

Google always requests https://yourdomain.com/robots.txt before crawling your site. If the file returns a 404 or 500 status, Google will assume there are no restrictions and will crawl everything.

Should I block all bots to save server resources?

No. Blocking all bots, including Googlebot, will de-index your site. Instead, use directives to block only the bots that consume excessive resources (e.g., some scraper bots).

Does robots.txt affect indexing or crawling?

It affects crawling: Google will not fetch URLs you disallow. However, if other pages link to a disallowed URL, Google may still index it (without seeing the content). To prevent indexing entirely, use the noindex meta tag in combination with robots.txt.

What should I do if I accidentally block a critical section?

Fix the robots.txt file immediately and re-submit the affected URLs in Google Search Console. Google will recrawl them within a few days. Monitor the Crawl stats to confirm recovery.

30-Day robots.txt Optimization Checklist

Use this checklist to systematically improve your file over a month.

Article Summary

In this article, you learned how to generate robots.txt for 2026, including the critical addition of AI crawler directives. You discovered a 5-Point Validation Framework to score your file and a 30-day checklist for ongoing optimization. The key takeaway is that robots.txt is not set-and-forget; it requires periodic review, especially as new AI agents emerge.

Useful Tool for This Task

If you need to control crawling rules, use the SMARTCHAINE Robots.txt Generator to create clear robots directives for search engines.

Conclusion

Generating robots.txt in 2026 is simple in execution but strategic in planning. You need to balance crawl efficiency with content protection. Block the right paths, manage AI crawlers explicitly, and validate your work in Google Search Console. Avoid the common pitfalls of over-blocking or under-blocking. Use the 5-Point Framework to evaluate your file today, and revisit it whenever you change your site structure.

Expert Tip: If you run a large site (10k+ pages), run a crawl with a tool like Screaming Frog before and after updating robots.txt. Compare the number of blocked URLs. A sudden jump in blocked URLs often indicates a mistake in your directives.

Recommended Resources

About the Author

The SMARTCHAINE Editorial Team specializes in SEO, AI Search Optimization, GEO (Generative Engine Optimization), AI Overviews, Structured Data, Technical SEO, and search visibility strategies for modern search engines and AI-powered discovery platforms.

Continue Learning SEO