WordPress robots.txt: What to Put In It in 2026 (and What 123 Real Sites Do)

The WordPress robots.txt file tells search engine and AI crawlers which parts of your site they may fetch. WordPress serves a virtual one automatically: it blocks /wp-admin/, allows admin-ajax.php and lists your sitemap. Most sites need only a few more lines. It controls crawling, not indexing, and it is not a security tool.

This guide shows the exact file WordPress generates (taken from the core source code), four ways to edit it, a safe 2026 template, and how to block or allow AI crawlers such as GPTBot and ClaudeBot without breaking your search rules. It also includes something no other guide has: we fetched the robots.txt of 128 real WordPress sites from the WordPress.org Showcase on 29 October 2026 and counted what they actually do, including a mistake that silently cancels rules on about one site in ten.

What is the WordPress robots.txt file?

A robots.txt file is a plain text file at the root of your domain, at https://yoursite.com/robots.txt, that tells crawlers which URLs they may request. Google describes it as a file that "tells search engine crawlers which URLs the crawler can access on your site", mainly so crawlers do not overload your server (Google Search Central). The format is an internet standard, RFC 9309, published in September 2022.

WordPress is unusual because you do not have to create the file. If no physical robots.txt exists in your site's root folder, WordPress answers the request itself and generates one on the fly. That is called the virtual robots.txt. If you, your host or a plugin uploads a real file, the web server sends that file instead and WordPress never sees the request. The reason is the standard WordPress rewrite rule: the line RewriteCond %{REQUEST_FILENAME} !-f in the default .htaccess only passes a request to WordPress when no real file matches it.

Both work equally well for crawlers. What matters is that you know which one you have, because editing the virtual file does nothing while a physical file exists.

What does the default WordPress robots.txt contain?

On a public site, WordPress outputs exactly this (your domain in place of example.com):

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml

That comes from two places in core. The function do_robots() in wp-includes/functions.php writes the first three lines, using the real path of your admin folder, then passes the text through the robots_txt filter. The sitemap system added in WordPress 5.5 hooks into that filter and appends the Sitemap: line, but only when the site is public (class-wp-sitemaps.php).

Each line has a job:

• User-agent: * means the rules below apply to every crawler that has no group of its own.
• Disallow: /wp-admin/ keeps crawlers out of the dashboard, which is useless to them and only shows a login page.
• Allow: /wp-admin/admin-ajax.php makes one exception. Many themes and plugins load content on public pages through this file, so blocking it can stop Google from seeing what visitors see. Google applies the most specific (longest) matching rule, so the Allow wins for that one URL (Google robots.txt specification).
• Sitemap: points crawlers at the list of your URLs. Google, Bing and other major engines support this line, and it is not tied to any user agent.

What happens when you tick "Discourage search engines"?

Less than most people think. Until WordPress 5.3 that setting (Settings, Reading, "Search engine visibility") added Disallow: / to robots.txt. Since 5.3 it does not. WordPress now adds a noindex, nofollow robots meta tag to pages instead, because blocking crawling does not stop a URL from being listed when other sites link to it (Make WordPress Core, 2019). The current code is wp_robots_noindex() in robots-template.php. The setting also removes the Sitemap line, because core sitemaps are switched off for non-public sites.

The practical lesson: if a site went live with that box still ticked, robots.txt will look normal but every page carries noindex. Check the page source for <meta name='robots' content='noindex, nofollow' before you blame robots.txt for missing rankings.

How do I check my WordPress robots.txt?

Open https://yoursite.com/robots.txt in a private browser window. Whatever you see there is what crawlers see. Then work out whether it is virtual or physical, and whether Google can fetch it.

1. Read it in the browser. If it shows the four default lines above, it is almost certainly the virtual file. If it has comments, a previous agency's name, or long lists of bots, someone wrote it.
2. Check from the command line. The status code and headers tell you more than the browser does:

   # status, content type and caching headers
   curl -sI https://example.com/robots.txt
   
   # the file itself, as Googlebot would request it
   curl -s -A "Googlebot" https://example.com/robots.txt

   You want HTTP/2 200 and content-type: text/plain. A 404 is not a disaster: Google treats a missing file as "no restrictions". A 5xx error is worse: Google stops crawling the site for the first 12 hours while it retries, then falls back to its last cached copy for up to 30 days (Google).
3. Find out whether a physical file exists. With SSH or the host's file manager, look in the folder that holds wp-config.php:

   ls -la robots.txt
   # with WP-CLI, print the site root path first if you are unsure
   wp eval 'echo ABSPATH, PHP_EOL;'

   No file means you have the virtual one.
4. Use Search Console's robots.txt report. Under Settings, the report shows which robots.txt files Google found for your top hosts, when it last fetched each one, and any errors. You can also "Request a recrawl" after an urgent fix (Search Console Help). Google normally caches robots.txt for up to 24 hours, so a change is not seen instantly.
5. Test a specific URL. Use the URL Inspection tool in Search Console on a page you care about. If it says the page is blocked by robots.txt, the next sections show how to find the rule responsible.

Free online "robots.txt checker" and "validator" tools are fine for spotting typos, but they apply their own parsers. When a checker and Google disagree, Google's own report is the one that counts for Google.

How to edit robots.txt in WordPress: four ways

Pick the method that matches what you already have. Do not mix them: a physical file overrides everything a plugin does to the virtual one.

Yoast SEO

In the dashboard go to Yoast SEO, Tools, File editor, then "Create robots.txt file". Yoast's own help page notes that the editor may not work if the file is not writable or file editing is disabled, in which case you edit the file on the server instead (Yoast help).

Rank Math

Go to Rank Math SEO, General Settings, Edit robots.txt (Rank Math knowledge base). If the box is greyed out, a physical file usually exists; delete or rename it first.

A physical file

Create a text file named exactly robots.txt (lower case), save it as plain UTF-8, and upload it to the site root. Keep a copy of the old one before you overwrite anything:

cp robots.txt robots.txt.bak-$(date +%F)
nano robots.txt
curl -s https://example.com/robots.txt

The robots_txt filter (for developers)

This adds lines to the virtual file without a plugin. Put it in a small must-use plugin, for example wp-content/mu-plugins/robots.php, so a theme change cannot remove it:

<?php
/**
 * Plugin Name: Site robots.txt rules
 */
add_filter( 'robots_txt', function ( $output, $public ) {
    if ( ! $public ) {
        return $output; // site is set to discourage search engines
    }
    $output .= "
# Keep internal search result pages out of crawl budget
";
    $output .= "User-agent: *
";
    $output .= "Disallow: /?s=
";
    $output .= "Disallow: /search/
";
    return $output;
}, 20, 2 );

Note that this appends a second User-agent: * group. Google combines groups that name the same user agent, so this works for Google, but cleaner output is easier to read. If you prefer, rebuild the whole string instead of appending. The filter is documented at developer.wordpress.org.

What should a WordPress robots.txt look like in 2026?

For most business sites, the default plus your sitemap is already right. Add rules only when you can name the problem they solve. Here is a template with comments; delete what you do not need.

# robots.txt for example.com, reviewed 2026-10-29
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
# Optional: internal search results waste crawl on thin pages
Disallow: /?s=
Disallow: /search/
# Optional for WooCommerce: cart, checkout and account pages
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/

# Optional: opt out of AI model training (see the AI section)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
Disallow: /

Sitemap: https://example.com/sitemap_index.xml

Three notes on that template:

• Use your real sitemap URL. Core uses /wp-sitemap.xml; Yoast and Rank Math use /sitemap_index.xml. Open the URL before you paste it. A Sitemap line pointing at a 404 does no harm, but it does no good either.
• The WooCommerce paths depend on your page slugs. Check Settings, Advanced in WooCommerce for the actual cart, checkout and account pages before copying them.
• Several user agents can share one group by stacking User-agent lines, as in the AI block. That is valid under RFC 9309 and shorter than repeating the rules.

What you should not block

• /wp-content/ or /wp-includes/. These hold your images, CSS and JavaScript. Google's guidance is that you should not block resources when a page is harder to understand without them (Google). Blocking them was common advice around 2010 and still survives in old files.
• /wp-content/uploads/. That blocks every image in the media library from image search.
• Pages you want removed from Google. A blocked page cannot be crawled, so Google never sees the noindex on it. More on that below.
• Feeds and tag pages "for SEO" without a measured reason. If they are thin, a noindex in your SEO plugin is the cleaner fix.

What do real WordPress sites put in robots.txt?

We checked. On 29 October 2026 we took every site listed in the WordPress.org Showcase archive (159 entries), confirmed 128 of them still run WordPress, and fetched each robots.txt once. 123 returned a valid file. The method and its limits are at the end of this article. These are large, professionally run sites (news publishers, universities, museums, government departments, brands), so treat the numbers as "what well-resourced WordPress teams do", not as an average of all WordPress sites.

What this tells you:

• Short files are normal. The median file had 5 rule lines. Big, well-run sites do not carry long lists of folder blocks.
• The old /wp-includes/ block survives on 8 sites (6.5%). Nobody in the sample blocked all of /wp-content/, which suggests that particular piece of bad advice has mostly been cleaned up.
• AI opt-outs are a minority choice even among publishers. 13.0% block at least one AI crawler. Most of those are news and media sites, which have a licensing interest in their archives. A typical business site has the opposite interest: it wants to be quoted.
• Nobody used noindex in robots.txt, which is good, because Google stopped honouring it in 2019.

How do I block or allow AI crawlers in robots.txt?

Add a group for each AI crawler's user agent token with Disallow: / to block it, or leave it out to allow it. The tokens and what they control are set by each company, so here they are from each company's own documentation:

Should a business site block AI crawlers?

It depends on what you sell. If your content is the product (a publisher, a course library, a paid database), blocking training crawlers is a reasonable business decision. If your content exists to bring in customers (a service business, a shop, a clinic), you generally want AI assistants to read and cite you, the same way you want Google to. A middle path many sites choose is to block the training tokens (GPTBot, ClaudeBot, CCBot, Google-Extended) and leave the search and user tokens open.

# Opt out of training, stay visible in AI search answers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
Disallow: /

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap_index.xml

Two limits to be honest about. First, robots.txt is a request, not a lock: well-behaved crawlers follow it, others do not. Second, the newer Content-Signal line that Cloudflare introduced on 24 September 2025 (for example Content-Signal: search=yes, ai-train=no) expresses a preference about how content is used after it is fetched. Cloudflare itself says these signals are "not technical countermeasures against scraping" and that some companies may ignore them (Cloudflare blog). 7 sites in our sample used it.

The group trap: how adding one bot can cancel your rules

A crawler obeys only one group: the one with the most specific user agent that matches it. If a crawler has its own group, it ignores the User-agent: * group completely. Google's specification puts it plainly: user-agent-specific groups and the global group "are not combined" (Google).

That causes a quiet mistake. Someone wants to "make sure Bingbot can crawl" and adds this:

User-agent: *
Disallow: /wp-admin/
Disallow: /checkout/
Allow: /wp-admin/admin-ajax.php

User-agent: Bingbot
Allow: /

Bingbot now reads only its own group, which allows everything. The /wp-admin/ and /checkout/ rules no longer apply to it. The same thing happens when a file welcomes AI crawlers with Allow: /. In our sample, 13 sites (10.6%) had at least one named group with no Disallow while their wildcard group had some. Several look deliberate, for example advertising crawlers given full access on purpose. Others name search engines or AI crawlers in a way that suggests the author expected the wildcard rules to still apply. We found the same mistake in our own files during an audit earlier this year, so it is easy to make.

The fix is to repeat the rules you want in every named group:

User-agent: Bingbot
Disallow: /wp-admin/
Disallow: /checkout/
Allow: /wp-admin/admin-ajax.php

Or simply delete the named group if all it says is Allow: /. Crawlers are allowed by default; you never need to grant permission. The exception is Google-Extended, which is a usage control rather than a crawler, so a bare group for it does not change what Googlebot crawls.

robots.txt rules explained: allow all, deny all, folders, wildcards, crawl delay

How do I allow all crawlers?

An empty Disallow allows everything, and so does having no robots.txt at all:

User-agent: *
Disallow:

On WordPress you would normally keep the /wp-admin/ pair rather than allow literally everything, because there is nothing useful for crawlers in the dashboard.

How do I block all crawlers (deny all)?

User-agent: *
Disallow: /

Use this only on a staging or development copy, and even there it is not enough on its own: a blocked URL can still be listed if someone links to it. For a staging site, password protection is the reliable option. Never copy a staging robots.txt to the live site; it is one of the most common causes of a site vanishing from search after a launch or migration.

How do I disallow a folder?

User-agent: *
Disallow: /private-folder/

Paths are case sensitive and must start with /. The trailing slash matters: /private also matches /private-offers/ and /private.html, while /private/ matches only the folder.

Can I use wildcards?

Yes, for Google and most major engines. * matches any characters and $ marks the end of the URL:

# every URL with a query string that sorts products
Disallow: /*?orderby=
# every PDF file
Disallow: /*.pdf$

Does Crawl-delay work?

Not for Google. Google's specification lists the fields it supports and says "other fields such as crawl-delay aren't supported". Some other crawlers do honour it; Anthropic, for example, says its bots support it. 20 sites in our sample (16.3%) use it. If Googlebot is overloading your server, the cause is usually something else, such as infinite filter or calendar URLs, and the fix is to stop generating those URLs or block the pattern.

Should I add my sitemap to robots.txt?

Yes. It costs nothing and helps every crawler that does not use your Search Console account. Use a full URL including https://. You can list more than one Sitemap line. 88.6% of the sites we checked had one.

Can I use noindex in robots.txt?

No. Google retired support for unpublished rules such as noindex in robots.txt on 1 September 2019 and recommends the noindex robots meta tag or HTTP header instead (Google Search Central blog). In WordPress, your SEO plugin sets that meta tag per page.

How big can robots.txt be?

Google reads the first 500 kibibytes and ignores anything after that. RFC 9309 says crawlers must read at least 500 KiB. No WordPress site needs anywhere near that.

How to fix "Blocked by robots.txt" in Search Console

Search Console's Page indexing report has two statuses that sound alike but mean opposite problems.

Both recommendations come from Google's Page indexing report help. To find the rule responsible:

1. Copy the blocked URL from the report.
2. Open your robots.txt and find every Disallow line in the group that applies to Googlebot (its own group if it has one, otherwise *).
3. Find the longest rule that matches the start of the URL path. That is the one Google applies, unless an Allow rule of equal or greater length also matches.
4. Edit or remove it, then use "Request a recrawl" in the robots.txt report and "Validate fix" in the indexing report.

A common WordPress cause: a rule like Disallow: /*?, added to stop crawling of filter URLs, also blocks every page that a plugin or campaign adds a query string to.

What robots.txt does NOT do

WordPress robots.txt checklist

• Open /robots.txt: it returns 200 and plain text.
• You know whether it is virtual or physical, and only one method is used to edit it.
• The wildcard group keeps Disallow: /wp-admin/ and Allow: /wp-admin/admin-ajax.php.
• Nothing blocks /wp-content/, /wp-includes/, uploads, CSS or JavaScript.
• The Sitemap line points at a sitemap URL that actually loads.
• Every named group repeats the rules it still needs, or is deleted.
• Your AI crawler choice is deliberate and written down, with the date.
• No noindex or Disallow: / left over from staging.
• Search Console's robots.txt report shows no errors.
• Settings, Reading, "Discourage search engines" is unticked on the live site.

robots.txt is one small part of technical SEO. For the rest, see our WordPress SEO guide; if you are moving hosts, the migration guide covers the robots and noindex settings that most often go wrong; and if your login page is the bigger worry, see how to find and protect your WordPress login URL.

How we measured this

Sample. All 160 entries in the WordPress.org Showcase archive (pages 1 to 9 of wordpress.org/showcase/archives/), collected on 29 October 2026. For each entry we took the site address linked from its Showcase page. 159 entries had one.

WordPress check. We fetched each home page once and counted a site as WordPress if the HTML had a WordPress generator tag, the api.w.org REST API link, or /wp-content/ or /wp-includes/ asset paths. 128 sites passed. Sites that blocked our request (for example with a 403 bot challenge) or have moved off WordPress were excluded rather than guessed.

robots.txt. One GET request per site to /robots.txt, following redirects, with a user agent naming wpdeveloper.ca and a contact URL, four requests at a time. 123 sites returned a plain-text file with status 200. We parsed groups as RFC 9309 describes (consecutive User-agent lines form one group) and matched rules exactly as written.

Limits. The Showcase is a curated list of prominent sites, so it over-represents publishers, universities and large brands and under-represents small business sites. A sample of 123 gives a rough picture, not a precise market share: one site is about 0.8 percentage points. "Fully blocks" means a group naming that token with Disallow: /. We did not check whether crawlers obey the files. The script and raw results are kept so we can repeat the count; we plan to rerun it each quarter and will update this page when the numbers change.