robots txt

What Is Robots.txt: The Complete Guide to Setting It Up

Last reviewed: May 2026 Want to control which pages of your website Google crawls? The robots.txt file is the tool that lets you tell search engines which parts of your site they should visit and which they should ignore. At our SEO agency in Barcelona we configure robots.txt properly for every client so Google only crawls what matters. In this guide we explain what robots.txt is, how it works, how to create it correctly and the most common mistakes to avoid. A badly configured file can block access to important content and wreck your search rankings.

What you will learn in this guide

  • What robots.txt is: Definition and main purpose
  • Syntax and directives: User-agent, Disallow, Allow, Sitemap
  • How to create it: Step by step, with examples
  • Common mistakes: What NOT to do
  • Testing: How to check that it works properly

Contents

What is the robots.txt file?

The robots.txt is a plain text file placed in the root of a website that tells search engine crawlers (such as Googlebot) which pages or sections they may or may not crawl.

It is part of the Robots Exclusion Protocol, a standard that search engines follow voluntarily. It is important to understand that it is a recommendation, not a binding order and not a security measure.

Where is it located?

Always in the root of the domain: https://yourdomain.com/robots.txt

You can view any site’s robots.txt by adding /robots.txt to its domain. Try it on well-known sites to see real examples.

What is it for?

  • Managing crawling: Stopping bots from wasting time on irrelevant pages
  • Protecting resources: Preventing the crawling of admin or development areas
  • Optimising crawl budget: Prioritising the crawling of important content
  • Pointing to the sitemap: Telling crawlers where your sitemap is

robots.txt syntax and directives

robots.txt uses a simple syntax based on directives. Each directive takes one line:

User-agent

Specifies which crawler the instructions are aimed at:

  • User-agent: * → All crawlers
  • User-agent: Googlebot → Google’s bot only
  • User-agent: Bingbot → Bing’s bot only

Disallow

Specifies which paths should NOT be crawled:

  • Disallow: /admin/ → Blocks the admin folder
  • Disallow: /privado.html → Blocks one specific page
  • Disallow: / → Blocks the WHOLE site
  • Disallow: (empty) → Blocks nothing

Allow

Allows crawling of a path inside a blocked one:

  • Disallow: /carpeta/
  • Allow: /carpeta/publico.html → Allows only that page

Sitemap

Gives the location of the XML sitemap:

Sitemap: https://yourdomain.com/sitemap.xml

robots.txt configuration examples

Here are some practical robots.txt setups for different needs:

Allow everything (minimum setup)

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Block a folder

User-agent: *
Disallow: /admin/
Disallow: /wp-admin/

Sitemap: https://yourdomain.com/sitemap.xml

Block pages with parameters

User-agent: *
Disallow: /*?*
Disallow: /search/

Sitemap: https://yourdomain.com/sitemap.xml

Typical WordPress setup

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-includes/
Disallow: /trackback/
Disallow: /feed/
Disallow: /*?s=
Disallow: /*?p=

Sitemap: https://yourdomain.com/sitemap_index.xml

Block the whole site (maintenance)

User-agent: *
Disallow: /

Careful! This stops Google crawling any page at all. Only use it temporarily.

que es robots txt

How to create a robots.txt step by step

Creating a robots.txt is simple. All you need is a text editor:

Step 1: Create the file

Open Notepad, TextEdit or any plain text editor. Do not use Word or any rich text editor.

Step 2: Write the directives

Start with the basic setup:

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Then add whatever Disallow directives your case needs.

Step 3: Save the file

Save it exactly as robots.txt (all lowercase, with no extra extension).

Step 4: Upload it to the root

Upload the file to the root of your domain via FTP or your hosting file manager. It must be reachable at: yourdomain.com/robots.txt

Step 5: Check it

Open yourdomain.com/robots.txt in your browser to confirm it works.

robots.txt in WordPress

If you use WordPress, you have several ways to manage robots.txt:

With Yoast SEO

  1. Go to Yoast SEO → Tools → File editor
  2. You will see the current robots.txt content
  3. Edit the directives to suit your needs
  4. Save the changes

With Rank Math

  1. Go to Rank Math → General Settings → Edit robots.txt
  2. Change the content
  3. Save the changes

Virtual vs physical robots.txt

WordPress generates a virtual robots.txt by default. SEO plugins usually create a physical one, which takes priority. If something is not working, check there is no conflict between the two.

Watch out for the visibility setting

Under Settings → Reading, ticking “Discourage search engines from indexing this site” makes WordPress add a Disallow that blocks the entire site. Make sure it is unticked in production.

Crawling vs indexing: the key difference

It is essential to understand that robots.txt controls crawling, not indexing:

Crawling vs Indexing

  • Crawling: The bot visits and reads the page
  • Indexing: Google adds the page to its results index

If you block a URL in robots.txt, Google will not crawl it. But if external links point to it, Google may index it with no content, showing something like “No information is available for this page” in the results.

So how do you prevent indexing properly?

If you want a page NOT to appear in Google, use the meta noindex tag:

<meta name="robots" content="noindex">

Y do not block it in robots.txt, because then Google will not be able to read the noindex tag.

In short

  • I do not want Google to crawl it: Use robots.txt
  • I do not want Google to index it: Use meta noindex
  • I want neither: Use noindex (without blocking it in robots.txt)

How to test your robots.txt

robots.txt is a basic but fundamental part of technical SEO. A wrong setup can block important content and seriously hurt your rankings, while a good one helps Google crawl your site efficiently.

At JRC Web we review and optimise every technical aspect of your website, including robots.txt, sitemaps, canonicals and URL structure. As an SEO agency in Barcelona, we make sure your site is properly configured for search engines.

A full SEO audit covers all of these technical elements to find and fix the problems that may be holding your rankings back.

Not sure about the technical setup of your website? Get in touch and we will review your case with no commitment.

Common robots.txt mistakes

Google provides tools to check that your robots.txt works properly:

The robots.txt tester in Search Console

  1. Open Google Search Console
  2. Go to Settings → Open robots.txt
  3. Or use the old tool: search.google.com/test/robots-testing-tool
  4. Enter a URL from your site to test whether it is blocked or allowed

What to check

  • Correct syntax: The tool flags formatting errors
  • Important URLs allowed: Test your main pages
  • Private URLs blocked: Check that /admin/ and similar are blocked
  • Sitemap reachable: The sitemap URL must work

Coverage report

In Search Console, the Pages report shows you whether any URLs are blocked by robots.txt. If you see important pages in that category, review your setup.

robots.txt and crawl budget

These mistakes can seriously damage your SEO:

1. Blocking the whole site by accident

Disallow: / blocks absolutely everything. Sometimes this line is left over from a development environment and pushed to production by mistake. Always check before uploading.

2. Blocking CSS and JavaScript files

Google needs access to CSS and JS to render pages properly. Blocking them affects how Google interprets your content and can hurt your rankings.

3. Blocking pages and expecting them not to be indexed

As we saw, blocking in robots.txt does not prevent indexing. If external links exist, Google can index the URL and show an empty result.

4. Using robots.txt to hide sensitive content

The file is public. Anyone can read it and see what you are “hiding”. It is not a security measure.

5. Not including the sitemap

Adding the sitemap reference to robots.txt helps search engines find it. It is good practice and costs nothing.

The main search engine user-agents

The idea of a crawl budget is especially relevant for large sites:

What is crawl budget?

It is the number of pages Google is willing to crawl on your site in a given period. It depends on how important your site is and on how much your server can handle.

When does it matter?

  • Sites with more than 10,000 pages
  • Sites with many dynamically generated URLs
  • Sites whose content changes frequently

How to optimise it with robots.txt

  • Block low-value pages: Internal searches, filters, sorting
  • Block tracking parameters: UTMs, session IDs
  • Block deep pagination: /page/50/, /page/100/
  • Allow what matters: Make sure your key content is not blocked

For small sites (under 1,000 pages), crawl budget is rarely a problem.

robots.txt and other SEO tools

Every search engine bot has its own name (user-agent). The most important ones are:

Google bots

  • Googlebot: Google Search’s main crawler
  • Googlebot-Image: Image crawler
  • Googlebot-Video: Video crawler
  • Googlebot-News: Google News crawler
  • AdsBot-Google: Ad quality crawler

Bots from other search engines

  • Bingbot: Microsoft Bing
  • Slurp: Yahoo
  • DuckDuckBot: DuckDuckGo
  • Baiduspider: Baidu (China)
  • Yandex: Yandex (Russia)

Per-bot configuration

You can give different instructions to each bot:

User-agent: Googlebot
Disallow: /privado/

User-agent: Bingbot
Disallow: /privado/
Disallow: /otro/

Frequently asked questions about robots.txt

It is not compulsory. If there is no file, search engines assume they can crawl the whole site. Even so, it is good practice to have one so you can point to your sitemap and block areas that do not need crawling.
Not necessarily. Robots.txt prevents crawling, but if external links point to that page, Google can index it without the content. To prevent indexing, use the noindex meta tag on the page (and do not block it in robots.txt, so Google can read that tag).
No. The robots.txt file is public and anyone can read it. If you block /admin/, you are revealing that the folder exists. To protect sensitive content, use authentication (a password) or server-level restrictions.
Google caches robots.txt and re-reads it periodically (generally every 24 hours). If you make significant changes, you can request a fresh read from Google Search Console, although changes are usually reflected within a day.
Search engines try to interpret the file as best they can, but they may ignore lines containing errors. Use Google Search Console’s robots.txt tester to check that the syntax is correct before you upload any changes.

Click to rate this post!

How useful did you find this content?

Click a star to rate it!

Average rating!0/ 5. Vote count0

No votes yet! Be the first to rate this post.

Visit our company
Contact us
More News