How to Create a Robots.txt File That Works

Many site owners don’t realize they can block search engines from important pages with a single line of text. That line usually lives in robots.txt, a small file at the root of a website that tells crawlers where they can and cannot go. If you are trying to learn how to create a robots.txt file, the goal is not just to write one. The goal is to write one that helps search engines crawl your site efficiently without accidentally hiding content you want discovered.

This guide is for website owners, marketers, and anyone managing a site’s SEO basics. You’ll learn what a robots.txt file does, when you need one, how to create it step by step, which directives matter, and which mistakes can cause crawl problems. By the end, you should be able to publish a safe, practical robots.txt file with confidence.

Quick answer: how to create a robots.txt file

To create a robots.txt file, open a plain text editor, add crawler rules such as User-agent and Disallow, save the file as robots.txt, and upload it to the root of your domain, such as https://example.com/robots.txt. Then test it to make sure you are not blocking important pages, CSS, JavaScript, or image files by mistake.

What a robots.txt file is and what it actually does

A robots.txt file is a public text file that gives crawling instructions to search engine bots and other web crawlers. It follows the Robots Exclusion Protocol, which is designed to communicate crawl preferences, not to protect private data.

Here’s the important part: robots.txt controls crawling, not indexing in every case, and not access security. A crawler may avoid fetching a blocked URL, but that does not make the page private. If the URL is linked elsewhere, it can still sometimes appear in search results without full content.

If you need a page kept out of search results, you may need other methods too, such as noindex directives where supported, authentication, or removing public access entirely.

A good robots.txt file improves crawl efficiency. It is not a security tool.

Why robots.txt matters for SEO

Robots.txt matters because it helps search engines spend crawl time on the pages that actually matter. On larger sites, that can mean fewer wasted requests on internal search pages, filtered URLs, duplicate parameter combinations, or admin sections.

For smaller websites, the file is often simpler, but it still plays an important role. It can:

• prevent crawling of low-value areas
• reference your XML sitemap
• reduce crawler noise on duplicate or utility URLs
• help avoid unnecessary server load in some situations

What it should not do is block essential resources blindly. Search engines often need access to CSS, JavaScript, and images to understand how pages render. Blocking too much can hurt crawling and page understanding.

Suggested Illustration: Simple diagram showing crawler access allowed vs blocked through robots.txt

When you should create a robots.txt file

Not every site needs a complicated robots.txt file, but most live websites benefit from having a basic one. The best approach depends on what you’re trying to accomplish.

Good reasons to use robots.txt

• You want to block crawling of admin or login areas.
• You want to discourage crawling of search result pages on your own site.
• You want to reduce crawling of duplicate parameter URLs.
• You want to include a sitemap location for search engines.
• You run a large site where crawl efficiency matters.

Bad reasons to use robots.txt

• You want to hide sensitive files from the public.
• You want to stop users from accessing a page directly.
• You assume blocked pages will never appear in search results.
• You want to fix indexing problems without checking whether crawling is the real issue.

Basic robots.txt syntax you need to know

Before you write the file, it helps to understand the main directives. These are simple, but using them incorrectly can block a whole site.

Directive	What it means	Example
User-agent	Specifies which crawler the rule applies to	User-agent: Googlebot
Disallow	Tells the crawler not to fetch matching paths	Disallow: /admin/
Allow	Permits crawling of a specific path inside a blocked section	Allow: /assets/public-file.css
Sitemap	Points crawlers to your XML sitemap	Sitemap: https://example.com/sitemap.xml

A basic example looks like this:

User-agent: *
Disallow: /admin/
Disallow: /search/

Sitemap: https://example.com/sitemap.xml

In practice, this means all crawlers are asked not to crawl the /admin/ and /search/ folders, while still being shown where the sitemap lives.

How to create a robots.txt file step by step

If you want a practical workflow, this is the simplest way to do it safely.

1. List the site areas you do not want crawled.
   Common examples include admin paths, cart steps, internal search results, test folders, and duplicate filtered URLs. Be specific. Avoid broad rules unless you are sure they are safe.
2. Open a plain text editor.
   Use a plain text editor, not a word processor. The file should contain only plain text. You can use a code editor, your CMS file manager, or a hosting control panel editor.
3. Write the crawler rules.
   Start with a User-agent line, then add Disallow or Allow rules. If the same rule should apply to most crawlers, use User-agent: *.
4. Add your sitemap URL.
   If you have an XML sitemap, include its full absolute URL. This helps search engines find important pages faster.
5. Save the file as robots.txt.
   The filename should be exactly robots.txt, all lowercase. Save it in plain text format.
6. Upload it to the root of your site.
   The file should be accessible at the domain root, such as https://example.com/robots.txt. For subdomains, each one needs its own robots.txt file if you want separate rules.
7. Test the file before and after publishing.
   Check whether important pages remain crawlable. Review any blocked patterns carefully, especially on ecommerce sites or large content sites.
8. Monitor search performance and crawl behavior.
   After publishing, keep an eye on crawl reports, coverage signals, and unexpected drops in discoverability.

A beginner-friendly robots.txt template

If you only need a simple starting point, this template is usually easier to work from than writing from scratch.

User-agent: *
Disallow: /admin/
Disallow: /login/
Disallow: /search/

Sitemap: https://example.com/sitemap.xml

This template is intentionally conservative. It does not block your whole site, and it does not try to manage every possible crawler behavior. Replace the paths with sections that make sense for your site.

Here’s where many beginners get confused: a blank Disallow: means nothing is blocked for that user-agent. A slash by itself, Disallow: /, means the entire site is blocked for that crawler.

Examples of safe and risky rules

Small syntax changes can have big effects. The comparison below helps explain why careful formatting matters.

Rule	Result	Risk level
Disallow: /admin/	Blocks crawling of the admin folder	Usually low if admin pages should stay out of crawl paths
Disallow: /	Blocks crawling of the entire site	Very high on a live site
Disallow:	Blocks nothing	Low, but only if intentional
Allow: /	Explicitly allows crawling	Low, though often unnecessary

How robots.txt works with crawling, indexing, and sitemaps

One of the most common SEO misunderstandings is assuming these are all the same thing. They are not.

• Crawling means a bot fetches a URL.
• Indexing means a search engine stores and may use that page in search results.
• Sitemaps help search engines discover URLs you consider important.

If a page is blocked in robots.txt, some search engines may not crawl it, which means they also may not see on-page directives on that URL. That is why using robots.txt as an indexing control can backfire.

For official guidance, Google explains the difference in its robots.txt documentation and related crawling resources.

Where to place the robots.txt file

The file belongs at the top-level root of the host it applies to. That means:

• https://example.com/robots.txt applies to example.com
• https://blog.example.com/robots.txt applies to blog.example.com
• https://example.com/folder/robots.txt is not the standard location for site-wide crawler instructions

If your site uses multiple subdomains, treat each subdomain separately. A robots.txt file on the main domain does not automatically control a different subdomain.

Best practices when creating a robots.txt file

If you’re doing this regularly, there’s a simpler workflow: keep the file minimal, intentional, and easy to audit. Overly complex robots.txt files often create confusion later.

• Block only what you truly want crawlers to avoid. Every extra rule adds risk.
• Include your sitemap. This is one of the most useful and low-risk additions.
• Do not block CSS and JavaScript without a strong reason. Search engines often need these resources for rendering.
• Review staging and development rules before going live. Entire sites are sometimes blocked because a temporary rule was forgotten.
• Use comments carefully if supported in your workflow. They can make maintenance easier, especially on larger teams.
• Keep a backup copy. If an update causes crawl issues, rollback is much easier.

To better understand your sitemap strategy, it can also help to review official guidance from Sitemaps.org.

Common robots.txt mistakes that can hurt your site

Most robots.txt problems are not advanced technical failures. They are simple mistakes made during launches, redesigns, migrations, or plugin changes.

Blocking the entire site by accident

This usually happens when a staging rule like Disallow: / is left in place after launch. The impact can be serious because crawlers may stop fetching your public pages.

Blocking important assets

Some site owners block asset folders without realizing they contain CSS, JavaScript, or images needed for rendering. That can reduce how clearly search engines understand the page.

Using robots.txt to hide private content

Because robots.txt is public, listing a sensitive folder there can actually draw attention to it. Use authentication or access controls instead.

Blocking pages that need on-page directives

If a page is blocked from crawling, search engines may not be able to fetch the page and read instructions on it. That can make indexing control harder, not easier.

Forgetting to update rules after a site migration

Paths change during redesigns and CMS migrations. Old disallow rules may no longer make sense, or new sections may be left unmanaged.

Practical robots.txt examples for different site types

The right setup depends on your site structure. Here are realistic examples, not one-size-fits-all rules.

Small business website

You may only need to block admin and login areas while listing the sitemap.

User-agent: *
Disallow: /wp-admin/
Disallow: /login/

Sitemap: https://example.com/sitemap.xml

Content website with internal search

If your site has public search result pages, blocking crawl access there can reduce wasted crawl activity.

User-agent: *
Disallow: /search/
Disallow: /results/

Sitemap: https://example.com/sitemap.xml

Ecommerce site

Be careful here. Many ecommerce sites generate parameter-heavy URLs. Broad blocking can interfere with product discovery if done poorly.

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/

Sitemap: https://example.com/sitemap.xml

For online stores, test every rule against category pages, product pages, filters, and pagination before publishing.

How to test and validate your robots.txt file

Publishing the file is only half the job. You also need to confirm it behaves the way you expect.

1. Open the live file in your browser.
   Visit yourdomain.com/robots.txt and confirm the file loads correctly.
2. Check a few important URLs manually.
   Compare your file rules against important page paths to see whether they appear blocked or allowed.
3. Use official documentation and webmaster tools where available.
   Google’s crawler guidance and related tools can help verify behavior.
4. Review server behavior.
   Make sure the file is returning properly and not redirecting strangely.
5. Re-test after edits.
   Even a small change can affect thousands of URLs on a larger site.

Google’s official guide to creating a robots.txt file and robots.txt interpretation documentation are useful references if you want to validate edge cases.

Troubleshooting common robots.txt problems

If your pages are not being crawled as expected, start with the simplest possibilities first.

The file does not load

Check whether the file is uploaded to the correct root location and that the filename is exactly robots.txt. Also confirm there are no permission issues on the server.

Important pages stopped appearing in search workflows

Review whether a broad Disallow rule matches those paths. This can happen after a CMS update, migration, or copied staging configuration.

Search engines still show a blocked URL

This can happen because blocking crawl access does not always prevent indexing signals from other sources. If a page should not be publicly accessible, robots.txt alone is not enough.

Resources needed for rendering are blocked

Check asset folders such as theme, script, or image directories. Restore access to essential rendering resources if they were blocked unintentionally.

Limitations of robots.txt you should understand

There is an important limitation to understand: robots.txt is a voluntary instruction system for compliant crawlers. It does not function like a login wall, a firewall, or a legal access restriction.

• It does not secure confidential content.
• It may not control all bots equally.
• It does not guarantee a blocked URL will never be indexed.
• It cannot replace proper authentication.
• It needs to be maintained as your site structure changes.

For a more technical overview of the standard itself, MDN’s general HTTP references are useful for broader web context, and Google’s own robots documentation remains the most relevant source for SEO behavior. For protocol-level reading, the Robots Exclusion Protocol RFC explains the standard in detail.

A simple workflow for larger site reviews

If you manage many website sections, migrations, or competitor research lists, it helps to separate robots.txt review from page-level SEO review. A simple workflow looks like this:

• export or collect important site URLs
• group them by section such as blog, product, category, admin, search, and account
• compare those sections against your robots.txt rules
• verify which areas should be crawlable
• test the live file after updates

If you regularly review many URLs during audits, a tool like OpenMultiURL can help you open multiple URLs at once for manual verification. That can be useful when checking how different sections, templates, or resource paths behave in the browser. As always, verify unfamiliar URLs before opening them, especially when working from exported lists.

Suggested Image: Minimal SEO illustration of a site audit workflow comparing URL groups with robots.txt rules

Frequently asked questions

1. Do all websites need a robots.txt file?

No, but most public websites benefit from having one. A small brochure site may only need a basic file with a sitemap reference and perhaps one or two blocked utility paths. Larger sites usually benefit more because they often generate admin, search, account, or duplicate URLs that do not need crawler attention. The key is not complexity. It is having a file that matches your site structure and SEO goals.

2. Can robots.txt stop a page from appearing in Google?

Not reliably on its own. Robots.txt mainly controls crawling, not guaranteed removal from search results. If a blocked URL is linked from other places, Google may still know it exists and may show the URL without full content. If the page must stay out of search, use the appropriate indexing controls where supported and make sure the content is not publicly accessible if privacy is required.

3. What is the difference between Disallow and noindex?

Disallow in robots.txt tells compliant crawlers not to fetch matching URLs. noindex is an indexing instruction used in page-level directives where supported. The difference matters because if a page is blocked from crawling, a search engine may not be able to read the page and see its indexing instructions. That is why these methods should not be confused or used interchangeably without checking the actual goal.

4. Where should I upload the robots.txt file?

Upload it to the root of the host it controls, such as https://example.com/robots.txt. If your site uses subdomains like blog.example.com or shop.example.com, each subdomain needs its own robots.txt file if you want separate crawler instructions there. Placing the file inside a subfolder usually will not provide proper site-wide control.

5. Can I use robots.txt to block bad bots?

You can include rules for specific user-agents, but robots.txt is not a dependable security control against malicious bots. Compliant crawlers often respect it. Bad actors may ignore it completely. If bot abuse is a real issue, use server-side protections, rate limiting, bot management tools, firewalls, or authentication measures instead of relying on robots.txt.

6. Should I block CSS and JavaScript files?

Usually no. Search engines often need access to CSS and JavaScript to render pages correctly and understand layout, content loading, and page behavior. Blocking those assets can make it harder for crawlers to interpret the page. Only block resource files if you have a very specific reason and have tested the outcome carefully afterward.

7. What happens if I accidentally use Disallow: / ?

That rule tells the targeted crawler not to crawl any URL on the site. On a live website, this can seriously affect discoverability because important pages may stop being fetched. It is a common staging mistake. If you find it on a public site by accident, remove or correct it quickly, then request re-crawling through the relevant webmaster tools and monitor recovery.

8. How often should I review my robots.txt file?

Review it whenever your site structure changes, especially after redesigns, migrations, plugin changes, ecommerce platform updates, or staging-to-live launches. Even if nothing major changes, a periodic review is still wise. Many site owners only revisit robots.txt after a traffic issue appears, but checking it proactively can help catch mistakes before they affect crawling.

Conclusion

Learning how to create a robots.txt file is less about writing a few directives and more about making careful decisions about crawl access. Keep the file simple, place it at the root of your domain, include your sitemap, and avoid broad blocks unless they are intentional and tested. Most importantly, remember that robots.txt guides crawlers. It does not secure content.

If you maintain larger audits or review many site sections at once, streamline the manual checking process with a clean URL workflow. When you need to open multiple URLs for verification, OpenMultiURL can help reduce repetitive browser work without changing how websites themselves function.