Robots.txt, what is it?
A robots text file (robots.txt) informs search engines which URLs they can access on your site. It's critical this is correctly configured, or valuable content may be blocked from the search engines
The robots.txt file on your website is regularly checked by the major search engines (such as Google, Bing, and Yahoo) for instructions on how they should crawl your website. If there isn't a robots.txt file on your website, the search engines will crawl the entire website, which could include private or sensitive directories that you may wish to hide from the search engine results pages (SERPs).
The majority of the search engines do follow the instructions (directives) in robots.txt, but it’s important to understand that following your directives is not a strict requirement of the search engines. That is, search engines can ignore your robots.txt file, although the majority of 'ethical' search engines do adhere to robots.txt directives.
Is robots.txt important for SEO?
To a degree, this depends on the size and nature of your website. If your website is under 20 pages and you want all search engines to see everything on your website, then you don't technically 'need' a robots.txt file as you don't need to instruct the search engines on what to allow or disallow.
This said, if your website is huge, with hundreds of pages and you need to be economical with your crawl budget (a crawl budget is the number of pages a search engine will crawl on your website site on any given day) it is advisable to configure your robots.txt file to disallow pages of lower importance and allow the pages that are most important. By doing this, your more likely to ensure your most important pages are crawled, which will have a positive impact on your SEO.
As briefly mentioned above, if have content that is sensitive or private that you want to keep out of the SERPs, then it is highly advisable to add a robots.txt file to your website in order to instruct the search engines on what to allow and what to disallow.
So yes, in most cases, a robots.txt file is important for SEO.
Is your website robots.txt correct?
All our website solutions are built and configured individually for each customer - click below to find out more.
Is your SEO working for you?
Get help with SEOCreativity Sells Ltd
