Crawlability refers to the ability of a search engine to access and crawl content on a web page. If a site has good crawlability, search engines can easily access content by following the website links throughout the site. So what's crawling? A search engine, such as Google, has a three stage approach to adding your site to its SERPs (Search Engine Page Results). Firstly, it sends its 'crawler' (also called 'robot', 'bot' or 'spider') across the Internet, discovering new and updated content on the literally billions of website pages. Secondly, upon arriving at website, the bot saves the website HTML back to its database (called the 'Index'). Thirdly, and perhaps most importantly when it comes to SEO, your content is processed by its algorithm before updating it's search engine database. The SEO Ranking Factors influence the final SERPs that you see online.
In order to have a highly crawlable website, its important to ensure your website has a good 'site URL structure' and well 'structured data'. In terms of URL structure, a well structured navigation is a great place to start. Arrange your navigation / page hierarchy so it follows an intuitive structure, starting at the broader top-level categories, then narrowing down onto sub-categories and ultimately the individual pages. A well structured navigation not only help your crawlability, but also provides your users with a better User Experience (UX / UE) which is another valuable SEO ranking factor.
A sitemap file provides link information about the pages and other files that exist on your website. Search engines use the sitemap to crawl and index your site more efficiently. Sitemaps should be submitted to the various Search Console Apps that exist for the major search providers. In Google's case, this is called 'Google Search Console' (GSC).
Robots.txt is a text file that instructs search engine bots how to crawl a website. The robots.txt files allows you to specify which user agents can or can't crawl parts of a website by 'allowing' or 'disallowing' the activity of specific (or all) user agents.
Do not change your robots.txt file unless you are 100% confident. Get this wrong and you could block your whole website from the search engine bots.
We can help you optimise your website for SEO crawlability - click below to find out more.