Crawl Budget is the number of URLs that search engines are willing and able to crawl on a website within a given period. It is influenced by the size of the site, server performance, and the overall quality of the pages. Managing crawl budget is especially important for large websites with thousands of URLs.
Crawl History is a record of when search engines and SEO tools last visited a website and which pages they discovered. It helps show whether the site is being crawled regularly and whether important pages are accessible. Reviewing crawl history can reveal technical issues and changes in site activity over time.
View definitionRobots.txt is a text file placed in the root of a website that tells search engines which pages and folders they are allowed to crawl. It is commonly used to block unimportant or private sections of a site. Robots.txt controls crawling, but it does not guarantee that blocked pages will be removed from search results.
View definitionSitemap.xml is a file that lists the important URLs on a website. It helps search engines discover pages and understand when they were last updated. A sitemap is especially useful for large websites, new websites, and sites with pages that are difficult to find through normal internal links.
View definitionDefinitions of the most important terms used across the domain industry.