robots.txt
robots.txt tells crawlers which paths they may fetch. What belongs in it, what a missing one costs, and why Disallow: / is the line to check first.
What it is
robots.txt is a plain-text file at the root of a site — /robots.txt, never anywhere else — listing which paths crawlers may and may not fetch. It is a convention that well-behaved crawlers follow voluntarily; it is not an access control, and it does not stop anyone determined from reading a page.
This check makes one request for it and asks only whether the server answers with a success status. A separate check reads the contents and fails a file that disallows everything.
Why it matters
A missing robots.txt is a 404, and a 404 is ambiguous: the crawler cannot distinguish "this site has no rules" from "this server is misconfigured". A permissive file that says so explicitly is a clearer signal than no file at all, and it is two lines.
It is also where the Sitemap: line goes. A crawler that finds your sitemap through robots.txt does not have to discover every page by following links, which matters most for the pages nothing links to yet.
How to fix it
with a reference docUser-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlRelated checks
4This is one of the 75 checks the scanner runs. See what we check for the full list, every severity weight, and how the score is computed from them.