The fix, in one snippet
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xmlIllustrative values. Change the paths, hostnames and options to match your own site before using it.
The sections below explain what is tested, why it matters and the alternatives.
What we test
Scan.now requests /robots.txt before crawling anything else, parses it with the same rules a search engine uses (user-agent groups, Allow, Disallow, Sitemap), and then honours it for the rest of the crawl. The check passes when the file is served as plain text with a 200 status and contains at least one directive.
It fails when the file is missing, returns an HTML error page under a 200 status (a "soft 404", which crawlers treat as a file full of nonsense), or cannot be fetched at all. We record how many directive lines it holds and whether a Sitemap: line is present, because that line is how most crawlers discover a sitemap in the first place.
Why it matters
Without robots.txt every crawler falls back to crawling whatever it can reach. On a small site that is harmless. On a large one it means crawl budget is spent on faceted search URLs, print views, session parameters and staging paths instead of the pages you want indexed.
The file is also the only site-wide place to point at a sitemap. Google, Bing and every other major crawler read Sitemap: from robots.txt, which means a site with no robots.txt is relying on each engine finding the sitemap by convention.
A soft 404 is worse than a missing file: a crawler that receives HTML with a 200 status may try to parse it as directives, and the result is undefined. Serve a real 404, or serve a real robots.txt.
How to fix it
Publish a plain-text file at the root. The minimum useful version allows everything and points at the sitemap:
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xmlAdd Disallow lines only for paths that genuinely should not be crawled: internal search results, faceted filter combinations, cart and checkout URLs, duplicate print views. Never use robots.txt to hide something sensitive, because the file is public and reads as a list of the paths you care about. Use authentication for that.
Remember that Disallow stops crawling, not indexing: a blocked URL with external links can still appear in results without a snippet. To keep a page out of the index, allow crawling and serve noindex.
Where this fits
robots.txt present and usable is check 10 of 10 that the seo & site health audit runs under indexing and crawlability, ordered the way they are worth fixing. That ordering is the point: Fixing this one while the check above it still fails buys less than it looks like.
Fix this one first
Above it in the same category sits XML sitemap published and declared (medium), where an XML sitemap lists the URLs you want crawled and when they last changed. An attacker who has that does not need this, so it is the better use of the same hour.
Found in the same scan
The seo & site health audit reports this alongside checks from other categories that are at least as serious, including Broken internal links, where an internal link that returns an error is a dead end for the reader and a wasted request for the crawler, and Duplicate body content, where two pages with identical body text are one page at two addresses, and search engines will pick one of them for you. A single run of seo & site health audit answers all of them at once.
The first few lines
You are a senior web engineer. I ran a security and SEO scanner against my site and it reported the finding below. Fix it properly rather than suppressing the symptom.
Finding: robots.txt present and usable (medium severity)
Scanner check id: seo-robots-txtThe rest of this prompt names the pages and line numbers we found the problem on, the configuration to change, and the constraints a good answer has to respect. It is free, it just needs an account so the work is not scraped wholesale.
Sign in with GoogleSigning in is free and takes one click. We store your email address and nothing else.