The fix, in one snippet
User-agent: *
Disallow: /cgi-bin/
Sitemap: https://example.com/sitemap.xmlIllustrative values. Change the paths, hostnames and options to match your own site before using it.
The sections below explain what is tested, why it matters and the alternatives.
What we test
Scan.now requests /robots.txt and, if it exists, parses it per RFC 9309. It reports as information: Disallow paths that look like private or administrative areas (containing admin, backup, private, config, tmp, old, staging, api, cgi-bin), Sitemap directives, and a blanket Disallow: / (an SEO concern rather than a security one, but often unintended). It also flags configuration problems: the path returns HTML with a 200 status (a soft 404, meaning crawlers will parse your homepage as robots rules), a 5xx status (Google treats a persistent 5xx as "disallow all"), a non-text content type, or a file over 500 KiB, which is the limit Google enforces. The check never fails hard; a missing file is simply "no robots.txt", which is a valid state.
Why it matters
robots.txt is a request to well-behaved crawlers, not an access control. Anything listed in it is readable by everyone and is, in practice, the first place an attacker looks for interesting paths. Disallowing /wp-admin/ costs nothing because everyone knows it exists, but disallowing /internal-reports-2024/ tells the world that path exists and that you would rather they did not look. Attack tooling and bug-bounty hunters parse the file for exactly that. There is no security benefit to listing a path that is properly protected by authentication, and no protection at all for one that is not. The right control for "reachable but not indexed" is a noindex signal, which does not disclose the path to non-crawlers; the right control for "not reachable" is authentication. The exposed files guide discusses where this fits.
How to fix it
Keep robots.txt to crawl-management of public content and remove references to anything private. A typical minimal file:
User-agent: *
Disallow: /search
Disallow: /cart/
Sitemap: https://example.com/sitemap.xmlFor pages that must remain reachable but should not be indexed, send a header instead of naming them in robots.txt. nginx:
location /account/ {
add_header X-Robots-Tag "noindex, nofollow" always;
}Apache:
<Location "/account">
Header always set X-Robots-Tag "noindex, nofollow"
</Location>For areas that should not be reachable, add authentication (HTTP basic auth, an IP allowlist or the application's own login) and confirm the path returns 401 or 403 to an anonymous request. Make sure /robots.txt is served as text/plain with a 200 or 404 status, never as a soft-404 page. Cloudflare can serve a static robots.txt from a Worker or a Snippet if the origin cannot; it can also block known aggressive crawlers, but that is a separate control.
Where this fits
robots.txt review is check 13 of 14 that the website vulnerability scanner runs under exposed files and information disclosure, ordered the way they are worth fixing. That ordering is the point: Fixing this one while the check above it still fails buys less than it looks like.
Fix this one first
Above it in the same category sits Redirect parameters on the page (info), where links or forms on the page carry parameters whose names and values suggest a redirect target (next=, return_to=, url=). An attacker who has that does not need this, so it is the better use of the same hour.
What fixing this still leaves open
robots.txt review closes one route in. The next one down is security.txt (RFC 9116) present, where no security.txt file was found at /.well-known/security.txt, or the one present is invalid or expired.
Found in the same scan
The website vulnerability scanner reports this alongside checks from other categories that are at least as serious, including Certificate chain and hostname validation, where the certificate presented for this hostname did not validate: The chain does not reach a trusted root, an intermediate is missing, the name does not match, or the certificate is self-signed or expired, and HTTPS is available, where the site could not be reached over HTTPS on port 443, or the TLS handshake failed. A single run of website vulnerability scanner answers all of them at once.
The first few lines
You are a senior web engineer. I ran a security and SEO scanner against my site and it reported the finding below. Fix it properly rather than suppressing the symptom.
Finding: robots.txt review (info severity)
Scanner check id: robots-txtThe rest of this prompt names the pages and line numbers we found the problem on, the configuration to change, and the constraints a good answer has to respect. It is free, it just needs an account so the work is not scraped wholesale.
Sign in with GoogleSigning in is free and takes one click. We store your email address and nothing else.
