The fix, in one snippet
grep -rn "/old-url-path" templates/ content/
# only if it genuinely moved:
location = /old-url-path { return 301 /new-url-path; }Illustrative values. Change the paths, hostnames and options to match your own site before using it.
The sections below explain what is tested, why it matters and the alternatives.
What we test
Every URL the crawler requests is recorded with its final status code after redirects. This check collects everything that returned 4xx or 5xx, or failed to respond at all, and reports the status alongside the page that linked to it where we can determine that.
Only URLs reachable from the entry point within the crawl limit are included, so this is a sample of the site rather than a complete inventory.
Why it matters
A 404 reached from your own navigation is a broken promise to the visitor and a wasted request for the crawler. At scale it eats crawl budget that would otherwise go to pages that work.
A 5xx is more serious. Search engines back off when they see server errors, reducing crawl rate across the whole site, and pages that return 5xx repeatedly are dropped from the index faster than pages that return 404, because the engine assumes the site is in trouble.
Both are usually symptoms of the same thing: content was removed or renamed and the links to it were not updated.
How to fix it
For each error, fix the link or fix the target. Updating the link is always better than adding a redirect, because a redirect costs a round trip on every visit forever.
# find the offenders in your templates and content
grep -rn "old-url-path" templates/ content/
# if the page genuinely moved, redirect once, permanently
location = /old-url-path { return 301 /new-url-path; }Serve a real 404 status for pages that are gone; a "not found" page returning 200 is a soft 404 and keeps the URL in the index. For 5xx errors, read the server log for the specific request: the status is a symptom and the cause is in the application.
Where this fits
Crawl errors (4xx / 5xx) is check 1 of 10 that the seo & site health audit runs under indexing and crawlability, ordered the way they are worth fixing. That ordering is the point: Fixing this one while the check above it still fails buys less than it looks like.
What fixing this still leaves open
Crawl errors (4xx / 5xx) closes one route in. Immediately below it: Pages excluded with noindex, where a noindex directive removes a page from search results entirely; Redirect loops, where a URL that redirects back to something already in its own chain never resolves; Canonical URL missing, where a canonical link tells search engines which URL is the real one when the same content is reachable at several addresses.
Found in the same scan
The seo & site health audit reports this alongside checks from other categories that are at least as serious, including Broken internal links, where an internal link that returns an error is a dead end for the reader and a wasted request for the crawler, and Duplicate body content, where two pages with identical body text are one page at two addresses, and search engines will pick one of them for you. A single run of seo & site health audit answers all of them at once.
The first few lines
You are a senior web engineer. I ran a security and SEO scanner against my site and it reported the finding below. Fix it properly rather than suppressing the symptom.
Finding: Crawl errors (4xx / 5xx) (high severity)
Scanner check id: seo-http-status-errorsThe rest of this prompt names the pages and line numbers we found the problem on, the configuration to change, and the constraints a good answer has to respect. It is free, it just needs an account so the work is not scraped wholesale.
Sign in with GoogleSigning in is free and takes one click. We store your email address and nothing else.