Crawl a Website and Find Every SEO Problem

Crawls your site the way a search engine does, following internal links and your sitemap, and reports everything holding it back: missing, short, long and duplicate titles; thin main content; indexability, descriptions, headings, broken and orphaned links, image alt text, speed, structured data and international targeting. It is a sample rather than a full crawl, and the report says which fraction of your site it covered.

A sample, not a full crawl: We start at this URL, follow internal links and read the sitemap, up to the page budget you choose. On a large site that is a fraction of the pages, and the report says which fraction. How we scan · Acceptable use

One scan is a snapshot. Run the whole audit.

This page checks one thing. Premium points every scanner at the same domain in a single pass, then hands you the result in a form you can act on the same afternoon.

Every scanner, one run. Security, LLM key exposure, SEO, TLS, email authentication and WordPress against one domain, merged into one report rather than separate tabs.
Up to 5,000 pages. 25 pages find the pattern your templates repeat. 5,000 find the exceptions, which is where the real problems hide.
A PDF for the meeting, a text file for the AI. Every finding, its evidence and the exact lines, written so an assistant can work through it without you summarising anything.
Certificates watched, not remembered. Every host you scan is re-checked daily and you are told a week before one expires.

What is included

How it works

Enter your site
We start at the URL you give and follow internal links, honouring robots.txt.
Up to your chosen budget is crawled
30, 100 or 250 pages, seeded from your entry point and your sitemap. Each is recorded with its status, redirect chain, timing, size, headings, links and metadata.
Read the issues, most serious first
Short or overlong titles and thin main content are labelled as review signals; site-wide problems such as duplicate titles and orphan pages only appear across a crawl.

What this tool checks

47 checks across 7 areas. Each links to its reference page with the test, the risk and the fix.

Full reference

Titles, headings and content

13 checks

Indexing and crawlability

10 checks

International targeting

3 checks

Links and site structure

7 checks

Images and page weight

3 checks

Structured data and social

4 checks

Technical SEO and speed

7 checks

The ones that most often decide the grade:

High Pages excluded with noindexA noindex directive removes a page from search results entirely. Applied by accident, it is the fastest way to lose traffic.

High Crawl errors (4xx / 5xx)URLs reached during the crawl that returned a client or server error. Every one is a dead end for a visitor and for...

High Redirect loopsA URL that redirects back to something already in its own chain never resolves. The page is unreachable for everyone.

High Page title missingThe title element is the headline of every search result and the label on every browser tab. A page without one has no name.

High Duplicate body contentTwo pages with identical body text are one page at two addresses, and search engines will pick one of them for you.

High Broken internal linksAn internal link that returns an error is a dead end for the reader and a wasted request for the crawler.

Medium robots.txt present and usablerobots.txt is the first file a crawler asks for. It sets crawl rules for the whole site and is where a sitemap is...

Medium XML sitemap published and declaredAn XML sitemap lists the URLs you want crawled and when they last changed. Search engines use it to find pages that...

Medium Sitemap URLs are live and indexableEvery URL in a sitemap is a claim that the page exists and should be indexed. We cross-check the URLs we also...

Medium Linked pages blocked by robots.txtA page you link to internally but disallow in robots.txt sends contradictory signals and wastes the link.

Medium Canonical URL missingA canonical link tells search engines which URL is the real one when the same content is reachable at several addresses.

Medium Canonical points to another URLA canonical pointing at a different URL asks search engines to index that URL instead of this one. Correct for...

Medium Redirect chainsA URL that reaches its destination through two or more redirects costs every visitor and every crawler extra round trips.

Medium Title length outside the useful rangeTitles are truncated in search results at roughly 600 pixels. Too long and the end is cut; too short and the page...

Medium Duplicate titles across pagesTwo pages with the same title compete for the same queries and give searchers no way to tell them apart.

Medium Meta description missingThe meta description is the snippet under the headline. Without one, the engine writes it for you from whatever text...

Low Meta description lengthDescriptions much shorter than 70 characters waste the space; much longer than 160 are cut off mid-sentence.

Low Duplicate meta descriptionsThe same description on many pages tells a searcher nothing about which one to open.

Medium H1 missingThe H1 is the page's visible headline. Its absence usually means the page has no clear subject in its own markup.

Low More than one H1Several H1s on one page split its subject. HTML5 permits it; readers and assistive technology still do better with one.

Low Heading levels skippedHeading levels should descend one at a time. Jumping from H2 to H4 leaves a hole in the document outline.

Medium Thin contentPages with very little body text rarely satisfy the query that brought someone to them, and rarely rank for anything...

Info Title does not match the page's own wordsIf none of a page's most frequent words appear in its title, the title and the content are describing different things.

Low Generic or empty anchor text"Click here" describes the action, not the destination. Anchor text is how both readers and engines learn what a...

Medium Orphan pagesA page nothing links to is reachable only by knowing its address. Crawlers find it late, if at all.

Info Pages buried deep in the structurePages several clicks from the entry point are crawled less often and carry less internal authority.

Medium Internal links marked nofollowrel=nofollow on a link to your own page tells search engines not to pass authority through it. Inside a site that is...

Medium Images without alt textAlt text is what a screen reader announces and what an image search engine reads. Without it an image is invisible to both.

Medium Oversized HTML documentsA very large HTML document delays rendering and can be truncated by crawlers before the end of the page is read.

Low Images not lazy-loadedImages below the fold that load immediately compete for bandwidth with the content the reader can actually see.

High Viewport meta tag missingWithout a viewport meta tag, mobile browsers render the page at desktop width and shrink it. Google indexes the...

Low Language attribute missingThe lang attribute on <html> tells screen readers which pronunciation to use and search engines which language to...

Medium HTML served uncompressedHTML served without gzip or brotli transfers several times more bytes than it needs to.

Info No cache policy on HTMLHTML with no cache policy is re-downloaded in full on every visit, including every back-button navigation.

Medium Slow server responseHow long the server took to return each page. Slow responses delay everything that follows and limit how much of the...

Medium Render-blocking scripts in the headSynchronous scripts in the head stop the parser. Everything below them waits, including the text the reader came for.

Medium http:// references on HTTPS pagesLinks from an HTTPS page to http:// URLs cost a redirect on every click and expose the first request.

Medium Structured data (JSON-LD) missing or invalidStructured data describes a page in a vocabulary search engines parse directly. Without it, no rich result is possible.

Low Open Graph tags missingOpen Graph tags decide how a link looks when it is shared. Without them, a shared page is a bare URL.

Info Twitter card missingA twitter:card tag selects the card format on X. Without one, the platform falls back to Open Graph.

Info Site icon missingThe site icon appears in tabs, bookmarks, history and increasingly in search results themselves.

Medium hreflang annotations invalidhreflang tells search engines which language and region each version of a page is for. Invalid codes are ignored silently.

Medium hreflang return links missinghreflang must be reciprocated. If page A names B but B does not name A, search engines ignore the whole annotation.

Medium Pages with little internal link supportPages that almost nothing links to inherit almost no authority, however good they are.

Info Internal anchor text repetitionEvery link to a page using the same words tells search engines one thing about it, repeatedly.

Info Reading difficultyHow hard the page is to read, measured rather than guessed: Sentence length, word length and the Flesch score they produce.

Medium Language declarations disagreePages declaring different languages with nothing connecting them leaves search engines guessing which version...

What a multi-page SEO audit measures

The crawler combines page-level checks with site-wide patterns that cannot be seen from a single URL.

Technical SEO is partly about individual pages and partly about relationships across a site. A missing title is visible on one page, while duplicate titles, orphaned sitemap URLs and inconsistent canonicals appear only when many pages are compared. The audit preserves both levels of evidence.

Scan.now begins with the entered URL, follows same-site links, reads sitemap URLs and honours robots.txt until the selected page budget is reached. Each response is recorded with indexability, metadata, headings, main-content signals, links, images, structured data, language annotations, size and timing. The report states coverage instead of implying that a bounded sample represents the whole site.

Discover URLsCombine the start page, internal links and sitemap entries.

Inspect every pageRecord indexability, metadata, content, links, media and technical signals.

Compare the crawlFind duplicates, orphans and recurring template-level problems.

Put this result in context

This scanner answers one part of the review. Continue with Tech Stack Scanner for cms, frameworks, analytics, cdn and hosting, then Common Crawl Rank for web graph centrality and pagerank positions. These are separate checks because a strong result here does not prove that the adjacent surface is healthy.

For the background behind these signals, read Website Security: What a Vulnerability Scan Looks For and How to Fix It and How to Read a Security Scan Report: Severity, False Positives and What to Fix First. The robots.txt present and usable check reference explains the exact test, evidence and remediation used in the report. The Scan.now methodology explains request limits, scan boundaries and what the scanner deliberately does not attempt.

SEO audit sections and the questions they answer

The audit does not collapse every signal into one vague score. Findings remain grouped by the decision they support.

Audit area	Examples examined	Question answered
Discovery and indexability	Status codes, redirects, robots directives, canonical targets and sitemap membership.	Can a crawler reach and index the intended URL?
Search presentation	Titles, descriptions, headings and structured data.	Can a search engine understand and present the page clearly?
Content signals	Main-text length, repeated titles, heading structure and language annotations.	Does the page provide a distinct and coherent subject signal?
Link architecture	Internal links, broken targets, incoming-link counts and sitemap-only pages.	Can users and crawlers navigate to important content efficiently?
Media and delivery	Image alternative text, response size, timing and render-blocking clues.	Are accessibility and delivery problems weakening the page experience?

How to prioritise SEO findings

Fix blockers first, then repeated templates, then page-specific improvements.

Healthy sampled pattern

The sampled pages expose no major crawl or metadata failure in the tested area. Track later crawls because content and templates change.

Indexing or site-wide problem

Blocked important URLs, broken canonicals, server errors or a repeated template defect can affect many pages and deserve priority.

Editorial review signal

Length and wording checks identify pages worth reviewing. They do not decide content quality or search intent by themselves.

Turn the crawl into an SEO work plan

Resolve indexability, canonical, redirect and server-error problems before rewriting metadata.
Fix repeated issues in the shared template rather than editing hundreds of pages individually.
Use affected URL lists to assign work and verify each pattern in a follow-up crawl.
Increase the crawl budget when the reported coverage leaves important directories unsampled.

What this result does not prove

The audit is a bounded sample and explicitly reports its coverage of discovered and sitemap URLs.
JavaScript is not executed, so client-only content may look absent in the server response.
Automated checks cannot judge usefulness, expertise, intent satisfaction or whether a page deserves to rank.

Guides for this tool

Background reading written to go with these results.

13 min read · Updated Sep 13, 2026 Website Security: What a Vulnerability Scan Looks For and How to Fix It

A website vulnerability scan checks the parts of a site an attacker can reach without logging in: Transport security, HTTP headers,...

Read the guide

9 min read · Updated Sep 18, 2026 How to Read a Security Scan Report: Severity, False Positives and What to Fix First

A scan report is a list of findings with severities, and the hard part is deciding what matters. How severities are assigned, how to...

Read the guide

Frequently asked questions

How many pages do you crawl?

As many as the budget you pick: 30, 100 or 250. Breadth-first from the URL you enter, plus URLs taken from your sitemap, honouring robots.txt throughout. On a large site that is a small sample, and the report states the percentage it covered rather than implying it saw everything. The point is to find the patterns your templates repeat, which a sample shows perfectly well.

Is this the same as Lighthouse?

It overlaps deliberately. Lighthouse audits one page in a real browser and measures rendering; this crawls many pages and finds the problems that only exist across a site. The per-page SEO audits here follow Lighthouse's, and the crawl-wide issue model follows SEOnaut's.

Will the crawl slow my site down?

No. Requests are paced, run four at a time at most, and stop at the budget you chose. Even at 250 pages it is a fraction of the load a search engine crawler places on the same site in a day.

Do you run JavaScript?

No. We read the HTML the server sends, which is what a crawler indexes first. If your content only exists after hydration, this scan shows you what a crawler sees before it runs your scripts, which is usually the more useful view.

Can it find orphan pages?

Only where you have a sitemap. Following links can reach nothing but linked pages, so a crawl on its own can never find a page with no links to it. We seed the crawl with sitemap URLs and report the ones that turned out to have no inbound link from any page we saw. Without a sitemap the check is skipped rather than passed, because passing it would mean claiming something we did not test.

Why does word count matter?

It does not, on its own. It is a proxy for whether a page covers its subject, and it is reported so you can look. A 90-word contact page is fine; ninety 90-word pages generated from a template are the pattern search engines act on.