How to check whether a website crawl is complete

Crawl completeness needs a defined scope: which URLs or page types matter, which fields must be returned and when the material was collected. Without an expected population, you can describe what a crawler found, but you cannot prove that it found every page on the site.

Start with the research decision. A competitor offer review may need current product, pricing and service pages. A technical audit needs a different population and different evidence. An archive containing both kinds of page is not automatically adequate for either task. Compare search results versus page extraction before deciding which source records the brief actually needs.

• State the allowed host, subdomains, paths and languages.
• Name required page types and fields, such as main text, title, source URL and publication date when present.
• Record intentional exclusions, including duplicates, private areas or file formats outside the capture method.
• Define what would make the result unusable: for example, no pricing-page capture when the question is about pricing.

An expected URL list gives you a denominator independent of the crawl result. On a site you own, compare the relevant published content inventory, XML sitemap and internal-link discovery. On a site you do not own, record which of those sources are actually available.

A link-following crawler can miss a page that has no discoverable path from its starting URL. A sitemap can contribute URLs the crawl did not find, but the sitemap itself is not proof of a complete or current website inventory. Treat a search-engine result list as another discovery lead, not an exhaustive page list.

Record where each URL came from and how you handled it. For a redirect, keep the requested URL and the destination you actually reached. If two URLs appear to cover the same page, check that the returned content serves the same purpose for your research before counting them together. Record intentional exclusions, such as PDFs outside the scope, separately.

required URLs with no usable capture
= expected in-scope URLs − URLs with verified usable content

Review each difference as:
not discovered | not attempted | failed | content incomplete | excluded with reason

If the site population remains unknown, keep that limitation in the final research note. A complete list of discovered URLs is still only a discovered list.

Crawl counters describe different stages of collection. Read the tool’s definitions and inspect the URL-level records before treating the totals as a coverage measure. The most useful question is which required URLs have a usable result, rather than whether the run returned a large number.

Check whether the run stopped because its frontier was exhausted, its page or credit ceiling was reached, a time limit intervened, or errors prevented continuation. If the totals do not reconcile, investigate the actual statuses; some tools also track skipped, excluded or deduplicated URLs. Do not force every tool into one arithmetic formula.

A successful HTTP response shows that a server returned a response; it does not show that the intended content was extracted. Open the saved page representation and check the information your research needs. A login screen, empty application shell or error message can be returned where the useful page content should be.

Inspect at least one example of every materially different page type, plus every capture that could change the decision. For a competitor offer review, read the actual offer and pricing captures rather than only a homepage sample. Compare important returned fields with the source page when the extraction looks thin or inconsistent.

If the required fields appear only after a click or scroll and you are choosing where that automation will run, compare a managed browser with hosted scraper execution. Run the same interaction and check the returned records before expanding the crawl.

If you are considering replacing an existing page extraction or crawl workflow, compare Firecrawl alternatives by the job you need to keep running. Check the same required URLs and fields before moving a recurring job.

Check that the main text includes the relevant section, table, pagination or dynamically loaded content. Check source URL, collection time and field meaning. “Not present on the page” and “not captured by this method” must remain different states.

If the crawl found the right URL but the extracted records are incomplete, compare the raw response, rendered page, and scraper output to see where the records disappear.

Sampling is a quality check, not proof that every unsampled record is sound. Record the sample and its limits. When a missing field is essential to the decision, resolve that page individually or leave the conclusion incomplete.

Record the URL, capture status, required field, observed field result and next action. Keep a URL with an essential missing field unresolved even when the crawler counted the fetch as successful.

Diagnose a missing page by locating where the collection stopped: discovery, eligibility, fetching, rendering or extraction. That distinction tells you which check is likely to help and avoids repeating an unchanged crawl.

If you suspect a crawl restriction, check the robots.txt rule for the crawler and URL you used. For a Google indexing question, check the noindex directive separately. Google has to fetch the page to read that directive, so a page can be captured successfully while still carrying noindex. Once you have checked coverage and URL behavior, turn crawl findings into an SEO fix list with a retest for each issue. See Google’s noindex documentation.

Respect the source’s access controls and collection rules. A different rendering method does not grant access permission. If the required content cannot be acquired through an authorized route, record the gap rather than silently replacing it with a snippet.

In an MCP Scraper technical audit of mcpscraper.dev on November 6, 2026 Mountain time, the run discovered 28 URLs and recorded 26 successful captures. The run was partial. Its counters show why the successful-capture count alone would have been a poor completion claim.

Sources: [crawl-study]

The failed capture was /llms, which returned HTTP 200 with text/plain content. The HTML extraction workflow did not treat that response as a successful page capture. Calling the URL broken would have confused a capture-method limitation with a website failure.

Repeat the status and content-type check to verify how /llms responds now. Keep that result separate from the historical extraction record and the unnamed remaining URL.

We checked the original job’s status against its export manifest. The status counted 28 discovered URLs. The manifest listed 27 attempted URLs, with 26 successes and one failure. The status counted one remaining URL without naming it, so we could not tell whether it was a page the research needed. The run finished below its 200-page cap.

Run: Partial. Discovered 28, attempted 27, succeeded 26, failed 1, remaining 1. Page cap 200.
Failed URL: /llms returned HTTP 200 with text/plain. Classify as an HTML-extraction mismatch.
Remaining URL: Unnamed in the returned status. Whether it was required remains unknown.
Next check: Compare your required URL list with the manifest. Verify any missing priority URL directly.

Use the same check on your crawl. Match your required URLs to the manifest, then open the successful captures that contain the fields you need. If a required URL has no usable capture, record that gap before relying on the crawl for a decision.

Read the public crawl example for the recorded scope, counters and interpretation. This single run does not establish completeness of every route on the website or performance on other sites.

Recovery starts by inspecting the original run and keeping its successful captures. A partial result may already answer a bounded research question; an unresolved critical page may prevent the answer. Let that distinction determine the next collection step.

If a request times out before you know its outcome, check the original job status before submitting replacement work. Where the tool provides an idempotency key, follow its documented retry contract. A timeout does not prove that no work occurred.

Read the retained failure details and artifacts. Retry only the failed or missing scope when the tool supports it, and change the condition that caused the failure. A full rerun with the same settings is unlikely to explain an undiscoverable URL or an unsupported content type.

Keep the original and recovered results separately dated. If a retry collects changed content, do not present it as one simultaneous snapshot. If no supported resume route exists, create a clearly scoped follow-up capture and reconcile it against the original inventory.

A crawl is ready to use when its remaining uncertainty cannot overturn the specific conclusion you intend to draw. Record that decision against the required URL and field list, with a reason for each exclusion. There is no universal page count or completion percentage that can make that judgment for you.

• Complete for the stated scope: every required URL has a verified usable capture or an explicitly justified exclusion, and essential fields have been checked.
• Usable partial evidence: the captured subset supports a narrower conclusion, and the missing material is named beside that conclusion.
• Not ready for the decision: an essential page, field or source boundary remains unresolved. Continue collection or withhold the affected answer.

Attach a coverage note to the research output: the question, expected URL sources, collection date, limits, captured URL list, checks performed, failures, exclusions and next action. Preserve the result that supports the conclusion, rather than only the conclusion itself.

For a workflow that feeds these results into an agent, see web data for AI agents. The same coverage boundary should travel with the data.

Use MCP Scraper’s collection tools according to the evidence you need: URL discovery, page content or a technical audit. Then inspect the original job and its returned page records before using the result.

1. Use map_site_urls when the immediate job is discovering URLs. Treat its list as the result of that discovery method.
2. Choose extract_site for a multi-page content capture, or audit_site for technical SEO evidence. Set a scope and page ceiling suited to the task.
3. For extract_site or audit_site, keep the job ID and poll check_site_export. Complete and partial exports both offer readback, so check the counters and status before treating the result as complete.
4. Use site_export_read to list the export manifest and read an individual page’s retained JSON, HTML, or Markdown. If the manifest spans several responses, follow the manifest assembly and hash check before counting its records. If map_site_urls returns an artifact ID, use report_artifact_read for the full URL inventory.
5. Reconcile the required URLs and fields, record limitations, and deliberately retain evidence when a later task will need it.

Explore the website collection tools and use the current tool documentation for supported inputs, limits and returned readers. Site behavior and access rules vary; no tool connection guarantees a complete capture.

This guide draws on MCP Scraper documentation, the linked guidance on missing crawl content and Google crawling, and one documented audit of MCP Scraper’s public website. Use the checklist to decide whether the capture supports your research task.

The worked example preserves its original partial status and counters. It is not a multi-site benchmark, a customer result or proof that every route was discovered. The discussion of captured-content inspection does not imply that every record was manually verified.

A third-party crawl, an HTTP response, extracted content and Google’s indexed version are separate evidence surfaces. Use the appropriate tool for each question. This article does not diagnose why a particular page is absent from Google or claim that a complete research capture will improve rankings.