Large sites are collected in parts you control
A site crawl or technical audit now collects up to 1,000 pages, or about eight minutes of crawling, per call. When it finishes, the status answer tells you how many known pages remain and gives you the job to continue from. Say continue and the next call picks up where the last one stopped, skips everything already collected, and is charged on its own, so you decide after each part whether to keep going. Each part comes with its own downloadable files.