Troubleshooting6 min read
When a crawl stalls, fails, or returns partial results
Crawls depend on your site being reachable, fast enough to respond, and allowed by robots policy. Partial results usually mean some URLs timed out or were blocked while others succeeded.
Crawls, exports, and when something does not finish as expected.
Steps
- Open the job detail page and read the status message and any error summary.
- Verify the seed URL loads in a browser from a network similar to the public internet (not only VPN-corporate split tunnel).
- Check robots.txt and meta robots for accidental disallow or noindex on important sections.
- Look for rate limiting or bot protection (WAF, Cloudflare) that may challenge automated clients.
- Retry with a smaller scope (single path or fewer URLs) to isolate whether the issue is global or section-specific.
- If still blocked, contact support with job id, approximate time, and example URLs that failed.
Common pitfalls
- Assuming Inseo is down when the origin server returns 5xx or times out for every URL.
- Running aggressive parallel crawls against a tiny shared host that cannot handle concurrency.
Frequently asked questions
Does Inseo respect robots.txt?
Yes for standard web crawling behavior. If you need a different arrangement for a controlled staging host, discuss options with support.