Skip to main content
SecurityApril 28, 20269 min read

Threat modeling for public sites: crawlers, APIs, and bot traffic

A practical framework for prioritizing fixes when your marketing site is scanned constantly by search engines, vendors, and opportunistic bots.

Threat modeling for public sites: crawlers, APIs, and bot traffic

Marketing sites rarely look like “critical infrastructure,” yet they sit on the same edges as APIs: public URLs, predictable forms, and predictable admin paths. The first step is not buying another tool—it is writing down who can reach what, without authentication.

Start with assets: origin hosts, edge headers, form endpoints, webhooks, and any authenticated islands on the same registrable domain. Then enumerate actors: major search crawlers, SEO/AI crawlers you’ve opted into, unknown agents, and scriptable clients that ignore robots.txt. For each actor, decide what “allowed” means in policy vs. what your logs actually show.

Finally, connect gaps to measurable controls: rate limits at the edge, WAF rules tuned with observed false positives, dependency and TLS hygiene, and tight CSP where your stack allows it. Revisit quarterly; crawl and bot ecosystems change faster than most roadmap cycles.

Related articles