Tool directory
Screaming Frog SEO Spider
The right tool the moment you care about more than one page. Free up to 500 URLs.
Last reviewed 09 August 2026. This subject moves monthly — products ship, crawlers change and documentation is rewritten without announcement. For anything newer than this page, Screaming Frog’s own site is the source of truth.
- Category
- Site crawler
- Roughly
- Free for small crawls, licensed beyond that
- Best for
- Auditing an entire site at once
- Weak at
- Telling you what to write — it finds problems, not words
- Overlaps with us
- We check one page deeply; it checks every page broadly
Why it belongs here
Our checker does one page at a time, which is right for diagnosis and wrong for a forty-page site. Screaming Frog crawls everything and answers questions like: which pages have no structured data, which images have no alt text, which titles exceed 60 characters, which pages are orphaned.
The three configurations that matter
1. Structured data validation. Configuration → Spider → Extraction → tick JSON-LD, Microdata, and both validation options. The Structured Data tab then lists every validation error on the site.
2. JavaScript rendering. Configuration → Spider → Rendering → JavaScript. Crawl once with it off and once with it on. If the two crawls differ significantly, the difference is content that AI crawlers probably cannot see. This is the single most useful thing the tool does for AI visibility and almost nobody runs it.
3. Custom extraction. Pull any element by XPath across every page — for example, checking whether your author byline actually appears everywhere you think it does.
The free tier
500 URLs, no saved configurations, no scheduling. That covers most small business sites entirely.
Verdict
Once you are past a handful of pages, this is the highest-leverage tool in the directory, and the JavaScript-versus-raw crawl comparison has no substitute.
Commercial disclosure. No link in this directory earns us a commission. If that ever changes, the link will be marked (commercial link), carry rel="sponsored", and this note will say so. What we write does not depend on it: nothing here is a paid placement, the order is not for sale, we review tools that pay us nothing, and we say plainly when a tool is better than ours or when you should spend nothing at all. If that ever stops being true, this directory is worthless and so is the rest of the site.
On pricing. We describe what a tool is for, not what it charges. Prices move, free tiers get withdrawn or introduced, plans get renamed, and what you pay differs by region and currency. Anything we wrote about cost could be wrong by the time you read it, so treat it as a rough shape and check the vendor's own pricing page before you buy. Spotted something out of date? Tell us and we will fix it.
Take this to your assistant
Paste it into ChatGPT, Copilot, Claude or Gemini and apply it to your own website.
Nothing is sent anywhere. The text is copied to your clipboard.