/bot
The SEOtto crawler
If this string is in your access log, somebody who runs that site — or works for whoever does — asked SEOtto to watch its pages for changes that affect search: titles, canonicals, robots directives, the sitemap. We read what any visitor can read, slowly, and we identify ourselves on every request.
access.log
07:12:00 GET /robots.txt 200 "SEOtto/1.0 (+https://seotto.com/bot)"
07:12:01 GET /sitemap.xml 200 "SEOtto/1.0 (+https://seotto.com/bot)"
07:12:02 GET /pricing 200 "SEOtto/1.0 (+https://seotto.com/bot)"
07:12:03 GET /blog/sizing-guide 200 "SEOtto/1.0 (+https://seotto.com/bot)"
User-Agent
Always the same string
SEOtto/1.0 (+https://seotto.com/bot)
concurrency
Eight requests at a time, at most
Per site, across the whole crawl. Customers can add a pause between batches for a fragile origin, and we lower it on request.
If-Modified-Since
Never a conditional request
A 304 has no body, so every fetch is a plain GET. Responses are cut off at 2 MB.
Location
Redirects followed by hand
Up to five hops, each one recorded, and each one re-checked so we never follow a redirect into a private network.
robots.txt
How to slow it down, or stop it
What robots.txt does and does not cover, said plainly.
robots.txt governs discovery. When we find pages by following links, a rule for our token stops us following links into the paths it disallows — as does a rule for *, when there is no group naming us:
User-agent: SEOtto
Disallow: /It does not govern what the site's owner asked for by name. Pages listed in your own sitemap, pages the owner added to their monitor by hand, and the handful of site-level files a monitor exists to read — /robots.txt, /sitemap.xml, and the home page with its http:// and www variants, to see that they redirect — are fetched regardless. A monitor that could be told not to look at the page it is monitoring would be no use on the day that page broke.
If you run the origin and did not ask for this, write to [email protected] with the hostname. Adding a site we are not authorised to crawl is against our terms; we will confirm with the account that added it, and stop crawling it either way while we do. If it is fetching too fast for your server, say that instead and we will slow it down.
Blocking the user agent at your edge works immediately and is fine by us. The customer will see every page fail, which is the correct thing for them to see.
What we do with what we read
We keep the signals that decide a search result — status, redirects, title, description, headings, canonical, robots directives, structured data types, link and word counts — and show them, with their history, to the account that is monitoring the site. We do not keep page bodies, we do not follow forms, we do not run your scripts, and we do not fetch anything behind a login.