Our crawler

What you're looking at

isitsellingme/1.0 (+https://isitsellingme.com/bot)

If that's in your logs, it's us. This site grades how companies handle a refusal of advertising data collection, and every grade quotes the company's own published policy, word for word and dated. The crawler exists to fetch and store those policy pages so a quotation can be checked against a document instead of taken on trust.

What it fetches, and how often

• 65 URLs in total, across every company on the site: privacy policies, cookie notices and advertising-preference pages. Public documents, all of them.
• One pass a week, on Monday mornings shortly after 06:15 UTC. That's one request per URL, not a crawl. It never follows links, never discovers new pages, and never touches anything but the addresses on its list.
• No logins, no forms, no search. It reads what you serve to anyone.
• A page is stored only when its content hash changes, but it'sfetched every week either way, so the record can say "we looked, and it was the same" instead of leaving a gap.

It sometimes sends a browser User-Agent

We ask under our own name first. If a host refuses that, the crawler tries once more with an ordinary Chrome User-Agent, and if that also fails it looks for a copy in the Internet Archive instead.

Better to say that here than leave it for someone to find in their logs. Several companies on this site serve their privacy policy to a browser and refuse it to anything identifying itself as automated. That's a document they're legally required to publish, withheld from the only kind of reader that would check it systematically. We're not willing to let that be the reason a company goes ungraded.

Every attempt is recorded either way, under its own name, so the archive shows per host whether an identified crawler was allowed to read the policy. If you'd prefer we always identified ourselves to you, say so and we'll pin your host to the honest User-Agent.

It doesn't consult robots.txt

Most crawlers should. This one fetches a fixed list of 65 public legal documents once a week, and honouring a Disallow would let a company opt out of having its published policy quoted accurately. That's the behaviour this site exists to document.

That's a deliberate position, not an oversight, and you're entitled to disagree with it. Blocking us by User-Agent or IP works, we won't route around it, and the entry will say that the document couldn't be retrieved.

How to stop it, or reach a person

• Block it. Deny on the User-Agent above. One request a week will stop.
• Tell us the page moved. The commonest problem is a stale URL on our side, which wastes your bandwidth and gets us the wrong document. We'd prefer to fix it.
• Dispute what we quoted. Every company gets a right of reply, published word for word beside its entry, uncut. See the methodology for how grades are derived, and corrections for what we've got wrong.

Contact: the about page has the address, and security.txt covers anything that needs a security contact and not an editorial one.