Skip to content

About RankwellBot

The crawler behind technical monitoring and the free URL tools.

RankwellBot is the crawler behind technical monitoring in Rankwell Insights and the free URL tools on this site. If you are seeing it in your logs, a customer who has confirmed they are authorised to monitor your website has scheduled a crawl, or someone checked one of your URLs with a free tool.

How it behaves

  • It identifies itself with a user agent beginning RankwellBot/1.0.
  • It obeys robots.txt, including Crawl-delay.
  • It requests one page at a time with a pause between requests and stops at a page limit.
  • It only requests public web pages over HTTP and HTTPS.

How to block it

Add this to your robots.txt:

User-agent: RankwellBot
Disallow: /

If you believe it is being used against a site without authorisation, please use the contact page and we will investigate.