OpenLinkProfiler.org offers information about the inbound links of websites. It answers the question: Who links to whom on the Internet?
To get this kind of information, OpenLinkProfiler.org uses a web crawler with the name spbot. Basically, it starts with a list of known URLs from across the entire Internet, then it fetches local links found as it goes. There are several advantages to this approach, most importantly that it creates the least possible disruption to the sites being crawled.
We will not index anything you would like to remain private. All you have to do is tell us. How? By using the official Robots exclusion standard. Example:
User-agent: spbot Disallow: /
You can find detailed instructions on how to block web crawlers on Wikipedia and on the robots.txt page. Make sure that you don't inadvertently block other crawlers that you don't want to block, for example, the Googlebot. Our bot also supports the crawl-delay directive. We recommend that you use this directive if you want to limit the number of requests:
User-agent: spbot Crawl-delay: 10
Before you block spbot, consider the following:
Here's our offer: you let our web crawler find out to which web pages your website links and in return we help you to find the pages that link to your website, how they link to your website and how these links can be improved to get better rankings on Google for your website.
Just enter your domain name in the search box at the top of this page to get an analysis.
Other web crawlers can spoof the spbot user agent to make them seem legitimate. They seem to be coming from us but they don't. You can verify the IP addresses to make sure that the spbot that visits your site is actually from OpenLinkProfiler.org. We're currently using the hosting company Digital Ocean for our crawlers. Therefore, our IP addresses start with the following numbers:
Digital Ocean may add new IP ranges at any time. The best way to block us is using the robots.txt file and the user agent name spbot.
Feel free to contact us: spbot _AT_ seoprofiler -DOT- com