.onion Mapper

Initially I had the idea of developing this during Codebits, but I ended up in developing this for fun before the event, invalidating any idea of using it.

.onion is a “non official” top-level domain suffix, the big thing about this TLD is that you can only access domains in it over the TOR network.

Due to the anonymity characteristics of the TOR network and the necessity of using it to access this TLD, very often this type of networks are called deep web.

The idea here was to crawl the .onion network, but instead of crawling and data mining its contents I just wanted to crawl its server’s relationships.

The stack is very simplistic, at the infrastructure level this was built using a main control node. Which runs node.js and a Redis instance.

Additionally there were multiple crawlers running a onion tweaked version of crawler4j, each crawler grabs the *.onion links in the html code and saves them (domains relationships) in Redis using Jedis. At the network level Polipo was used as a proxy and obviously tor client.

Each domain relationship is displayed in a graph which is rendered using sigma.js, all data is delivered to the browser using socket.io.

In two days, it crawled 1.5M urls finding and mapping relationships between 440 domains. Keep in mind that this crawling was done inside the tor network, which sometimes have very high latency times.

Finally here it is.

Advertisements
Previous Post
Leave a comment

2 Comments

  1. Hi there,

    I’m currently researching the possibility to predict networks attacks based on chatter on various sources. For that I would like to index and map the darknet. I was wondering if you would share your sources / instructions for your onion mapper with me.

    Kind regards.

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s