Usage

See and personalize configuration files priv/*.conf

Then start crawlers: see src/ebot_test.erl for details

You can monitor how many urls are wating in MQ server with

rabbitmq_server/sbin/rabbitmqctl list_queues

the output will be something like

Listing queues … ebot.url.new.0 3446 ebot.url.new.1 24061 ebot.url.new.2 5716 ebot.url.new.3 0

In ebot.url.new.0 there will be urls with depth == 0 (ie http://www.redaelli.org, http://www.redaelli.org/index.html, …)
In ebot.url.new.1 there will be urls with depth == 1 (ie http://www.redaelli.org/matteo/, http://www.redaelli.org/matteo/index.html, …)