Usage

= Setup =
See and personalize configuration files under priv/*.conf

Then start crawlers: see src/ebot_test.erl for details

You can monitor how many urls are wating in MQ server with

rabbitmq_server/sbin/rabbitmqctl list_queues

the output will be something like

@
ebot.url.refused 37
ebot.url.new.1 8830
ebot.url.new.0 3238
ebot.url.new.2 5683
ebot.url.new.3 0
ebot.url.new.4 0
@

In ebot.url.new.0 there will be urls with depth == 0 (ie http://www.redaelli.org, http://www.redaelli.org/index.html, …)
In ebot.url.new.1 there will be urls with depth == 1 (ie http://www.redaelli.org/matteo/, http://www.redaelli.org/matteo/index.html, …)