Usage

Setup

See and personalize configuration files under priv/*.conf

Managing Crawlers

• http://localhost:8000/crawlers/start
• http://localhost:8000/crawlers/stop
• http://localhost:8000/crawlers/check_recover

And see src/ebot_test.erl for details

AMQP

You can monitor how many urls are wating in MQ server with

rabbitmq_server/sbin/rabbitmqctl list_queues

the output will be something like

@
ebot.url.refused 37
ebot.url.new.1 8830
ebot.url.new.0 3238
ebot.url.new.2 5683
ebot.url.new.3 0
ebot.url.new.4 0
@

In ebot.url.new.0 there will be urls with depth == 0 (ie http://www.redaelli.org, http://www.redaelli.org/index.html, …)
In ebot.url.new.1 there will be urls with depth == 1 (ie http://www.redaelli.org/matteo/, http://www.redaelli.org/matteo/index.html, …)