-
Notifications
You must be signed in to change notification settings - Fork 55
Usage
See and personalize configuration files under priv/*.conf, especially priv/ebot_web.conf
Start Rabbitmq wiith rabbitmq_server/sbin/rabbitmq-server
Start Couchdb/Riak server
Start statistics with ebot/utils/update_rrd.sh
Start Ebot with ebot/start.sh
- http://localhost:8000/crawlers/start
- http://localhost:8000/crawlers/stop
- http://localhost:8000/crawlers/check_recover
- http://localhost:8000/crawlers/add_url?url=http://www.redaelli.org/
And see src/ebot_test.erl for details
You can monitor how many urls are wating in MQ server with
http://localhost:8000/stats/show
or running
rabbitmq_server/sbin/rabbitmqctl list_queuesthe output will be something like
ebot.url.refused 37
ebot.url.new.1 8830
ebot.url.new.0 3238
ebot.url.new.2 5683
ebot.url.new.3 0
ebot.url.new.4 0
In ebot.url.new.0 there will be urls with depth == 0 (ie http://www.redaelli.org, http://www.redaelli.org/index.html, …)
In ebot.url.new.1 there will be urls with depth == 1 (ie http://www.redaelli.org/matteo/, http://www.redaelli.org/matteo/index.html, …)