Skip to content
matteoredaelli edited this page Aug 28, 2010 · 31 revisions

Setup

See and personalize configuration files under priv/*.conf, especially priv/ebot_web.conf

Start

Start Rabbitmq wiith rabbitmq_server/sbin/rabbitmq-server
Start Couchdb/Riak server
Start statistics with ebot/utils/update_rrd.sh
Start Ebot with ebot/start.sh

Managing Crawlers

  • http://localhost:8000/crawlers/start
  • http://localhost:8000/crawlers/stop
  • http://localhost:8000/crawlers/check_recover
  • http://localhost:8000/crawlers/add_url?url=http://www.redaelli.org/

And see src/ebot_test.erl for details

AMQP

You can monitor how many urls are wating in MQ server with

http://localhost:8000/stats/show

or running

rabbitmq_server/sbin/rabbitmqctl list_queues

the output will be something like

ebot.url.refused 37
ebot.url.new.1 8830
ebot.url.new.0 3238
ebot.url.new.2 5683
ebot.url.new.3 0
ebot.url.new.4 0

In ebot.url.new.0 there will be urls with depth == 0 (ie http://www.redaelli.org, http://www.redaelli.org/index.html, …)
In ebot.url.new.1 there will be urls with depth == 1 (ie http://www.redaelli.org/matteo/, http://www.redaelli.org/matteo/index.html, …)

Clone this wiki locally