BotBotMe · unbracketed · Sep 12, 2014 · Jun 5, 2014 · Jul 9, 2014 · Jul 9, 2014
diff --git a/.env.example b/.env.example
@@ -0,0 +1,23 @@
+# Required
+SECRET_KEY=supersecretkeyhere
+WEB_PORT=8000
+EMAIL_BACKEND='django.core.mail.backends.console.EmailBackend'
+GOPATH=$VIRTUAL_ENV
+WEB_SECRET_KEY=somerandomstring
+STORAGE_URL=postgres://user:pass@localhost:5432/botbot
+REDIS_PLUGIN_STORAGE_URL=redis://localhost:6379/0
+REDIS_PLUGIN_QUEUE_URL=redis://localhost:6379/1
+REDIS_SSEQUEUE_URL=redis://localhost:6379/2
+SSE_ENDPOINT_URL=http://localhost:3000/
+SSE_HOST=localhost:3000
+
+# Optional
+# MEMCACHE_URL=127.0.0.1:11211
+# STATIC_ROOT=/var/www/botbot/static
+# MEDIA_ROOT=/var/www/botbot/uploads
+# DEBUG=True
+# SMTP_URL=smtp://user:pass@host:port
+# SMTP_TLS=True
+# ALLOWED_HOSTS=host1,host2
+# INCLUDE_DJANGO_ADMIN=False
+# EXCLUDE_NICKS=nick1,nick2
diff --git a/.travis.yml b/.travis.yml
@@ -18,6 +18,6 @@ before_script:
   - "psql -c 'create database botbot;' -U postgres"
   - "psql -c 'create extension hstore;' -U postgres botbot"
   - "psql -c 'create extension hstore;' -U postgres template1"
-  - "echo 'DATABASE_URL=postgres://postgres@localhost:5432/botbot' >> .env"
+  - "echo 'STORAGE_URL=postgres://postgres@localhost:5432/botbot' >> .env"
   - "manage.py collectstatic --noinput"
 script: "manage.py test"
diff --git a/README.md b/README.md
@@ -13,62 +13,3 @@ Botbot is collection of tools for running IRC bots. It has primarily been
 used with Freenode channels but works with other IRC networks or servers.
 
 [Documentation](http://botbot.readthedocs.org/en/latest/)
-
-
-Quickstart Guide:
-=================
-
-```
-mkvirtualenv botbotenv
-cdvirtualenv
-
-# Set necessary environment variables e.g. in:
-vi bin/postactivate
-
-export GOPATH=$VIRTUAL_ENV
-export WEB_SECRET_KEY=somerandomstring
-export STORAGE_URL=postgres://user:pass@localhost:5432/botbot
-export REDIS_PLUGIN_STORAGE_URL=redis://localhost:6379/0
-export REDIS_PLUGIN_QUEUE_URL=redis://localhost:6379/1
-export REDIS_SSEQUEUE_URL=redis://localhost:6379/2
-export SSE_ENDPOINT_URL=http://localhost:3000/
-export SSE_HOST=localhost:3000
-
-source bin/postactivate
-
-# Clone and install the Django part
-mkdir src
-cd src
-git clone [email protected]:BotBotMe/botbot-web.git
-pip install -e botbot-web
-pip install -r botbot-web/requirements.txt
-
-# Install the Go related part
-cd ..
-go get github.com/BotBotMe/botbot-bot
-go get github.com/BotBotMe/botbot-eventsource
-
-# Setup the database
-manage.py dbshell
-CREATE EXTENSION hstore;
-CREATE EXTENSION unaccent;
-ALTER FUNCTION unaccent(text) IMMUTABLE;
-
-manage.py syncdb
-manage.py migrate
-manage.py collectstatic
-manage.py createsuperuser
-
-# Run all of the services at once with Foreman
-ln -s src/botbot-web/Procfile
-foreman start
-
-# Or each one manually
-botbot-bot
-botbot-eventsource
-manage.py runserver
-manage.py run_plugins
-```
-
-Go to http://127.0.0.1:8000/admin/ and setup a bot and a channel.
-Finally open http://127.0.0.1:8000/.
diff --git a/botbot/apps/bots/admin.py b/botbot/apps/bots/admin.py
@@ -37,6 +37,7 @@ class MembershipInline(admin.TabularInline):
 
 
 class ChatBotAdmin(admin.ModelAdmin):
+    exclude = ('connection', 'server_identifier')
     list_display = ('__unicode__', 'is_active')
     list_editable = ('is_active',)
     list_filter = ('is_active',)

diff --git a/docs/developers.rst b/docs/developers.rst
@@ -1,8 +1,75 @@
-Developing with Botbot
-========================
+**********************
+Developing with BotBot
+**********************
+
+Architecture
+============
+
+Several loosely coupled pieces make up the whole of BotBot:
+
+1. **botbot-bot:** An IRC client capable of connecting to multiple IRC networks, and connecting multiple channels and nicks per network. (Go)
+2. **botbot-plugins:** A plugin framework - plugins receive messages from IRC channels and can respond within the channel. (Python)
+3. **botbot-web:** A web site for managing bots/channels as well as a beautiful public interface for channel logs. (Python/Django)
+4. **botbot-eventsource:** A basic SSE provider that the web site can connect to for real-time logs. (Go)
+
+.. image:: /images/botbot-architecture.png
+
+Configuration / Environment Variables
+-------------------------------------
+
+At Lincoln Loop we've found the 12-factor style to be helpful and use those guidelines. As recommended, we rely on environment variables whenever possible for determining site-specific configuration values.
+
+The .env file
+^^^^^^^^^^^^^
+
+We've adopted a convention used by Heroku and others of creating a ``.env`` file in the project root that can be used as a base for managing environment variables. Tools like Honcho and Foreman will use the ``.env`` file to bootstrap the environment for each process it starts up. If you prefer to use these tools, all you have to worry about is editing the ``.env`` file as needed. The ``.env`` file can also be used as a template for other methods of populating environment variables.
+
+We don't want make assumptions about how developers prefer to configure their environments. There are some handy open source libs out there that we could use to easily make each service utilize the ``.env`` file, but this won't work well for everyone or every environment. If you need to run a service individually, you'll need to make sure the proper environment variables are configured. The next section describes some easy ways to handle this.
+
+
+Running Services
+----------------
+
+Run All Services
+^^^^^^^^^^^^^^^^
+
+Honcho is great for getting everything started quickly. Running this command will load all services defined in the ``Procfile``::
+
+    honcho start
+
+Run Services Individually
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+When working on code changes or debugging, it is often desirable to manage one or more of the services (bot, plugin runner, runserver, eventsource) independently.
+
+If you'd like to make use of the ``.env`` file, you can still start services individually using Honcho::
+
+    honcho start web  # Starts Django site
+    honcho start bot  # Starts IRC client
+    honcho start plugins  # Starts plugin runner
+    honcho start realtime # Starts event source (SSE)
+
+If you would prefer not to use Honcho, you'll need to manage the environment variables. Many developers use the virtualenv ``postactivate`` feature to set required environment variables whenever a virtualenv is activated. An alternative approach could be to attempt to set variables directly from the ``.env`` file. As an example, you could put the following into a ``set_env.sh`` file::
+
+    export $(cat .env | grep -v ^# | xargs)
+
+Then you could invoke commands and individual services like::
+
+    source set_env.sh && manage.py runserver
+    source set_env.sh && botbot-bot
+    source set_env.sh && botbot-eventsource
+    source set_env.sh && manage.py run_plugins
+
+If you've explicitly set the environment through your own methods, services can be invoked like usual::
+
+    manage.py runserver
+    botbot-bot
+    botbot-eventsource
+    manage.py run_plugins
+
 
 Go IRC Client (bot)
--------------------
+~~~~~~~~~~~~~~~~~~~
 
 Execution starts in main.go, in function "main". That starts the chatbots (via NetworkManager), the goroutine which listens for commands from Redis, and the mainLoop goroutine, then waits for a Ctrl-C or kill to quit.
 
@@ -29,10 +96,27 @@ And now, in ASCII art::
 
 
 Django Site
-------------
+~~~~~~~~~~~~
+
+You can run commands within the Honcho environment using the ``run`` command::
+
+    honcho run manage.py dbshell
+    honcho run manage.py syncdb
+
+If you're using the ``set_env`` method::
+
+    source set_env.sh && manage.py dbshell
+    source set_env.sh && manage.py syncdb
+
+If you've explicitly set the environment variables, run commands like usual::
+
+    manage.py dbshell
+    manage.py syncdb
+
+
 
 Working with LESS
-~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~
 
 LESS requires Node.js. There are shortcuts in the Makefile for installing everything necessary:
 
@@ -49,8 +133,10 @@ From this point forward, if you need to compile LESS run:
 To automatically compile whenever you save a change:
 
 .. code-block:: bash
+
     make less-watch
 
+
 Plugins
 --------
 

diff --git a/docs/getting_started.rst b/docs/getting_started.rst
@@ -42,21 +42,30 @@ Now that your bot is connected to a network or server, you can start having it j
 
 1. Go to ``http://localhost:8000/admin/bots/channel/add/``
 2. Select your bot from the dropdown
-3. **Channel**: ``#botbot-test``
-4. Select **'Is public'**
-5. In the **Active Plugins** section click *Add another Active Plugin* and select **Ping**.
-6. Save. In the ``honcho`` console output you should see messages similar to::
-
-    12:14:42 bot.1     | 2013/09/19 12:14:42 Command:  REFRESH
-    12:14:42 bot.1     | 2013/09/19 12:14:42 Reloading configuration from database
-    12:14:42 bot.1     | 2013/09/19 12:14:42 config.Channel: [#botbot-test ]
-    12:14:42 bot.1     | 2013/09/19 12:14:42 [RAW1] -->JOIN #botbot-test
-
-7. In your IRC client, join `#botbot-test <irc://irc.freenode.net:6667/botbot-test>`_. Try issuing a `ping` command (using your bot's nick in place of "mybot"). The bot should respond with a friendly message.
-8. Go back to the home page ``http://localhost:8000``, you should see the channel listed as a public channel.
-9. **Add another Active Plugin** and this time select **Logger**.
-10. **Save**.  Your ``honcho`` console should once again show a refresh
-11. In your IRC client, go to `#botbot-test <irc:irc.freenode.net:6667/botbot-test>`_ and post a message. You should now have a log available at ``http://localhost:8000/freenode/botbot-test``. Each message you post in the channel shows up in the ``honcho`` console.
+3. **Channel**: ``#botbot-warmup``. This is a channel where we test channel bots.
+4. If you'd like the channel to be listed on the site home page, Select **'Is public'**
+5. Several useful plugins will already be configured. At a minimum, ``ping`` and ``logger`` will be helpful for testing the bot.
+6. Save your channel. You should be directed back to the Channel list view.
+7. On the Channel list view, select your channel. From the ``Actions`` dropdown select "Reload botbot-bot configuration" and press "Go". You should see something similar in the ``honcho`` console output (edited for brevity)::
+
+    14:15:07 bot.1      | I0711 14:15:07.557470 61493 botbot.go:67] Command:  REFRESH
+    14:15:07 bot.1      | I0711 14:15:07.557546 61493 botbot.go:124] HandleCommand: REFRESH
+    14:15:07 bot.1      | I0711 14:15:07.557557 61493 botbot.go:153] Reloading configuration from database
+    14:15:07 bot.1      | I0711 14:15:07.557564 61493 network.go:49] Entering in NetworkManager.RefreshChatbots
+    14:15:07 bot.1      | I0711 14:15:07.558013 61493 storage.go:121] config.Id: 1
+    14:15:07 bot.1      | I0711 14:15:07.558753 61493 storage.go:145] config.Channel: [#botbot-warmup]
+    ...
+    14:15:07 bot.1      | I0711 14:15:07.559072 61493 irc.go:460] [Info] The channels the bot is connected to need to be updated
+    14:15:07 bot.1      | I0711 14:15:07.559083 61493 irc.go:473] [Info] Joining new channel:  #botbot-warmup
+    14:15:07 bot.1      | I0711 14:15:07.559096 61493 network.go:98] Exiting NetworkManager.RefreshChatbots
+    14:15:07 bot.1      | I0711 14:15:07.559111 61493 irc.go:228] [RAW thahslkd334558 on chat.freenode.net:6667 (0xc208028750) ] --> JOIN #botbot-warmup
+
+
+8. In your IRC client, join `#botbot-warmup <irc://irc.freenode.net:6667/botbot-warmup>`_. Try issuing a `ping` command (using your bot's nick in place of "mybot"). The bot should respond with a friendly message.
+9. Go back to the home page ``http://localhost:8000``, you should see the channel listed as a public channel.
+10. **Add another Active Plugin** and this time select **Logger**.
+11. **Save** and "Reload botbot-bot configuration" as before.  Your ``honcho`` console should once again show a refresh
+12. In your IRC client, go to `#botbot-warmup <irc:irc.freenode.net:6667/botbot-warmup>`_ and post a message. You should now have a log available at ``http://localhost:8000/freenode/botbot-warmup``. Each message you post in the channel shows up in the ``honcho`` console.
 
 
 .. warning:

diff --git a/docs/images/botbot-architecture.png b/docs/images/botbot-architecture.png
diff --git a/docs/install.rst b/docs/install.rst
@@ -25,7 +25,7 @@ Postgresql with hStore extension
 Go
 ~~
 
-`botbot-bot` requires a go version greater than 1.2 
+Version 1.2 or higher required
 
 * **OS X**: ``brew install go``
 * **Ubuntu**: ``apt-get install golang-go``
@@ -51,7 +51,7 @@ Run in a terminal:
     make dependencies
 
     # Adjust ``.env`` file if necessary. Defaults are chosen for local debug environments.
-    # If your Postgres server requires a password, you'll need to override DATABASE_URL
+    # If your Postgres server requires a password, you'll need to override STORAGE_URL
     # The default database name is 'botbot'
     $EDITOR .env
 
@@ -80,91 +80,9 @@ You should now be able to access the site at ``http://localhost:8000``. Log in w
 
 See :doc:`getting_started` for instructions on configuring a bot.
 
-Environment Variables to override
----------------------------------::
+If you plan make code changes, please read through the :doc:`developers` doc.
 
-    # Django (required)
-    WEB_SECRET_KEY=supersecretkeyhere
-
-    # DB Storage where channel/bot information is stored
-    STORAGE_URL=postgres://user:pass@localhost:5432/botbot
-
-    # Pipes
-    REDIS_PLUGIN_STORAGE_URL=redis://localhost:6379/1
-    REDIS_PLUGIN_QUEUE_URL=redis://localhost:6379/2
-    REDIS_SSEQUEUE_URL=redis://localhost:6379/3
-
-    # The host and port eventsource is delivering and the browser is listening
-    # for live changes.
-    SSE_ENDPOINT_URL=http://localhost:3000/
-
-    # Specific Django settings to override
-    # MEMCACHE_URL=127.0.0.1:11211
-    # STATIC_ROOT=/var/www/botbot/static
-    # MEDIA_ROOT=/var/www/botbot/uploads
-    # DEBUG=True
-    # SMTP_URL=smtp://user:pass@host:port
-    # SMTP_TLS=True
-    # ALLOWED_HOSTS=host1,host2
-    # INCLUDE_DJANGO_ADMIN=False
-    # EXCLUDE_NICKS=nick1,nick2
-
-Serving In Production
----------------------
-
-When you deploy botbot to production, we recommend that you do not use the Procfile. Instead, serve three pieces individually:
-
-* **botbot-web**: should be served as a wsgi application, from the ``wsgi.py`` file located at ``src/botbot/botbot/wsgi.py`` from `uwsgi <https://uwsgi-docs.readthedocs.org/en/latest/>`_, `gunicorn <http://gunicorn.org/>`_, `mod_wsgi <https://code.google.com/p/modwsgi/>`_, or any other wsgi server.
-* **botbot-plugins**: should be run as an application from botbot's manage.py file. Use `upstart <http://upstart.ubuntu.com/>`_, `systemd <http://freedesktop.org/wiki/Software/systemd/>`_, `init <http://www.sensi.org/~alec/unix/redhat/sysvinit.html>`_, or whatever your system uses for managing long-running tasks. An example upstart script is provided below.
-* **botbot-bot**: should also be run as an application from your system's task management system. An example upstart script is provided below.
-
-Example upstart scripts
------------------------
-
-``botbot-plugins.conf``:
-
-.. code-block:: bash
-
-    # BotBot Plugins
-    # logs to /var/log/upstart/botbot_plugins.log
-
-    description "BotBot Plugins"
-    start on startup
-    stop on shutdown
-
-    respawn
-    env LANG=en_US.UTF-8
-    exec /srv/botbot/bin/manage.py run_plugins
-    setuid www-data
-
-``botbot-bot.conf``:
-
-.. code-block:: bash
-
-    # BotBot-bot
-    # logs to /var/log/upstart/botbot.log
-
-    description "BotBot"
-    start on startup
-    stop on shutdown
-
-    respawn
-    env LANG=en_US.UTF-8
-    env DATABASE_URL=postgres://yourdburl
-    env REDIS_PLUGIN_QUEUE_URL=redis://localhost:6379/0
-
-    exec /srv/botbot/bin/botbot-bot
-    setuid www-data
-
-Running In A Subdirectory
--------------------------
-
-If you intend to run botbot in a subdirectory of your website, for example at ``http://example.com/botbot`` you'll need to add two options to your ``settings.py``:
-
-.. code-block:: python
-
-    FORCE_SCRIPT_NAME = '/botbot'
-    USE_X_FORWARDED_HOST = True
+If you plan to run BotBot in a production environment please read the :doc:`production` doc.
 
 
 Running Tests