Problem: sit-git-hook is insufficient

While [sit-git-hook](https://github.com/sit-fyi/sit-git-hook)
has enabled exchange of sit repository information, it is
still not a full solution for setting up an exchange point,
it requires a further setup (http or ssh endpoint; or an email
processing endpoint), as it is just a hook.

As a matter of fact, even current setups, being hosted
on gitolite, suffer from being served over SSH as this
requires a private key (hence, trickier git setup) and
that key needs secure permissions, which is something Git
can't preserve.

Solution: develop sit-inbox to automate this whole setup
in one box (specifically, a Docker container, but can be
extended to other things later)

Its initial focus on decentralized workflows (like email)
with the intent to extend it to other workflows shortly.

The idea is that this "box" can be run on a maintainer's
machine where it'll be polling its inboxes and forwarding
valid additions into repositories. It can also run on a
server for either the email workflow or (future) Git endpoint
serving workflow.

.dockerignore

@@ -0,0 +1,5 @@
+example
+doc
+sit-inbox
+.git
+.sit

.gitignore

@@ -0,0 +1,5 @@
+*.swp
+*.swo
+example
+node_modules
+package-lock.json

Dockerfile

@@ -0,0 +1,34 @@
+FROM alpine:edge
+
+RUN apk update
+RUN apk add getmail daemontools ansible~=2.6 --repository http://dl-3.alpinelinux.org/alpine/edge/testing/ --allow-untrusted
+# maildrop
+RUN wget -c https://downloads.sourceforge.net/project/courier/maildrop/2.9.3/maildrop-2.9.3.tar.bz2 && \
+    wget -c https://downloads.sourceforge.net/project/courier/courier-unicode/2.0/courier-unicode-2.0.tar.bz2 && \
+    apk add g++ make pcre-dev perl file && \
+    tar xjvfp courier-unicode-2.0.tar.bz2 && rm -f courier-unicode-2.0.tar.bz2 && \
+    cd courier-unicode-2.0 && ./configure && make && make install && \
+    cd .. && rm -rf courier-unicode-2.0 && \
+    tar xjvfp maildrop-2.9.3.tar.bz2 && rm -f maildrop-2.9.3.tar.bz2 && \
+    cd maildrop-2.9.3 && mkdir /var/spool/mail && ./configure && make && make install-strip && \
+    cd .. && rm -rf maildrop-2.9.3 && \
+    apk del g++ make perl
+RUN apk add git
+RUN apk add py-pip && pip install -e git+https://github.com/dbohdan/[email protected]#egg=Remarshal 
+RUN apk add bash coreutils msmtp dovecot shadow dialog ncurses util-linux
+RUN git clone https://github.com/bats-core/bats-core && \
+    cd bats-core && git checkout v1.1.0 && ./install.sh /usr/local && cd .. && rm -rf bats-core
+RUN apk add curl && curl -s https://sit.fyi/install.sh | sh
+RUN apk add nodejs npm && npm install -g ajv-cli
+RUN echo "export PATH=/root/.sit-install:\$PATH" >> /root/.bashrc
+
+VOLUME [ "/etc/sit-inbox", "/var/lib/repositories", "/var/run/oldmail" ]
+
+ADD startup.sh /usr/bin/startup
+ADD email-ingress /usr/bin/email-ingress
+ADD playbook /playbook
+ADD ansible /etc/ansible
+ADD dovecot.conf /etc/dovecot/dovecot.conf
+ADD schema.yaml /
+
+CMD [ "startup" ]

Makefile

@@ -0,0 +1,10 @@
+all:
+	docker build -t sit-inbox .
+
+test:
+	docker build -qt sit-inbox-test .
+	docker run -v $(shell pwd)/tests:/tests -ti sit-inbox-test bash -ic "source /tests/init.bash && bats /tests"
+
+install:
+	mkdir -p /usr/local/bin
+	install sit-inbox /usr/local/bin/sit-inbox

README.md

@@ -0,0 +1,63 @@
+# sit-inbox
+
+*sit-inbox* is a tool that helps managing inbound additions for
+[SIT](https://sit.fyi) repositories over different transports. A common case
+would be accepting patches for a Git repository for a SIT repository portion of
+it, allowing others to contribute their additions to other parties in an
+automated way.
+
+This tool is built with decentralized scenarios in mind. It can enable workflows that
+don't require hosting it on a publicly accessible server. For example, by piggying back
+on e-mail, it can be operated on a regular end-user computer, sporadically connected
+to the internet.
+
+Currently, sit-inbox supports:
+
+* receiving patches for **Git-hosted** SIT repositories over **email** and **Maildir** 
+
+This list is expected to grow.
+
+It is packaged as a Docker container to make its installation and operation easier.
+
+## Building
+
+In order to build it, you'll need `make` and `docker`:
+
+```
+make
+```
+
+## Usage
+
+sit-inbox comes with a small convenience helper that can be used either directly
+from the source root or installed using `make install`:
+
+```
+sit-inbox new path/to/new/setup
+```
+
+This will create default structure for an inbox. You will need to edit
+`etc/config.toml` [configuration](#configuration) and possibly add more arguments
+to the docker container (most likely, for extra volumes) in `.dockerargs.
+
+In order to run the inbox, simply do this:
+
+```
+sit-inbox run path/to/setup
+```
+
+After (successful) [re-]provisioning, you should see something like this:
+
+<img src="doc/sit-inbox.png" alt="Screenshot" width="300px">
+
+Upon startup, sit-inbox will attempt to retrieve updates
+from inboxes that don't have `autostart` disabled.
+
+The interface allows the operator to observe logs and manually invoke
+operations such as mail retrieval.
+
+## Configuration
+
+Please refer to [schema.yaml](schema.yaml) for *almost-human-interpretable*
+schema for the `config.toml` file. It'd be great to convert it into Markdown
+or something like it, so if you're up to it -- please contribute.

dovecot.conf

@@ -0,0 +1,9 @@
+protocols = imap
+mail_location = maildir:~
+userdb {
+  driver = passwd
+  args = blocking=no
+}
+passdb {
+  driver = shadow 
+}

email-ingress

@@ -0,0 +1,10 @@
+#! /usr/bin/env bash
+
+repository=${1:-$REPOSITORY}
+
+file=$(mktemp)
+cat <&0 > "${file}"
+
+cat "${file}" | setlock -n "/var/run/email.${repository}" "/usr/bin/email.${repository}" 2>&1
+
+rm -f "${file}"

playbook/playbook.yml

@@ -0,0 +1,84 @@
+---
+
+- name: Provision sit-inbox
+  hosts: 127.0.0.1
+  connection: local
+  become: yes
+  roles:
+          - common
+  tasks:
+          - name: start dovecot
+            command: "bash -c 'dovecot || dovecot reload'"
+          - name: prepare mailbox users
+            user:
+                    name: "mb_{{ item.key }}"
+                    password: "$6$P0SvmyMSldk$.FTEdn/rBtDYuN4dWMGpICoLtQSJCltyjMpwk9AqwPIbPZJvxlI/CTdPTD4p5VnpR2kH.i6AOhtzqKvkP2B.7/"
+                    home: "{{ item.value.path }}"
+            loop: "{{ inbox | dict2items }}"
+            when: item.value.type == "email" and item.value.retriever == "MailboxRetriever"
+          - name: .getmail directory 
+            file:
+                  path: /root/.getmail
+                  state: directory
+                  mode: 0700
+          - name: configure getmail
+            template:
+                  src: templates/getmailrc.j2
+                  dest: "/root/.getmail/{{ item.key }}"
+            loop: "{{ inbox | dict2items }}"
+            when: item.value.type == "email"
+          - name: getmail scripts
+            template:
+                  src: templates/getmail.j2
+                  dest: "/usr/bin/getmail.{{ item.key }}"
+                  mode: 0700
+            loop: "{{ inbox | dict2items }}"
+            when: item.value.type == "email"
+          - name: getmail on start
+            cron:
+                    name: "get email for {{ item.key }} at startup"
+                    special_time: reboot
+                    job: "getmail.{{ item.key }} 2>&1 | logger -t getmail.{{ item.key }}"
+            loop: "{{ inbox | dict2items }}"
+            when: item.value.type == "email" and (item.value.autostart | default(true))
+          - name: check-email script
+            template:
+                    src: templates/check-email.j2
+                    dest: "/usr/bin/check-email"
+                    mode: 0777
+          - name: configure maildrop
+            template:
+                    src: templates/maildrop.j2
+                    dest: "/root/.getmail/maildrop.{{ item.key }}"
+                    mode: 0600
+            loop: "{{ inbox | dict2items }}"
+            when: item.value.type == "email"
+          - name: clone git repositories
+            git:
+                    dest: "/var/lib/repositories/{{ item.key }}"
+                    repo: "{{ item.value.source }}"
+            loop: "{{ repository | dict2items }}"
+            when: item.value.type == "git"
+          - name: configure git repository username
+            git_config:
+                    name: user.name
+                    value: "{{ item.value.git_username | default('sit-inbox') }}"
+                    scope: local
+                    repo: "/var/lib/repositories/{{ item.key }}"
+            loop: "{{ repository | dict2items }}"
+            when: item.value.type == "git"
+          - name: configure git repository email
+            git_config:
+                    name: user.email
+                    value: "{{ item.value.git_email | default('inbox@sit') }}"
+                    scope: local
+                    repo: "/var/lib/repositories/{{ item.key }}"
+            loop: "{{ repository | dict2items }}"
+            when: item.value.type == "git"
+          - name: git ingress
+            template:
+                    src: templates/git_email.j2
+                    dest: "/usr/bin/email.{{ item.key }}"
+                    mode: 0700
+            loop: "{{ repository | dict2items }}"
+            when: item.value.type == "git"

playbook/templates/check-email.j2

@@ -0,0 +1,7 @@
+#! /usr/bin/env bash
+
+{% for item in (inbox | dict2items) -%}
+{% if item.value.type == "email" -%}
+getmail.{{ item.key }} 2>&1 | logger -t getmail.{{ item.key}}
+{% endif -%}
+{% endfor -%}

playbook/templates/getmail.j2

@@ -0,0 +1,6 @@
+#! /usr/bin/env bash
+{% if item.value.default_repository is defined -%}
+export REPOSITORY={{ item.value.default_repository }}
+{% endif -%}
+setlock "/var/run/getmail.{{ item.key }}" /usr/bin/getmail --rcfile="$HOME/.getmail/{{ item.key }}" \
+        --getmaildir=/var/run/oldmail -n

playbook/templates/getmailrc.j2

@@ -0,0 +1,27 @@
+[retriever]
+{% if item.value.retriever == 'MailboxRetriever' -%}
+type = SimpleIMAPRetriever
+server = 127.0.0.1
+username = mb_{{ item.key }}
+password = 
+use_peek = false
+{% else -%}
+type = {{ item.value.retriever | mandatory }}
+server = {{ item.value.server | mandatory }}
+username = {{ item.value.username | mandatory }}
+{% if item.value.port is defined -%}
+port = {{ item.value.port }}
+{% endif -%}
+{% if item.value.password is defined -%}
+password = {{ item.value.password }}
+{% endif -%}
+{% if item.value.password_command is defined -%}
+password_command = {{ item.value.password_command | to_json | regex_replace("^\[", "(") | regex_replace("\]$", ")")}}
+{% endif -%}
+{% endif -%}
+
+[destination]
+type = MDA_external
+path = /usr/local/bin/maildrop
+arguments = ( "-f", "%(sender)", "/root/.getmail/maildrop.{{ item.key }}", )
+allow_root_commands = true

playbook/templates/git_email.j2

@@ -0,0 +1,79 @@
+#! /usr/bin/env bash
+email_reponse=${EMAIL_RESPONSE:-{{ item.value.email_response | default('') }}}
+file=$(mktemp)
+cat <&0 > "${file}"
+temp=$(mktemp -d)
+
+cd "/var/lib/repositories/{{ item.key }}"
+
+git pull -f -q 2>/dev/null
+
+original="${temp}/original"
+git clone -q "/var/lib/repositories/{{ item.key }}" "${original}" 2>/dev/null 1>/dev/null
+
+git branch -q -D inbox 2>/dev/null >/dev/null
+git checkout -b inbox 2>/dev/null
+
+cat ${file} | reformail -X Date: -X From: -X Subject: | \
+        logger -s -t "git.{{ item.key }}" 
+git_am=$(git am -s -3 "${file}" 2>&1)
+result=$?
+
+if [ "${result}" == "0" ]; then
+        files=$(git diff --name-only origin/{{ item.value.branch | default('master') }} inbox)
+        # Figure out if we're using .sit/items or deprecated .sit/issues
+       if [ -d "/.sit/issues" ]; then
+                target=issue
+                target_dir=.sit/issues
+        else
+                target=item
+                target_dir=.sit/items
+        fi
+        target_dir_len=$(echo -n $target_dir | wc -m) # not using bash string length to avoid clashing with template syntax
+        outside_files=
+        for file in ${files}; do
+                if [ "${file:0:${target_dir_len}}" != "${target_dir}" ]; then
+                        outside_files="${outside_files}${file} is outside of ${target_dir}\n"
+                        continue
+                fi
+
+                item=$(echo "${file}" | cut -d'/' -f 3)
+                record=$(echo "${file}" | cut -d'/' -f 4)
+                record_path=$(echo "${file}" | cut -d'/' -f 1-4)
+
+                if [ -f "${original}/${file}" ]; then
+                        outside_files="${outside_files}File ${file} already exists in the target repository\n"
+                        continue
+                fi
+
+                if [ -d "${original}/${record_path}" ]; then
+                        if ! [[ "${outside_files}" =~ "Record ${item}/${record} already exists in the target repository" ]]; then
+                                outside_files="${outside_files}Record ${item}/${record} already exists in the target repository\n"
+                        fi
+                        continue
+                fi
+
+        done
+        if [ "${outside_files}" == "" ]; then
+                git push "{{ item.value.source }}" "inbox:{{ item.value.branch | default('master') }}" 2>/dev/null
+                logger -s -t "git.{{ item.key }}" "Patch applied and pushed"
+        else
+                logger -s -t "git.{{ item.key }}" "Patch rejected"
+                printf "${outside_files}" | logger -s -t "git.{{ item.key }}"
+        fi
+else
+        if [[ "${git_am}" =~ "Patch is empty" ]]; then
+                logger -s -t "git.{{ item.key }}" "No patch found, skipping"
+        else
+                logger -s -t "git.{{ item.key }}" "Applying patch failed ${git_am}"
+        fi
+        git am --abort 2>/dev/null
+fi
+
+
+git checkout "{{ item.value.branch | default('master') }}" 1>/dev/null 2>/dev/null
+git branch -q -D inbox 1>/dev/null 2>/dev/null
+
+rm -rf "${temp}"
+rm -f "${file}"
+

playbook/templates/maildrop.j2

@@ -0,0 +1,3 @@
+{{ item.value.maildrop | default('') }}
+
+to "|email-ingress"