Skip to content

Commit

Permalink
Port 9.6 to develop (#89)
Browse files Browse the repository at this point in the history
* Part one of porting work from 9.6 to 10

* Backported more scripts from 9.6 branch

* Added missing apt update in dockerfile

* Updates to entrypoint to reference image and update docker-compose to reference 10 pg

* Added sample and docs from 9.6 branch

* Removed my diagram as Rizky had already added one

* Fix env paths for pg 10

* Fixes for backporting work from 9.6 to 10 - dbb now spins up and accepts connections properly
  • Loading branch information
timlinux authored Mar 21, 2018
1 parent 1250909 commit cd66fea
Show file tree
Hide file tree
Showing 21 changed files with 1,340 additions and 310 deletions.
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,2 +1,5 @@
.idea
*.*~
*/replication/pg-*
*/replication/docker-compose.override.yml
.DS_Store
29 changes: 19 additions & 10 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -6,31 +6,40 @@ RUN export DEBIAN_FRONTEND=noninteractive
ENV DEBIAN_FRONTEND noninteractive
RUN dpkg-divert --local --rename --add /sbin/initctl

RUN apt-get -y update
RUN apt-get -y install gnupg2 wget ca-certificates rpl pwgen
RUN apt-get -y update; apt-get -y install gnupg2 wget ca-certificates rpl pwgen
RUN sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ stretch-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
RUN wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -
RUN apt-get -y update
#RUN apt-get -y upgrade

#-------------Application Specific Stuff ----------------------------------------------------

# We add postgis as well to prevent build errors (that we dont see on local builds)
# on docker hub e.g.
# The following packages have unmet dependencies:
RUN apt-get install -y postgresql-client-10 postgresql-common postgresql-10 postgresql-10-postgis-2.4 postgresql-10-pgrouting netcat
RUN apt-get update; apt-get install -y postgresql-client-10 postgresql-common postgresql-10 postgresql-10-postgis-2.4 postgresql-10-pgrouting netcat

# Open port 5432 so linked containers can see them
EXPOSE 5432

# Run any additional tasks here that are too tedious to put in
# this dockerfile directly.
ADD env-data.sh /env-data.sh
ADD setup.sh /setup.sh
RUN chmod 0755 /setup.sh
RUN chmod +x /setup.sh
RUN /setup.sh

# We will run any commands in this when the container starts
ADD start-postgis.sh /start-postgis.sh
RUN chmod 0755 /start-postgis.sh

CMD /start-postgis.sh
ADD docker-entrypoint.sh /docker-entrypoint.sh
ADD setup-conf.sh /
ADD setup-database.sh /
ADD setup-pg_hba.sh /
ADD setup-replication.sh /
ADD setup-ssl.sh /
ADD setup-user.sh /
ADD postgresql.conf /tmp/postgresql.conf
RUN chmod +x /docker-entrypoint.sh

# Optimise postgresql
RUN echo "kernel.shmmax=543252480" >> /etc/sysctl.conf
RUN echo "kernel.shmall=2097152" >> /etc/sysctl.conf

ENTRYPOINT /docker-entrypoint.sh
143 changes: 126 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,10 @@ environment (though probably not for heavy load databases).
**Note:** We recommend using ``apt-cacher-ng`` to speed up package fetching -
you should configure the host for it in the provided 71-apt-cacher-ng file.

There is a nice 'from scratch' tutorial on using this docker image on Alex Urquhart's
blog [here](https://alexurquhart.com/post/set-up-postgis-with-docker/) - if you are
just getting started with docker, PostGIS and QGIS, we really recommend that you use it.

## Tagged versions

The following convention is used for tagging the images we build:
Expand All @@ -30,7 +34,7 @@ kartoza/postgis:[postgres_version]-[postgis-version]

So for example:

``kartoza/postgis:9.5-2.2`` Provides PostgreSQL 9.5, PostGIS 2.2
``kartoza/postgis:9.6-2.4`` Provides PostgreSQL 9.6, PostGIS 2.4

**Note:** We highly recommend that you use tagged versions because
successive minor versions of PostgreSQL write their database clusters
Expand Down Expand Up @@ -101,30 +105,24 @@ the container will allow connections only from the docker private subnet.
* -e ALLOW_IP_RANGE=<0.0.0.0/0>


## Convenience run script
## Convenience docker-compose.yml

For convenience we have provided a bash script for running this container
that lets you specify a volume mount point and a username / password
for the new instance superuser. It takes these options:
For convenience we have provided a ``docker-compose.yml`` that will run a
copy of the database image and also our related database backup image (see
[https://github.com/kartoza/docker-pg-backup](https://github.com/kartoza/docker-pg-backup)).

```
OPTIONS:
-h Show this message
-n Container name
-v Volume to mount the Postgres cluster into
-l local port (defaults to 25432)
-u Postgres user name (defaults to 'docker')
-p Postgres password (defaults to 'docker')
-d database name (defaults to 'gis')
```
The docker compose recipe will expose PostgreSQL on port 25432 (to prevent
potential conflicts with any local database instance you may have).

Example usage:

```
./run-postgis-docker.sh -p 6789 -v /tmp/foo/ -n postgis -u foo -p bar
docker-compose up -d
```

**Note:** The docker-compose recipe above will not persist your data on your local
disk, only in a docker volume.

## Connect via psql

Connect with psql (make sure you first install postgresql client tools on your
Expand Down Expand Up @@ -159,7 +157,118 @@ docker run -d -v $HOME/postgres_data:/var/lib/postgresql kartoza/postgis`
You need to ensure the ``postgres_data`` directory has sufficient permissions
for the docker process to read / write it.

## Postgres Replication Setup

This image were provided with replication abilities. In some sense, we can
categorize an instance of the container as `master` or `slave`. A `master`
instance means that a particular container have a role as a single point of
database write. A `slave` instance means that a particular container will
mirror database content from a designated master. This replication scheme allows
us to sync database. However a `slave` is only of read-only transaction, thus
we can't write new data on it.

To experiment with the replication abilities, you can see a (docker-compose.yml)[sample/replication/docker-compose.yml]
sample provided. There are several environment variables that you can set, such as:

Master settings:
- ALLOW_IP_RANGE: A pg_hba.conf domain format which will allow certain host
to connect into the container. This is needed to allow `slave` to connect
into `master`, so specifically this settings should allow `slave` address.
- Both POSTGRES_USER and POSTGRES_PASS will be used as credentials for slave to
connect, so make sure you changed this into something secure.

Slave settings:
- REPLICATE_FROM: This should be the domain name, or ip address of `master`
instance. It can be anything from docker resolved name like written in the sample,
or the IP address of the actual machine where you expose `master`. This is
useful to create cross machine replication, or cross stack/server.
- REPLICATE_PORT: This should be the port number of `master` postgres instance.
Will default to 5432 (default postgres port), if not specified.
- DESTROY_DATABASE_ON_RESTART: Default is `True`. Set to otherwise to prevent
this behaviour. A slave will always destroy its current database on
restart, because it will try to sync again from `master` and avoid inconsistencies.
- PROMOTE_MASTER: Default none. If set to any value, then the current slave
will be promoted to master.
In some cases when `master` container has failed, we might want to use our `slave`
as `master` for a while. However promoted slave will break consistencies and
is not able to revert to slave anymore, unless the were destroyed and resynced
with the new master.

To run sample replication, do the following instructions:

Do manual image build by executing `build.sh` script

```
./build.sh
```

Go into `sample/replication` directory and experiment with the following Make
command to run both master and slave services.

```
make up
```

To shutdown services, execute:

```
make down
```

To view logs for master and slave respectively, use the following command:

```
make master-log
make slave-log
```

You can try experiment with several scenarios to see how replication works

### Sync changes from master to slave

You can use any postgres database tools to create new tables in master, by
connecting using POSTGRES_USER and POSTGRES_PASS credentials using exposed port.
In the sample, master database were exposed in port 7777.
Or you can do it via command line, by entering the shell:

```
make master-shell
```

Then made any database changes using psql.

After that, you can see that slave follows the changes by inspecting
slave database. You can, again, uses database management tools using connection
credentials, hostname, and ports for slave. Or you can do it via command line,
by entering the shell:

```
make slave-shell
```

Then view your changes using psql.

### Promoting slave to master

You will notice that you cannot make changes in slave, because it was read-only.
If somehow you want to promote it to master, you can specify `PROMOTE_MASTER: 'True'`
into slave environment and set `DESTROY_DATABASE_ON_RESTART: 'False'`.

After this, you can make changes to your slave, but master and slave will not
be in sync anymore. This is useful if slave needs to take over a failover master.
However it was recommended to take additional action, such as creating backup from
slave, so a dedicated master can be created again.

### Preventing slave database destroy on restart

You can optionally set `DESTROY_DATABASE_ON_RESTART: 'False'` after successful sync
to prevent the database from destroyed on restart. With this settings, you can
shutdown your slave and restart it later and it will continue to sync using existing
database (as long as there is no consistencies conflicts).

However, you should note that this option doesn't mean anything if you didn't
persist your database volumes. Because if it is not persisted, then it will be lost
on restart because docker will recreate the container.

## Credits

Expand Down
1 change: 1 addition & 0 deletions build.sh
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
docker build -t kartoza/postgis:manual-build .
docker build -t kartoza/postgis:10.0-2.4 .
38 changes: 38 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# docker-compose build
# docker-compose up -d web
version: '2'
volumes:
dbbackups:
postgis-data:

services:

db:
image: kartoza/postgis:10.0-2.4
volumes:
- 'postgis-data:/var/lib/postgresql'
- 'dbbackups:/backups'
environment:
- POSTGRES_DB=gis
- POSTGRES_USER=docker
- POSTGRES_PASS=docker
- ALLOW_IP_RANGE=0.0.0.0/0
ports:
- 25432:5432
restart: unless-stopped

dbbackups:
image: kartoza/pg-backup:10.0
hostname: pg-backups
volumes:
- dbbackups:/backups
links:
- db:db
environment:
- DUMPPREFIX=demo
- PGUSER=docker
- PGPASSWORD=docker
- PGDATABASE=gis
- PGPORT=5432
- PGHOST=db
restart: unless-stopped
54 changes: 54 additions & 0 deletions docker-entrypoint.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
#!/usr/bin/env bash

# This script will run as the postgres user due to the Dockerfile USER directive
set -e

# Setup postgres CONF file
source /setup-conf.sh

# Setup ssl
source /setup-ssl.sh

# Setup pg_hba.conf
source /setup-pg_hba.sh

if [ -z "$REPLICATE_FROM" ]; then
# This means this is a master instance. We check that database exists
echo "Setup master database"
source /setup-database.sh
else
# This means this is a slave/replication instance.
echo "Setup slave database"
source /setup-replication.sh
fi

# Running extended script or sql if provided.
# Useful for people who extends the image.

echo
for f in /docker-entrypoint-initdb.d/*; do
case "$f" in
*.sh) echo "$0: running $f"; . "$f" ;;
*.sql) echo "$0: running $f"; "${psql[@]}" < "$f"; echo ;;
*.sql.gz) echo "$0: running $f"; gunzip -c "$f" | "${psql[@]}"; echo ;;
*) echo "$0: ignoring $f" ;;
esac
echo
done

# If no arguments passed to entrypoint, then run postgres by default
if [ $# -eq 0 ];
then
echo "Postgres initialisation process completed .... restarting in foreground"
cat /tmp/postgresql.conf > ${CONF}
su - postgres -c "$SETVARS $POSTGRES -D $DATADIR -c config_file=$CONF"
fi

# If arguments passed, run postgres with these arguments
# This will make sure entrypoint will always be executed
if [ "${1:0:1}" = '-' ]; then
# append postgres into the arguments
set -- postgres "$@"
fi

exec su - "$@"
Binary file added docs/diagram.graffle
Binary file not shown.
Binary file added docs/execution_flow.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
64 changes: 64 additions & 0 deletions env-data.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
#!/usr/bin/env bash

DATADIR="/var/lib/postgresql/10/main"
ROOT_CONF="/etc/postgresql/10/main"
CONF="$ROOT_CONF/postgresql.conf"
RECOVERY_CONF="$ROOT_CONF/recovery.conf"
POSTGRES="/usr/lib/postgresql/10/bin/postgres"
INITDB="/usr/lib/postgresql/10/bin/initdb"
SQLDIR="/usr/share/postgresql/10/contrib/postgis-2.4/"
SETVARS="POSTGIS_ENABLE_OUTDB_RASTERS=1 POSTGIS_GDAL_ENABLED_DRIVERS=ENABLE_ALL"
LOCALONLY="-c listen_addresses='127.0.0.1'"
PG_BASEBACKUP="/usr/bin/pg_basebackup"
PROMOTE_FILE="/tmp/pg_promote_master"
PGSTAT_TMP="/var/run/postgresql/"
PG_PID="/var/run/postgresql/10-main.pid"

# Make sure we have a user set up
if [ -z "${POSTGRES_USER}" ]; then
POSTGRES_USER=docker
fi
if [ -z "${POSTGRES_PASS}" ]; then
POSTGRES_PASS=docker
fi
if [ -z "${POSTGRES_DBNAME}" ]; then
POSTGRES_DBNAME=gis
fi
# SSL mode
if [ -z "${PGSSLMODE}" ]; then
PGSSLMODE=require
fi
# Enable hstore and topology by default
if [ -z "${HSTORE}" ]; then
HSTORE=true
fi
if [ -z "${TOPOLOGY}" ]; then
TOPOLOGY=true
fi
# Replication settings
if [ -z "${REPLICATE_PORT}" ]; then
REPLICATE_PORT=5432
fi
if [ -z "${DESTROY_DATABASE_ON_RESTART}" ]; then
DESTROY_DATABASE_ON_RESTART=true
fi
if [ -z "${PG_MAX_WAL_SENDERS}" ]; then
PG_MAX_WAL_SENDERS=8
fi
if [ -z "${PG_WAL_KEEP_SEGMENTS}" ]; then
PG_WAL_KEEP_SEGMENTS=100
fi


# Compatibility with official postgres variable
# Official postgres variable gets priority
if [ ! -z "${POSTGRES_PASSWORD}" ]; then
POSTGRES_PASS=${POSTGRES_PASSWORD}
fi
if [ ! -z "${PGDATA}" ]; then
DATADIR=${PGDATA}
fi

if [ ! -z "$POSTGRES_DB" ]; then
POSTGRES_DBNAME=${POSTGRES_DB}
fi
Loading

0 comments on commit cd66fea

Please sign in to comment.