Skip to content

Commit

Permalink
Merge pull request #1 from CrunchyData/master
Browse files Browse the repository at this point in the history 
Re-sync Crunchy
  • Loading branch information
@rkhandha
rkhandha authored Jan 17, 2020
2 parents 686f591 + 317ef16 commit 1308e3b
Show file tree
Hide file tree
Showing 23 changed files with 356 additions and 38 deletions.
4 changes: 2 additions & 2 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@

If you are interested in learning more about this repository, please speak with https://github.com/thesteve0

The majority of the PostGIS Scenarios come from [this repository](https://github.com/postgis/postgis-workshops
The majority of the PostGIS Scenarios come from [this repository](https://github.com/postgis/postgis-workshops).

License
-------

This content is licensed under the [Creative Commons Attribution-ShareAlike 4.0](https://creativecommons.org/licenses/by-sa/4.0/).)
This content is licensed under the [Creative Commons Attribution-ShareAlike 4.0](https://creativecommons.org/licenses/by-sa/4.0/).
12 changes: 9 additions & 3 deletions branded-ui/katacoda/templates/header.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,15 @@
<head>
<title>CrunchyData Interactive Learning Portal</title>
<link rel="shortcut icon" href="/static/favicon.ico" />

<meta name="robots" content="noindex">
<meta name="description" content="">
<meta name="description" content="Learn PostgreSQL by example with interactive tutorials and courses.">
<meta name="google-site-verification" content="5VJxGXV3AJcfxGuLj7crXYglZpu-E1wG_k-GgqU5oPw" />
<meta property="og:image" content="https://cdn2.hubspot.net/hubfs/2283855/Learning%20Portal%20Share%20Image.jpg">
<meta name="twitter:card" content="summary_large_image">
<meta name="twitter:site" content="@crunchydata">
<meta name="twitter:creator" content="@crunchydata">
<meta name="twitter:title" content="Crunchy Data Interactive Learning Portal for Developers">
<meta name="twitter:description" content="Learn PostgreSQL by example with interactive tutorials and courses.">
<meta name="twitter:image" content="https://cdn2.hubspot.net/hubfs/2283855/Learning%20Portal%20Share%20Image.jpg">
<link href="/static/vendor/bootstrap/css/bootstrap.min.css" rel="stylesheet" type="text/css" media="screen, print">
<link href="/static/css/main.css" rel="stylesheet" type="text/css" media="screen, print">
<link href="/static/css/katacoda.css" rel="stylesheet" type="text/css" media="screen, print">
Expand Down
12 changes: 4 additions & 8 deletions pg-administration/basic-postgresql-for-dbas/pgbackrest/10-pg-setup.md
View file
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
`pg_basebackup` is included with PostgreSQL and is great for many filesystem backup scenarios. However, it can be lacking in many advanced features that enterprise backup solutions require; incrementals & differentials, retention management, parallelism, and more. pgBaseBackup is a third-party tool that aims to provide these advanced features and much more!
`pg_basebackup` is included with PostgreSQL and is great for many filesystem backup scenarios. However, it can be lacking in many advanced features that enterprise backup solutions require; incrementals & differentials, retention management, parallelism, and more. pgBackRest is a third-party tool that aims to provide these advanced features and much more!

pgBackrest is available in the PGDG Yum repositories. These have already been set up in this scenario, so it's just a matter of installing the package

```
sudo yum install -y pgbackrest
```{{execute T1}}
Next some settings in the postgresql.conf must be updated to ensure WAL archiving is working. `archive_mode` must be turned on and the `archive_command` must be set to use backrest's `archive-push` command. A stanza name for the pgbackrest repo must also be given and that will be set up in the next step. Since we're changing these values on a running system, the ALTER SYSTEM command can be used
Next some settings in the postgresql.conf must be updated to ensure WAL archiving is working. `archive_mode` must be turned on and the `archive_command` must be set to use pgBackRest's `archive-push` command. A stanza name for the pgBackRest repo must also be given and that will be set up in the next step. Since we're changing these values on a running system, the ALTER SYSTEM command can be used
```
psql -c "ALTER SYSTEM SET archive_mode = 'on'"
```{{execute T1}}
Expand All @@ -16,10 +16,6 @@ Archive mode isn't typically turned on by default and changing this setting requ
```
sudo systemctl restart postgresql-11
```{{execute T1}}
You may see errors in the postgresql logs about the archive_command failing. This is normal and should resolve itself once the pgbackrest repository is set up. Do note that until the errors subside, the database will retain all WAL files generated in the `pg_wal` folder until it succeeds. In the case where the pgbackrest repo may not be available yet for an extended period of time, the easiest thing to do is to set `archive_command = '/bin/true'` until then. The command itself only requires a reload of the database to change, so that allows you to at least enable `archive_mode` until it's ready.
Next is configuring pgbackrest itself.
You may see errors in the postgresql logs about the archive_command failing. This is normal and should resolve itself once the pgBackRest repository is set up. Do note that until the errors subside, the database will retain all WAL files generated in the `pg_wal` folder until it succeeds. In the case where the pgBackRest repo may not be available for an extended period of time, the easiest thing to do is to set `archive_command = '/bin/true'` until then. The command itself only requires a reload of the database to change, so that allows you to at least enable `archive_mode` until it's ready.
Next is configuring pgBackRest itself.
13 changes: 5 additions & 8 deletions pg-administration/basic-postgresql-for-dbas/pgbackrest/20-backrest-setup.md
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
A minimal `/etc/pgbackrest.conf` file is created by the package installation. We will be configuring a global section for settings that are common to all stanzas. And then some stanza specific settings. It's also possible to group settings into sections for specific commands as well. You can either run the command below or manually edit the config file yourself
A minimal `/etc/pgbackrest.conf` file is created by the package installation. We will be configuring a global section for settings that are common to all stanzas. And then some stanza specific settings. It's also possible to group settings into sections for specific commands as well. You can either run the command below or manually edit the config file yourself
```
sudo bash -c "cat > /etc/pgbackrest.conf << EOF
[global]
Expand All @@ -11,19 +11,16 @@ retention-full=2
EOF"
```{{execute T1}}
The `global` section has settings for the pgbackrest repository that will be common for any stanzas that are created. This is very useful when you have a dedicated backup system for many databases that should have some common settings such as the repository location and logging levels. The log level is also slightly increased from the default to give slightly more feedback when running commands manually.
The `global` section has settings for the pgBackRest repository that will be common for any stanzas that are created. This is very useful when you have a dedicated backup system for many databases that should have some common settings such as the repository location and logging levels. The log level is also slightly increased from the default to give slightly more feedback when running commands manually.
Each stanza's settings are then placed under a `[stanza-name]` heading. In this case, it contains the path to our database's data directory and our retenion policy. Here it is keeping 2 full backups. When the next ***successful*** backup exceeds this number, pgbackrest will automatically expire older backups.
Each stanza's settings are then placed under a `[stanza-name]` heading. In this case, it contains the path to our database's data directory and our retention policy. Here it is keeping 2 full backups. When the next ***successful*** backup exceeds this number, pgBackRest will automatically expire older backups.
With the configuration in place, the stanza can now be created in our repository. You'll typically be wanting to run the backups as the postgres system user, not root. The archive_command is run as the postgres system user, so it needs to be able to write to the repository. And then this allows the backups to be run as the postgres user as well, without requiring root.
```
sudo -Hiu postgres pgbackrest --stanza=main stanza-create
```{{execute T1}}
It's then good to run the check command to ensure pgbackrest is configured and working properly
It's then good to run the check command to ensure pgBackRest is configured and working properly
```
sudo -Hiu postgres pgbackrest --stanza=main check
```{{execute T1}}
If either of these commands fail, you can check the pgbackrest logs located in `/var/log/pgbackrest` by default. If the issue is with the archive command, additional informaton as to why it's failing can also be found in the postgresql logs themselves (`/var/lib/pgsql/11/data/log`).
If either of these commands fail, you can check the pgBackRest logs located in `/var/log/pgbackrest` by default. If the issue is with the archive command, additional information as to why it's failing can also be found in the postgresql logs themselves (`/var/lib/pgsql/11/data/log`).
4 changes: 0 additions & 4 deletions pg-administration/basic-postgresql-for-dbas/pgbackrest/30-running-backup.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,3 @@ Run the above two commands a few more times. If you cause more than 2 fulls to b
```
sudo -Hiu postgres pgbackrest info
```{{execute T1}}
10 changes: 5 additions & 5 deletions pg-administration/basic-postgresql-for-dbas/pgbackrest/40-backrest-restore.md
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
If we need to restore our backups at any time, say due to a dropped table, pgbackrest allows that to be done very efficiently. And with Point-In-Time recovery, we can restore the database to a specific time we know that it was in a good state.
If we need to restore our backups at any time, say due to a dropped table, pgBackRest allows that to be done very efficiently. And with Point-In-Time recovery, we can restore the database to a specific time when we know that it was in a good state.

We'll need to know the time before our table was dropped for this example, so we'll make note of that now using an environment variable to be used later. It is important to represent the time as reckoned by PostgreSQL and to include timezone offsets. This reduces the possibility of unintended timezone conversions and an unexpected recovery result.
We'll need to know the time before our table was dropped for this example, so we'll make note of that now using an environment variable to be used later. It is important to represent the time as reckoned by PostgreSQL and to include timezone offsets. This reduces the possibility of unintended timezone conversions and an unexpected recovery result.
```
export RESTORETIME=$(psql -Atc "select current_timestamp")
echo $RESTORETIME
Expand All @@ -15,7 +15,7 @@ We're now in disaster recovery mode, so we'll want to shut down our primary data
```
sudo systemctl stop postgresql-11
```{{execute T1}}
Note that the last backup is always used by default when running the restore command. If the last backup happened after the desired point in time, the `--set` command can be given to pgbackrest to tell it to use an earlier backup that still exists in the repository. Also by default, the restore location must be empty. But if we're restoring to an existing cluster, the `--delta` option can be used to just overwrite what has changed since the last backup. This can GREATLY speed up recovery times.
Note that the last backup is always used by default when running the restore command. If the last backup happened after the desired point in time, the `--set` command can be given to pgBackRest to tell it to use an earlier backup that still exists in the repository. Also by default, the restore location must be empty. But if we're restoring to an existing cluster, the `--delta` option can be used to just overwrite what has changed since the last backup. This can GREATLY speed up recovery times.
```
sudo -Hiu postgres pgbackrest --stanza=main --delta --type=time "--target=$RESTORETIME" --target-action=promote restore
```{{execute T1}}
Expand All @@ -39,6 +39,6 @@ sudo -Hiu postgres chmod 700 /var/lib/pgsql/11/br_restore
```
sudo -Hiu postgres pgbackrest restore --stanza=main --db-path=/var/lib/pgsql/11/br_restore
```{{execute T1}}
If this is done, it is critically important to ***TURN OFF THE ARCHIVE COMMAND***. Otherwise, if your target destination also has write access to the pgbackrest repository, you will have two locations writing there and likely corrupt the entire backup. As stated earlier, the easiest way to disable the archive_command on a linux system is to set it to `/bin/true`.
If this is done, it is critically important to ***TURN OFF THE ARCHIVE COMMAND***. Otherwise, if your target destination also has write access to the pgBackRest repository, you will have two locations writing there and likely corrupt the entire repository. As stated earlier, the easiest way to disable the archive_command on a linux system is to set it to `/bin/true`.
The Restore section of the pgbackrest userguide has much more thorough explanations of these restore situations - https://pgbackrest.org/user-guide-centos7.html#restore
The Restore section of the pgBackRest user guide has much more thorough explanations of these restore situations - https://pgbackrest.org/user-guide-centos7.html#restore
10 changes: 5 additions & 5 deletions pg-administration/basic-postgresql-for-dbas/pgbackrest/index.json
View file
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
{
"title": "PGBackRest",
"description": "PGBackRest",
"title": "pgBackRest",
"description": "pgBackRest",
"difficulty": "beginner",
"time": "20 minutes",
"details": {
"steps": [
{"title": "Setting up PostgreSQL to use PGBackRest", "text": "10-pg-setup.md"},
{"title": "Setting up PGBackrest", "text": "20-backrest-setup.md"},
{"title": "Setting up PostgreSQL to use pgBackRest", "text": "10-pg-setup.md"},
{"title": "Setting up pgBackRest", "text": "20-backrest-setup.md"},
{"title": "Running Backups", "text": "30-running-backup.md"},
{"title": "Restoring with PGBackrest", "text": "40-backrest-restore.md"}
{"title": "Restoring with pgBackRest", "text": "40-backrest-restore.md"}
],
"intro": {
"courseData": "env-init.sh",
Expand Down
6 changes: 3 additions & 3 deletions pg-administration/basic-postgresql-for-dbas/pgbackrest/intro.md
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# PGBackRest
This scenario will go over the basics of using pgbackrest to perform backups and restoration. It has many, many more options available within it for customizing your backup processes, but this scenario should help you get started with it.
# pgBackRest
This scenario will go over the basics of using pgBackRest to perform backups and restoration. It has many, many more options available within it for customizing your backup processes, but this scenario should help you get started with it.

Full documentation is available at https://pgbackrest.org/

Let's get started!
6 changes: 6 additions & 0 deletions postgresql-devel/basics-pathway.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,5 +11,11 @@
"course_id": "basicpgadmin",
"title": "Using PgAdmin4 as a GUI to your PostgreSQL Server"
}
,
{
"external_link": "https://learn.crunchydata.com/postgresql-devel/courses/basics/intropsql",
"course_id": "intropsql",
"title": "Introduction to Using the Command Line with PostgreSQL"
}
]
}
43 changes: 43 additions & 0 deletions postgresql-devel/basics/intropsql/01-intro-psql.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
## Introduction to `psql`

`psql` is described in the [docs](https://www.postgresql.org/docs/12/app-psql.html) as a "terminal-based front-end to PostgreSQL." This means that instead of having to use a GUI tool such as [pgAdmin](https://www.pgadmin.org/) to create, access, or manipulate a database, you can use text commands via a command line interface (CLI).

We'll get started by connecting to a database.

### Connect to the `workshop` database

The database contains storm events data, which is public domain data from the National Weather Service.

In your interactive terminal, you're currently logged in as root user, as indicated by the command line prompt:

![Terminal prompt](./assets/01_-_command_line_prompt.PNG)

> **Note**
>
> Depending on the system you're using and the shell (i.e. program you're in),
> the prompt might look a little different. You'll see this below when you're
> in the `psql` shell.
We want to connect to the PostgreSQL database as **groot**, so we'll enter the `psql` command, accompanied by some flags:

```
psql -h localhost -U groot -d workshop
```{{execute}}
Each command line flag or option begins with a `-` or `--`, and what follows after the space is the parameter you're passing in with the command.
1. The `-h` option specifies the host server (e.g. `localhost`), or socket directory.
2. `-U` specifies the user you're connecting as (`groot`).
3. `-d` specifies the database you're connecting to (`workshop`).
The CLI shows a response whenever a command is run. In this case, since you are logging in to a database as the user **groot**, the command line will then prompt you for the password (enter `password`). You won't see the password displayed as you type, but the terminal is recording your keystrokes.
![Terminal prompt for password](./assets/01_-_password_prompt.PNG)
Once you're logged in to the database server, the prompt in the terminal should change. The psql prompt indicates the database you're connected to, along with `=>`.
![Postgres prompt](01_-_postgres_prompt.PNG)
The beauty of the command line is that there are often shortcuts or tips and tricks. For example, you could shorten the command used above even further to:
`psql -h localhost workshop groot`
54 changes: 54 additions & 0 deletions postgresql-devel/basics/intropsql/02-metacommands.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
## Meta-commands

The `psql` shell comes with meta-commands, which begin with an unquoted backslash.

### Get help

An example of one is `\?`, which will bring up help information about commands available in `psql`.

`\?`{{execute}}

> **Tip**
> Press the spacebar to scroll through the list, or press `q` any time to return > to the `psql` prompt.
You should see the list of available meta-commands, including the one you just used. You'll see that the `\?` command can also include parameters that allow you to look up specific topics, e.g. `\? options` will bring up help information on psql options (flags).

The `\h` command will bring up a list of SQL commands.

`\h`{{execute}}

And if you add the name of a command, you'll be shown more detailed information:

`\h ALTER INDEX`{{execute}}

### Describe a table

`\d` with the name of a table will display table metadata, e.g. columns, data types, and other attributes.

`\d se_details`{{execute}}

`\d` used without any parameters will show a list of all available tables in the current database. If `+` is appended, you'll also see extended information such as each table's size on disk.

`\d+`{{execute}}

Give it a try with a table name as well, and see the difference:

`\d+ se_details`{{execute}}

There are more in the `\d` set of meta-commands that you can use. Examples of more common ones are `\dn` (all schemas), `\dv` (all available views), `\du` (all users), `\df` (all functions).

## Options

As you saw in the earlier step, the `psql` command can be run with options.

One useful option is `--help`, which displays help information about `psql`, and then exits (i.e. you don't stay in the `psql` shell).

Let's log out of PostgreSQL first (by entering `\q`, or typing `CTRL`+`d`), and then run `psql` again with `--help` this time.

`\q`{{execute}}

`psql --help`{{execute}}

<!---
psql -l
-->
23 changes: 23 additions & 0 deletions postgresql-devel/basics/intropsql/03-run-query.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
## Running queries in `psql`

Once you're in the `psql` shell (i.e. logged in), you can run SQL queries.

Let's connect to the database again:

```
psql -h localhost -U groot -d workshop
```{{execute}}
```
SELECT cz_name FROM se_details LIMIT 5;
```{{execute}}
Note that SQL statements need to end in a semicolon. The `psql` command or other command-line statements, however, do not.
### Multiline queries
You can press `ENTER` in the `psql` shell if you want the SQL statement to continue on a new line -- `psql` won't process the statement until you add the semicolon at the end of a line.
![Multiline query in terminal](./assets/03_-_multiline.PNG)
You'll see that the command line prompt starts with a `=>`, but changes to a `->` if the statement continues on to the next line (i.e. if you press enter without ending with a `;`). The prompt doesn't change back to `=>` until the statement ends and runs.
Loading

0 comments on commit 1308e3b

Please sign in to comment.