Skip to content

Latest commit

 

History

History
115 lines (75 loc) · 2.84 KB

playbook.md

File metadata and controls

115 lines (75 loc) · 2.84 KB
Raw

Expedition Production Playbook

If something's amiss in production, look at this page for debugging tools and tricks.

DO THIS FIRST

Check prod dyno metrics for failures.

If you see errors at the serving level (e.g. not a particular part of the quest creator or app), first check that heroku does not have an active incident:

http://status.heroku.com/

Check newrelic monitoring for abnormal traffic patterns that indicate problems with specific requests or users.

Actions

Stop all payments

In an emergency (runaway payments for instance), stop all new payment transactions in prod:

heroku config:set ENABLE_PAYMENT=false -a expedition-quest-creator

Maintenance Page (for extended outages)

In a hard outage where

  • the quest creator site is not visible at all
  • users cannot interact with the SQL database

OR for scheduled downtime, put the site in maintenance mode:

heroku maintenance:on -a expedition-quest-creator

After the outage is resolved, turn off maintenance mode:

heroku maintenance:off -a expedition-quest-creator

Viewing logs

heroku logs -a expedition-quest-creator

Deploy to prod

git checkout ${BRANCH_TO_DEPLOY}
git push heroku ${BRANCH_TO_DEPLOY}:master

If you want to be especially cautious, you can even set up your default heroku remote to point to dev, and create an explicit prod remote for pushing to prod:

heroku git:remote -a expedition-quest-creator-dev -r heroku
heroku git:remote -a expedition-quest-creator -r prod

Rollback

Get prod release list (versions are on the left column):

heroku releases -a expedition-quest-creator

Rollback prod to version:

heroku rollback v### -a expedition-quest-creator

For dev, replace expedition-quest-creator with expedition-quest-creator-dev.

DB Backups

Listing backup schedule:

heroku pg:backups:schedules -a expedition-quest-creator

Listing backups:

heroku pg:backups --app expedition-quest-creator

Downloading backups:

heroku pg:backups:url ${BACKUP} -a expedition-quest-creator

Restoring backups:

heroku pg:backups:restore ${BACKUP} DATABASE_URL -a expedition-quest-creator

Quest DB

Connecting to the DB:

heroku pg:psql -a expedition-quest-creator

Monthly Actions

  • Check prod datastore is below 80% of row quota and has recent backups.
  • Backup prod heroku config variables to lastpass.
  • Test manual deployment via git push heroku master, and rollback with heroku rollback in dev environment.