instructions.md

Using

git clone https://github.com/18F/autoapi.git
cd autoapi
export AUTOAPI_NAME=(a unique identifier)

The remaining steps depend on where your files and API server will 'live'.

Server at	CSV at	Directions
local (via Docker)	local and/or S3	Install Docker (through Kitematic, Docker Beta, or a Linux package manager) Place spreadsheet files in a source_data folder or on S3 If using S3, set environment with S3 credentials Use docker-machine ip to get your Docker machine's IP address From the parent directory to `source_data`, docker run \ -p 5000:5000 \ -e "AUTOAPI_NAME=$AUTOAPI_NAME" \ -e "AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID" \ -e "AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY" \ -e "AUTOAPI_BUCKET=$AUTOAPI_BUCKET" \ -v `pwd`/data_sources:/data_sources \ --rm -it 18fgsa/autoapi (The repo contains this as a run_docker.sh script) View page at IP address from docker-machine, typically http://192.168.99.100:5000
local	local	Set up local environment invoke apify my_table.csv invoke serve
local	your S3 bucket	Set up local environment Get S3 credentials Set environment with S3 credentials invoke fetch_bucket invoke serve
cloud.gov	cloud.gov	Push your app to cloud.gov Set up cloud.gov S3 bucket cf restage $AUTOAPI_NAME Place your files on S3 bucket Access at https://AUTOAPI_NAME.apps.cloud.gov cloud.gov troubleshooting
cloud.gov	your S3 bucket	Push your app to cloud.gov Get S3 credentials Set cloud.gov environment cf restage $AUTOAPI_NAME Access at https://AUTOAPI_NAME.apps.cloud.gov cloud.gov troubleshooting

Instruction details

Set up local environment

1. Follow these instructions to

   • install virtualenv

   • create a virtualenv

   • activate the virtualenv in your session, and use that session for the remainder of these steps.

   • pip install -r requirements.txt

2. Install Swagger static assets:

npm install ```

Importing local files

A table name differing from the filename can be specified:

invoke apify <filename> <tablename>

apify supports wildcards - but only if tablename is not specified:

invoke apify "allmydatafiles/*.*"

When wildcards are used, enclose the file path in quote marks - otherwise, shell expansion will make the second filename look like a tablename and confuse apify.

Get S3 credentials

Only necessary if you want to use an AWS S3 bucket you set up yourself - not to serve from local files, or from an S3 bucket set up through cloud.gov.

In [Identity and Access Management](https://console.aws.amazon.com/iam), create a user and assign a policy like "AmazonS3ReadOnlyAccess" to it.

For that user, create an access key and download its credentials.

Set local environment variables for S3

Using your S3 credentials,

export AWS_ACCESS_KEY_ID=(Access Key ID from IAM user)
export AWS_SECRET_ACCESS_KEY=(Secret Access Key from IAM user)
export AUTOAPI_BUCKET=(your s3 bucket name)

Push app to cloud.gov

1. Follow these instructions to

   1. Create a cloud.gov account
   2. Install the cf command-line tool
   3. cf login
   4. cf target -o (your organization) -s (your workspace)

2. Push the app to cloud.gov, and bind a database service to it:

   cf push $AUTOAPI_NAME
   cf set-env $AUTOAPI_NAME AUTOAPI_NAME $AUTOAPI_NAME
   cf create-service aws-rds shared-psql ${AUTOAPI_NAME}-db
   cf bind-service $AUTOAPI_NAME ${AUTOAPI_NAME}-db
   

Troubleshooting cloud.gov

• cf logs $AUTOAPI_NAME --recent

• cloud.gov docs

Set up an S3 bucket on cloud.gov

1. Arrange payment! Cloud.gov doesn't have a free tier for S3.

2. Run:

cf create-service s3 basic ${AUTOAPI_NAME}-s3 cf bind-service ${AUTOAPI_NAME} ${AUTOAPI_NAME}-s3 ```

Place your files in cloud.gov S3 bucket

You can transfer files to the bucket using a tool like Cyberduck. Your bucket's name, set above, is ${AUTOAPI_NAME}-s3. You will also need the Access Key ID and Secret Access Key, which you can get from

cf env $AUTOAPI_NAME

under System-Provided: > VCAP_SERVICES > s3 > credentials, as access_key_id and secret_access_key.

Set cloud.gov environment for S3

cf set-env $AWS_ACCESS_KEY_ID AWS_ACCESS_KEY_ID (Access Key ID from IAM user)
cf set-env $AWS_SECRET_ACCESS_KEY AWS_SECRET_ACCESS_KEY (Secret Access Key from IAM user)
cf set-env $AUTOAPI_NAME AUTOAPI_BUCKET (your s3 bucket name)

Refreshing data

When the app is run locally, changes to the tables can be reflected in the app by repeating invoke apify command and, if a table has been deleted, by restarting the app.

When the app is run on cloud.gov, visit the https://AUTOAPI_NAME.apps.cloud.gov/quick_refresh endpoint to load a new file from S3, or https://AUTOAPI_NAME.apps.cloud.gov/refresh to incorporate all changes in the files in S3 - including file deletions or changes in the data in existing files.

The autoapi_table_refresh_log table is used to track the progress of any refresh requests currently being handled.

Optional configuration

• If your API will not be served from https://api.18f.gov/(API name),

  export AUTOAPI_HOST=(actual hostname)
  cf set-env $AUTOAPI_NAME AUTOAPI_HOST $AUTOAPI_HOST
  

• The Swagger API interface offers the default API key of DEMO_KEY1 by default. To change:

  export AUTOAPI_DEMO_KEY=(actual demo key)
  cf set-env $AUTOAPI_NAME AUTOAPI_DEMO_KEY $AUTOAPI_DEMO_KEY
  

• By default, an in-process refresh is abandoned after one hour. To change (for example, to 10 minutes):

  export AUTOAPI_REFRESH_TIMEOUT_SECONDS=600
  cf set-env $AUTOAPI_NAME AUTOAPI_REFRESH_TIMEOUT_SECONDS $AUTOAPI_REFRESH_TIMEOUT_SECONDS
  

• By default, when serving locally, relational forms of your spreadsheets will be stored in an autoapi.sqlite SQLite3 database file. To change, supply an alternate valid SQLAlchemy database URL; for example,

  export DATABASE_URL=postgresql://myusername:mypassword@localhost/apidb