Primarily installation instructions and readme cleanup (#37)

Primarily installation instructions and readme cleanup, and update the cloud sdk version in the DockerFile. 
Expose Flask port in docker image.

INSTALL.md

@@ -19,6 +19,8 @@
     - [Transfer learning example](#transfer-learning-example)
   - [Optional: Clone/Download the TensorFlow repo from GitHub](#optional-clonedownload-the-tensorflow-repo-from-github)
 
+* * *
+
 You can set up for the workshop in two different, mutually-exclusive ways:
 
 - [Running in a docker container](#docker-based-installation).
@@ -64,8 +66,8 @@ Create a directory (called, say, `workshop-data`) to mount as a volume when you
 Once you've downloaded the container image, you can run it like this:
 
 ```sh
-$ docker run -v `pwd`/workshop-data:/root/tensorflow-workshop-master/workshop-data -it \
-    -p 6006:6006 -p 8888:8888 gcr.io/google-samples/tf-workshop:v5
+docker run -v `pwd`/workshop-data:/root/tensorflow-workshop-master/workshop-data -it \
+    -p 6006:6006 -p 8888:8888 -p 5000:5000 gcr.io/google-samples/tf-workshop:v5
 ```
 
 Edit the path to the directory you're mounting as appropriate. The first component of the `-v` arg is the local directory, and the second component is where you want to mount it in your running container.
@@ -103,27 +105,48 @@ $ docker exec -it <container_id> bash
 
 ### Running the Docker container on a VM
 
-It is easy to set up a Google Compute Engine (GCE) VM in which to run the Docker container.
+It is easy to set up a Google Compute Engine (GCE) VM on which to run the Docker container. We sketch the steps below, or see [TLDR_CLOUD_INSTALL.md](TLDR_CLOUD_INSTALL.md) for more detail.
 
 First, make sure that your project has the GCE API enabled. An easy way to do this is to go to the [Cloud Console](https://console.cloud.google.com/), and visit the Compute Engine panel.  It should display a button to enable the API.
 
 - Connect to your project's [Cloud Shell](https://cloud.google.com/shell/).
-- From the Cloud Shell, create a container-optimized image as described [here](https://cloud.google.com/container-optimized-os/docs/how-to/create-configure-instance).
-- SSH to that image.  You can do this from the Cloud Console by visiting the Compute Engine panel, and clicking on the 'SSH' pulldown to the right of your instance.
+
+- From the Cloud Shell, create a container-optimized image as described [here](https://cloud.google.com/container-optimized-os/docs/how-to/create-configure-instance). E.g.:
+
+```
+gcloud compute instances create mlworkshop \
+    --image-family gci-stable \
+    --image-project google-containers \
+    --zone us-central1-b --boot-disk-size=100GB \
+    --machine-type n1-standard-1
+```
+
+- Set up a default network firewall rule for: tcp:8888, tcp:6006, and tcp:5000.  (This opens the ports we'll use for jupyter, Tensorboard, and Flask.) E.g:
+
+```shell
+gcloud compute firewall-rules create mlworkshop --allow tcp:8888,tcp:6006,tcp:5000
+```
+
+- SSH to your image.  You can do this from the Cloud Console by visiting the Compute Engine panel, and clicking on the 'SSH' pulldown to the right of your instance.
 
 Then, once you've ssh'd to the VM, follow the instructions above to download and run the Docker container there.
 Note: **Docker is already installed** on the 'container-optimized' VMs.
 
+When you're done with the workshop, you may want to *stop* or *delete* your VM instance.
+
+You can also delete your firewall rule:
+
+```shell
+gcloud compute firewall-rules delete mlworkshop
+```
 
+* * *
 ## Virtual environment-based installation
 
 (These steps are not necessary if you have already completed the instructions for running the Docker image.)
 
 We highly recommend that you use a virtual environment for your TensorFlow installation rather than a direct install onto your machine.  The instructions below walk you through a `conda` install, but a `virtualenv` environment will work as well.
 
-Note: The 'preprocessing' stage in the [Cloud ML transfer learning](workshop_sections/transfer_learning/cloudml)
-example requires installation of the Cloud ML SDK, which requires Python 2.7. Otherwise, Python 3 should likely work.
-
 ### Install Conda + Python 2.7 to use as your local virtual environment
 
 Anaconda is a Python distribution that includes a large number of standard numeric and scientific computing packages. Anaconda uses a package manager called "conda" that has its own environment system similar to Virtualenv.
@@ -196,32 +219,14 @@ Because we have limited workshop time, we've saved a set of
 generated as part of the [Cloud ML transfer learning](workshop_sections/transfer_learning/cloudml)
 example. To save time, copy them now to your own bucket as follows.
 
-Copy a zip of the generated records to some directory on your local machine:
-
-```shell
-gsutil cp gs://oscon-tf-workshop-materials/transfer_learning/cloudml/hugs_preproc_tfrecords.zip .
-```
-
-and then expand the zip:
-
-```shell
-unzip hugs_preproc_tfrecords.zip
-```
-
-Set the `BUCKET` variable to point to your GCS bucket (replacing `your-bucket-name` with the actual name):
+Set the `BUCKET` variable to point to your GCS bucket (replacing `your-bucket-name` with the actual name), then copy the records to your bucket.  Then, set the GCS_PATH variable to the newly copied subdir.
 
 ```shell
 BUCKET=gs://your-bucket-name
-```
-
-Then set the `GCS_PATH` variable as follows, and copy the unzipped records to a `preproc` directory under that path:
-
-```shell
+gsutil cp -r gs://tf-ml-workshop/transfer_learning/hugs_preproc_tfrecords $BUCKET
 GCS_PATH=$BUCKET/hugs_preproc_tfrecords
-gsutil cp -r hugs_preproc_tfrecords/ $GCS_PATH/preproc
 ```
 
-Once you've done this, you can delete the local zip and `hugs_preproc_tfrecords` directory.
 
 ## Optional: Clone/Download the TensorFlow repo from GitHub
 

README.md

@@ -7,10 +7,11 @@ The accompanying slides are
 
 Contributions are not currently accepted.  This is not an official Google product.
 
-This document points to more information for each step in the workshop.
+This document points to more information for each workshop lab.
 
 - [Installation](INSTALL.md)
 - [Building a small starter TensorFlow graph](workshop_sections/starter_tf_graph/README.md)
+- [XOR: A minimal training example](workshop_sections/xor/README.md)
 - [Introducing MNIST, and building a simple linear classifier in TensorFlow](workshop_sections/mnist_series/01_README_mnist_simple.md).
 - [Using TensorFlow's high-level APIs to build an MNIST DNN Classifier, and introducing TensorBoard](workshop_sections/mnist_series/02_README_mnist_tflearn.md).
 - [Building the same classifier using TensorFlow's "low-level" APIs](workshop_sections/mnist_series/the_hard_way).

TLDR_CLOUD_INSTALL.md

@@ -0,0 +1,148 @@
+
+# TensorFlow on Cloud Machine Learning: Setup in a cloud-based environment
+
+If you’re unfamiliar with Google Cloud Platform, and/or would rather do the exercises in this workshop in a cloud environment, follow these steps for initial setup.
+
+Alternative setup flows for local installation can be found [here](INSTALL.md).
+
+## Initial setup
+
+### 1. Set Up Your GCP Project
+
+#### 1.1 Create a Cloud Platform project
+
+Create a Google Cloud Platform (GCP) account by [signing up for the free trial](https://cloud.google.com/free-trial/).
+You will be asked to enter a credit card number, but you will get $300 of credits, and won't be billed.
+
+If you already have an account, and have run through your trial credits, see one of the workshop instructors.
+
+#### 1.2 Enable the necessary APIs
+
+1. Click on the “hamburger” menu at upper-left, and then “API Manager”.
+1. Click on “+ Enable API” (top-middle of page).
+1. Enter “Compute Engine API” in the search box and click it when it appears in the list of results.
+1. Click on “Enable” (top-middle of page).
+1. Repeat steps 2 through 4 for: “Cloud Machine Learning” and “Dataflow”.
+
+![Hamburger menu](./assets/hamburger.png)  
+
+![API Manager](./assets/api_manager.png)
+
+
+### 2. Connect to your project's Cloud Shell
+
+Click on the Cloud Shell icon (leftmost icon in the set of icons at top-right of the page).
+
+![Cloud Shell](./assets/cloudshell.png)
+
+
+Run commands 3-6 below in the Cloud Shell.
+
+### 3. Initialize Cloud ML for your project
+
+```shell
+gcloud beta ml init-project
+```
+
+### 4. Set up your Cloud Storage Bucket
+
+```shell
+PROJECT_ID=$(gcloud config list project --format "value(core.project)")
+BUCKET_NAME=${PROJECT_ID}-ml
+gsutil mb -l us-central1 gs://$BUCKET_NAME
+```
+
+### 5. Create a container-optimized image in GCE
+
+```shell
+gcloud compute instances create mlworkshop \
+    --image-family gci-stable \
+    --image-project google-containers \
+    --zone us-central1-b --boot-disk-size=100GB \
+    --machine-type n1-standard-1
+```
+
+### 6. Set up a firewall rule for your project that will allow access to the web services we will run
+
+```shell
+gcloud compute firewall-rules create mlworkshop --allow tcp:8888,tcp:6006,tcp:5000
+```
+
+### 7. SSH into the new GCE instance, in a new browser window
+
+- Click on the “hamburger” menu at upper-left, and then “Compute Engine”
+- Find your instance in the list (mid-page) and click on the “SSH” pulldown menu on the right. Select “Open in browser window”.
+- A new browser window will open, with a command line into your GCE instance.
+
+### 8. Start the Docker container in the GCE image (in the newly opened SSH browser window):
+
+```shell
+docker pull gcr.io/google-samples/tf-workshop:v5
+mkdir workshop-data
+docker run -v `pwd`/workshop-data:/root/tensorflow-workshop-master/workshop-data -it \
+    -p 6006:6006 -p 8888:8888 -p 5000:5000 gcr.io/google-samples/tf-workshop:v5
+```
+
+Then, run the following in the Docker container.
+
+### 9. Configure the Docker container. You’ll need your project ID for this step.
+
+```shell
+gcloud config set project <your-project-ID>
+PROJECT_ID=$(gcloud config list project --format "value(core.project)")
+BUCKET=gs://${PROJECT_ID}-ml
+```
+
+```shell
+gcloud auth login
+```
+(and follow the subsequent instructions)
+
+```shell
+gcloud beta auth application-default login
+```
+(and follow the subsequent instructions)
+
+### 10. Copy some data for the 'transfer learning' example to your bucket:
+
+```shell
+gsutil cp -r gs://tf-ml-workshop/transfer_learning/hugs_preproc_tfrecords $BUCKET
+GCS_PATH=$BUCKET/hugs_preproc_tfrecords
+```
+
+## If you need to restart the container later
+
+If you later exit your container and then want to restart it again, you can find the container ID by running the following in your VM:
+
+```shell
+docker ps -a
+docker start <container_id>
+```
+Once the workshop container is running again, you can exec back into it like this:
+
+```shell
+docker exec -it <container_id> bash
+```
+
+## Cleanup
+
+Once you’re done with your VM, you can stop or delete it. If you think you might return to it later, you might prefer to just stop it. (A stopped instance does not incur charges, but all of the resources that are attached to the instance will still be charged).  You can do this from the [cloud console](https://console.cloud.google.com), or via command line from the Cloud Shell as follows:
+
+```shell
+gcloud compute instances delete --zone us-central1-b mlworkshop
+```
+Or:
+
+```shell
+gcloud compute instances stop --zone us-central1-b mlworkshop
+```
+Then later:
+
+```shell
+gcloud compute instances start --zone us-central1-b mlworkshop
+```
+Delete the firewall rule as well:
+
+```shell
+gcloud compute firewall-rules delete mlworkshop
+```

workshop_image/Dockerfile

@@ -26,7 +26,7 @@ RUN pip install --upgrade --force-reinstall https://storage.googleapis.com/cloud
 
 # RUN python -c "import nltk; nltk.download('punkt')"
 
-RUN curl https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-132.0.0-linux-x86_64.tar.gz | tar xvz
+RUN curl https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-138.0.0-linux-x86_64.tar.gz | tar xvz
 RUN ./google-cloud-sdk/install.sh -q
 RUN ./google-cloud-sdk/bin/gcloud components install beta
 
@@ -36,5 +36,7 @@ ADD download_git_repo.py download_git_repo.py
 EXPOSE 6006
 # IPython
 EXPOSE 8888
+# Flask
+EXPOSE 5000
 
 CMD ["sh", "-c", "python download_git_repo.py ; /bin/bash"]

workshop_sections/README.md

@@ -11,6 +11,8 @@ This directory contains the workshop labs.
 - [Building custom `Estimator`s for a version of MNIST that uses CNNs](mnist_series/mnist_cnn/README.md).
 - [Running distributed MNIST using Cloud ML](mnist_series/cloudml).
 - [Transfer learning: using a trained model to 'bootstrap' learning new classifications](transfer_learning/README.md).
+    + [Using Cloud ML](transfer_learning/cloudml)
+    + [Using a custom Estimator](transfer_learning/TF_Estimator)
 - [Building a word2vec model using a Custom Estimator, and exploring the learned embeddings](word2vec/README.md). Introducing [TFRecords](https://www.tensorflow.org/versions/r0.11/api_docs/python/python_io.html#data-io-python-functions).
 
 In addition, there is an [extras](extras/README.md) directory, that contains some older labs not currently used in this workshop, but which may be of interest.

workshop_sections/transfer_learning/cloudml/README.md

@@ -1,6 +1,18 @@
 
 # Classifying your own images using *transfer learning*
 
+  - [The "hugs/not-hugs" image classification task](#the-hugsnot-hugs-image-classification-task)
+    - [1. Image Preprocessing](#1-image-preprocessing)
+      - [1.1 Deploy the preprocessing job to Cloud Dataflow](#11-deploy-the-preprocessing-job-to-cloud-dataflow)
+    - [2. Modeling: Training the classifier](#2-modeling-training-the-classifier)
+    - [2.1 For the workshop, use pre-generated TFRecords for training](#21-for-the-workshop-use-pre-generated-tfrecords-for-training)
+    - [2.2 Run the training script](#22-run-the-training-script)
+    - [2.3 Monitor the training](#23-monitor-the-training)
+    - [3. Prediction: Using the trained model](#3-prediction-using-the-trained-model)
+      - [3.1 Prediction from the command line using gcloud](#31-prediction-from-the-command-line-using-gcloud)
+      - [3.2 Prediction using the Cloud ML API: A prediction web server](#32-prediction-using-the-cloud-ml-api-a-prediction-web-server)
+  - [Appendix: Running training locally](#appendix-running-training-locally)
+
 The [Google Vision API](https://cloud.google.com/vision/) is great for identifying labels, or categories, for a given
 image. However, sometimes you want to further classify your own images, in more specialized categories that the Google
 Vision API hasn't been trained on.