MNIST_Vanilla.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MNIST Vanilla\n",
    "The purpose of this notebook is to act as a refresher for training a simple neural network on the mnist dataset.\n",
    "We'll accomplish this without yet utilizing TensorFlow, although we'll cheat and grab the MNIST data from TensorFlow. This should end up looking very similar to the network defined in the first chapter of Neural Networks and Deep Learning at http://neuralnetworksanddeeplearning.com/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load up useful libraries\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tasks\n",
    "A lot of Deep Learning resources espouse how a simple neural network can be written in 70 lines, 30 lines, even 11 lines, but that does not do justice to information packed in each line of code. The major tasks include:\n",
    "* Preparing the data (collecting, labeling, preprocessing, encoding, etc)\n",
    "* Defining the network\n",
    "* Training\n",
    "* Tuning hyper-parameters\n",
    "\n",
    "Each one of these tasks is monumental and the list is not exhaustive."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preparing the Data\n",
    "\n",
    "First, the dataset needs to be downloaded and extracted. Then, it is divided into train/validation/test sets. Labels for the train/validation data must be one-hot encoded. All of this is handled via the read_data_sets function call."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n",
      "(55000, 784)\n",
      "(55000, 10)\n"
     ]
    }
   ],
   "source": [
    "''' \n",
    "Download the mnist dataset and validate the size/shape \n",
    "Note the image dimensions. Once we get to using TensorFlow, they will become 4D, rather than 2D: N x H x W x C, where:\n",
    "    N = # of samples,\n",
    "    H = rows/height of the image,\n",
    "    W = columns/width of the image, and\n",
    "    C = channels (colors) in the image\n",
    "'''\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets('MNIST_data', one_hot=True)\n",
    "print(mnist.train.images.shape)\n",
    "print(mnist.train.labels.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Shuffling the Data\n",
    "This handy function will later be used to shuffle the training data. It ensures that images and their labels will be shuffled in the same order. Another approach is to tuple your data and shuffle the tuples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def shuffle(a, b):\n",
    "    assert len(a) == len(b)\n",
    "    state = np.random.get_state()\n",
    "    np.random.shuffle(a)\n",
    "    np.random.set_state(state)\n",
    "    np.random.shuffle(b)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's give it a shuffle in order to verify that it works as intended."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAADU1JREFUeJzt3V2IHfUZx/Hf021DwEaNhiwxjUYl\nKLoXWlbpxVpb2pRUC4ki0qCSgmYFFSr0olEvGvWmSF+oIJUNDU1KalKoxSCljQ0RI5aaRKxmkya+\nkGBiNtsYNQqraZKnFzux27jzP2fPmTMzu8/3A8ueM8/MmYfD/nbmnHn5m7sLQDxfqLoBANUg/EBQ\nhB8IivADQRF+ICjCDwRF+IGgCD8QFOEHgvpimSszM04nBDrM3a2Z+dra8pvZIjPbY2ZvmtmKdl4L\nQLms1XP7zaxL0l5JCyUdkLRN0lJ335VYhi0/0GFlbPmvlfSmu7/t7sclrZe0uI3XA1CidsI/V9I7\nY54fyKb9HzPrN7PtZra9jXUBKFjHv/Bz9wFJAxK7/UCdtLPlPyhp3pjnX8mmAZgE2gn/NkkLzOxi\nM5sm6fuSNhbTFoBOa3m3391PmNl9kv4qqUvSancfLKwzAB3V8qG+llbGZ36g40o5yQfA5EX4gaAI\nPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGg\nCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUC0P0S1JZrZP0keSTko64e69\nRTQFoPPaCn/mm+5+pIDXAVAidvuBoNoNv0vaZGY7zKy/iIYAlKPd3f4+dz9oZrMlPWdm/3L3F8bO\nkP1T4B8DUDPm7sW8kNlKSR+7+88S8xSzMgC53N2ama/l3X4zO8vMZpx+LOk7kna2+noAytXObn+3\npD+Z2enX+b27/6WQrgB0XGG7/U2tjN3+lsydOzdZP/fcc3NrO3d2dmfs4YcfTtbXrl3b8msPDg4m\n69OnT0/WU3/b2UYr14oVK5L1xx9/PFkfGRlJ1jup47v9ACY3wg8ERfiBoAg/EBThB4Ii/EBQHOqr\ngZ6enmR9w4YNyfpll11WZDtowsKFC5P1LVu2lNTJ53GoD0AS4QeCIvxAUIQfCIrwA0ERfiAowg8E\nVcTde8O7/PLLk/W+vr5k/a677krWOY5fvq1btybr7777bkmddA5bfiAowg8ERfiBoAg/EBThB4Ii\n/EBQhB8IiuP8BWh0C+mlS5cm6729jGxeN41uOb5nz56SOukctvxAUIQfCIrwA0ERfiAowg8ERfiB\noAg/EFTD4/xmtlrS9yQNu3tPNu08SRskzZe0T9Kt7v5+59qst5kzZybrs2fPLqmTiduxY0eyvmvX\nrmT9jjvuKLKdQr300ku5tSeeeCK57LZt24pup3aa2fL/VtKiM6atkLTZ3RdI2pw9BzCJNAy/u78g\n6egZkxdLWpM9XiNpScF9AeiwVj/zd7v7oezxkKTugvoBUJK2z+13d0+NwWdm/ZL6210PgGK1uuU/\nbGZzJCn7PZw3o7sPuHuvu3P1ClAjrYZ/o6Rl2eNlkp4pph0AZWkYfjN7StLfJV1mZgfM7E5JP5W0\n0MzekPTt7DmAScTccz+uF7+yxHcDdZe65n79+vXJZefPn19wN8176623kvXHHnssWX/ooYeS9Qsv\nvHDCPRVlcHAwWb/55ptza43el8nM3a2Z+TjDDwiK8ANBEX4gKMIPBEX4gaAIPxAUt+7OXHTRRcn6\npk2bcmtnn3120e1MyJVXXplbGx7OPflSknTdddcl61Ueyjty5Eiyfv311yfrH3zwQZHtTDls+YGg\nCD8QFOEHgiL8QFCEHwiK8ANBEX4gKI7zZ7q6upL1qo/lp7z33nu5tfffn7x3VN+9e3eyznH89rDl\nB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgOM4/BSxadOYgyv9z7Nix5LLXXHNN0e007fnnn0/Wb7zx\nxnIaCYotPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8E1XCIbjNbLel7kobdvSebtlLSckn/zmZ70N3/\n3HBlNR6i+5JLLknW9+7dW1InU8vGjRtza/fcc09y2aGhoaLbCaHIIbp/K2m8s0h+6e5XZT8Ngw+g\nXhqG391fkHS0hF4AlKidz/z3mdlrZrbazGYW1hGAUrQa/l9LulTSVZIOSfp53oxm1m9m281se4vr\nAtABLYXf3Q+7+0l3PyVplaRrE/MOuHuvu/e22iSA4rUUfjObM+bpTZJ2FtMOgLI0vKTXzJ6S9A1J\ns8zsgKSfSPqGmV0lySXtk3R3B3sE0AENj/MXurIaH+c/55xzkvVHH300t9boeHVkS5Ysya09++yz\nJXYSR5HH+QFMQYQfCIrwA0ERfiAowg8ERfiBoLh1d+bDDz9M1h955JHc2qZNm5LL3n777cn6Lbfc\nkqwDncCWHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeC4pLeEnR3dyfrs2bNStbXrVuXrPf09Ey4p7Jw\nSW/5uKQXQBLhB4Ii/EBQhB8IivADQRF+ICjCDwTFcf5JYPr06cn6/v37c2vnn39+0e1MyPHjx3Nr\nJ0+eTC47e/bsZH1kZKSlnqY6jvMDSCL8QFCEHwiK8ANBEX4gKMIPBEX4gaAa3rffzOZJWiupW5JL\nGnD3X5nZeZI2SJovaZ+kW939/c61Gtcnn3ySrJd5rsZETZs2reoWkKOZLf8JST9y9yskfU3SvWZ2\nhaQVkja7+wJJm7PnACaJhuF390Pu/kr2+CNJuyXNlbRY0ppstjWS8m/ZAqB2JvSZ38zmS7pa0j8k\ndbv7oaw0pNGPBQAmiabH6jOzL0v6o6T73f2Y2f9OH3Z3zztv38z6JfW32yiAYjW15TezL2k0+Ovc\n/els8mEzm5PV50gaHm9Zdx9w91537y2iYQDFaBh+G93E/0bSbnf/xZjSRknLssfLJD1TfHsAOqXh\nJb1m1idpq6TXJZ3KJj+o0c/9f5B0oaT9Gj3Ud7TBa9X3mNQkNjQ0lFtrdFvwOrvtttuS9Q0bNpTU\nyeTS7CW9DT/zu/uLkvJe7FsTaQpAfXCGHxAU4QeCIvxAUIQfCIrwA0ERfiAobt09BUzV4/zDw+Oe\nNPqZCy64oKROJhdu3Q0gifADQRF+ICjCDwRF+IGgCD8QFOEHgmr6Nl5A2R544IGqW5jS2PIDQRF+\nICjCDwRF+IGgCD8QFOEHgiL8QFBczz8F9PX15dYWLFiQXHbVqlVFt1OYGTNmJOsjIyMldTK5cD0/\ngCTCDwRF+IGgCD8QFOEHgiL8QFCEHwiq4fX8ZjZP0lpJ3ZJc0oC7/8rMVkpaLunf2awPuvufO9Uo\n8r344ou5tZdffjm5bFdXV7L+5JNPttTTacuXL8+tbdmyJbnsp59+2ta6kdbMzTxOSPqRu79iZjMk\n7TCz57LaL939Z51rD0CnNAy/ux+SdCh7/JGZ7ZY0t9ONAeisCX3mN7P5kq6W9I9s0n1m9pqZrTaz\nmTnL9JvZdjPb3lanAArVdPjN7MuS/ijpfnc/JunXki6VdJVG9wx+Pt5y7j7g7r3u3ltAvwAK0lT4\nzexLGg3+Ond/WpLc/bC7n3T3U5JWSbq2c20CKFrD8JuZSfqNpN3u/osx0+eMme0mSTuLbw9ApzS8\npNfM+iRtlfS6pFPZ5AclLdXoLr9L2ifp7uzLwdRrcUkv0GHNXtLL9fzAFMP1/ACSCD8QFOEHgiL8\nQFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0E1c/feIh2RtH/M81nZtDqqa291\n7Uuit1YV2dtFzc5Y6vX8n1u52fa63tuvrr3VtS+J3lpVVW/s9gNBEX4gqKrDP1Dx+lPq2ltd+5Lo\nrVWV9FbpZ34A1al6yw+gIpWE38wWmdkeM3vTzFZU0UMeM9tnZq+b2atVDzGWDYM2bGY7x0w7z8ye\nM7M3st/jDpNWUW8rzexg9t69amY3VNTbPDPbYma7zGzQzH6YTa/0vUv0Vcn7Vvpuv5l1SdoraaGk\nA5K2SVrq7rtKbSSHme2T1OvulR8TNrOvS/pY0lp378mmPSbpqLv/NPvHOdPdf1yT3lZK+rjqkZuz\nAWXmjB1ZWtISST9Qhe9doq9bVcH7VsWW/1pJb7r72+5+XNJ6SYsr6KP23P0FSUfPmLxY0prs8RqN\n/vGULqe3WnD3Q+7+Svb4I0mnR5au9L1L9FWJKsI/V9I7Y54fUL2G/HZJm8xsh5n1V93MOLrHjIw0\nJKm7ymbG0XDk5jKdMbJ0bd67Vka8Lhpf+H1en7t/VdJ3Jd2b7d7Wko9+ZqvT4ZqmRm4uyzgjS3+m\nyveu1RGvi1ZF+A9Kmjfm+VeyabXg7gez38OS/qT6jT58+PQgqdnv4Yr7+UydRm4eb2Rp1eC9q9OI\n11WEf5ukBWZ2sZlNk/R9SRsr6ONzzOys7IsYmdlZkr6j+o0+vFHSsuzxMknPVNjL/6nLyM15I0ur\n4veudiNeu3vpP5Ju0Og3/m9JeqiKHnL6ukTSP7Ofwap7k/SURncD/6PR70bulHS+pM2S3pD0N0nn\n1ai332l0NOfXNBq0ORX11qfRXfrXJL2a/dxQ9XuX6KuS940z/ICg+MIPCIrwA0ERfiAowg8ERfiB\noAg/EBThB4Ii/EBQ/wWoZU8WnS+K9QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f79e2a08b00>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "shuffle(mnist.train.images, mnist.train.labels)\n",
    "# Sanity check that our shuffling is properly associating the labels with the images\n",
    "img = np.reshape(np.copy(mnist.train.images), (55000, 28, 28))\n",
    "\n",
    "plt.imshow(img[0], cmap='gray')\n",
    "print(np.argmax(mnist.train.labels[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One more for good measure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAADaxJREFUeJzt3X+sVPWZx/HPUywGKCpK9nJjcWkR\nG6t/0PXGmEgW1q6NVRJsYgxozG226e0fkGyjiXt1g5ps1h+bbTf9h5rbgOCmtW0CRqJVKKRRNq7N\nvRhXAW1hGxq4QaihCFWS7r332T/m3N0r3vOdYebMnDM871dyMzPnmTPz5IQP55z5npmvubsAxPOZ\nshsAUA7CDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gqIs6+WZmxuWEQJu5uzXyvJb2/GZ2m5n9\nxswOmdlgK68FoLOs2Wv7zWyGpN9KulXSUUnDkta4+4HEOuz5gTbrxJ7/RkmH3P137v5nST+VtKqF\n1wPQQa2E/0pJR6Y8Ppot+wQzGzCzETMbaeG9ABSs7R/4ufuQpCGJw36gSlrZ849KWjjl8eezZQC6\nQCvhH5a0xMy+YGYzJa2WtL2YtgC0W9OH/e4+ZmbrJO2QNEPSJnffX1hnANqq6aG+pt6Mc36g7Tpy\nkQ+A7kX4gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQRF+\nICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUE1P0S1JZnZY\n0hlJ45LG3L2viKaAqps5c2ay/sADDyTrl156aW5tcHCwqZ7OV0vhz/yNu39QwOsA6CAO+4GgWg2/\nS9ppZnvNbKCIhgB0RquH/cvcfdTM/kLSL83sPXd/beoTsv8U+I8BqJiW9vzuPprdnpD0vKQbp3nO\nkLv38WEgUC1Nh9/M5pjZ3Mn7kr4maV9RjQFor1YO+3skPW9mk6/zE3d/pZCuALSduXvn3sysc28G\ntGD27NnJ+tNPP52s33vvvcn6qVOncmtXXHFFct163N0aeR5DfUBQhB8IivADQRF+ICjCDwRF+IGg\nivhWH9B1Fi9enKyvX78+Wa83lLdu3bpkfc+ePcl6J7DnB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGg\n+EovLlizZs3Krb300kvJdZcvX56sb9myJVlfu3Ztsn727NlkvRV8pRdAEuEHgiL8QFCEHwiK8ANB\nEX4gKMIPBMX3+Sugt7c3WV+xYkWyfvDgwdzayMhIMy1dEB588MHcWr1x/OHh4WS93vf12zmOXxT2\n/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QVN1xfjPbJGmlpBPufn227HJJP5O0SNJhSXe7+x/b12Z3\nW7p0abK+Y8eOZH3+/PnJ+oYNG3JrF/I4/5w5c5L1+++/P7e2c+fO5LqPPvposv7xxx8n692gkT3/\nZkm3nbNsUNJud18iaXf2GEAXqRt+d39N0slzFq+SNPlTJlsk3VlwXwDarNlz/h53P5bdf19ST0H9\nAOiQlq/td3dP/TafmQ1IGmj1fQAUq9k9/3Ez65Wk7PZE3hPdfcjd+9y9r8n3AtAGzYZ/u6T+7H6/\npBeKaQdAp9QNv5k9J+k/JX3JzI6a2bckPSnpVjM7KOlvs8cAukjdc353X5NT+mrBvXStiy5Kb8bB\nwfRIaL1x/HrGx8dbWr+qbr755mS93na94447cmtvvPFGct2xsbFk/ULAFX5AUIQfCIrwA0ERfiAo\nwg8ERfiBoJiiu0GzZ8/OrdUbNrrmmmuS9ZMnz/3e1Cc988wzyfqTT+ZfZnHmzJnkumWqN8S5f//+\nZP306dPJ+pIlS867pwsBU3QDSCL8QFCEHwiK8ANBEX4gKMIPBEX4gaCYortBGzduzK1dd911yXXr\nTfd80003NdVTN1i0aFFubdeuXcl19+7dm6yvXLmymZaQYc8PBEX4gaAIPxAU4QeCIvxAUIQfCIrw\nA0Exzt+g1atX59YmJiaS63744YfJ+n333Zes33XXXcl6laV+fnvevHnJdetdH4HWsOcHgiL8QFCE\nHwiK8ANBEX4gKMIPBEX4gaDqjvOb2SZJKyWdcPfrs2WPSfq2pD9kT3vY3X/Rriar4OWXX86tLV68\nOLnutddem6xv3ry5mZa63pEjR5L1AwcOdKiTmBrZ82+WdNs0y//N3Zdmfxd08IELUd3wu/trktJT\nygDoOq2c868zs7fNbJOZpa/TBFA5zYb/h5IWS1oq6Zik7+U90cwGzGzEzEaafC8AbdBU+N39uLuP\nu/uEpB9JujHx3CF373P3vmabBFC8psJvZr1THn5D0r5i2gHQKY0M9T0naYWk+WZ2VNKjklaY2VJJ\nLumwpO+0sUcAbWDu3rk3M+vcmxVsxowZubWrr746ue4rr7ySrF911VVN9TTpvffey62tWbMmue4l\nl1ySrL/66qtN9TQpNZZf73f39+3jgLIZ7m6NPI8r/ICgCD8QFOEHgiL8QFCEHwiK8ANBMdTXAZdd\ndlmyfvHFF7f0+mfPns2tnT59OrnuggULkvXR0dFkfcOGDcn6+vXrc2unTp1KrovmMNQHIInwA0ER\nfiAowg8ERfiBoAg/EBThB4JinP8CN2vWrGR9165dyfrChQuT9VtuuSVZP3ToULKO4jHODyCJ8ANB\nEX4gKMIPBEX4gaAIPxAU4QeCqvu7/ai+5cuX59aeeuqp5Lo33HBDsr569epknXH87sWeHwiK8ANB\nEX4gKMIPBEX4gaAIPxAU4QeCqjvOb2YLJT0rqUeSSxpy9x+Y2eWSfiZpkaTDku529z+2r9W45s2b\nl6w/8sgjubW+vr7kuk888USyvnXr1mQd3auRPf+YpAfc/cuSbpK01sy+LGlQ0m53XyJpd/YYQJeo\nG353P+bub2b3z0h6V9KVklZJ2pI9bYukO9vVJIDindc5v5ktkvQVSb+W1OPux7LS+6qdFgDoEg1f\n229mn5O0VdJ33f202f//TJi7e97v85nZgKSBVhsFUKyG9vxm9lnVgv9jd9+WLT5uZr1ZvVfSienW\ndfchd+9z9/QnTwA6qm74rbaL3yjpXXf//pTSdkn92f1+SS8U3x6Adqn7091mtkzSHknvSJrIFj+s\n2nn/zyVdJen3qg31nazzWvx0dxO2bduWrK9atSq39vjjjyfXTU2hje7U6E931z3nd/f/kJT3Yl89\nn6YAVAdX+AFBEX4gKMIPBEX4gaAIPxAU4QeCYoruCliwYEGy/vrrryfrw8PDubV77rknue74+Hiy\nju7DFN0Akgg/EBThB4Ii/EBQhB8IivADQRF+ICim6K6A/v7+ZH1sbCxZf+ihh3JrjOMjD3t+ICjC\nDwRF+IGgCD8QFOEHgiL8QFCEHwiK7/NXwNy5c5P1iYmJZP2jjz4qsh10Ob7PDyCJ8ANBEX4gKMIP\nBEX4gaAIPxAU4QeCqjvOb2YLJT0rqUeSSxpy9x+Y2WOSvi3pD9lTH3b3X9R5Lcb5gTZrdJy/kfD3\nSup19zfNbK6kvZLulHS3pD+5+7822hThB9qv0fDX/SUfdz8m6Vh2/4yZvSvpytbaA1C28zrnN7NF\nkr4i6dfZonVm9raZbTKzeTnrDJjZiJmNtNQpgEI1fG2/mX1O0quS/tndt5lZj6QPVPsc4J9UOzX4\nuzqvwWE/0GaFnfNLkpl9VtKLkna4+/enqS+S9KK7X1/ndQg/0GaFfbHHzEzSRknvTg1+9kHgpG9I\n2ne+TQIoTyOf9i+TtEfSO5Imv1v6sKQ1kpaqdth/WNJ3sg8HU6/Fnh9os0IP+4tC+IH24/v8AJII\nPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQdX9Ac+CfSDp91Me\nz8+WVVFVe6tqXxK9NavI3v6y0Sd29Pv8n3pzsxF37yutgYSq9lbVviR6a1ZZvXHYDwRF+IGgyg7/\nUMnvn1LV3qral0RvzSqlt1LP+QGUp+w9P4CSlBJ+M7vNzH5jZofMbLCMHvKY2WEze8fM3ip7irFs\nGrQTZrZvyrLLzeyXZnYwu512mrSSenvMzEazbfeWmd1eUm8LzexXZnbAzPab2d9ny0vddom+Stlu\nHT/sN7MZkn4r6VZJRyUNS1rj7gc62kgOMzssqc/dSx8TNrO/lvQnSc9OzoZkZv8i6aS7P5n9xznP\n3f+hIr09pvOcublNveXNLP1Nlbjtipzxughl7PlvlHTI3X/n7n+W9FNJq0roo/Lc/TVJJ89ZvErS\nluz+FtX+8XRcTm+V4O7H3P3N7P4ZSZMzS5e67RJ9laKM8F8p6ciUx0dVrSm/XdJOM9trZgNlNzON\nnikzI70vqafMZqZRd+bmTjpnZunKbLtmZrwuGh/4fdoyd/8rSV+XtDY7vK0kr52zVWm45oeSFqs2\njdsxSd8rs5lsZumtkr7r7qen1srcdtP0Vcp2KyP8o5IWTnn8+WxZJbj7aHZ7QtLzqp2mVMnxyUlS\ns9sTJffzf9z9uLuPu/uEpB+pxG2XzSy9VdKP3X1btrj0bTddX2VttzLCPyxpiZl9wcxmSlotaXsJ\nfXyKmc3JPoiRmc2R9DVVb/bh7ZL6s/v9kl4osZdPqMrMzXkzS6vkbVe5Ga/dveN/km5X7RP//5b0\nj2X0kNPXFyX9V/a3v+zeJD2n2mHg/6j22ci3JF0habekg5J2Sbq8Qr39u2qzOb+tWtB6S+ptmWqH\n9G9Leiv7u73sbZfoq5TtxhV+QFB84AcERfiBoAg/EBThB4Ii/EBQhB8IivADQRF+IKj/BSCIcU3U\n3iCPAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f79e2a08cc0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(img[1], cmap='gray')\n",
    "print(np.argmax(mnist.train.labels[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Defining the Network\n",
    "The structure of a basic neural network is as follows:\n",
    "* An input layer and an output layer (dimensions match input data and number of labels, respectively\n",
    "* One or more hidden layers \n",
    "* Weight matrices (the parameters)\n",
    "* Bias matrices\n",
    "* Feed Forward function\n",
    "* Activation function for each layer (can be different for each layer, but we'll keep it the same for this example)\n",
    "* Cost/Loss Function\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Activation Function and Cost Function\n",
    "\n",
    "The activation function is used during the forward pass to add a non-linearity to the graph/network. After the forward pass, the cost function tells us how wrong the network was. Additionally, we'll need the derivative of each during the backpropagation phase of training.\n",
    "\n",
    "Note about activation: here we use **sigmoid**, but we can return later to play with relu\n",
    "\n",
    "Note about cost: For now, we're using **Mean Squared Error**, but there exist many cost functions, such as Cross Entropy, which behave better for categorization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid(z):\n",
    "    return 1 / (1 + np.exp(-z))\n",
    "\n",
    "def sigmoid_prime(z):\n",
    "    return sigmoid(z) * (1 - sigmoid(z))\n",
    "\n",
    "def cost(y, y_pred):\n",
    "    return 0.5 * (1 / y.shape[0]) * np.sum(np.abs(y - y_pred) ** 2)\n",
    "\n",
    "def cost_prime(y, y_pred):\n",
    "    return y_pred - y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Network Object\n",
    "Store the network object in a simple python class called Network. It will be initialized by passing a list of integers defining each layer size. The network object itself will handle initializing the weights (basically, we just want them to be small, non-zero numbers)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Network:\n",
    "    \n",
    "    def __init__(self, *layers):\n",
    "        self.layers = [l for l in layers]\n",
    "        self.W = [np.random.rand(layers[i], layers[i+1]) * 0.001 for i in range(len(layers) - 1)]\n",
    "        self.B = [np.random.rand(1, layers[i+1]) for i in range(len(layers) - 1)]  # Input layer doesn't have bias\n",
    "        \n",
    "    def feedforward(self, x):\n",
    "        a = x  # re-label input to simplify loop\n",
    "        activations = [a]\n",
    "        z_products = []\n",
    "        for b, w in zip(self.B, self.W):\n",
    "            z = np.dot(a, w) + b\n",
    "            a = sigmoid(z)\n",
    "            activations.append(a)\n",
    "            z_products.append(z)\n",
    "        return activations, z_products"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Go ahead and give the network a spin. Note that position in the output & z array corresponds to the layer in the network where it was calculated (therefore, the last element in the array is the output of the network)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(55000, 10)\n",
      "1.95800059112\n"
     ]
    }
   ],
   "source": [
    "network = Network(784, 15, 10)\n",
    "output, z = network.feedforward(mnist.train.images)\n",
    "print(output[-1].shape)\n",
    "print(cost(mnist.train.labels, output[-1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training the Network\n",
    "\n",
    "To train the network, we'll use Stochastic Gradient Descent (mini-batch). Achieving this will require shuffling the training data and labels and dividing the training data into mini-batches. Next, we have to do a forward pass on the network and then execute backpropagation to update the weights and biases of the network. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Network(Network):  # Extend the Network class defined above\n",
    "    def test(self):\n",
    "        # Stubbed. Gets defined further down\n",
    "        return\n",
    "    \n",
    "    def train(self, images, labels, batch_size=200, learning_rate=0.01, epochs=10):\n",
    "        assert len(images) == len(labels)\n",
    "        \n",
    "        for i in range(epochs):\n",
    "            shuffle(images, labels)\n",
    "            # Divide training data into batches\n",
    "            img_batches = [images[i:i+batch_size] for i in range(0, len(images), batch_size)]\n",
    "            label_batches = [labels[i:i+batch_size] for i in range(0, len(labels), batch_size)]\n",
    "            n_batches = len(img_batches)\n",
    "\n",
    "            training_cost = []\n",
    "            for imgs, lbls in zip(img_batches, label_batches):\n",
    "                a, z = self.feedforward(imgs)\n",
    "\n",
    "                # Calculate Output Error\n",
    "                dL = np.multiply((a[-1] - lbls),sigmoid_prime(z[-1]))\n",
    "\n",
    "                # Update biases and weights for output layer using output error\n",
    "                dBL = np.reshape(np.sum(dL, axis=0), (1,dL.shape[1]))\n",
    "                dWL = np.dot(a[-2].T, dL)\n",
    "                # Being verbose to show that we're building a list of deltas for each layer in the network (excluding input layer)\n",
    "                dBl = [dBL]\n",
    "                dWl = [dWL]\n",
    "                dl = dL\n",
    "                \n",
    "                # Calculate Hidden Error\n",
    "                for l in range(len(a) - 2, 0, -1):  # -1 for zero-indexing, -1 more because output layer already calculated\n",
    "                    dl = np.multiply(np.dot(dl, self.W[l].T), sigmoid_prime(z[l-1]))\n",
    "                    dBl.insert(0, np.reshape(np.sum(dl, axis=0), (1, dl.shape[1])))\n",
    "                    dWl.insert(0, np.dot(a[l-1].T, dl))\n",
    "\n",
    "                # Update weights and biases\n",
    "                for j in range(len(self.W)):\n",
    "                    self.W[j] -= (learning_rate / batch_size) * dWl[j]\n",
    "                    self.B[j] -= (learning_rate / batch_size) * dBl[j]\n",
    "\n",
    "\n",
    "            # Cost after training this epoch\n",
    "            a, z = self.feedforward(images)\n",
    "            training_cost.append(cost(labels, a[-1]))\n",
    "            result = str(training_cost[-1])\n",
    "\n",
    "            if (i + 1) % 5 == 0:\n",
    "                result += ' ' + str(self.test()) +  ' / 10000'\n",
    "            print(result)\n",
    "        return training_cost\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Executing Training\n",
    "With (mostly) everything defined, the network is ready to start training. First, let's define a simple network and see what its cost looks like"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.7686093702\n"
     ]
    }
   ],
   "source": [
    "network = Network(784, 15, 10)\n",
    "a, z = network.feedforward(mnist.train.images)\n",
    "print(cost(mnist.train.labels, a[-1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, training can begin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.449512576095\n",
      "0.399371706367\n",
      "0.339003654516\n",
      "0.262533907116\n",
      "0.198346654812 None / 10000\n",
      "0.151840758107\n",
      "0.121685275456\n",
      "0.105677567164\n",
      "0.0954982060348\n",
      "0.0884731201589 None / 10000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.088473120158866644]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "network.train(mnist.train.images, mnist.train.labels, learning_rate=1, batch_size=200, epochs=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Testing and Verification\n",
    "Let's add a function that explicitly tells us how many images we've correctly labeled. As the network is just using MSE at the moment, we'll simply pick the label that has the highest confidence for each image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Network(Network):  # Extend the Network class defined above\n",
    "    def test(self):\n",
    "        img = mnist.test.images\n",
    "        a, z = self.feedforward(img)\n",
    "        a[-1].shape\n",
    "        prediction = np.argmax(a[-1], axis=1)\n",
    "        truth = np.argmax(mnist.test.labels, axis=1)\n",
    "        result = np.sum([p == i for p, i in zip(prediction, truth)])\n",
    "        return result\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can conveniently see both the cost and the test results while training. Let's go ahead and make a new network to show it off:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "deep_network = Network(784,100,20,10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.023243093668\n",
      "0.0228427627962\n",
      "0.0227634320302\n",
      "0.0224295491061\n",
      "0.022164345061 9652 / 10000\n",
      "0.0218719834621\n",
      "0.0213202846866\n",
      "0.0210423460174\n",
      "0.0209766956108\n",
      "0.0205507511025 9672 / 10000\n",
      "0.0202910638104\n",
      "0.0198748999097\n",
      "0.0196906197979\n",
      "0.019430119096\n",
      "0.0191744574822 9676 / 10000\n",
      "0.0190347612555\n",
      "0.0186285660688\n",
      "0.0187447499589\n",
      "0.0182625280044\n",
      "0.0178989877822 9695 / 10000\n",
      "0.017819695328\n",
      "0.017606082905\n",
      "0.0172492833402\n",
      "0.0169827529661\n",
      "0.0167888319138 9697 / 10000\n",
      "0.0166801378872\n",
      "0.0166505082546\n",
      "0.0162444945122\n",
      "0.0161528727794\n",
      "0.0159099343575 9701 / 10000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.015909934357499763]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "deep_network.train(mnist.train.images, mnist.train.labels, learning_rate=1, batch_size=200, epochs=30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Final Thoughts\n",
    "\n",
    "With this basic implementation, the network was able to hit **97% accuracy** on the MNIST dataset. Not bad! Next up, we'll review some improvements to the nextwork, including using ReLU for activations, Cross Entropy for cost, L1/L2 normalization (maybe), and dropout. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load up useful libraries\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "def shuffle(a, b):\n",
    "    assert len(a) == len(b)\n",
    "    state = np.random.get_state()\n",
    "    np.random.shuffle(a)\n",
    "    np.random.set_state(state)\n",
    "    np.random.shuffle(b)\n",
    "    \n",
    "def sigmoid(z):\n",
    "    return 1 / (1 + np.exp(-z))\n",
    "\n",
    "def sigmoid_prime(z):\n",
    "    return sigmoid(z) * (1 - sigmoid(z))\n",
    "\n",
    "def cost(y, y_pred):\n",
    "    return 0.5 * (1 / y.shape[0]) * np.sum(np.abs(y - y_pred) ** 2)\n",
    "\n",
    "def cost_prime(y, y_pred):\n",
    "    return y_pred - y\n",
    "\n",
    "\n",
    "class Activation:\n",
    "    def __init__(self, activation, derivative):\n",
    "        self.activation = activation\n",
    "        self.derivative = derivative\n",
    "        \n",
    "    def activate(z):\n",
    "        return self.activation(z)\n",
    "    \n",
    "    def derive(z):\n",
    "        return self.derivative(z)\n",
    "    \n",
    "\n",
    "class Network:\n",
    "    \n",
    "    def __init__(self, *layers, activation=Activation(sigmoid, sigmoid_prime)):\n",
    "        assert activation is not None\n",
    "        self.layers = [l for l in layers]\n",
    "        self.W = [np.random.rand(layers[i], layers[i+1]) * 0.001 for i in range(len(layers) - 1)]\n",
    "        self.B = [np.random.rand(1, layers[i+1]) for i in range(len(layers) - 1)]  # Input layer doesn't have bias\n",
    "        self.activation = activation\n",
    "        \n",
    "    def feedforward(self, x):\n",
    "        a = x  # re-label input to simplify loop\n",
    "        activations = [a]\n",
    "        z_products = []\n",
    "        for b, w in zip(self.B, self.W):\n",
    "            z = np.dot(a, w) + b\n",
    "            a = self.activation.activate(z)\n",
    "            activations.append(a)\n",
    "            z_products.append(z)\n",
    "        return activations, z_products\n",
    "    \n",
    "    def train(self, images, labels, batch_size=200, learning_rate=0.01, epochs=10):\n",
    "        assert len(images) == len(labels)\n",
    "        \n",
    "        for i in range(epochs):\n",
    "            shuffle(images, labels)\n",
    "            # Divide training data into batches\n",
    "            img_batches = [images[i:i+batch_size] for i in range(0, len(images), batch_size)]\n",
    "            label_batches = [labels[i:i+batch_size] for i in range(0, len(labels), batch_size)]\n",
    "            n_batches = len(img_batches)\n",
    "\n",
    "            training_cost = []\n",
    "            for imgs, lbls in zip(img_batches, label_batches):\n",
    "                a, z = self.feedforward(imgs)\n",
    "\n",
    "                # Calculate Output Error\n",
    "                dL = np.multiply((a[-1] - lbls),self.activation.derive(z[-1]))\n",
    "\n",
    "                # Update biases and weights for output layer using output error\n",
    "                dBL = np.reshape(np.sum(dL, axis=0), (1,dL.shape[1]))\n",
    "                dWL = np.dot(a[-2].T, dL)\n",
    "                # Being verbose to show that we're building a list of deltas for each layer in the network (excluding input layer)\n",
    "                dBl = [dBL]\n",
    "                dWl = [dWL]\n",
    "                dl = dL\n",
    "                \n",
    "                # Calculate Hidden Error\n",
    "                for l in range(len(a) - 2, 0, -1):  # -1 for zero-indexing, -1 more because output layer already calculated\n",
    "                    dl = np.multiply(np.dot(dl, self.W[l].T), self.activation.derive(z[l-1]))\n",
    "                    dBl.insert(0, np.reshape(np.sum(dl, axis=0), (1, dl.shape[1])))\n",
    "                    dWl.insert(0, np.dot(a[l-1].T, dl))\n",
    "\n",
    "                # Update weights and biases\n",
    "                for j in range(len(self.W)):\n",
    "                    self.W[j] -= (learning_rate / batch_size) * dWl[j]\n",
    "                    self.B[j] -= (learning_rate / batch_size) * dBl[j]\n",
    "\n",
    "\n",
    "            # Cost after training this epoch\n",
    "            a, z = self.feedforward(images)\n",
    "            training_cost.append(cost(labels, a[-1]))\n",
    "            result = str(training_cost[-1])\n",
    "\n",
    "            if (i + 1) % 5 == 0:\n",
    "                result += ' ' + str(self.test()) +  ' / 10000'\n",
    "            print(result)\n",
    "        return training_cost\n",
    "    \n",
    "    def test(self):\n",
    "        img = mnist.test.images\n",
    "        a, z = self.feedforward(img)\n",
    "        a[-1].shape\n",
    "        prediction = np.argmax(a[-1], axis=1)\n",
    "        truth = np.argmax(mnist.test.labels, axis=1)\n",
    "        result = np.sum([p == i for p, i in zip(prediction, truth)])\n",
    "        return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}