MNIST_Enhancements.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MNIST Enhancements\n",
    "For this notebook, the goal is to provide a number of improvements to the Neural Network defined in the MNIST Vanilla notebook; this is to refamiliarize us with the following topics:\n",
    "\n",
    "* ReLU (and other) Activation Functions\n",
    "* Cross Entropy & Softmax Cost Functions\n",
    "* Regularization Techniques (L1, L2, Dropout)\n",
    "* Learning Rate Algorithms (TBD)\n",
    "\n",
    "## Getting Started\n",
    "With some small modifications to the Network class defined in the MNIST Vanilla notebook, we can now import all of our previous work. See the following files in the **libs** folder to examine the changes; primarily, the changes allow us to supply different activation functions and cost functions for the network. It should be noted that different layers in a neural network can (and often do) have different activation functions, but to keep the scope from getting out of hand, we simply define one activation function for the entire network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import Necessary Libraries\n",
    "Nothing too surprising here. We load the **network** python file mentioned above, provide some Jupyter configuration, and reload the MNIST dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "sys.path.append('/notebooks/projects/MNIST/libs')  # Jupyter import fix\n",
    "import numpy as np\n",
    "import network\n",
    "import network_ce\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "# Reload the mnist dataset\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets('MNIST_data', one_hot=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Verify It Works\n",
    "Here, we'll define and train another network, just like at the end of the last notebook. *Note the new keyword arguments specifying the test data & labels*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.449975839762 1135 / 10000\n",
      "0.449912287983 1135 / 10000\n",
      "0.449968891986 1135 / 10000\n",
      "0.44992055809 1135 / 10000\n",
      "0.449945373114 1135 / 10000\n",
      "0.449903742943 1135 / 10000\n",
      "0.449905197755 1135 / 10000\n",
      "0.449974896014 1135 / 10000\n",
      "0.450009697038 1135 / 10000\n",
      "0.449924198004 1010 / 10000\n",
      "0.450052785236 974 / 10000\n",
      "0.449992236052 1135 / 10000\n",
      "0.45010733942 1135 / 10000\n",
      "0.449913259588 1135 / 10000\n",
      "0.449933833457 1032 / 10000\n",
      "0.449899320266 1135 / 10000\n",
      "0.449835065275 1135 / 10000\n",
      "0.449848109852 1135 / 10000\n",
      "0.449770891838 1135 / 10000\n",
      "0.44976420732 1135 / 10000\n",
      "0.44963018526 1135 / 10000\n",
      "0.44966462176 1135 / 10000\n",
      "0.44940681508 1135 / 10000\n",
      "0.448229398708 2758 / 10000\n",
      "0.439544282468 1698 / 10000\n",
      "0.406620377293 2397 / 10000\n",
      "0.394376587611 2935 / 10000\n",
      "0.379867674322 3312 / 10000\n",
      "0.367365942826 3803 / 10000\n",
      "0.359096510296 4025 / 10000\n"
     ]
    }
   ],
   "source": [
    "deep_network = network.Network(784,100,20,10)\n",
    "mse_cost = deep_network.train(mnist.train.images, mnist.train.labels, learning_rate=1.0, batch_size=200, epochs=30, test_epoch=1, test_images=mnist.test.images, test_labels=mnist.test.labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ReLU\n",
    "For fully-connected layers in a network, ReLU has become the current gold-standard, due to its simplicity and efficiency. Many variants of it now exist, but we'll keep it simple for now. ReLU is wonderful not only because it's simple, but because its derivative is constant, meaning that gradient descent doesn't slow down with larger networks (no sigmoid derivative, which we'll see again with Cross Entropy next)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def relu(z):\n",
    "    return np.maximum(0, z)  # For each element, return 0 if z[i] <= 0, else return z[i]\n",
    "\n",
    "def relu_prime(z):  # Similar to above, but return 1 if z[i] > 0 instead of z\n",
    "    out = np.copy(z)\n",
    "    out[out<=0] = 0\n",
    "    out[out>0] = 1\n",
    "    return out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define the Activation\n",
    "To use ReLU in our network, we simply define an Activation object and supply it when defining our Network object. \n",
    "\n",
    "*Note*: To support future changes to the cost function, along with many other reasons not listed here, it's better to define activations for each layer, so you can support multiple activation fuctions in one network (e.g. need the output layer to be sigmoid or softmax, while the rest can be ReLU)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Matrix:\n",
      " [[-0.62005269 -1.23466063 -0.66512931]\n",
      " [-1.09545393 -1.51667587 -0.13271472]\n",
      " [ 0.89529023  0.19272295 -0.35767351]]\n",
      "ReLU:\n",
      " [[ 0.          0.          0.        ]\n",
      " [ 0.          0.          0.        ]\n",
      " [ 0.89529023  0.19272295  0.        ]]\n",
      "ReLU Derivative:\n",
      " [[ 0.  0.  0.]\n",
      " [ 0.  0.  0.]\n",
      " [ 1.  1.  0.]]\n"
     ]
    }
   ],
   "source": [
    "from activations import Activation\n",
    "ReLU = Activation(relu, relu_prime)\n",
    "\n",
    "# Test and verify that the activation and derivations are working as expected\n",
    "rand = np.random.randn(3, 3)\n",
    "print('Test Matrix:\\n',rand)\n",
    "print('ReLU:\\n',ReLU.activate(rand))\n",
    "print('ReLU Derivative:\\n',ReLU.derive(rand))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, define the network object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "relu_network = network.Network(784,100,20,10, activation=ReLU)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And train. Note the *much* lower **learning_rate** parameter. An issue that crops up with using ReLU is exploding gradients. This issue can be mitigated a number of ways, such as regularization, or using leaky ReLU (not explored here). Run this multiple times (or incrase the epochs parameter) to perform additional training. \n",
    "\n",
    "The highest classification accuracy I've seen so far at this point is **97.45%**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.451603112615 980 / 10000\n",
      "0.450474032984 1010 / 10000\n",
      "0.449003687443 965 / 10000\n",
      "0.409254326695 2631 / 10000\n",
      "0.358449908965 3957 / 10000\n",
      "0.304928381028 5058 / 10000\n",
      "0.257471421608 6688 / 10000\n",
      "0.183885502053 8045 / 10000\n",
      "0.135608899491 8561 / 10000\n",
      "0.10964011665 8799 / 10000\n",
      "0.0955454650436 8985 / 10000\n",
      "0.0837568039805 9105 / 10000\n",
      "0.0744007392146 9224 / 10000\n",
      "0.068150035903 9285 / 10000\n",
      "0.0632452424595 9326 / 10000\n",
      "0.0594159802821 9373 / 10000\n",
      "0.0564065444697 9407 / 10000\n",
      "0.0552886646541 9423 / 10000\n",
      "0.0505586739784 9466 / 10000\n",
      "0.0498097105838 9467 / 10000\n",
      "0.0461729622482 9493 / 10000\n",
      "0.0447580297368 9514 / 10000\n",
      "0.0437520719267 9519 / 10000\n",
      "0.0419841134356 9557 / 10000\n",
      "0.0410312715218 9553 / 10000\n",
      "0.0386511323437 9586 / 10000\n",
      "0.0376058877161 9574 / 10000\n",
      "0.0365443368989 9592 / 10000\n",
      "0.0350452992383 9606 / 10000\n",
      "0.0343573134317 9625 / 10000\n"
     ]
    }
   ],
   "source": [
    "relu_cost = relu_network.train(mnist.train.images, mnist.train.labels, learning_rate=0.1, batch_size=200, epochs=30, test_epoch=1, test_images=mnist.test.images, test_labels=mnist.test.labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Classification accuracy increases much earlier and each epoch executes faster. Occasionally, the network can get \"stuck\" and not be able to update the cost at all; one possibility is due to \"dead\" neurons."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cross Entropy\n",
    "An alternative cost function to Mean Squared Error\n",
    "## What it does\n",
    "With MSE, when an error is very wrong, it can take a long time to \"get the ball rolling\"; that is, it can take a long time for efficient learning to occur. By taking advantage of a couple assumptions, it can reduce training time significantly. It does so by removing the sigmoid derivative from the error calculation, which is the source of the slowdown (i.e. when the network is very wrong, the sigmoid derivative becomes very small which slows training)\n",
    "## Assumptions\n",
    "There are two primary requirements/assumptions\n",
    "* We're solving a classification problem (e.g. output should be 0 or 1)\n",
    "* Our output layer has a sigmoid activation function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Code\n",
    "We won't redefine the entire training function here; however, the cost function looks like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cross_entropy_cost(y, y_pred):\n",
    "    ce = y * np.log(y_pred) + (1 - y) * np.log(1 - y_pred)\n",
    "    return - np.sum(ce) / len(ce)\n",
    "\n",
    "def cross_entropy_cost_prime(y, y_pred, hidden_a=None):\n",
    "    return y_pred - y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the relevant change to the training function looks like this:  \n",
    "\n",
    "`dL = self.cost.calculate_derivative(lbls, a[-1])  # Cross Entropy doesn't have hadamard product`  \n",
    "\n",
    "Everything else remains the same! Let's see how our network performs with the new cost function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.77992724319 2651 / 10000\n",
      "2.08224189649 6010 / 10000\n",
      "0.965064546221 8669 / 10000\n",
      "0.604118612583 9089 / 10000\n",
      "0.463529882357 9304 / 10000\n",
      "0.376623235391 9395 / 10000\n",
      "0.314187868762 9473 / 10000\n",
      "0.276056932678 9522 / 10000\n",
      "0.250019038613 9580 / 10000\n",
      "0.235056172552 9590 / 10000\n",
      "0.19703173752 9649 / 10000\n",
      "0.176004525389 9663 / 10000\n",
      "0.158284065704 9676 / 10000\n",
      "0.154031141363 9706 / 10000\n",
      "0.140709593203 9701 / 10000\n",
      "0.129701771578 9709 / 10000\n",
      "0.12325422139 9714 / 10000\n",
      "0.111839168112 9717 / 10000\n",
      "0.0996104735285 9740 / 10000\n",
      "0.0951556211666 9738 / 10000\n",
      "0.0875607648239 9735 / 10000\n",
      "0.0794790858646 9755 / 10000\n",
      "0.0832229196043 9724 / 10000\n",
      "0.0738754574135 9737 / 10000\n",
      "0.0683708828262 9748 / 10000\n",
      "0.0611208665733 9748 / 10000\n",
      "0.0564042226876 9752 / 10000\n",
      "0.054477306965 9742 / 10000\n",
      "0.0511211544207 9743 / 10000\n",
      "0.0488402806444 9750 / 10000\n"
     ]
    }
   ],
   "source": [
    "ce_network = network_ce.NetworkCE(784,100,20,10)\n",
    "ce_cost = ce_network.train(mnist.train.images, mnist.train.labels, learning_rate=1.0, batch_size=200, epochs=30, test_epoch=1, test_images=mnist.test.images, test_labels=mnist.test.labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results\n",
    "With everything else held constant--network shape, activation functions, learning rate--it's obvious the benefit that cross entropy provides to the speed at which our network learns. \n",
    "\n",
    "*Note*: Really, just the shape of this plot matters. Notice how long it takes for MSE to begin significantly decreasing as opposed to Cross Entropy. Even though MSE cost starts at *only* 0.44, it's incomparable to Cross Entropy--in fact, at cost = 0.45, Cross Entropy was already correctly classifying 93% of images, whereas MSE hadn't figured anything out yet! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f7dbf5434a8>]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEICAYAAACzliQjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xuc3HV97/HXe2YvSTYJkOxySbJh\nFwiXAAq4xBsKRVQuBWypXDytaNtD9Yi1RU9BaxXR1uKx2HOU2tJztKjVCF5qKCjSKlpELhuuhhAI\nJJAEAhtyv212dz7nj99vksmyl0l2k99c3s/HYx4zv8v85vOd3+57vvP9/WZGEYGZmdWHXNYFmJnZ\n/uPQNzOrIw59M7M64tA3M6sjDn0zszri0DczqyMOfbNxJOl9ku7Jug6z4Tj0DUnLJe2Q1Dpo/sOS\nQlJHBjV9QtIySZslrZT03f1dw3iT1JE+n5sHXS7JoJajJd0qaY2kDZIek3SVpPwYtnmtpG+NZ502\n/hz6VrQMuKw4IelEYFIWhUi6HPgD4KyImAx0Af+ZQR0N+2jTB0bE5JLLkC9oQwXwnobyUG2QdCRw\nP7ACODEiDgDeTfI8T9mT7Vv1cehb0TeB95ZMXw58o3QFSc2SvijpeUkvSfpHSRPTZQdJ+ndJPZLW\npbdnldz3bkmflfQrSZsk/XTwO4sSpwJ3RsQzABGxOiJuKtlWp6RfpNu5S9JXij1MSWdIWjmo7uWS\nzkpvz5P0a0nrJb2Y3repZN2Q9CFJTwNPp/OOTR9nraQlki4uWX+6pAWSNkp6ADiy7Gd8EEn/Iumr\nku6QtAX4rWHmHSDpG+lz/ZykT0rKpdt4X/ocf0nSK8C1QzzUZ4B7I+KqiHgxfY6XRMR7ImJ9up0L\nJC1Kn6e7JR1XUufVklalz/8SSW+TdDbwCeCS9N3Lo3v7PNg+FhG+1PkFWA6cBSwBjgPywErgcCCA\njnS9LwELgGkkPcLbgM+ny6YDF5G8O5gC3Ar8W8lj3A08AxwNTEyn/3aYen4fWAv8T5LeZ37Q8l8D\nNwDNwFuBTcC30mVnACuHal96+3XAG4AGoANYDPxZyboB3JW2cSLQQtIjfn96n5OBNcDcdP35wC3p\neicAq4B7hmlXR7r9hmGW/wuwAXgzSYdswjDzvgH8KH2eO4CngD9Kt/E+oB/4cFrvxCEeZzXw/hH+\nHo4GtgBvBxqBvwCWAk3AMenzMaOkTUemt68t7gdfKvfinr6VKvb2304ShquKCyQJuAL484hYGxGb\ngL8BLgWIiFci4vsRsTVd9tfA6YO2//WIeCoitpEE5UlDFRER3yIJrXcCvwBelnR1WsdskncCfxUR\nvRHxS5IXn7JExMKIuC8i+iNiOfBPQ9T5+bSN24DfBpZHxNfT+zwMfB94dzrUchHwqYjYEhG/AW4u\no4w1aQ+6eDmuZNmPIuJXEVGIiO2D5wF9JM/5xyNiU9qGvyMZDit6ISK+nNa7bYjHnw68OEJ9lwC3\nR8RdEdEHfJHkBfBNwADJi+1cSY0RsTzSd2RWHfbVmKVVp28CvwQ6GTS0A7SR9OIXJvkPgEjeFSBp\nEsk7gbOBg9LlUyTlI2IgnV5dsr2twOThComIfwX+VVIj8K709iMkvd51EbGlZPXngPZyGijpaJJ3\nCV1pexqAhYNWW1Fy+3Dg9ZLWl8xrIHmu2tLbpes/V0YZrRHRP8yyFaPMayXpfZc+znPAzFG2UeoV\n4LARls8o3X5EFCStAGZGxN2S/oykV3+8pDuBqyLihVEe0yqEe/q2U0Q8R3JA91zgB4MWrwG2AcdH\nxIHp5YBIDrQCfJTkrf/rI2IqybALJC8MY6mpLyJuBR4jGT55EThIUkvJarNLbm+h5AB02htvK1n+\nVeBJYE5a5yeGqLH0q2dXAL8oaXPxIOwHgR6SoZTSF5zSWvbGUF97WzpvDUlv//BBj7lqmPWH8h8k\n71CG80Lp9tN3ee3Fx4iIb0fEaewa/ru+zMe1CuDQt8H+CDhzUE+adGjhn4EvSToYQNJMSe9MV5lC\n8qKwXtI04NN7W0B6MPI8SVMk5SSdAxwP3J++MHUDn5HUJOk04PySuz8FTEjv3wh8kmQ4omgKsBHY\nLOlY4IOjlPPvwNGS/kBSY3o5VdJx6TuYHwDXSpokaS7JAfB9Jn3MW4C/Tp+fw4GrgD05VfLTwJsk\n/S9JhwJIOkrStyQdmG7/vPQAbSPJC3ovcK+kYySdKakZ2E6yzwvpdl8COooHla0yeefYbiLimYjo\nHmbx1SQH9O6TtJGkx3hMuuzvScZ91wD3AT8ZQxkbSXrgzwPrgS8AH4yI4oee3gO8nuRg76cpGYqK\niA3A/wD+L0nPdAvJQemij6X330TyIjbi+f/p8Yl3kIyjv0AyRHU9u15IriQZplpNctD162W0b712\nP0//qjLuU+rDJO16FrgH+DbwtXLvnI7Bv5HkIOwiSRtIjlN0A5siYgnJwfQvk+zP84HzI2IHSbv/\nNp2/GjgY+Hi66VvT61ckPbSHbbL9RBF+R2bVTdK1wFER8ftZ12JW6dzTNzOrIw59M7M64uEdM7M6\n4p6+mVkdqbgPZ7W2tkZHR0fWZZiZVZWFCxeuiYi20daruNDv6Oigu3u4MwbNzGwoksr5NLiHd8zM\n6olD38ysjjj0zczqiEPfzKyOOPTNzOqIQ9/MrI449M3M6kjFnae/t7b09vPk6k3kcyIvJdc5kc9B\nPpdL5uWTZbkcCFGIYKBQcomgkF4PFIJCAQoR5CQkyOdETsk2cyre3rWsuM2g+NvDJBeCQqTzSObl\ntGsbpdvOCXIlt1Xmb5AEu9rRXxj6djJd2FlLIZL2FUpq3W2aIJ/L0ZBTcsmLhlyOfE405ovXyXMQ\nATsGCuzoTy596e3egd2n+wYKSMl+aMjveg5z2rW/ckoepxBB30B634GgL73/zul0m/2FYEJjjomN\n+eTSNPx1PicIdu6HoNjWXfusVLIvkpoESNp936X1lv7N5ZSsZ1aJaib0l768mYu+em/WZZgB7Op0\n7NYBEcXvuiq+6MCuzsDOBYAEDekLa0N6313X6fz8rk5DUfFm6YtO8VY+J5oacjQ35GhqyNGUz9Hc\nkE9ul8xrasjtfAEsdgx27yQUOw1BU0OOluYGpjQ3MHlCAy1NyfWU5sZkujnPlOZGJjTm/EJYIWom\n9DvbWrj5D+cxUCgwUOBVvff+wq7rgbRbm0v/kXK79dIGXyf/nAPpH/5AofR2+o9QiJ3/DEkvL/mn\nK+0ZKu21l/7dFyJ5NzFQ8k81UHj1P1i5/yoN+d2DJpnO7ZrO7f7OpBgYxV5srqSXmksLHSgE/QMF\nBgpBXyEYKBToG0ja3pfO7y8kSdVcEhpNDTka86Xhkkw3NuR2Pl+l+6j4zmqgZH4+R3Kf9NKUz9HY\noN2n0zb39hfY3jfA1h0DbOsbYNsQ11t3DDAQke6XXfujdJqd0yoJvtgt/ILiPt/1zmggfVfYP1Bs\nT/HvsLBrWaGw299A8XGKds3f9S60P33Oi+/Y+gvBwMDu8wcrfbdSfDmJgP5CsGl7P6/0F3a+K+vt\nH9j57mzHQLJvh1J8nkrfnUqwoz955zia5oYcr20/kDd0TmNe53ROOfxAJjXVTPxUlYr7ls2urq7w\n1zCYZaNQCPrSF6edw1ojDFdFBNv7Cmzq7WNL7wCbt/fvut3bx+bt/WzuHeDlTdvpXr6ORS9soBDQ\nkBMnzjqAeZ3TeEPndF7XcRBTJzTu59bWFkkLI6JrtPX8UmtmO+VyojmXL3t9Sckxk6Z88uvDo9i0\nvY+Fz63j/mVreWDZWr52zzL+6RfPkhPMnTGVeR3Tec/rZ3PUwZPH0AobiXv6ZpaZbTsGePj5XS8C\nDz2/joOnNvMfV51Oc0P5Lz7mnr6ZVYGJTXnedFQrbzqqFYBfPtXDe7/2AN+49zn++1uPyLi62uTz\n9M2sYrz16DZOP7qNL//sadZt2ZF1OTXJoW9mFeUT5x7H5t5+vvyzpVmXUpMc+mZWUY45dAqXnNrO\nN+9bzvI1W7Iup+Y49M2s4vz524+mMZ/j+p88mXUpNcehb2YV5+ApE/jA6Ufy49+s5sHla7Mup6Y4\n9M2sIv3xWzo5ZGozn7t9MZV2ank1c+ibWUWa1NTAR99xDI+uWM9tj72YdTk1w6FvZhXrolNmcdxh\nU7n+x0+yvW8g63JqgkPfzCpWPif+8tzjWLV+GzffuzzrcmqCQ9/MKtppc1r5rWPa+MrPl7LWH9ga\nM4e+mVW8j597HFt6+/k///l01qVUPYe+mVW8ow+ZwqXzZvOt+57j2Z7NWZdT1Rz6ZlYV/vyso2lu\n8Ae2xqqs0Jd0tqQlkpZKumaE9S6SFJK6Bs2fLWmzpI+NtWAzq09tU5r54BlHcueil7j/2VeyLqdq\njRr6kvLAjcA5wFzgMklzh1hvCvAR4P4hNnMD8OOxlWpm9e6PTjuCQ6dO4G/uWEyhnN9ptFcpp6c/\nD1gaEc9GxA5gPnDhEOt9Frge2F46U9K7gGXAojHWamZ1bmJTno+98xgeXbmB2x57IetyqlI5oT8T\nWFEyvTKdt5OkU4D2iLh90PzJwNXAZ0Z6AElXSOqW1N3T01NW4WZWn3735JnMPWwqX/jJEn9gay+M\n+UCupBzJ8M1Hh1h8LfCliBjxcHtE3BQRXRHR1dbWNtaSzKyG5XLiw2cexar123h81Yasy6k65fxc\n4iqgvWR6VjqvaApwAnC3JIBDgQWSLgBeD/yepC8ABwIFSdsj4ivjUbyZ1ae5M6YCsGzNFk7tmJZx\nNdWlnNB/EJgjqZMk7C8F3lNcGBEbgNbitKS7gY9FRDfwlpL51wKbHfhmNlYzD5xIQ04s84+s7LFR\nh3cioh+4ErgTWAzcEhGLJF2X9ubNzParhnyO2dMn+Ze19kI5PX0i4g7gjkHzPjXMumcMM//aPazN\nzGxYndNb3NPfC/5ErplVpc7WFpa/ssXn6+8hh76ZVaWO1ha29xVYvXH76CvbTg59M6tKR7S2AHiI\nZw859M2sKnW2OfT3hkPfzKrSIVMmMKEx59DfQw59M6tKuZzomN7i0zb3kEPfzKpWZ6tP29xTDn0z\nq1qdrS08v3Yr/QOFrEupGg59M6taHa0t9BeCleu2ZV1K1XDom1nV8mmbe86hb2ZVq9Ohv8cc+mZW\ntaa1NDFlQoNDfw849M2sakniiPQ7eKw8Dn0zq2odrS082+PQL5dD38yqWmdrCy9s2Obfyy2TQ9/M\nqlpnawsR8PzarVmXUhUc+mZW1Ypn8HiIpzwOfTOrah1p6Ptgbnkc+mZW1aZOaKR1chPL3NMvi0Pf\nzKpeZ2sLy9zTL4tD38yqXod/JL1sDn0zq3qdbS30bOpl0/a+rEupeA59M6t6ndOTg7nPveLTNkfj\n0Dezqlf8vdxnPcQzKoe+mVW9jrSn759OHJ1D38yq3oTGPDMOmOCDuWVw6JtZTehsa/HwThkc+mZW\nEzqmt7CsZzMRkXUpFc2hb2Y1obO1hY3b+1m31adtjsShb2Y14Yg2/3RiORz6ZlYTimfwOPRH5tA3\ns5rQPm0S+Zx82uYoHPpmVhMa8znaD5ronv4oHPpmVjM6W33a5mjKCn1JZ0taImmppGtGWO8iSSGp\nK51+u6SFkh5Pr88cr8LNzAbraG1h+ZotPm1zBKOGvqQ8cCNwDjAXuEzS3CHWmwJ8BLi/ZPYa4PyI\nOBG4HPjmeBRtZjaUI1pb2NY3wEsbe7MupWKV09OfByyNiGcjYgcwH7hwiPU+C1wPbC/OiIiHI+KF\ndHIRMFFS8xhrNjMbUmfrZMBn8IyknNCfCawomV6ZzttJ0ilAe0TcPsJ2LgIeigi/BJvZPtHROglw\n6I+kYawbkJQDbgDeN8I6x5O8C3jHMMuvAK4AmD179lhLMrM6NeOAiTQ15Pwj6SMop6e/CmgvmZ6V\nziuaApwA3C1pOfAGYEHJwdxZwA+B90bEM0M9QETcFBFdEdHV1ta2560wMwNyOdExfRLP+kfSh1VO\n6D8IzJHUKakJuBRYUFwYERsiojUiOiKiA7gPuCAiuiUdCNwOXBMRv9oH9ZuZ7aaztYVlazZnXUbF\nGjX0I6IfuBK4E1gM3BIRiyRdJ+mCUe5+JXAU8ClJj6SXg8dctZnZMDpaW3h+7VYGCj5tcyhljelH\nxB3AHYPmfWqYdc8ouf054HNjqM/MbI8c0dpC30Cwat02Zk+flHU5FcefyDWzmrLztE0fzB2SQ9/M\nasrO0zZ7PK4/FIe+mdWUtsnNTG5u8Ln6w3Dom1lNkURH6ySWvbI161IqkkPfzGpOZ+tkn7Y5DIe+\nmdWczumTWLVuG739A1mXUnEc+mZWczrbWigErFjrIZ7BHPpmVnN2fdumQ38wh76Z1ZzOnT+S7nH9\nwRz6ZlZzDpjUyLSWJp+2OQSHvpnVpI7pkxz6Q3Dom1lNSk7bdOgP5tA3s5p0RFsLL23sZUtvf9al\nVBSHvpnVpI70YK5/RWt3Dn0zq0mdrWno+7TN3Tj0zawm7fqRdJ+2Wcqhb2Y1aVJTA4dOncCzPpi7\nG4e+mdWsjtZJLHfo78ahb2Y1y6dtvppD38xq1hGtLazb2sf6rTuyLqViOPTNrGZ1tBa/g8e9/SKH\nvpnVrE6H/qs49M2sZs2eNomc8MHcEg59M6tZTQ05Zh00icWrN2VdSsVw6JtZTTvz2IP5xZIe1m7x\nwVxw6JtZjbt0Xjs7Bgr84KGVWZdSERz6ZlbTjj10Kie1H8h3HnieiMi6nMw59M2s5l02r51nerbQ\n/dy6rEvJnEPfzGreb79mBi1Neb7zwPNZl5I5h76Z1byW5gYuOGkmdzz+Ihu29WVdTqYc+mZWF94z\nbzbb+wr86JFVWZeSKYe+mdWFE2cdwPEzpvKdB1bU9QFdh76Z1Y1L581m8YsbeWzlhqxLyYxD38zq\nxoUnzWBiY575D9bvAV2HvpnVjakTGjnvNYex4JEX2NLbn3U5mSgr9CWdLWmJpKWSrhlhvYskhaSu\nknkfT++3RNI7x6NoM7O9ddm8drbsGOC2R1/IupRMjBr6kvLAjcA5wFzgMklzh1hvCvAR4P6SeXOB\nS4HjgbOBf0i3Z2aWiVNmH8ScgyfznQdXZF1KJsrp6c8DlkbEsxGxA5gPXDjEep8Frge2l8y7EJgf\nEb0RsQxYmm7PzCwTkrh03mweXbGexS9uzLqc/a6c0J8JlL4krkzn7STpFKA9Im7f0/um979CUrek\n7p6enrIKNzPbW7978kya8jnm1+EndMd8IFdSDrgB+OjebiMiboqIrojoamtrG2tJZmYjOqilibNP\nOJQfPryK7X0DWZezX5UT+quA9pLpWem8oinACcDdkpYDbwAWpAdzR7uvmVkmLp3Xzsbt/dzx+ItZ\nl7JflRP6DwJzJHVKaiI5MLuguDAiNkREa0R0REQHcB9wQUR0p+tdKqlZUicwB3hg3FthZraH3njE\ndDqmT2L+A/V1QHfU0I+IfuBK4E5gMXBLRCySdJ2kC0a57yLgFuAJ4CfAhyKivt5LmVlFksQlp87m\ngeVrWfry5qzL2W9Uad9B0dXVFd3d3VmXYWZ1oGdTL2/8/H/y/jd38JfnvepM9KoiaWFEdI22nj+R\na2Z1q21KM2cddwjff2gVvf31MQjh0DezunbpvHbWbtnBXU+8lHUp+4VD38zq2lvmtDHzwIl1c0DX\noW9mdS2fExd3tXPP0jU8/8rWrMvZ5xz6Zlb3Lj51FjnBd7tr/xO6Dn0zq3uHHTCRM445mFu7V9I/\nUMi6nH3KoW9mBlxyajsvb+rlnqVrsi5ln3Lom5kBpx/dxuTmhpr/WgaHvpkZMKExz9vnHsKdi16i\nr4aHeBz6Zmapc088jA3b+vhVDQ/xOPTNzFJvmdNa80M8Dn0zs1RxiOenT9TuEI9D38ysxLknHsb6\nrX3c+8wrWZeyTzj0zcxKFId4bn/shaxL2Scc+mZmJSY05jnruINrdojHoW9mNsh5r5lRs0M8Dn0z\ns0F2nsXzWO2dxePQNzMbpDjEc+cTq2tuiMehb2Y2hFod4nHom5kNoVaHeBz6ZmZDqNUhHoe+mdkw\navGDWg59M7NhvLX4dcs1NMTj0DczG0YtDvE49M3MRlBrQzwOfTOzEdTaEI9D38xsBLU2xOPQNzMb\nRXGI59c1MMTj0DczG0VxiOf2GhjiceibmY2iloZ4HPpmZmWolSEeh76ZWRl2nsVT5T+a7tA3MyvD\nhMY8bzvuYH6yqLqHeBz6ZmZlOq8GhnjKCn1JZ0taImmppGuGWP4BSY9LekTSPZLmpvMbJd2cLlss\n6ePj3QAzs/2lFoZ4Rg19SXngRuAcYC5wWTHUS3w7Ik6MiJOALwA3pPPfDTRHxInA64A/kdQxTrWb\nme1XxSGeO6t4iKecnv48YGlEPBsRO4D5wIWlK0TExpLJFiCKi4AWSQ3ARGAHULqumVlVOe/Ew1hX\nxUM85YT+TGBFyfTKdN5uJH1I0jMkPf0/TWd/D9gCvAg8D3wxItaOqWIzswwVh3j+/bEXsi5lr4zb\ngdyIuDEijgSuBj6Zzp4HDAAzgE7go5KOGHxfSVdI6pbU3dPTM14lmZmNuwmNec4+4VDueHw1W3f0\nZ13OHisn9FcB7SXTs9J5w5kPvCu9/R7gJxHRFxEvA78CugbfISJuioiuiOhqa2srr3Izs4xccmo7\nm3v7q/JrGcoJ/QeBOZI6JTUBlwILSleQNKdk8jzg6fT288CZ6TotwBuAJ8datJlZlroOP4gj2lq4\npXvF6CtXmFFDPyL6gSuBO4HFwC0RsUjSdZIuSFe7UtIiSY8AVwGXp/NvBCZLWkTy4vH1iHhs3Fth\nZrYfSeLirnYeXL6OZ3o2Z13OHlFEjL7WftTV1RXd3d1Zl2FmNqKXN23njZ//GX/8lk4+fs5xWZeD\npIUR8arh88H8iVwzs71w8JQJnHnswXx/4aqqOmffoW9mtpcu6WpnzeZefv7ky1mXUjaHvpnZXjrj\nmDbapjRX1QFdh76Z2V5qyOe46JRZ/HxJDy9v3J51OWVx6JuZjcHFXbMYKATfe2hl1qWUxaFvZjYG\nR7RNZl7HNG7tXkmlnQ05FIe+mdkYXXxqO8vWbOGBZZX/1WIOfTOzMTr3xEOZ3NzAd6vggK5D38xs\njCY1NXD+a2dwx+MvsnF7X9bljMihb2Y2Di45tZ3tfQVue7Syv3LZoW9mNg5eO+sAjjlkCrc8WNlD\nPA59M7NxIImLT23n0ZUbeHJ15f5AoEPfzGyc/M7JM2nMi+9WcG/foW9mNk6mtTTxjrmH8sOHV9Hb\nP5B1OUNy6JuZjaOLT21n/dY+7nripaxLGZJD38xsHJ12VCszDpjALd2V+bUMDn0zs3GUz4nf62rn\nv57uYdX6bVmX8yoOfTOzcfbu180iAr5Xgb19h76Z2ThrnzaJNx81nVsXrqBQqKwvYXPom5ntAxd3\ntbNy3TbufeaVrEvZjUPfzGwfeOfxh3LAxMaK+xI2h76Z2T4woTHPu06awZ2/Wc2azb1Zl7OTQ9/M\nbB9575s6KETwdz99KutSdnLom5ntI0e2TebyN3Uw/8Hn+c2qDVmXAzj0zcz2qT992xymTWri2gWL\nKuLnFB36Zmb70AETG/mLs4+h+7l1LKiA79p36JuZ7WPvfl07r5l1AJ+/40m29PZnWotD38xsH8vl\nxKfPP57VG7fzD3cvzbaWTB/dzKxOvO7wg/jdk2fyz79cxnOvbMmsDoe+mdl+cvU5x9KQF5+7fXFm\nNTj0zcz2k0OmTuDDZ87hride4hdP9WRSg0PfzGw/+sPTOuiYPonrbltE30Bhvz++Q9/MbD9qbsjz\nV789l2d6tnDzvcv3++M79M3M9rMzjz2Y049u43//x9P0bNq/38vj0Dcz288k8anz57Ktb4Av3rlk\nvz52WaEv6WxJSyQtlXTNEMs/IOlxSY9IukfS3JJlr5H0a0mL0nUmjGcDzMyq0ZFtk/nD0zq5ZeEK\nHl2xfr897qihLykP3AicA8wFLisN9dS3I+LEiDgJ+AJwQ3rfBuBbwAci4njgDKBv/Mo3M6teHz7z\nKKa3NHPtbYv22y9sldPTnwcsjYhnI2IHMB+4sHSFiNhYMtkCFKt/B/BYRDyarvdKRAyMvWwzs+o3\nZUIjV599DA8/v55/e2TVfnnMckJ/JlD60y8r03m7kfQhSc+Q9PT/NJ19NBCS7pT0kKS/GOoBJF0h\nqVtSd09PNueumpll4aJTZvHa9gP5/I+fZPN++F6ecTuQGxE3RsSRwNXAJ9PZDcBpwH9Lr39H0tuG\nuO9NEdEVEV1tbW3jVZKZWcXL5cS158+lZ1MvX/nZvv9ennJCfxXQXjI9K503nPnAu9LbK4FfRsSa\niNgK3AGcsjeFmpnVqpNnH8T739zBjAP3/Xku5YT+g8AcSZ2SmoBLgQWlK0iaUzJ5HvB0evtO4ERJ\nk9KDuqcDT4y9bDOz2vLp84/nvW/s2OeP0zDaChHRL+lKkgDPA1+LiEWSrgO6I2IBcKWks0jOzFkH\nXJ7ed52kG0heOAK4IyJu30dtMTOzUagSfr6rVFdXV3R3d2ddhplZVZG0MCK6RlvPn8g1M6sjDn0z\nszri0DczqyMOfTOzOuLQNzOrIw59M7M6UnGnbErqAZ4bwyZagTXjVE4lcHsqX621qdbaA7XXpqHa\nc3hEjPo9NhUX+mMlqbucc1WrhdtT+WqtTbXWHqi9No2lPR7eMTOrIw59M7M6Uouhf1PWBYwzt6fy\n1Vqbaq09UHtt2uv21NyYvpmZDa8We/pmZjYMh76ZWR2pmdCXdLakJZKWSrom63rGg6Tlkh6X9Iik\nqvu+aUlfk/SypN+UzJsm6S5JT6fXB2VZ454apk3XSlqV7qdHJJ2bZY17QlK7pJ9LekLSIkkfSedX\n5X4aoT3VvI8mSHpA0qNpmz6Tzu+UdH+aed9Nf+Rq9O3Vwpi+pDzwFPB2kp9ofBC4LCKq+le6JC0H\nuiKiKj9UIumtwGbgGxFxQjrvC8DaiPjb9MX5oIi4Oss698QwbboW2BwRX8yytr0h6TDgsIh4SNIU\nYCHJz52+jyrcTyO052Kqdx8JaImIzZIagXuAjwBXAT+IiPmS/hF4NCK+Otr2aqWnPw9YGhHPRsQO\nkt/pvTDjmupeRPwSWDto9oURTwZFAAACOElEQVTAzentm9n1e8pVYZg2Va2IeDEiHkpvbwIWAzOp\n0v00QnuqViQ2p5ON6SWAM4HvpfPL3ke1EvozgRUl0yup8h2dCuCnkhZKuiLrYsbJIRHxYnp7NXBI\nlsWMoyslPZYO/1TFUMhgkjqAk4H7qYH9NKg9UMX7SFJe0iPAy8BdwDPA+ojoT1cpO/NqJfRr1WkR\ncQpwDvChdGihZkQytlj944vwVeBI4CTgReDvsi1nz0maDHwf+LOI2Fi6rBr30xDtqep9FBEDEXES\nMItkZOPYvd1WrYT+KqC9ZHpWOq+qRcSq9Ppl4IckO7vavZSOuxbHX1/OuJ4xi4iX0n/KAvDPVNl+\nSseJvw/8a0T8IJ1dtftpqPZU+z4qioj1wM+BNwIHSmpIF5WdebUS+g8Cc9Kj2U3ApcCCjGsaE0kt\n6YEoJLUA7wB+M/K9qsIC4PL09uXAjzKsZVwUwzH1O1TRfkoPEv4/YHFE3FCyqCr303DtqfJ91Cbp\nwPT2RJITVhaThP/vpauVvY9q4uwdgPQUrL8H8sDXIuKvMy5pTCQdQdK7B2gAvl1tbZL0HeAMkq+B\nfQn4NPBvwC3AbJKv0L44IqrmwOgwbTqDZNgggOXAn5SMh1c0SacB/wU8DhTS2Z8gGQevuv00Qnsu\no3r30WtIDtTmSTrqt0TEdWlGzAemAQ8Dvx8RvaNur1ZC38zMRlcrwztmZlYGh76ZWR1x6JuZ1RGH\nvplZHXHom5nVEYe+mVkdceibmdWR/w9IbubG+QcC1gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f7dbf543358>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.title('Mean Squared Error Cost')\n",
    "plt.plot(mse_cost)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f7dbd478828>]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xl4XHd97/H3dxYto12WvEqykzgk\ndYKdxTiBpC23a+CmhFK2pCxpS9P2wi08lNv29rYQuF1poZSyhi1hCYGyJIEnXEpv6Q0N2WTHzuYE\nnMSOvEvyon2Zme/94xwpiqJlJI80mnM+r+eZZ86c89PM98zYnznz+53F3B0REYmWRKkLEBGR4lO4\ni4hEkMJdRCSCFO4iIhGkcBcRiSCFu4hIBCncRUQiSOEui2Jm15lZp5kNmNkRM/uemV1ZwnpuNrOx\nsJ6J254C//ZGM/vyUte4EEvx/prZfjP7pWLVKCubwl0WzMzeDXwE+GtgDdABfAK4Zpb2qWUq7YPu\nXjvltq0YT2qBZfu/stD3V2RG7q6bbgXfgAZgAHjdHG1uBL4BfBnoA94GVBIE1uHw9hGgMmzfAnwX\nOAWcAH4EJMJlfwIcAvqBJ4FfnOU1bwb+cpZlmwAH3go8C/QA/ytcdhUwBoyH67UnnP8fwF8B9wDD\nwGZgPXBnWOM+4HdnWOevhbXuAraFy/4H8M1pNX0U+KdFvr8Lfi+BLwH5cF0GgD8u9b8l3Zb2VvIC\ndCuvWxiGWSA1R5sbw7B8dRgs1cAHgPuA1UAr8GPgf4ft/wb4FJAObz8LGHAe0AWsD9ttAs6Z5TUL\nCffPhLVsA0aBn5lS75en/c1/hF8EFwCpsK67Cbagq4CLgG7gF6at82vDtu8Bngmn1wGDQGPYNgUc\nBy5d5Pu74PcyXLYf+KVS/xvSbXlu6paRhVoF9Lh7dp5297r77e6ed/dh4DeBD7j7cXfvBt4PvDls\nO04QgBvdfdzdf+TuDuQItlK3mFna3fe7+1NzvOZ7zOzUlNst05a/392H3X0PsIcg5Odys7s/Fq7r\nWuAK4E/cfcTddwOfBd4ypf1Od/+Gu48DHyb4Erjc3Y8QfDG8Lmx3FcF7uHOG1yzk/V3Meykxo3CX\nheoFWgroR++a9ng9cGDK4wPhPIC/J+jm+Fcze9rM/hTA3fcB7yLYKj5uZreZ2Xpm9w/u3jjl9tZp\ny49OmR4CahewDuuBE+7eP20dNszU3t3zwMEp63gL8KZw+k0E3SQzKeT9XfB7KfGjcJeFupegS+PV\n87SbvrV4GNg45XFHOA9373f3P3L3s4FXAe82s18Ml93q7leGf+vA3535Ksxb60zzDwPNZlY3ZV4H\nwXjAhPaJiXAAti38O4Dbga1mdiFwNfCVWV6zkPd3Ue8ls6+nRJDCXRbE3U8D7wU+bmavNrOMmaXN\n7BVm9sE5/vSrwJ+bWauZtYTP8WUAM7vazDabmQGnCbpj8mZ2npn9gplVAiMEg4H5JVitY8CmufaI\ncfcugr7tvzGzKjPbCvzOxDqELjWz14Rb3e8iCOn7wr8fIRhwvRV4wN2fneV1Cnl/F/xeTlnPsxf4\n3kiZUrjLgrn7h4B3A39OMKjYBbyDYOt0Nn8JdAIPA48Q7E3yl+Gyc4F/I9iL417gE+7+Q4L+9r8l\n2LvlKMEA4v+c4zX+eNp+7j0FrtK/hPe9ZrZrjnbXEgzOHga+DbzP3f9tyvI7gDcAJwn6wF8T9r9P\nuAV4MbN3yQAFvb+LeS8hGGz983A84j1z1SDlzzTWInLmzOxGYLO7v2mONh3AE8Bad+9brtoknrTl\nLrIMwi6fdwO3KdhlOSzXkYMisWVmNQT93QcIdoMUWXLqlhERiSB1y4iIRFDJumVaWlp806ZNpXp5\nEZGytHPnzh53b52vXcnCfdOmTXR2dpbq5UVEypKZHZi/lbplREQiSeEuIhJBCncRkQhSuIuIRJDC\nXUQkghTuIiIRpHAXEYmgsgv3J4/289d37WVobL6rvImIxFfZhfvBk0PcdPfTPHpIJ9YTEZlN2YX7\n1rZGAB4+eKrElYiIrFxlF+6tdZWsb6hiz8HTpS5FRGTFKrtwh2DrXVvuIiKzK8tw39beyIHeIU4O\njpW6FBGRFak8w72tAYCHD6lrRkRkJmUZ7hdOhHuXumZERGZSluFeX5Xm7NYaDaqKiMyiLMMdYFtb\nI3sOnkLXgBUReaEyDvcGuvtHOdo3UupSRERWnLIN963twcFMe7rUNSMiMl3ZhvuWdfWkEqb93UVE\nZlC24V6VTnLe2joe1qCqiMgLlG24w3NHqubzGlQVEZmqrMP9ovYG+kay7O8dLHUpIiIrSlmH+3Nn\niFTXjIjIVGUd7ueurqUqnWCPBlVFRJ5n3nA3s3Yz+6GZPW5mj5nZO2do83IzO21mu8Pbe5em3OdL\nJRNcuL5BW+4iItOkCmiTBf7I3XeZWR2w08x+4O6PT2v3I3e/uvglzm1rWyO3PnCA8VyedLKsf4iI\niBTNvGno7kfcfVc43Q/sBTYsdWGF2tbewMh4np8c6y91KSIiK8aCNnXNbBNwMXD/DItfamZ7zOx7\nZnZBEWoryDYNqoqIvEDB4W5mtcA3gXe5+/SrU+8CNrr7NuCfgdtneY4bzKzTzDq7u7sXW/PzbFyV\noaE6rSNVRUSmKCjczSxNEOxfcfdvTV/u7n3uPhBO3wWkzaxlhnY3uft2d9/e2tp6hqVP1sbWtgad\nY0ZEZIpC9pYx4HPAXnf/8Cxt1obtMLMd4fP2FrPQuWxta+DJY/2MjOeW6yVFRFa0QvaWuQJ4M/CI\nme0O5/0Z0AHg7p8CXgv8gZllgWHgjb6MJ1rf2tZILu88dvg0l25sXq6XFRFZseYNd3f/T8DmafMx\n4GPFKmqhLppy+l+Fu4hImR+hOmFNfRVr6is1qCoiEopEuMPEGSI1qCoiAhEK921tDTzdM8jp4fFS\nlyIiUnKRCfeJM0Q+oq13EZEohXsDgM4QKSJChMK9MVPBplUZDaqKiBChcAcNqoqITIhYuDdw5PQI\nx/tGSl2KiEhJRSrct00czKStdxGJuUiF+wXr60kmTP3uIhJ7kQr3TEWKc1fXastdRGIvUuEOwcU7\nHj54imU8b5mIyIoTuXDf2t7AqaFxuk4Ml7oUEZGSiVy4T1x2TwcziUicRS7cz1tbR0UqwZ4uhbuI\nxFfkwj2dTHDB+nodzCQisRa5cIega+bRw6fJ5TWoKiLxFMlw39rWwNBYjn3HB0pdiohISUQ03DWo\nKiLxFslwP7ulhrrKlAZVRSS2IhnuiYTx4rYGDaqKSGxFMtwh6Jp54mgfo9lcqUsREVl2kQ33bW0N\njOecvUf6S12KiMiyi2y4bw1P/6szRIpIHEU23Nc3VNFSW8GeLvW7i0j8RDbczYxzWms50DtY6lJE\nRJZdZMMdoL05Q9fJoVKXISKy7CId7h3NGY71jTIyrj1mRCRe5g13M2s3sx+a2eNm9piZvXOGNmZm\nHzWzfWb2sJldsjTlLkx7czUAB0/q3O4iEi+FbLlngT9y9y3A5cDbzWzLtDavAM4NbzcAnyxqlYvU\n3pQBUNeMiMTOvOHu7kfcfVc43Q/sBTZMa3YN8EUP3Ac0mtm6ole7QO3NQbgfPKFwF5F4WVCfu5lt\nAi4G7p+2aAPQNeXxQV74BYCZ3WBmnWbW2d3dvbBKF6G1tpLKVIJnFe4iEjMFh7uZ1QLfBN7l7n2L\neTF3v8ndt7v79tbW1sU8xYIkEkZbU7WupyoisVNQuJtZmiDYv+Lu35qhySGgfcrjtnBeyWl3SBGJ\no0L2ljHgc8Bed//wLM3uBN4S7jVzOXDa3Y8Usc5Fa2/K0KVuGRGJmVQBba4A3gw8Yma7w3l/BnQA\nuPungLuAVwL7gCHgt4pf6uK0N1fTN5Ll9NA4DZl0qcsREVkW84a7u/8nYPO0ceDtxSqqmDqan9sd\nsiHTUOJqRESWR6SPUAVom9jXXV0zIhIjkQ/39mYdyCQi8RP5cG+oTlNfldLukCISK5EPd4COVRkd\nyCQisRKLcG9v0r7uIhIv8Qj35gwHTw6Tz3upSxERWRbxCPemasayeboHRktdiojIsohHuId7zKjf\nXUTiIlbhrn3dRSQuYhHuGxqDKzJpd0gRiYtYhHtVOsma+krtMSMisRGLcIdgd0j1uYtIXMQm3Dua\nM7rcnojERmzCva05w5G+Ecay+VKXIiKy5GIT7u1N1bjD4VMaVBWR6ItPuGtfdxGJkdiEe4dO/Ssi\nMRKbcF9TX0U6adrXXURiITbhnkwYGxqrteUuIrEQm3CH8OyQ6nMXkRiIVbi36UAmEYmJWIV7R3OG\nk0PjDIxmS12KiMiSilW4tzdPnEBMW+8iEm3xCvcmnfpXROIhXuGuA5lEJCZiFe5NmTS1lSkOntS+\n7iISbbEKdzOjrala3TIiEnnzhruZfd7MjpvZo7Msf7mZnTaz3eHtvcUvs3jamzM6kElEIq+QLfeb\ngavmafMjd78ovH3gzMtaOu1NGbpODOPupS5FRGTJzBvu7n43cGIZalkWHc3VDI/n6BkYK3UpIiJL\nplh97i81sz1m9j0zu2C2RmZ2g5l1mllnd3d3kV56Ydp1dkgRiYFihPsuYKO7bwP+Gbh9tobufpO7\nb3f37a2trUV46YWbDHcNqopIhJ1xuLt7n7sPhNN3AWkzaznjypZIW1NwlKp2hxSRKDvjcDeztWZm\n4fSO8Dl7z/R5l0qmIkVLbQXP9mrLXUSiKzVfAzP7KvByoMXMDgLvA9IA7v4p4LXAH5hZFhgG3ugr\nfFcU7Q4pIlE3b7i7+7XzLP8Y8LGiVbQM2psyPNR1stRliIgsmVgdoTqhvbmaw6dGyObypS5FRGRJ\nxDPcmzLk8s6R0yOlLkVEZEnEMtw7tDukiERcLMNdBzKJSNTFMtzXNVSRTBhdJ7Svu4hEUyzDPZVM\nsK6hShftEJHIimW4Q9Dvrm4ZEYmq2Ib7xKl/RUSiKL7h3lxNz8Aow2O5UpciIlJ0MQ537TEjItEV\n23Bva9K+7iISXbENdx3IJCJRFttwb6mtoDqdpEvndReRCIptuJsZbU3V2nIXkUiKbbhDMKiqA5lE\nJIpiHe4dzRkOnhxmhV9bRERkwWId7m1N1QyMZjk1NF7qUkREiirW4a593UUkquId7uG+7up3F5Go\niXe4N1cD6BwzIhI5sQ73uqo0TZm0umVEJHJiHe4Q9LtrX3cRiRqFe5PCXUSiJ/bh3tZczaFTw+Ty\n2tddRKIj9uHe0ZxhPOcc6xspdSkiIkUT+3Bv16l/RSSCFO6TBzJpd0gRiY55w93MPm9mx83s0VmW\nm5l91Mz2mdnDZnZJ8ctcOusbqzDTgUwiEi2FbLnfDFw1x/JXAOeGtxuAT555WcunMpVkXX0VBxXu\nIhIh84a7u98NnJijyTXAFz1wH9BoZuuKVeByaGvO6EAmEYmUYvS5bwC6pjw+GM57ATO7wcw6zayz\nu7u7CC9dHMG+7upzF5HoWNYBVXe/yd23u/v21tbW5XzpObU3V3O0b4SR8VypSxERKYpihPshoH3K\n47ZwXtnYuCrYY+anxwZKXImISHEUI9zvBN4S7jVzOXDa3Y8U4XmXzctftJqKVIJ/2dk1f2MRkTJQ\nyK6QXwXuBc4zs4Nm9jtm9vtm9vthk7uAp4F9wGeA/7Zk1S6RppoKXnnhWr696xBDY9lSlyMicsZS\n8zVw92vnWe7A24tWUYlcd9lGbt99mO/uOcLrX9I+/x+IiKxgsT9CdcJLNjWxeXUttz7wbKlLERE5\nYwr3kJlx7Y4Odned4vHDfaUuR0TkjCjcp/iNSzZQkUpw6wMHSl2KiMgZUbhP0Zip4OoXr+P2hw4z\nOKqBVREpXwr3aa67rIOB0Szf2XO41KWIiCyawn2aSzc28aI1GlgVkfKmcJ/GzLhuRwcPHzzNo4dO\nl7ocEZFFUbjP4NcvbqMyldDWu4iULYX7DBoyaa7eup47HjrEgAZWRaQMKdxncd1lHQyO5bhztwZW\nRaT8KNxncUlHI+evrdM+7yJSlhTuszAzrrusg0cP9fHwwVOlLkdEZEEU7nN49cUbqE4n+aoGVkWk\nzCjc51BflebXtq3jjt2H6R8ZL3U5IiIFU7jP49odHQyN5bhDA6siUkYU7vO4qL2Rn1lXz633P0tw\n6noRkZVP4T6PiYHVx4/0seegjlgVkfKgcC/Aqy9aT6Yiya33a7dIESkPCvcC1FWledW29XxnzxH6\nNLAqImVA4V6g6y7rYHg8xx0PHSp1KSIi81K4F2hrWyMXbqjnKxpYFZEyoHBfgGt3dPDE0X4e6tIR\nqyKysincF+CaizZQU5Hkg//nCUbGc6UuR0RkVgr3BaitTHHjqy7g/mdO8Ns3P8jQmE4HLCIrk8J9\ngV63vZ0PvW4b9z3dy1s//4BOSyAiK5LCfRFec0kbH732Yh569hRv/twDnB5WwIvIyqJwX6Srt67n\nE795CY8dPs1vfvY+Tg6OlbokEZFJBYW7mV1lZk+a2T4z+9MZll9vZt1mtju8va34pa48v3LBWm56\ny3Z+cmyAaz9zHz0Do6UuSUQEKCDczSwJfBx4BbAFuNbMtszQ9GvuflF4+2yR61yx/st5q/nC9S9h\nf+8gb/j0vRzrGyl1SSIiBW257wD2ufvT7j4G3AZcs7RllZcrNrdwy2/t4OjpEd7w6Xs5fGq41CWJ\nSMwVEu4bgK4pjw+G86b7DTN72My+YWbtMz2Rmd1gZp1m1tnd3b2Icleuy85exZfedhm9g2O8/tP3\n0nViqNQliUiMFWtA9TvAJnffCvwAuGWmRu5+k7tvd/ftra2tRXrpleOSjiZufdvl9I9kef2n7+WZ\nnsFSlyQiMVVIuB8Cpm6Jt4XzJrl7r7tPjCZ+Fri0OOWVnxe3NfDV372csWye133qXr7/2FGdi0ZE\nll0h4f4gcK6ZnWVmFcAbgTunNjCzdVMevgrYW7wSy8+W9fV87fcuZ1VNBb/3pZ1c/4UHtRUvIstq\n3nB39yzwDuD7BKH9dXd/zMw+YGavCpv9oZk9ZmZ7gD8Erl+qgsvF5tV1fPcPr+Qvrt7CzgMn+dV/\nvJt/+P6TDI/pnDQisvSsVF0G27dv987OzpK89nI73jfC33zvCb790CE2NFbzF1dv4VcvWIOZlbo0\nESkzZrbT3bfP105HqC6D1fVV/OMbLuJrN1xOXVWK3//yTt76hQd5unug1KWJSEQp3JfRZWev4rv/\n/Uree/UWHjpwkqs+8iP+/vtP6OySIlJ0Cvdllkom+O0rz+L/vufnuXrrOj7+w6f4pQ/9P76x8yCj\nWfXHi0hxqM+9xB545gTvu/Mx9h7po6W2gusu28ibLutgdX1VqUsTkRWo0D53hfsK4O7cs6+XL9zz\nDP/+5HGSZvzXreu4/mWbuLijqdTlicgKUmi4p5ajGJmbmXHluS1ceW4L+3sG+eK9B/iXzi7u2H2Y\ni9ob+a0rNvGKC9dRkVIvmogURlvuK9TAaJZv7TrIzffs5+meQVrrKnnTZRu57rIOWusqS12eiJSI\numUiIp937v5pNzf/eD//8WQ3yYSxta2BKze38LJzWrhkYyOVqWSpyxSRZaJwj6Cnuwf41q5D3PNU\nD3u6TpF3qEoneMmmZq7Y3MIV57SwZX09yYQOjhKJKoV7xPWNjHP/0ye4Z18P9+zr4afHgwOiGjNp\nXnr2Kl62uYVLO5rYvLpWffUiEaIB1Yirr0rzy1vW8Mtb1gDBKQ5+/FTvZNh/79GjAKQSxjmttZy/\nro7z19Zz/ro6fmZtPWvqK3X6A5EI05Z7BLk7B3qHeOTQaZ442scTR/p54mg/h6ZcIaoxk+b8tUHg\nv2hNHesaq1hTV8XahiqaMmkFv8gKpS33GDMzNrXUsKmlhl/btn5y/unhcZ482s8TR/vYeyS4/3pn\nF0PTzlRZkUywur6SNfVVrJm8D6bPX1vPeWvqSKhfX2RFU7jHSEN1mh1nNbPjrObJefm8c/j0MMf6\nRjnWN8KxvhGO9o1wPHz8xNF+7v5JDwOjz53/ZlVNBS/b3MIV56ziis0ttDdnSrE6IjIHhXvMJRJG\nW1OGtqa5A3pgNMvR08PsevYUP97Xwz1P9fKdPYcBaG+u5opzWrhicwsvO2cVq2q1H75IqanPXRbF\n3dl3fCAYwH2ql/ue7qV/JNi6P39tHZefvYrNq2s5K+weWldfpa4ckSJQn7ssKTPj3DV1nLumjuuv\nOItsLs+jh/u4Z18PP36qh9sefJaR8fxk+8pUgo2rMmxaVcNZLTWToX9WSw2r67TnjkixactdlkQ+\n7xztG2F/zyDP9A4G9z1DPNMzQNeJYcZyzw/+DY3VbGiqpq2pesp0hg2N1aypr9KBWSIhbblLSSUS\nxvrGatY3VvOyzS3PW5bLO4dPDfNMzyD7ewfpOjHEoVPDHDo5zL8e7qN3cOx57VMJY21DFesaqqhK\nJ6lMJahIJahMTZ1+bl5FKkFDdZrW2kpa6ippraukpbZCp2mQWFG4y7JLJoz25gztzRl+jtYXLB8e\nywVhHwb+wZNB+B/rG2FgNEvvQJ6xXJ7RbI6xbJ7RbH7yPpef/ZdofVUqCPuJ0K8Ngr8xk6Y5U0Fj\npoLmmgqaMmkaMxU6slfKmsJdVpzqiiSbV9eyeXXtgv82l3dGszlODY3T3T9Kz8DotPsxuvtHefxw\nHz39o/SPzn6Jw9rKFE01aZoyFTRlKqitSpFJJ6mpTFFdkSSTTpKpTJGpSIa3YLqmMkVzpoKmmjS1\nlSmNJ0hJKNwlUpIJC0M2xfrG6nnbj4wHXwQnBsc4NTTGiaExTg6Nc2owmJ66rOvkEMNjOYbGcgyN\nZRnPzT9elU4aTeEvguaaCppqKsLgr6A5k6a5tpKWmgqaaytYVVNJUyZNKqlfDHLmFO4Sa1XpJGsb\nkqxtWPhlDcdz+cmgHxrLMTyWY3A0y8BolpND45wMvyBODIRfGoNj7D3SF35ZjM/4nGbQWJ1mVW0l\nzTUVtNROdBVVTI43VE6MN6SnTKcS4eMk1RVJaitT1FQGvzS0C2o8KdxFFimdTNBQHQzeLlQ2l+fU\ncPAF0DMwxonBMXoHR8PpUXoHxugdHOPJo/30zvFlMB8zqKlIUVMZBP5E6E9M11WlqK1KUVuZpq4q\nfFyZoq4q/dzyyhRmMJYNxjom7sezzlgux1jWJ+e7O3VVaRqq09RXp2ioVtdUqSjcRUoglUzQUltJ\nS20l566Zv727h4PIeUbHwwHl8VzwOBtMj+XyjIznGRoLfj0MjmYZGMkyMPrcL4qJ24nBIfpHnns8\n10D0mUoY1FenqZ8W+pmKFFXpBNXpJNXpJFUVSarCXx7V6SRV6QRV4bKJL6RM+CVVnU7qC2MeCneR\nMmBmYfdLEhbegzQnd2dkPE//yDj9k18I2eBxOA3BL5WKVIKKKffpycdGRTKJWXCtgb7hcfqGs5we\nHqdvZDy4Hx6nbySYd6xvgOGxHMPjOUbGg/uFHHKTmPxF8lzg11QEA90TXVeT3VjpJFXh/cTjymSC\nRMJIJey5ezOSM8yrSBmpRCJcfyOdTEzegvcgmJdK2Ir6wlG4i8ScmQVbyxVJVpeoBndnNJufDPqR\n8fxk+A+P5RgYzTI0Fv4aCX+JDIaPB0dzk79U+vvHGR3PTz5X8Msm97yjpZdSMhF8QaTD+1QyMfmF\nMfX+2h0dvO1nz17SWgoKdzO7CvgnIAl81t3/dtrySuCLwKVAL/AGd99f3FJFJKrMjKp0kqp0ksYl\neP7p3Vqj2Rz5POTcyeXz5PKQzefJT9y7T87L5pzxXJ7xXJ6xnJOdMj2ezT9vWS6fJ5t3cjkP7vMT\n9/nnPW5ZhpPrzRvuZpYEPg78MnAQeNDM7nT3x6c0+x3gpLtvNrM3An8HvGEpChYRWail7NZaqQrZ\noXYHsM/dn3b3MeA24Jppba4BbgmnvwH8oq2kzicRkZgpJNw3AF1THh8M583Yxt2zwGlg1fQnMrMb\nzKzTzDq7u7sXV7GIiMxrWQ+Fc/eb3H27u29vbX3hOUVERKQ4Cgn3Q0D7lMdt4bwZ25hZCmggGFgV\nEZESKCTcHwTONbOzzKwCeCNw57Q2dwJvDadfC/y7l+pE8SIiMv/eMu6eNbN3AN8n2BXy8+7+mJl9\nAOh09zuBzwFfMrN9wAmCLwARESmRgvZzd/e7gLumzXvvlOkR4HXFLU1ERBZL5xYVEYmgkl1D1cy6\ngQOL/PMWoKeI5awEUVunqK0PRG+dorY+EL11mml9Nrr7vLsblizcz4SZdRZygdhyErV1itr6QPTW\nKWrrA9FbpzNZH3XLiIhEkMJdRCSCyjXcbyp1AUsgausUtfWB6K1T1NYHordOi16fsuxzFxGRuZXr\nlruIiMxB4S4iEkFlF+5mdpWZPWlm+8zsT0tdTzGY2X4ze8TMdptZZ6nrWSgz+7yZHTezR6fMazaz\nH5jZT8P7plLWuFCzrNONZnYo/Jx2m9krS1njQphZu5n90MweN7PHzOyd4fyy/JzmWJ9y/oyqzOwB\nM9sTrtP7w/lnmdn9YeZ9LTzH1/zPV0597uFVoX7ClKtCAddOuypU2TGz/cB2dy/Lgy/M7OeAAeCL\n7n5hOO+DwAl3/9vwS7jJ3f+klHUuxCzrdCMw4O7/UMraFsPM1gHr3H2XmdUBO4FXA9dThp/THOvz\nesr3MzKgxt0HzCwN/CfwTuDdwLfc/TYz+xSwx90/Od/zlduWeyFXhZJl5u53E5wwbqqpV+e6heA/\nXtmYZZ3Klrsfcfdd4XQ/sJfgIjtl+TnNsT5lywMD4cN0eHPgFwiucAcL+IzKLdwLuSpUOXLgX81s\np5ndUOpiimSNux8Jp48Ca0pZTBG9w8weDrttyqILYzoz2wRcDNxPBD6naesDZfwZmVnSzHYDx4Ef\nAE8Bp8Ir3MECMq/cwj2qrnT3S4BXAG8PuwQiIzy3f/n0/83uk8A5wEXAEeBDpS1n4cysFvgm8C53\n75u6rBw/pxnWp6w/I3fPuftFBBdF2gGcv9jnKrdwL+SqUGXH3Q+F98eBbxN8qOXuWNgvOtE/erzE\n9Zwxdz8W/ufLA5+hzD6nsB/3m8BX3P1b4eyy/ZxmWp9y/4wmuPsp4IfAS4HG8Ap3sIDMK7dwL+Sq\nUGXFzGrCASHMrAb4FeDRuf9ooSDUAAAA80lEQVSqLEy9OtdbgTtKWEtRTIRg6Ncpo88pHKz7HLDX\n3T88ZVFZfk6zrU+Zf0atZtYYTlcT7DiylyDkXxs2K/gzKqu9ZQDCXZs+wnNXhfqrEpd0RszsbIKt\ndQgunnJrua2TmX0VeDnB6UmPAe8Dbge+DnQQnNr59e5eNgOUs6zTywl+7juwH/i9Kf3VK5qZXQn8\nCHgEyIez/4ygn7rsPqc51udayvcz2kowYJok2PD+urt/IMyI24Bm4CHgTe4+Ou/zlVu4i4jI/Mqt\nW0ZERAqgcBcRiSCFu4hIBCncRUQiSOEuIhJBCncRkQhSuIuIRND/B5kQg+yXUiytAAAAAElFTkSu\nQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f7dbf4d1d30>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.title('Cross Entropy Cost')\n",
    "plt.plot(ce_cost)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Softmax Activation and Cost\n",
    "Softmax (aka Log Likelihood) allows our network to represent its output as a probability distribution--that is, summing all outputs for a single sample will equal 1. This can **only be used for disjoint categories/classes** (e.g. if an image can be classified as a cat *and* an animal, softmax won't work).\n",
    "\n",
    "## A Note About Vectorization\n",
    "Softmax definitions typically are written with respect to a single sample. In this case, we wish to vectorize it to work on the entire mini-batch. To that end, the numpy max function is used to reduce all z values to prevent softmax from exploding and giving us NaN/inf errors; for both the max function and the summation, the axis must be set and the result must keep the original dimensions. By doing this, the functions will operate **per sample**, giving us the maximum value in a sample and summing the outputs of all nodes **in each given sample**. \n",
    "\n",
    "## Activation Function\n",
    "To use Softmax, change the activation function of *just* the output layer to the softmax function below (we won't bother defining the derivative, as it's not needed):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from activations import Activation\n",
    "from cost import Cost\n",
    "def softmax(z):\n",
    "    stable_z = z - np.max(z,axis=-1,keepdims=True)\n",
    "    z_exp = np.exp(stable_z)\n",
    "    return z_exp / np.sum(z_exp, axis=-1, keepdims=True)\n",
    "    #exp_max = np.exp(z - np.max(z,axis=-1,keepdims=True))\n",
    "    #out = exp_max/np.sum(exp_max,axis=-1,keepdims=True)\n",
    "    #return out\n",
    "\n",
    "def sigmoid(z):\n",
    "    return 1 / (1 + np.exp(-z))\n",
    "\n",
    "def sigmoid_prime(z):\n",
    "    return sigmoid(z) * (1 - sigmoid(z))\n",
    "\n",
    "sigmoid_activation = Activation(sigmoid, sigmoid_prime)\n",
    "softmax_activation = Activation(softmax, None)\n",
    "\n",
    "softmax_activations = [sigmoid_activation, sigmoid_activation, softmax_activation]\n",
    "sigmoid_activations = [sigmoid_activation, sigmoid_activation, sigmoid_activation]\n",
    "relu_ce_activations = [ReLU, ReLU, sigmoid_activation]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cost Function\n",
    "The cost function used with Softmax is the Log-Likelihood function; the derivative becomes the same as the MSE and CE derivatives:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def softmax_cost(y, y_pred):\n",
    "    return - np.sum(np.max(np.log(y_pred),axis=-1, keepdims=True)) / len(y_pred)\n",
    "\n",
    "def softmax_cost_prime(y, y_pred):\n",
    "    return y_pred - y\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing it out\n",
    "We can now build our network, making sure to use the softmax activation function in the output layer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.6495844819 2810 / 10000\n",
      "1.29744131288 3501 / 10000\n",
      "0.650390158902 7024 / 10000\n",
      "0.253691101758 8663 / 10000\n",
      "0.164069410632 8990 / 10000\n",
      "0.137037144103 9108 / 10000\n",
      "0.108962983453 9311 / 10000\n",
      "0.0915231322642 9444 / 10000\n",
      "0.0810809596267 9500 / 10000\n",
      "0.0719934952742 9523 / 10000\n"
     ]
    }
   ],
   "source": [
    "softmax_network = network_ce.NetworkCE(784,100,20,10, activations=softmax_activations, cost=Cost(softmax_cost, softmax_cost_prime))\n",
    "softmax_cost = softmax_network.train(mnist.train.images, mnist.train.labels, learning_rate=1.0, batch_size=200, epochs=10, test_epoch=1, test_images=mnist.test.images, test_labels=mnist.test.labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results seem to line up well with what we had when using Cross Entropy, so we'll just keep this one tucked in our toolbelt, ready if necessary."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regularization\n",
    "The following regularization will be explored:  \n",
    "* L2 \n",
    "* L1\n",
    "* Dropout  \n",
    "\n",
    "L1/L2 regularization operate simply and similarly: they introduce **weight decay**; L1 pulls the weights toward zero (reduces them by a constant amount) and L2 scales the weights down (e.g. each training pass multiplies the weights by something like 95%). They can be used individually or together.\n",
    "\n",
    "Dropout works entirely differently. When training, dropout randomly sets the output to zero for a percentage of the neurons. During backpropagation, we need to then nullify the activation (and gradient) for those neurons. During testing, dropout should not be used; instead, the outputs of the neurons should be scaled down. In this way, we're kind of creating many networks in one and choosing the output of the most likely solution. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ## L2 Regularization\n",
    " For both L2 and L1 regularization, we add a new term to the cost function being used; however, it is the same regardless of which cost function is used, so we'll add it to our training.\n",
    " \n",
    " \n",
    " `# Add L2 Cost to Cost Function\n",
    "l2_cost = np.sum(self.W**2) * l2_lambda / (2 * len(a))\n",
    "training_cost.append(self.cost.calculate(labels, a[-1]) + l2_cost`\n",
    "\n",
    "And update the derivative:\n",
    "\n",
    "`l2_loss = (1 - learning_rate * l2_lambda / batch_size)\n",
    "for j in range(len(self.W)):\n",
    "    self.W[j] = l2_loss * self.W[j] - learning_rate * dWl[j]\n",
    "    self.B[j] -= (learning_rate / batch_size) * dBl[j]`  \n",
    "    \n",
    "To utilize it, we simply declare a non-zero **l2_lambda** when training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.75680801475 2722 / 10000\n",
      "2.37774347763 3980 / 10000\n",
      "1.27174928841 7967 / 10000\n",
      "0.852060032577 8928 / 10000\n",
      "0.673231439795 9202 / 10000\n",
      "0.605198472388 9322 / 10000\n",
      "0.564355916083 9402 / 10000\n",
      "0.527382261693 9467 / 10000\n",
      "0.516726823119 9504 / 10000\n",
      "0.495622265807 9562 / 10000\n"
     ]
    }
   ],
   "source": [
    "l2_network = network_ce.NetworkCE(784,100,20,10, activations=sigmoid_activations, cost=Cost(cross_entropy_cost, cross_entropy_cost_prime))\n",
    "l2_cost = l2_network.train(mnist.train.images, mnist.train.labels, learning_rate=1.0, batch_size=200, epochs=10, test_epoch=1, test_images=mnist.test.images, test_labels=mnist.test.labels, l2_lambda=0.001)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not explicitly shown above is that a properly tuned regularizer should reduce the gap between training error and testing error, indicating that overfitting is being mitigated. We'll look at that below when implementing Dropout\n",
    "\n",
    "Note: softmax (or any other cost function) is **not** a requirement for using L2 or L1 regularization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## L1 Regularization\n",
    "Similar to L2, L1 gets added in like so:  \n",
    "\n",
    "Cost:  \n",
    "`l1_cost = l1_lambda * np.sum([np.sum(np.abs(w)) for w in self.W]) / len(a)\n",
    "training_cost.append(self.cost.calculate(labels, a[-1]) + l2_cost + l1_cost)`  \n",
    "\n",
    "Derivative:  \n",
    "`l1_loss = - learning_rate * l1_lambda / batch_size\n",
    "for j in range(len(self.W)):\n",
    "    self.W[j] = l2_loss * self.W[j] + l1_loss * np.sign(self.W[j]) - (learning_rate / batch_size) * dWl[j]\n",
    "    self.B[j] -= (learning_rate / batch_size) * dBl[j]`  \n",
    "\n",
    "Using L1 regularization also requires us to set a non-zero **l1_lambda** when training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.07132676416 2330 / 10000\n",
      "2.9220845948 3431 / 10000\n",
      "2.05098738308 7487 / 10000\n",
      "1.66078088763 8717 / 10000\n",
      "1.51627884548 9158 / 10000\n",
      "1.50957419485 9307 / 10000\n",
      "1.48261522797 9451 / 10000\n",
      "1.51693632422 9480 / 10000\n",
      "1.51147725509 9553 / 10000\n",
      "1.52291020413 9575 / 10000\n"
     ]
    }
   ],
   "source": [
    "l1_network = network_ce.NetworkCE(784,100,20,10, activations=sigmoid_activations, cost=Cost(cross_entropy_cost, cross_entropy_cost_prime))\n",
    "l1_cost = l1_network.train(mnist.train.images, mnist.train.labels, learning_rate=1.0, batch_size=200, epochs=10, test_epoch=1, test_images=mnist.test.images, test_labels=mnist.test.labels, l1_lambda=0.001)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A simple sanity test that L1/L2 is working as expected could be to set the lambda to a high value and see if the sum of the weights is close to zero; without regularization, this tends to happen less often (not to say it couldn't happen).\n",
    "\n",
    "**Note**: Nothing stops us from using both L1 and L2 at the same time. Simply set both lambdas to a positive value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dropout\n",
    "Vastly different from L1 & L2, Dropout requires that a random number of neurons have their activations set to zero. To do so, we'll simply use a flag for enabling it, which will drop 20% of the input data and 50% of each hidden layer's activations*.  \n",
    "\n",
    "First, we need a way to make the *mask*, a matrix of 1s with percentage p of its elements set to 0:\n",
    "\n",
    "***Note**: A more robust implementation of Dropout would allow you to tune the dropout probability, but for demonstration purposes, we'll use the values defined above; these values are used in the original paper describing Dropout and can be found at https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf .\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mask:\n",
      " [[ 1.  0.  1.  1.  0.]\n",
      " [ 0.  1.  1.  0.  1.]\n",
      " [ 1.  0.  1.  1.  0.]]\n",
      "Activations:\n",
      " [[ 0.90812353  0.          0.19343038  0.48488151  0.        ]\n",
      " [ 0.          0.53516386  0.9985114   0.          0.5264364 ]\n",
      " [ 0.30092327  0.          0.64862249  0.36775605  0.        ]]\n"
     ]
    }
   ],
   "source": [
    "def dropout_mask(shape, threshold=0.5):\n",
    "    assert len(shape) == 2\n",
    "    mask = np.random.rand(shape[0], shape[1])\n",
    "    mask[mask<threshold] = 0\n",
    "    mask[mask!=0] = 1\n",
    "    return mask\n",
    "\n",
    "mask = dropout_mask((3, 5))\n",
    "a = np.random.rand(3, 5)\n",
    "print('Mask:\\n',mask)\n",
    "print('Activations:\\n',np.multiply(mask, a))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll define a new network class called **DropoutNetwork**. It's based on the previous NetworkCE code, but includes everything necessary to make dropout work. Additionally, the return value from the train() function now returns an array of the cost, training accuracy, and testing accuracy for each epoch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "from misc import shuffle\n",
    "\n",
    "class DropoutNetwork:\n",
    "    \n",
    "    def __init__(self, *layers, activations=None, cost=Cost(cross_entropy_cost, cross_entropy_cost_prime), dropout=True):\n",
    "        # assert activation is not None\n",
    "        self.layers = [l for l in layers]\n",
    "        \n",
    "        if activations is not None:\n",
    "            assert len(activations) == len(layers) - 1\n",
    "            self.activations = [a for a in activations]\n",
    "        else:\n",
    "            self.activations = [Activation(sigmoid, sigmoid_prime) for i in range(len(layers) - 1)]\n",
    "            \n",
    "        self.W = [np.random.rand(layers[i], layers[i+1]) * 0.001 for i in range(len(layers) - 1)]  \n",
    "        self.B = [np.random.rand(1, layers[i+1]) for i in range(len(layers) - 1)]  # Input layer doesn't have bias\n",
    "        \n",
    "        if dropout:\n",
    "            self.drop = [0.8] + [0.5 for l in range(len(layers) - 2)] + [1] # \n",
    "        else:\n",
    "            self.drop = [1.0 for l in range(len(layers))]\n",
    "        self.cost = cost\n",
    "    \n",
    "    def feedforward(self, x, test=False):\n",
    "        masks = []\n",
    "        output = []\n",
    "        a = x\n",
    "        \n",
    "        if test == False:\n",
    "            masks += [dropout_mask(x.shape, threshold=1-self.drop[0])]\n",
    "            a = np.multiply(a, masks[0])\n",
    "        # Experiment with scaling inputs just as we do with activations\n",
    "        \n",
    "        output += [a]\n",
    "        z_products = []\n",
    "        \n",
    "        for b, w, p, activation in zip(self.B, self.W, self.drop[1:], self.activations):\n",
    "            if test == False:  # Only scale activations if we're testing\n",
    "                p = 1\n",
    "                \n",
    "            z = np.dot(a, w) + b\n",
    "            a = p * activation.activate(z)\n",
    "            if test == False:\n",
    "                masks += [dropout_mask(z.shape, threshold=1-p)]\n",
    "                output.append(np.multiply(a, masks[-1]))\n",
    "            else:\n",
    "                output.append(a)\n",
    "            z_products.append(z)\n",
    "        return output, z_products, masks\n",
    "\n",
    "    def train(self, images, labels, batch_size=200, learning_rate=0.01, epochs=10, test_epoch=5, test_images=None, test_labels=None, l2_lambda=0, l1_lambda=0):\n",
    "        assert len(images) == len(labels)\n",
    "        training_cost = []\n",
    "        output = []\n",
    "        for i in range(epochs):\n",
    "            shuffle(images, labels)\n",
    "            # Divide training data into batches\n",
    "            img_batches = [images[i:i+batch_size] for i in range(0, len(images), batch_size)]\n",
    "            label_batches = [labels[i:i+batch_size] for i in range(0, len(labels), batch_size)]\n",
    "            n_batches = len(img_batches)\n",
    "\n",
    "\n",
    "            for imgs, lbls in zip(img_batches, label_batches):\n",
    "                a, z, masks = self.feedforward(imgs)\n",
    "\n",
    "                # Calculate Output Error\n",
    "                # dL = np.multiply(self.cost.calculate_derivative(lbls, a[-1]),self.activations[-1].derive(z[-1]))  # MSE Cost Error\n",
    "                l2_loss = (1 - learning_rate * l2_lambda / batch_size)\n",
    "                l1_loss = - learning_rate * l1_lambda / batch_size\n",
    "                dL = self.cost.calculate_derivative(lbls, a[-1])  # Cross Entropy doesn't have hadamard product  # CE Cost Error\n",
    "                # Update biases and weights for output layer using output error\n",
    "                dBL = np.reshape(np.sum(dL, axis=0), (1,dL.shape[1]))\n",
    "                dWL = np.dot(a[-2].T, dL)\n",
    "                # Being verbose to show that we're building a list of deltas for each layer in the network (excluding input layer)\n",
    "                dBl = [dBL]\n",
    "                dWl = [dWL]\n",
    "                dl = dL\n",
    "                \n",
    "                # Calculate Hidden Error\n",
    "                for l in range(len(a) - 2, 0, -1):  # -1 for zero-indexing, -1 more because output layer already calculated\n",
    "                    dl = np.multiply(np.dot(dl, self.W[l].T), np.multiply(self.activations[l-1].derive(z[l-1]), masks[l]))\n",
    "                    dBl.insert(0, np.reshape(np.sum(dl, axis=0), (1, dl.shape[1])))\n",
    "                    dWl.insert(0, np.dot(a[l-1].T, dl))\n",
    "\n",
    "                # Update weights and biases\n",
    "                for j in range(len(self.W)):\n",
    "                    self.W[j] = l2_loss * self.W[j] + l1_loss * np.sign(self.W[j]) - (learning_rate / batch_size) * dWl[j]\n",
    "                    self.B[j] -= (learning_rate / batch_size) * dBl[j]\n",
    "\n",
    "\n",
    "            # Cost after training this epoch\n",
    "            a, z, masks = self.feedforward(images, test=True)\n",
    "            l2_cost = np.sum([np.sum(w ** 2) for w in self.W]) * l2_lambda / (2 * len(a))\n",
    "            l1_cost = l1_lambda * np.sum([np.sum(np.abs(w)) for w in self.W]) / len(a)\n",
    "            training_cost.append(self.cost.calculate(labels, a[-1]) + l2_cost + l1_cost)\n",
    "            result = [training_cost[-1]]\n",
    "\n",
    "            result += [self.test(images, labels) / len(images)]\n",
    "            if test_images is not None and test_labels is not None and (i + 1) % test_epoch == 0:\n",
    "                result += [self.test(test_images, test_labels) / len(test_images)]\n",
    "            else:\n",
    "                result += [None]\n",
    "            print(result)\n",
    "            output.append(result)\n",
    "        return np.asarray(output)\n",
    "    \n",
    "    def test(self, test_images, test_labels):\n",
    "        img = test_images\n",
    "        a, z, masks = self.feedforward(img, test=True)\n",
    "        a[-1].shape\n",
    "        prediction = np.argmax(a[-1], axis=1)\n",
    "        truth = np.argmax(test_labels, axis=1)\n",
    "        result = np.sum([p == i for p, i in zip(prediction, truth)])\n",
    "        return result\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the dropout network and begin testing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[4.9194439620448831, 0.090672727272727266, 0.089200000000000002]\n",
      "[3.2096470621846902, 0.43058181818181818, 0.43919999999999998]\n",
      "[1.7092278693120284, 0.79763636363636359, 0.79700000000000004]\n",
      "[1.4937725257727459, 0.87156363636363632, 0.86909999999999998]\n",
      "[1.1902994448313498, 0.92118181818181821, 0.9204]\n",
      "[1.0491747893205332, 0.93825454545454545, 0.93600000000000005]\n",
      "[1.1057856018673919, 0.92136363636363638, 0.91810000000000003]\n",
      "[0.92365473011777666, 0.94330909090909087, 0.93830000000000002]\n",
      "[0.84943093633738243, 0.95670909090909095, 0.95130000000000003]\n",
      "[0.8754342925088241, 0.94407272727272729, 0.9355]\n",
      "[0.81218000680149882, 0.95134545454545449, 0.94359999999999999]\n",
      "[0.8223347624551296, 0.93790909090909091, 0.93300000000000005]\n",
      "[0.70199913641145961, 0.96425454545454548, 0.95589999999999997]\n",
      "[0.70607483713376118, 0.95940000000000003, 0.9546]\n",
      "[0.64415788212795111, 0.96758181818181821, 0.96020000000000005]\n",
      "[0.70575946870537853, 0.95845454545454545, 0.95279999999999998]\n",
      "[0.58800717751191289, 0.97040000000000004, 0.96199999999999997]\n",
      "[0.60181483878643482, 0.96707272727272731, 0.95879999999999999]\n",
      "[0.57664205050779582, 0.97227272727272729, 0.96099999999999997]\n",
      "[0.52632048158979194, 0.97269090909090905, 0.96160000000000001]\n"
     ]
    }
   ],
   "source": [
    "dropout_network = DropoutNetwork(784,100,30,10, activations=relu_ce_activations, cost=Cost(cross_entropy_cost, cross_entropy_cost_prime))\n",
    "dropout_cost = dropout_network.train(mnist.train.images, mnist.train.labels, learning_rate=0.15, batch_size=100, epochs=20, test_epoch=1, test_images=mnist.test.images, test_labels=mnist.test.labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the results. Note the gap between the training data and test data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f7dbd3c1d68>]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZMAAAEWCAYAAACjYXoKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzs3Xl4VOX1wPHvyQok7DuEPWwJJGF3\nQXBBQBERtIiKUqs/92pt1S5u1bq2VqvW2qp1rSK4FRCtIIICghCFAAIJO4EQCEtC9m3O7487Cdkz\nyWQSEs/neebJzH3fe+87g86ZdxdVxRhjjPGGX0MXwBhjTONnwcQYY4zXLJgYY4zxmgUTY4wxXrNg\nYowxxmsWTIwxxnjNgonxGRHxF5EMEelZl3mbMhGZLCLfN3Q5jKkpsXkmpoiIZJR42QLIBQrdr29W\n1Xfrv1QNT0TOBT4teonz2WSWyDJAVZNqcd1QIB3oqqrJ3pazBvf9EJgKdFHVE/V1X9O0WTAxFRKR\nvcCNqvplFXkCVLWg/krV8EQkHNihqlIH16r3YCIi7YBDQAbwgKq+XB/3dd/7J/ffy0+JNXMZj4nI\nYyIyT0Tmikg6MFtEzhSRtSKSKiKHROQFEQl05w8QERWR3u7X/3Gnfy4i6SKyRkT61DSvO/0iEUkQ\nkTQReVFEVovIzysocw8RyRaR1iWOjRKRI+57DhCRb9zXOSoi79Xys2nvLnOyiOwXkQdERNxpke7y\npYlIioi84T7tG/ffXe4mvkvcj+0lrntURO4Ska3u898u+nzd6Q+730uiiNzs/gy7VFHUWcAe4K/A\nnDLvIVBEHhGRPSJyUkTWiUhHd9owEVkhIifc/86/ch//UER+V+IaFZX/1yKyFTjuPvaIiOx1/7tu\nFpGLSuQXEfmliMS70zeJSIT7nLfKlPd1EXm82n8cUz9U1R72KPcA9gITyhx7DMjDaSLxA5oDo4Ax\nQADQF0gA7nDnDwAU6O1+/R/gKDASCATmAf+pRd5OOL/op7nTfg3kAz+v5L18A1xf4vVzwN/dzz8A\nfut+P82As6v5XMKd/23KHV/qvm5zoBuwCbjGnbYI+BVOE1nzonsAoe733KXEdS4Btpd4fdRd/o7u\n970HmO1OuwLYD/R3X+ujsteroJzfAQ8BvQEXMLBE2iNArPvf0Q8YAbQG2gPHgFuAIPexUe5zPgR+\nV0351wJdgebuY7OALoA/cD2QBrRzp10P7Aai3Z/XIKA70A84WeIazd3nDazsvdqjfh9WMzE1tUpV\nF6mqS1WzVXW9qn6nqgWquht4BRhfxfkfqmqsquYD7wIxtch7CbBRVRe4057D+dKqzHvAVQAi4gdc\n6T4GThDqjdPUlKOqq6t68xURkX44AfVe92eSBLyI86VZdI8+QGd3ek3v8ayqpqjqEeBzTn0OM4F/\nqeoOVc0A/lRNOQcBo4H3VHUvsAa4rkSWG4Hfqupu97/v96qaBswAtqrqP1U1T1XTVHV9Dct/SFWz\nAVT1fVVNVtVCVX0DSAGGlSjD46oap47tqnpQVXcBccBl7nzTgHhVja9BOYwPWTAxNZVY8oWIDBKR\nxe7mnZPAo0CHKs4v2TeQhfOLuqZ5u5Ush6oqcKCK63wAnCMinYHzgBxV/dad9huc2k2su8llTmUX\nqUIvIAQ46m7uS8VpRursTr8LaAVsFJE4Ebmqhtf36HMo87wic4B1qrrT/fpd4FoR8RMRf5zaw64K\nzutRyXFPlf1v5ib3Z130WfXm1H8zVd3rLWC2+/ls4B0vymTqmAUTU1NlR2z8C9gChKtqK5wmFK87\np6txCAgreuHum+heWWZVPQZ8BfwMuBqYWyLtkKreqKpdgduBV0r2zXgoEafJpa2qtnE/WqnqaPc9\nElX1epwv618Db4tIN8p/ljVV6nPA+SKukLtGNhsY6g78yTg1mR7Aeapa6L5ev0reX0XHwRnV1qLE\n64r6a4rfp4hEAM8CN+A0bbXBaVIt+m+mqnvNx/lREAmcD7xfST7TACyYGG+1xPkizRSRwcDN9XDP\nT4HhIjJVRAJwfvl3rOac93B+mc/gVBMXIjJTRIoCUSrOF19h+dMrp6o7gA3AEyIS6v6lP0BEznbf\nY5aIdHXXoIrvoaqZOF/GfWtyvxLmAzeJSD9xRobdX0XeC3A+o5gSj0hgAaeaul4DnhSR3u73MNw9\ncOFjIMJdowgSkdYiMtJ9zkZgqvtYGHBHNWUOxemrSQH8ROQOnJpJkdeAP4hIlLszflDRv4+qngQW\n4/wY+FJVU6r7gEz9sWBivPUbnC/pdJxayjxf31BVD+P0ezyL0zHcD+fLPLeK0/4LRAD7VfXHEsfH\nAOtFJBPnS/N2Vd1fi2LNxGnWiscZtTSXUwHubOAHcebxzAVucL8HgAeBT9xNPlNqeM8Pgbdx+j62\nA1+7j1f0OcwB5qlqgru/Ilmd4cgvAJe7g9FjwBL3dVKBfwBB7prdhTg1mxRgG3CW+7qvAvtwahQL\nKRGoK6Kq64B/4/x7JeE01W0skeVNnP6mj3A63OfhNBEWeQsYijVxnXZsnolp9Nzt/UnAFaq6sqHL\n01BEZBSwQlVDGrosvuJuJvsWZ8RaTkOXx5xiNRPTKImz7EgbEQnG+XWfD6xr4GLVOxGZ4Z4f0hF4\nHPikocvkK+4fDXcDb1sgOf34LJi4JxQdEZEtlaSLOJPSdronJg0vkTZHRHa4H7UZXWOavrE48xFS\ngEnAdFWtqpmrqbobp6lvG3DC/brJEZFOOM1eo3CCpjnN+KyZS0TG4SzZ8LaqDqkg/WLgl8DFOO3W\nz6vqGHGWe4jFmaymwPfACLU1hIwx5rTls5qJqn6De/mESkzDCTSqqmuBNiLSFedX5lJVPe4OIEuB\nyb4qpzHGGO8FNOC9u1N6MtMB97HKjpcjIjcBNwGEhISMGDRokG9KaowxTdT3339/VFWrG1pfrYYM\nJl5T1Vdwlu9g5MiRGhsb28AlMsaYxkVE9tXFdRpyNNdBSs/YDXMfq+y4McaY01RDBpOFwHXuUV1n\nAGmqegj4ApgoIm1FpC0w0X3MGGPMacpnzVwiMhc4F+ggIgeAh3EW1ENV/wl8hjOSayfO4nXXu9OO\ni8ifgKJVSR9V1ao68o0xxjQwnwUTVa1yZVT3OkW3V5L2OvC6L8pljDGm7tkMeGOMMV6zYGKMMcZr\nFkyMMcZ4zYKJMcYYr1kwMcYY4zULJsYYY7xmwcQYY4zXLJgYY4zxmgUTY4wxXrNgYowxxmsWTIwx\nxnjNgokxxhivWTAxxhjjNQsmxhhjvGbBxBhjjNcsmBhjjPGaBRNjjDFes2BijDHGaxZMjDHGeM2C\niTHGGK9ZMDHGGOM1CybGGGO8ZsHEGGOM1yyYGGOM8ZoFkyYqNzeXK6+8kvDwcMaMGcPevXsrzVtY\nWMiwYcO45JJLio999dVXDB8+nCFDhjBnzhwKCgoAWLFiBa1btyYmJoaYmBgeffTR4nNSU1O54oor\nGDRoEIMHD2bNmjUAPPjgg0RFRRETE8PEiRNJSkoqPmfFihXExMQQGRnJ+PHjAcjJyWH06NFER0cT\nGRnJww8/XJxfVbn//vsZMGAAgwcP5oUXXgBgwYIFxfcYOXIkq1atKj5n//79TJw4kcGDBxMREVH8\nWdxwww1ER0cTFRXFFVdcQUZGRvE58+fPJyIigsjISK6++moA4uPjGTFiBFFRUcXvraCggAkTJpCV\nleX5P44xTZGqNonHiBEj1Jzy0ksv6c0336yqqnPnztWZM2dWmvevf/2rXnXVVTplyhRVVS0sLNSw\nsDCNj49XVdUHH3xQX3vtNVVVXb58eXG+sq677jp99dVXVVU1NzdXT5w4oaqqaWlpxXmef/754nKd\nOHFCBw8erPv27VNV1cOHD6uqqsvl0vT0dFVVzcvL09GjR+uaNWtUVfX111/Xa6+9VgsLC0udk56e\nri6XS1VV4+LidODAgcX3HD9+vC5ZsqQ4X2ZmZrly3X333frkk0+qqmpCQoLGxMTo8ePHS93j7rvv\n1pUrV2piYqLOmDFDVVVfeOEFfeONNyr9bI053QGxWgffwVYzqcBll13GiBEjiIyM5JVXXuGf//wn\n9957b3H6m2++yR133AHAn/70JwYOHMjYsWO56qqreOaZZyq97vr164t/Pd97770MGTIEgL1793LO\nOecwfPhwhg8fzrfffgs4v9rHjRvHlClTGDhwILfccgsul8uj97BgwQLmzJkDwBVXXMGyZctw/rsp\n7cCBAyxevJgbb7yx+NixY8cICgpiwIABAFx44YV89NFHVd4vLS2Nb775hhtuuAGAoKAg2rRpA0Cr\nVq2K82VmZiIiALz33nvMmDGDnj17AtCpUycARITQ0FAA8vPzyc/PLz7n5Zdf5qGHHsLPz6/UOaGh\nocV5St5j69atFBQUcOGFFxbna9GiRalyqSrZ2dnF57z66qvcfvvttG3bttQ9AgMDycrKIisri8DA\nQFJTU1m0aBHXXXddlZ+NMT8JdRGRKnsAk4F4YCfwuwrSewHLgE3ACiCsRNrTwBb348rq7lWXNZNj\nx46pqmpWVpZGRkZqcnKy9uvXrzh98uTJunLlSl23bp1GR0drdna2njx5UsPDw/Uvf/lLpdeNjIzU\nb7/9VlVVf/vb32pkZKSqqmZmZmp2draqOr+Ki97L8uXLNTg4WHft2qUFBQU6YcIE/eCDD1RVdebM\nmRodHV3u8dZbbxXfKzExsfjeffv21ZSUlHJluvzyyzU2NrZUjcPlcmnPnj11/fr1qqp655136pAh\nQ4rL1K5dO42KitLJkyfrli1bVFV1w4YNOmrUKJ0zZ47GxMToDTfcoBkZGcX3+cMf/qBhYWEaGRmp\nR44cUVXVu+66S2+77TYdP368Dh8+vLjsqqoFBQUaHR2tISEhet999xUfb9eunT722GM6YsQInTx5\nsiYkJBSnffzxxzpw4EBt27Zt8ef8ySef6JQpU3T69OkaExOj99xzjxYUFBSf8/Of/1w7deqk5557\nbnGNZdq0aXrvvffqWWedpWPGjNHPP/9cVVX37dun48eP1zPOOEPj4uL017/+tS5fvrzSf29jGgPq\nqGbiy0DiD+wC+gJBQBwQUSbPB8Ac9/PzgXfcz6cAS4EAIARYD7Sq6n51GUwefvhhjYqK0qioKG3V\nqpWuWbNGL7zwQl2zZo0ePXpUe/furS6XS5977jl96KGHis+7++67Kw0mJ06c0J49exa/jouLKw4m\nqampOnv2bB0yZIhGR0dr8+bNVdX54j7nnHOKz/n3v/+td911l0fvwZNgsmjRIr311luL71Wy+erb\nb7/VsWPH6qhRo/T+++/X6OhoVXWahoqaoBYvXqzh4eGqqrp+/Xr19/fXtWvXqqoTgB544IFy5Xri\niSeKP7Pbb79dx4wZoxkZGZqSkqLh4eHFTWslP7dzzz1XN2/erKqqISEh+swzz6iq6kcffaRjx44t\nd4+vv/5aL7jgAlVV/eCDD7RVq1a6a9cuzc/P1xkzZhQ32RUpKCjQW2+9VV9//XVVVZ0yZYpedtll\nmpeXp7t379awsLDiJrsiO3bs0JkzZ2pycrLOnj1bZ86cWa7sxjQGdRVMfNnMNRrYqaq7VTUPeB+Y\nViZPBPCV+/nyEukRwDeqWqCqmTg1l8k+LGuxFStW8OWXX7JmzRri4uIYNmwYOTk5zJo1i/nz5/PR\nRx8xffr04iaRuvDcc8/RuXNn4uLiiI2NJS8vrzit7H2KXl955ZXFneAlH2+//TYA3bt3JzExEXA6\nidPS0mjfvn2pa61evZqFCxfSu3dvZs2axVdffcXs2bMBOPPMM1m5ciXr1q1j3LhxxU1erVq1Km6C\nuvjii8nPz+fo0aOEhYURFhbGmDFjAKdp7Ycffij3Xq+55priJrOwsDAmTZpESEgIHTp0YNy4ccTF\nxZXK36ZNG8477zz+97//FZ8zY8YMAKZPn86mTZvK3WPcuHHs3r27uFwxMTH07duXgIAALrvssnLl\n8vf3Z9asWaXKdemllxIYGEifPn0YMGAAO3bsKHXO/fffz2OPPcYLL7zAjTfeyJ///GceeeSRcmUx\n5qfCl8GkO5BY4vUB97GS4oAZ7ufTgZYi0t59fLKItBCRDsB5QA8flrVYWloabdu2pUWLFmzfvp21\na9c6hZs+nQULFjB37lxmzZoFwNlnn82iRYvIyckhIyODTz/9tNLrtmnThpYtW/Ldd98B8P7775e6\nZ9euXfHz8+Odd96hsLCwOG3dunXs2bMHl8vFvHnzGDt2LADz5s1j48aN5R5F7feXXnopb731FgAf\nfvgh559/frnA9OSTT3LgwAH27t3L+++/z/nnn89//vMfAI4cOQI4o8KefvppbrnlFgCSk5OL+17W\nrVuHy+Wiffv2dOnShR49ehAfHw/AsmXLiIiIACj1RbxgwQIGDRoEwLRp01i1ahUFBQVkZWXx3Xff\nMXjwYFJSUkhNTQUgOzubpUuXFp9z2WWXsXz5cgC+/vrr4iC3c+fO4nL98MMP5Obm0r59e0aNGkVq\naiopKSmAM0otIiICVWXnzp2AUztfuHBhqXusWLECgKNHj5KQkEDfvn2L38PXX39Nt27d6N+/P1lZ\nWfj5+eHn52cjusxPW11Ubyp6AFcAr5V4fS3w9zJ5ugEfAxuA53ECTht32v3ARpzmrneBX1Vwj5uA\nWCC2ZBOSN3JycnTy5Mk6aNAgnTZtmo4fP764XXzKlCnap0+fUvkffvhh7d+/v44dO1ZnzJihr7zy\nSqXXXrt2rQ4dOlSjo6P1zjvv1LPOOktVnX6SoUOHalRUlN53330aEhKiqqeauS6++GIdMGCA3nzz\nzcWjmKqTnZ2tV1xxhfbr109HjRqlu3btUlXVgwcP6kUXXVQuf9lmrnvuuUcHDRqkAwYM0Oeee674\n+IsvvqgREREaFRWlY8aM0dWrVxenbdiwQUeMGKFDhw7VadOmFY+GmjFjhkZGRurQoUP1kksu0QMH\nDhSf8+c//1kHDx6skZGRxfeJi4vTmJgYHTp0qEZGRuojjzxSnP/EiRN68cUX65AhQ/SMM87QjRs3\nqqrqU089pRERERodHa1nnHGGrly5svicJUuW6NChQ3XIkCE6Z84czc3N1cLCQj3rrLN0yJAhGhkZ\nqVdffXXx6C6Xy6V33323Dh48WIcMGaJz584tvpbL5dIJEyYU96tt3bpVhw0bpkOHDtVVq1Z59G9j\nzOmEOmrmEtXyI3zqgoicCfxRVSe5X//eHbyerCR/KLBdVcMqSHsP+I+qflbZ/UaOHKmxsbF1Uvaa\nyMjIIDQ0lKysLMaNG8crr7zC8OHDq8wL8NRTT3Ho0CGef/75Sq+9YsUKnnnmmSprPMYY4w0R+V5V\nR3p7nYC6KEwl1gP9RaQPcBCYBVxdMoO7Ceu4qrqA3wOvu4/749RQjolIFBAFLPFhWWvtpptuYuvW\nreTk5DBnzpxKAwnA4sWLefLJJykoKKBXr168+eab9VdQY4zxIZ/VTABE5GLgbzgju15X1cdF5FGc\natVCEbkCeBJQ4BvgdlXNFZFmQFEv6UngFlXdWNW9GqpmUpHbb7+d1atXlzp21113cf311zdQiYwx\npmJ1VTPxaTCpT6dTMDHGmMairoKJzYA3xhjjNQsmxhhjvGbBxBhjjNcsmBhjjPGaBRNjjDFes2Bi\njDHGaxZMjDHGeM2CiTHGGK9ZMDHGGOM1CybGGGO8ZsHEGGOM1yyYGGOM8ZoFE2OMMV6zYGKMMcZr\nFkyMMcZ4zYKJMcYYr1kwMcYY4zULJsYYY7xmwcQYY4zXLJgYY4zxmgUTY4wxXrNgYowxxmsWTIwx\nxnjNgokxxhivWTAxxhjjNQsmxhhjvGbBxBhjjNcsmBhjjPGaT4OJiEwWkXgR2Skiv6sgvZeILBOR\nTSKyQkTCSqT9WUR+FJFtIvKCiIgvy2qMMab2fBZMRMQfeAm4CIgArhKRiDLZngHeVtUo4FHgSfe5\nZwFnA1HAEGAUMN5XZTXGGOMdX9ZMRgM7VXW3quYB7wPTyuSJAL5yP19eIl2BZkAQEAwEAod9WFZj\njDFe8GUw6Q4klnh9wH2spDhghvv5dKCliLRX1TU4weWQ+/GFqm4rewMRuUlEYkUkNiUlpc7fgDHG\nGM80dAf8PcB4EdmA04x1ECgUkXBgMBCGE4DOF5Fzyp6sqq+o6khVHdmxY8f6LLcxxpgSAnx47YNA\njxKvw9zHiqlqEu6aiYiEAperaqqI/B+wVlUz3GmfA2cCK31YXmOMMbXky5rJeqC/iPQRkSBgFrCw\nZAYR6SAiRWX4PfC6+/l+nBpLgIgE4tRayjVzGWOMOT34LJioagFwB/AFTiCYr6o/isijInKpO9u5\nQLyIJACdgcfdxz8EdgGbcfpV4lR1ka/Kaowxxjuiqg1dhjoxcuRIjY2NbehiGGNMoyIi36vqSG+v\n09Ad8MYYY5oACybGGGO8ZsHEGGOM1yyYGGOM8ZoFE2OMMV6zYGKMMcZrFkyMMcZ4zYKJMcYYr1kw\nMcYY4zULJsYYY7xWbTARkV+KSNv6KIwxxpjGyZOaSWdgvYjMd+/pbnuxG2OMKaXaYKKqDwD9gX8D\nPwd2iMgTItLPx2UzxhjTSHjUZ6LO0sLJ7kcB0Bb4UET+7MOyGWOMaSSq3WlRRO4CrgOOAq8B96pq\nvntTqx3Afb4tojHGmNOdJ9v2tgNmqOq+kgdV1SUil/imWMYYYxoTT5q5PgeOF70QkVYiMgZAVW0r\nXWOMMR4Fk5eBjBKvM9zHjDHGGMCzYCJaYm9fVXXhWfOYMcaYnwhPgsJuEbmTU7WR24DdviuSMcaY\nslSVk9kFpGTkkpKey9EM53Hx0K50btWsoYvnUTC5BXgBeABQYBlwky8LZYwxPwWqSlp2PkczcklJ\nz3P/PRUojmacOnYsI4+8Qle5a/TtGNo4gomqHgFm1UNZjDHmJ+FYRi43vfM9mw6kkl+o5dID/IT2\noUF0CA2mQ2gw/Tu1pEPLIDqGBtOxZXDx8Q6hQbRtEdQA76A8T+aZNANuACKB4vCnqr/wYbmMMaZJ\nys4r5Ia3Ytl26CS/OLsPnVo1o0OoEyg6uANFm+aB+Pk1rpWrPGnmegfYDkwCHgWuAWxIsDHG1FBB\noYtfzv2BTQdSeXn2CCZFdmnoItUZT0Zzhavqg0Cmqr4FTAHG+LZYxhjTtKgqDy74kS+3HeGRaUOa\nVCABz4JJvvtvqogMAVoDnXxXJGOMaXr+/tVO5q7bz23n9uPaM3o1dHHqnCfNXK+49zN5AFgIhAIP\n+rRUxhjThHwQm8hflyYwY3h37p00sKGL4xNV1kzcizmeVNUTqvqNqvZV1U6q+i9PLu7e/yReRHaK\nyO8qSO8lIstEZJOIrBCRMPfx80RkY4lHjohcVqt3aIwxJagquQWF9Xa/FfFH+N3HmzmnfweemhFF\nU90Sqspg4p7tXqtVgUXEH3gJuAiIAK4SkYgy2Z4B3lbVKJzO/Sfd912uqjGqGgOcD2QBS2pTDmOM\nKentNfsY+sclvPXtXkos7uETmw+kcdu7PzCwc0tenj2CoICmu1O6J+/sSxG5R0R6iEi7oocH540G\ndqrqblXNA94HppXJEwF85X6+vIJ0gCuAz1U1y4N7GmNMpVSVN7/dC8DDC3/kxrdiOZaR65N77T+W\nxfVvrqNtiyDevH4UocFNexUqT4LJlcDtwDfA9+5HrAfndQcSS7w+4D5WUhwww/18OtBSRNqXyTML\nmFvRDUTkJhGJFZHYlJQUD4pkjPkpW7fnOHuOZvL4ZUP449QIVu48yuTnV7JyR91+fxzPzGPOG+so\ncClv/WI0nU6DGeq+5sm2vX0qePSto/vfA4wXkQ3AeOAgUNyYKSJdgaHAF5WU7RVVHamqIzt27FhH\nRTLGNFXzYhMJDQ5gSlRXfn52HxbcfjZtmgdy7b/X8cRn28grKL9cSU1l5xXyizfXk5SazWvXjSS8\nU2gdlPz058kM+OsqOq6qb1dz6kGgR4nXYe5jJa+RhLtmIiKhwOWqmloiy0zgE1XNxxhjvHAyJ5/P\nNh9i+rAwWgQ5X32Du7Zi4R1jefyzrbzyzW7W7DrG87Ni6NuxdgHAmZS4gbgDqbx8zQhG9vakR6Bp\n8KSZa1SJxznAH4FLPThvPdBfRPqISBBOc9XCkhlEpIN7xBjA74HXy1zjKipp4jLGmJpYFJdETr6L\nK0f1KHW8eZA/j102lFeuHUHiiSwueXEV82MTa9w5r6o8tPBHvtx2mEcujWTykKY1KbE6niz0+MuS\nr0WkDU5nenXnFYjIHThNVP7A66r6o4g8CsSq6kLgXOBJEVGcPpnbS9ynN07N5mtP34wxxlRm/vpE\nBnZuSXRY6wrTJ0Z2ISqsDXfP28h9H27i64QUnpg+lNbNAz26/kvLd/Led/u59dx+XHdm7zoseeNQ\nm+EFmUAfTzKq6mfAZ2WOPVTi+YfAh5Wcu5fyHfbGGFNj25NPEncgjQcviahynkeX1s34z41j+Nc3\nu3h2SQIb96fyt1kxjKqmueqD2ESeWZLA9GHdua+JTkqsTrXNXCKySEQWuh+fAvHAJ74vmjHG1I15\n6xMJ9BemD6v+96m/n3DbueF8eOtZBPgLV/5rDc8tTaCggr1EAL5OSOH3H29mbHgHnr686U5KrI4n\nNZNnSjwvAPap6gEflccYY+pUbkEhn2w4yMSILrQL8Xzvj5gebVh85zk8tGALzy/bweqdR/nbrBjC\n2rYozrP5QBq3/ud7+nduycuzhzfpSYnV8eSd7we+U9WvVXU1cMzdn2GMMR7LL3Txf2/H8kFsYvWZ\n69DSrYdJzcpnZpmOd0+EBgfw7MwYnp8Vw/bkdC56fiWL4pIASDyexfVvri+elNiymWd9K02VJzWT\nD4CzSrwudB8b5ZMSGWOapHfX7mPp1sOs23OcSUO60KqevnznrU+kW+tmjA3vUOtrTIvpzrAebblr\n3gZ+OXcDK+JT2LD/BPmFLt6/acxpsW1uQ/OkZhLgXg4FAPfz02OfSGNMo5CWlc/flu2gf6dQ0rLz\neXP13nq574ETWazaeZQrRvbA38udC3u2b8H8m8/kl+eH8/GGAxxIzea1OSMJ79SyjkrbuHlSM0kR\nkUvdQ3kRkWnAUd8WyxjTlLz/Q9CYAAAgAElEQVTw1Q7SsvN578YzeHZpAq+u3M2cs3p7POy2tj78\n3une/dmIsDq5XqC/H7+ZOJAJgzujOP0qxuFJzeQW4A8isl9E9gO/BW72bbGMMU3FnqOZvL1mL1eO\n7EFEt1b8akJ/0nMKeH3VHp/e1+VSPog9wNn9OtCjXYvqT6iB6B5tLJCU4cnaXLtU9QycFX4jVPUs\nVd3p+6IZY5qCJz/bRpC/H7+eOACAId1bMzmyC6+v2kNalu9WSlq96ygHU7Nr1fFuas6TeSZPiEgb\nVc1Q1QwRaSsij9VH4YwxjduaXcdYsvUwt50XTqeWpzqp75rQn/TcAl5btdtn9563PpHWzQOZGNHZ\nZ/cwp3jSzHVRycUXVfUEcLHvimSMaQoKXcpji7fSvU1zbhhbetGMwV1bMWVoV15ftYcTmXmVXKH2\nTmTmseTHw0wf1p1mgf51fn1TnifBxF9EgoteiEhzILiK/MYYw0c/HODHpJPcN3lghV/od03oT1Z+\nIa+srPvayX83HiSv0MXMkdbEVV88CSbvAstE5AYRuRFYCrzl22IZYxqzzNwCnvkinmE923BpdLcK\n8wzo3JJLorrx1rd763S3Q1Vl3vpEhnZvTUS3VnV2XVM1TzrgnwYeAwYDA3FWAe7l43IZYxqxf329\niyPpuTwwpeqFFe+6IJzs/EJe+abuaiebD6axPTndOt7rmacLyRwGFPgZcD6wzWclMsY0akmp2byy\ncjdTo7sxolfbKvOGd2rJtOhuvL1mHynpdVM7mbc+keAAv0prRMY3Kg0mIjJARB4Wke3AizhrdImq\nnqeqf6+3EhpjvPb9vuNc89paklKzfX6vv3wRj0vht5M9W4r9zgv6k1tQyL++3uX1vbPzClm4MYmL\nh3b1+YRIU1pVNZPtOLWQS1R1rKq+SIn92Y0xjccXPx5m9c5jzH7tuzqrAVQkLjGVTzYc5MaxfUqt\nrluVvh1DuWxYd95Zu48jJ3O8uv/nWw6RnltgHe8NoKpgMgM4BCwXkVdF5ALgp7lQvzGNXHxyOh1b\nBnMoLYdr//0dqVl1PxxXVfnTp1vpEBrEref2q9G5d57fnwKX8rKXtZN56xPp1b4FZ/T96ey9frqo\nNJio6n9VdRYwCFgO/AroJCIvi8jE+iqgMcZ78cnpjA3vwKvXjWR3SiZzXl9Hek7dzj7/fEsysftO\n8JuJA2u8HHvvDiHMGNadd7/bz+Fa1k72Hs3kuz3HmTmyx092g6qG5MlorkxVfU9VpwJhwAac9bmM\nMY1AWlY+ySdzGNilJWP7d+Cla4azJekkN7wVS3Ze3bRc5+QX8uTn2xjUpWWtm5h+eX5/XC7lH8tr\nt1rT/NhE/AQuH143izqamqnRtmCqekJVX1HVC3xVIGNM3dqefBKAgV2cpdIvjOjMc1fGsH7vcW7+\nz/fkFngfUN76di+Jx7N5YEpErZd679m+BVeMCGPuukQOpdVsoEBBoYsPvz/AuQM70aW17S3SEH66\ne0wa8xMRfzgdgEFdTu27cWl0N56eEcU3CSncOXdDpfube+JoRi5//2on5w/qxNj+td+ACuD288JR\nlJdqWDv5OiGFI+m51vHegCyYGNPEbU9Op1WzALqU2Q1w5qgePDw1gi9+PMw9H8Thcmmtrv/c0gSy\n8gv5w8WDvS5rj3Yt+NnIHsxbn8jBGgxjnrc+kQ6hQVwwuJPXZTC1Y8HEmCYuPjmdQV1aVdgpff3Z\nfbh30kD+uzGJBxZsQbVmASXhcDpz1+1n9piehHcKrZPy3n5eOILw9688q52kpOfy1fYjzBgeRqC/\nB19pyVvgf7+HjCNeltSUZMHEmCZMVUlITmdAl8q/6G8/L5xbz+3He9/t5/HF22oUUB5bvI3Q4AB+\nNWFAXRQXgO5tmnPlqB58EJtI4vGsavN//MMBClzqWROXKiz+Daz9B/zjTNj2aR2UuAG4XJB2EPZ9\nCzlpDV0awLNte40xjdTB1GzScwsY2KXqBQ/vmzSQrNwCXlu1h5DgAO6+sPrgsCL+CN8kpPDAlMG0\nDQmqqyIDcNt5/ZgXm8jfv9rJ01dEVZpPVZkXm8iIXm09qxntXg6Ja+HMO2DvSph3DcTMhslPQrPT\nbFHI7BNwYi+c2Aep+0o/T90Phe65QtctgL7nNlw53SyYGNOEJVTQ+V4REeHhqZFk5RXy/LIdhAT7\nc9O4yiceFhS6eHzxNnq3b8F1Z/auyyID0LV1c64e3ZN31u7jtvP60at9SIX5vt93gt0pmfz5cg8m\nSarCiqegVRhc8BAg8PXTsOpZ2PsNXPZP6H123b6RqhTkOsHhxN4SwaLo+X7ILVPjaNYG2vaGzpEw\n8GLnedte0DW6/spcBQsmxjRh25OdYDKgc9XBBMDPT3jq8iiy8gt54rPttAgKYPYZFS8QPnd9IjuO\nZPCva0cQFOCb1vLbzu3H3HX7efGrnTzzs4q/MOetTyQkyJ8pUV2rv+CuryDxO5jyLAS4t2S64EEY\nMAk+vgnenAJn3QHnP3gq3RcyjsCal2D9vyEv/dTxgGbQpqcTJHqc4QSKtr2hTS/nebPWvitTHfBp\nMBGRycDzgD/wmqo+VSa9F/A60BE4DsxW1QPutJ7Aa0APnBWLL1bVvb4srzFNTXxyOt1aN/N40UN/\nP+G5mTHk5BXy4IIttAjyZ0aZSYBp2fk8tzSBMX3a+XRL3E6tmjH7jF68sXoPt58XTp8OpWsnGbkF\nLN58iKlR3QgJruarrGStZNjs0mk9RsMtq2DJA/Dti7DzK5jxCnQZUrdvKHU/rH4BNrzjNFFFXAYD\nJp+qYYR0Ar/G243ts5KLiD/wEnAREAFcJSIRZbI9A7ytqlHAo8CTJdLeBv6iqoOB0YANvTCmhuKT\n04snK3oqKMCPl64Zzpl923PPB3H8b8uhUun/WL6TE1l5PHhJ1XuVVCgv0+k89tAt4/sRFODHi8t2\nlEv7NC6JrLxCz/Yt2bUMDqyDcb+puNYRHApT/wZXz4fMFHjlXFj1HLjqYIWAlHj45FZ4YRh8/yZE\nzYQ7YuFnb0D0ldBzDLTs0qgDCfh2NNdoYKeq7lbVPOB9YFqZPBHAV+7ny4vS3UEnQFWXAqhqhqpW\nP6zDGFMsv9DFrpSMajvfK9Is0J9XrxtJTI82/HLuBlbEO7/l9h/L4o3Ve7l8eBhDutew2eXoDngu\nEhb90uNTOrYM5roze/PfjQfZlZJRKm1ebCLhnUIZ3rNN1RcpqpW07uF0tldlwCS4bS0MvAi+/KPT\n9HVir8flLSVpA8ybDS+NgR8/gdE3wV1xcOmL0L5mC2E2Br4MJt2BxBKvD7iPlRSHszoxwHSgpYi0\nBwYAqSLysYhsEJG/uGs6pYjITSISKyKxKSkpPngLxjReu1MyyS/UajvfKxMSHMAb149mQOeW3PzO\n96zdfYyn/rcNfz/h3kme7VVSLPsEvHclZKfChv/A7q89PvWmcX0JDvDnhRK1kx2H09mwP5UrPVnU\ncecyOLAezvkNBHgw6iykPcx8G6b/Cw7/CC+fDT+84wSl6qjC3lXwznSndrP7Gxh3D9y9xRkx1rrs\nV2DT0dD1qnuA8SKyARgPHMTZMyUAOMedPgroC/y87MnudcJGqurIjh071luhjWkMyq7JVRutmwfy\n9i9G06NdC37+xjo+25zMLeP70blVDda/KiyAD653+gyu/QTa9YVP74Z8z1YH7hAazJyzerMwLokd\n7tFp89YnEuAnTB9ezZezKqx4Elr3hJhrPC+zCETPgltXQ7dhsPAOeP9qyKjkR6sqJHwBr09yajPJ\nm2HCH50gcv4DEOLdMjONgS+DyUGczvMiYe5jxVQ1SVVnqOow4H73sVScWsxGdxNZAfBfYLgPy2pM\nkxOfnI6/n9C3Y8XDaj3VPjSYd28cQ6eWzejWuhk3jetbswssecCZ33HJs9DvPGc01fFdzpBcD900\nri8tAv15ftkO8gpcfLzhIBMGd6ZDaDWjrnZ+CQdj3X0ltZgL06YnXLcQJj3h1HD+cQZsX3wq3VUI\nWz6Cf54D782Ek4fg4mfgV5th7N2n39wVH/LlaK71QH8R6YMTRGYBV5fMICIdgOOq6gJ+jzOyq+jc\nNiLSUVVTcHZ8jPVhWY1pchIOp9O3QwjBAeVaiGusc6tmfH7XOeQWuGgeVIPrff8WfPcyjLkVhl/n\nHOt3HkRdCSufhSFXQMfqJ0i2Cwni52f35h8rdhHeKZTjmXlcWV3He8laSfTVVeetip8fnHk79D0P\nPrnJqaEMmw1ho2D183B8N3QY4MxTGXoF+P80twv2Wc3EXaO4A/gC2AbMV9UfReRREbnUne1cIF5E\nEoDOwOPucwtxmriWichmnB0eX/VVWY1pirbXYiRXVUKCA2hXk5nu+751li7pdz5MfKx02sTHISgE\nPv2VZ30RwP+d05eQoAD+9uUOurRqxrgB1TRt71gKB793+ixqUyspq3ME3PgVjP01bHwPFt0Fwa1g\n5jtw23cQc9VPNpCAj+eZqOpnwGdljj1U4vmHwIeVnLsUqHwdBWNMpTJyCzhwIptZngyb9YUT+5yR\nTG17wRVvgH+Zr5rQjnDho7DoTtj4bvm5HxVo0yKIX5zdmxe+2skVI8Kq3jdFFVY84TRTxXhRKykr\nIAgmPAxDLnfWxOp1ltO/YmwGvDFNUbx75ntthgV7LTfDaQoqLICr3ofmlQzdHXYtxM2FJQ/CgIuc\nUVTVuHFcX07mFDDnrN5VZ9yxxBmae+mLvqkt1PWExiagoUdzGWN8oCiY1HZYcK25XPDJzXBkqzMp\nr0P/yvP6+cElf4PcdKeT3gOtmgXyx0sj6diyio73or6SNr0g+qoavgFTWxZMjGmC4pNPEhLkT/c2\nzev3xiuegO2fOn0i4R7s7t1pEJx9F8S9B3u+qZsyJHzh1ErG3fuT7sOobxZMjGmCtienM6BLS/xq\nuR97rWz5CL75i9P/ccatnp837h5o26dGc08qVVQradvbmSdi6o0FE2OaGFUl/nB6/TZxHfwB/nsb\n9DzTmUdSk07pwObOHJRjO531sLyR8D84tNFqJQ3AgolpstKy85m/PrHWe5s3VkfSc0nNyvdo2fk6\nkZ4M718DIR2dYbK1Wb693/kw9GfORMaj5Rd19EhxraQPRFmtpL5ZMDFN1qvf7Oa+jzaxZvexhi5K\nvTo1kqsegkl+jjNyKycNrprrDPmtrUlPOLWUT+/2eO5JKfGfw6E4d63EBqrWNwsmpklSVRZtSgLg\n4x8OVpO7aTk1ksvHw4JVnXkiB7+HGf+CLkO9u15oJ2fuyd6VzpDhmpaluFZypXflMLViwcQ0SZsP\nprHvWBYdQoP4fMshsvIKGrpI9WZ7cjodWwaXnq1+MslZhn3nMu87uYusfh42zYPz7ofBU+vmmsOu\nc3YZ/OJ+yKxBjTL+M0jeBOPvs1pJA7FgYpqkRXFJBPoLT0wfSlZeIUt+PNzQRao38YdPlu98//5N\n55f7f2bA073h3Z/Bd/+CY7tq2aT0P2e/j8jpTrNSXfHzczapyj0JSx/07JyiWkm7fjB0Zt2VxdSI\nBRPT5LhcyqebDjF+QEcmDO5MWNvmfPTDgYYuVr0odCk7DmcwsGzne9IGZzHCaz6EEXOcIPL5ffDi\ncHghxllDK/5zZ/Z6dY5sg49ugK5RMO0fdb+cSKfBcJZ7mZU9K6vPv32xs+S71UoalAUT0+TE7jvB\nobQcpkZ3w89PmD6sO6t3HuXwyTpq3jmN7T2WSW6Bq3TnuyokbYTuI6H/hXDR03DnD3DnBme59I6D\nYeNcmDvLqbW8NdVpwjr8Y/laS9ZxJ19QCMyaC0EtfPNGxt3rzBX59FdQkFt5PpfLab5r189Zgdg0\nGAsmpslZFJdEs0A/JgzuDMD0Yd1xKSzY2PQ74ivsfD+ZBJlHoFtM6czt+sLo/4Or34ff7nH27Tjj\nVqevYulD8PJZ8OxgWHC7s+1sRgrMv87Zs+PKd327a2BQC5jyV/fck79Vni9+MRzeDON/a7WSBmaf\nvmlSCgpdfLb5EBcM7kxIsPOfd9+OocT0aMPHPxzkpnFNb+/tkrYnpyMC/TuHnjp4aKPzt9uwyk8M\nCIa+453HxD85AWjnMmdzqW2LnK12i0z/F/QY5Zs3UFL4BKe2sfIZZ5XeDuGl04tqJe3DnXTToKxm\nYpqUb3cd41hmHlOjupU6fvnw7mxPTmdr0skGKln9iE8+Se/2ITQLLLGBVdIGED/oXIOVblt1g+HX\nwsy34N7d8IslMO4+Z3Z7fS5TMukJCGhe8b4n2z+Fw1usVnKasGBimpRFcUmEBgdw7sDSk+cuiepG\noL/wcRPviE+orPO94+Da92/4B0DPMXD+/TDqBu8LWRMtO8OFf3TPPXn/1PHiWkl/q5WcJiyYmCYj\nt6CQ//2YzMTIzqV/mQNtQ4I4b2AnFsQlUVDoaqAS+lZ2XiF7j2VW3PleVRPX6W74zyFsNHzxh1Nz\nT7YthCM/OrUSP++3JTbes2BimoxvEo6SnlPApdHdKkyfMTyMlPRcVu08Ws8lqx87jqSjWmYPk7QD\nkHW0fOd7Y1Jq7slDTq3k66edoc5DZjR06YybNTSaJmNhXBJtWwRydniHCtPPG9SR1s0D+fiHg5w7\nsFM9l873tle0JlfSBudvY66ZAHSOhDPvgNV/g+BQZ/Oty/9ttZLTiNVMTJOQlVfAl1sPc9HQrgT6\nV/yfdXCAP1Oju7JkazLpOfn1XELfi09Op1mgH73ah5w6eGgj+AU4X8aN3fjfOnu6f/dP6DDQmX1v\nThsWTEyTsGzbEbLzC8uN4ipr+rAwcvJdfL4luZ5KVn/ik9Pp36kl/iU3xCrqfA+s5x0XfSGoBVzy\nHPgHwfkPWK3kNGPBxDQJi+KS6NwqmNF92lWZb3jPNvRu34JPmuBKwtuT0yvpfG/E/SVlhU+A+/ZA\nxKUNXRJThgUT0+ilZeezIj6FKUO7lf5VXgERYcbwMNbsPsaBE1n1VELfO5aRy9GM3NKd76n7Ift4\n4+8vKSs4tPo8pt5ZMDGN3pIfk8krdDE1uqtH+acPc5YBWbAxyZfFqlfxh53O91K7KxZ3vjehmok5\nbVkwMY3eok2H6NGuOTE92niUv0e7Fozu3Y6PfziA1mb59RoqdKnP73NqTa4SweTQRvALrNnMd2Nq\nyYKJadSOZeSyeudRpkZ1Q8ouhb7/O3htgrOlbBnTh3dnV0ommw6UT6tLmbkFXPjs1/z5i3if3ic+\nOZ22LQLp2LLE/utJG5zl3GuzJ7sxNWTBxDRqn29JptClTK1oouJ3L8OB9bBjabmki4d2JSjAj082\n+LYj/q9LEth9NJMPYhN9OvO+qPO9OKA2hZnvplHxaTARkckiEi8iO0XkdxWk9xKRZSKySURWiEhY\nibRCEdnofiz0ZTlN47UwLonwTqHldxbMOels9gQVBpPWzQO5MKIzC+OSyPfRl3xcYipvfruHAZ1D\nOZqRx9rdx31yH5dLSTicXnrZ+RN7ISfVgompNz4LJiLiD7wEXAREAFeJSESZbM8Ab6tqFPAo8GSJ\ntGxVjXE/bBygKedQWjbr9x6vuIlr+6dQkONMbtu5FFyF5c6fMaw7xzPz+Do+pc7Lll/o4ncfb6Zj\ny2DevfEMQoMDWBTnmw7/AyeyycorrGTmu3W+m/rhy5rJaGCnqu5W1TzgfWBamTwRwFfu58srSDem\nUos3HUKVikdxbZrv7NQ37l7IOgYHfyiXZdyAjrQPCeLjDXW/kvBrK/ew7dBJHrl0CB1bBjMxojOf\nbzlEXkHd14K2JzvL6pcLJv5B0Kns7zdjfMOXwaQ7kFji9QH3sZLigKKV2qYDLUWkvft1MxGJFZG1\nInKZD8tpGqlFmw4xpHsr+nYsM+8g/TDs+RqG/gzCL3D28tjxRbnzA/39mBrdjS+3HiEtq+6WV9l3\nLJO/fZnApMjOTB7SBYCp0d04mVPAyh11XwsqGslValjwoY3OEirW+W7qSUN3wN8DjBeRDcB44CBQ\n1B7RS1VHAlcDfxORclvkichN7oATm5JS9/+TmtPXvmOZxCWmVrx8ypaPQF1OMGnRDnqMgYTywQTg\n8uFh5BW6WLz5UJ2US1X5wyebCfL345FLTw3JPTu8A21aBPqkqWv74XR6tGtOqHtnSafzPQ66WhOX\nqT++DCYHgR4lXoe5jxVT1SRVnaGqw4D73cdS3X8Puv/uBlYA5XoSVfUVVR2pqiM7duxYNtk0YZ9u\ncr78p0RV0MS1eT50iYKOA53X/SdC8iZn7/IyhnRvRf9OoXW2adZHPxxk9c5j3HfRILq0blZ8PCjA\nj4uGdGHp1sNk55Xvv/FGQnJ66Q2xju+G3DTrfDf1ypfBZD3QX0T6iEgQMAsoNSpLRDqISFEZfg+8\n7j7eVkSCi/IAZwNbfVhW08gsiktiRK+2hLUts3vg0Z1Of0HUzFPH+k90/u4sP6pLRJg+vDux+06w\n71imV2U6lpHLY4u3MqJXW64Z3bNc+tSobmTmFbI8/ohX9ykpt6CQ3Uczm+ay86ZR8VkwUdUC4A7g\nC2AbMF9VfxSRR0WkaHTWuUC8iCQAnYHH3ccHA7EiEofTMf+UqlowMQAkHE5ne3J6xZtgbZ4PSOmt\nXDtHQqvulTZ1XRbTHRG8nnPyp0+3kplbwFMzhuJXwRphY/q2p0NocJ02de06kkmhSxlYclhw0gbw\nD3YmLBpTT3y6OZaqfgZ8VubYQyWefwh8WMF53wJDfVk203gtikvCT+CioV1KJ6g6o7j6nAOtSgQa\nEad2svkDKMgt1yndrU1zzuzbnk82HOSuC/qXH2bsga8TUvjvxiTuvKA//cvuwe7m7ydcEtWVuev2\nk56TT8tmgTW+T1nxh52RXKWXUYmDLkPA3/vrG+Ophu6AN6ZGVJVFcUmc2a89nVo2K5148Hs4sQeG\nzix/4oBJkJcB+76t8Lozhoex71gWP+w/UeMyZeUVcP8nm+nbMYTbzys3TqSUqdFdyS1w8eW2wzW+\nT0W2J6cT6C/06eDeEMvlcma+W+e7qWcWTEyjsuXgSfYey6p4FNem+U7zTkV7XfQZ56TtWFLhdScP\n6UKzQD8+qsU+J88tTeDAiWyemhFFcEDVGzYN69GW7m2asyiubkaPxSen069j6KndJY/vgrx06y8x\n9c6CiWlUFm1KItBfiudvFCssgB8/dmogzVqXPzEoxGn+qqTfJDQ4gMmRXfg0LoncAs9HW205mMa/\nV+3hqtE9q92YC8DP3dT1TUIKqVl5Ht+nMvHJ6aWbuJI2On8tmJh6ZsHENBoul9PENa5/R9q0CCqd\nuHsFZKaUHsVVVv9Jzi/3Y7sqTJ4xPIyTOQV8tc2z0VYFhS5++9Em2ocG87uLBnn4LpwJjAUu5X9e\nbh2clpXPobSc8p3vAc2go+flMaYuWDAxjcb3+09wKC2n4hWCN893aiRFw4Ar0v9C528lTV1nh3eg\nU8tgPvZwVNcbq/fyY9JJHrk0ktbNPe/sjuzWij4dQli0ybtRXUUbYpXbw6TLUPD36dgaY8qxYGIa\njUVxSQQH+DEhonPphLws2PYpREyrevmQdn2gw4BKm7r8/YTLhnVn+fYjHM+sugkq8XgWzy5NYMLg\nzlxUtsmtGiLC1KiurNl1jCPpOTU6t6Ti3RWLgomr0BnJZU1cpgFYMDGNQkGhi882H2LC4M6nlg0p\nEv8Z5GdWPIqrrP4TYd9qyM2oMHn6sO4UuJvTKlO0ZIqfwKPTIms1lHhqdDdcCp9vrn1TV3zySVo2\nC6Bb0Uz7YzudEWs2kss0AAsmplFYu/s4RzPyKl4hePMHzqTEXmdXf6EBk6Awz+ljqcDgrq0Y3LVV\nlU1dCzYmsXLHUe6bPIhubZp7+A5K69+5JYO6tPRqAmO8exmV4mBmne+mAVkwMY3CwriDhAYHcO7A\nTqUTMo/Bzi+dGe9+Hvzn3PNMCG5V4SrCRS4f3p24xFR2pZSvvRzPzOPRT7cyrGcbZp/Rq6Zvo5Sp\n0d2I3XeCg6nZNT5XVYt3VyyWtAECWzhNecbUMwsm5rSXW1DI/7YkMzGiM80Cy8zj2PoJuAqqHsVV\nkn8g9DvP2X1RtcIsl0Z3w0/gkwrmnDy+eBsns/N5csZQ/CtYMqUmLnEvUrm4Fh3xh9JySM8pKDMs\neIN1vpsGY8HEnPZWJhzlZE4BU2Mqmqj4AXQcDJ2HlE+rTP9JkH7IWUm4Ap1aNeOc/h35ZMNBXK5T\nAWfVjqN89MMBbhnfr/QWubXUq30I0WGtazWBsWgPk+Jhwa5C5/1YE5dpIBZMzGlv0aYk2rQIZGx4\nh9IJJ/ZB4lqI+pmz/panwic4fysZIgwwY3h3DqZms26vs297dl4hf/hkM306hHDH+eE1fQuVmhrd\njc0H09hztGYrFm8vCiZF64AdTYD8LOt8Nw3Ggok5rWXnFbJ062EuGtL11JIhRTZ/4Pwd+rOaXbRl\nZ+cXfELlwWRiRBdCgvyL9zl5ftkO9h/P4onpQ8s3tXmhaD+WT2vYER+ffJKurZvRuoV7fostO28a\nmAUTc1pbtv0wWXmF5UdxqTrBpOeZ0Kb83iHV6j8JDqx3OvAr0DzIn4uHduWzzcl8v+8Er67czZUj\ne3Bmv/YV5q+trq2bM7p3uxpPYIw/nFGm830jBIZAh/51Wj5jPGXBxJzWFsUl0allMGP6lPkST94M\nKdtrXispMmAioM5IsEpMH96djNwCfv7GOtq2COT3F/tmiZKp0V1JOJxR3A9SnfxCF7uOZJTeXTFp\nA3SNAr+6qzUZUxMWTMxp62ROPsvjU5gS1bX8yKnN88EvACKn1+7iXYdBSKcqhwif0ac93Vo3Iz2n\ngIenRpZfD6yOXDS0K36Cx3NO9h7NJK/QdapmUljgBFdr4jINyIKJOW0t+fEweQWu8mtxuQph80cQ\nfiG0qH6l3gr5+Tlrde380vkyrjCL8JuJA7lhbJ/iYby+0CE0mLPDO7BoUxJayXDlkoo734uCydF4\nKMi2YGIalAUTc9paFIoCk3oAAAzmSURBVJdEWNvmDOvRpnTCvtWQnuSM4vJG/4mQk+b0nVTi8hFh\nPHhJRK2WTAGgIA9ePR9WPFVltqlR3dh3LIvNB9OqvWR8cjr+fkJ4p1DnQFHnu43kMg3Igok5LaVm\n5bFq51GmRncr/0W+aT4EhcKAi7y7Sb/znKayKpq6vLb+VWcHyG//DjknK802KbILgf7iUVPX9uR0\n+nQIObURV9JG5/NoX3dDlo2pKQsm5rS0ITGVQpdyTv8yc0vyc2DrQhg8FYJaeHeTZq2d0WBVDBH2\nSuYxWPE0dBjo7H648b1Ks7ZuEcj4AR35dNOhUhMlKxJ/+GT5ZVS6xni2nIwxPmL/9ZnTUlxiKiIQ\nFVamiWvHEshNq/0orrL6T4QjP0JqYt1cr6QV/9/emQdJWVwB/PfYXQ5ZlGPDCmLEhfUAZAkS8Ayo\n1IJU0AQtxZgEr1hiSOkfxqBWWZaJWpqjEu9CxZCUSSAaEyqlEYJ4JBE8cJflXkAMC8sC4UZk2fXl\nj+6Fj2G+YXfn++ZI3q9qanq6+5vvTU9Pv+n3ul8/7KL4XjML+o2Exc86f08IEyv6Ur/7cz5KcQ79\nvoNNbNxxgLNaVnI1H/LOdzNxGdnFlImRk1Rv3EV57+Jjw83XzHGrsE4fHc2NzhjnnlPshm8XDSvg\nw5kw4ibofTacNxV2fpLyPmPPLqVzUYeUpq41DQnO922roPmgOd+NrGPKxMg5VJXqut1UJM5KDuxy\nB1sNuSq6YIYlZ0D306JVJqrwxr3QqRuMucflnT3Rhclf9HToZV07FXLZWaW8VlNPU/MXSeus2dJy\nuqKPyWXOdyNHMGVi5Bx1Ow+wY38jQxNXca2c684iSXcVVxARNztZ/zYcanso+KTUzoP1C2H0dOjq\nN1sWFMHI78En70DD8tBLJ1b0Yfu+Rhat35G0fNWWvZzQsYB+Pfw5KpurXEj9nmXRyG4Y7cSUiZFz\nVG3cBcCwxJnJ0jnQcwD0HR7tDcvHuX0aG/6Z/ns1H3Kzkl4D4au3HF02fAoUdoFFz4RePubM3hR3\nKgw1da3espfy0m50aNnEuflj6FNhzncj61gPNHKOpXW76FjY4egVS3s2w4Z/uHNL2rvnI4z+F7pB\nPoolwh88747PrXwIChN2zJ/QEyomO6W4f3vSyzsXFVA5qJTXl9XT2HS0qUtVWd2w94jzvakRGpaZ\n893ICUyZGDlH9cbdDO57Ih0LA92z5mVAo1vFFaSoC5SNdv6YVuxAD+WzHfDWI1B2yRHHfiKjbnMO\n849eDH2biRV92fN5E+/Wbjsqf9u+g+zY3xhwvq90Zj9zvhs5gCkTI6doav6Cmk1JnO81c+CUc6HX\ngHhuXF4Juz5154K0l7cegYN7YdzD4bOn3mfBgEvhgxfczCIJFw4sofsJRceYulYfdr57ZWJh540c\nIlZlIiLjRWS1iKwVkelJyk8TkQUislRE3hKRfgnlJ4pInYg8GaecRu5Qu3UfBw41MyzofN+6yu2l\nOKeVR/O2h/JK97ymnaauraucgjj3RigdlLruqKnupMcVf0la3LGwA5cPOZn5Kxo40HhkX8rqxJhc\nmz92Gy97nN4+mQ0jQmJTJiJSADwFXA4MAq4TkcRf2c+A36jqUOBB4JGE8h8D78Qlo5F7VHvne0VQ\nmdTMASmAIZPiu3H3U6H34PYvEZ53nwtpcsm9x687cKxz0C96OtSsNnFoX/Y3NrNw9dbDeau27KWk\nuBO9iju5jM1Vbklw1D4kw2gHcc5MRgJrVXW9qjYCfwCuTKgzCHjTpxcGy0XkXKAUiCnWhZGLVNft\n4sTOhfTv5UOltByCVTYGinvHe/MzKuHf77ngj22hdr6LPjz6buhacvz6HTo438nmJaFBJkeV9eJL\n3Toxt+qIqWtNw94jJq6mg26JsTnfjRxBWhPyul1vLHI1MF5Vb/GvvwOMUtVpgTq/Axar6q9EZBLw\nClAC7MQpmW8DY4ERwesC198K3OpfDgGWxfJhoqUESL6UJ7cwOaPF5IyWfJAzH2QEOFNVux2/Wmoi\n2kbcbu4CnhSRG3DmrE1AM3A78Jqq1qUK/a2qM4AZACLyoaqOiF3iNDE5o8XkjBaTMzryQUZwckbx\nPnEqk03AqYHX/XzeYVR1MzAJQESKgatUdZeInA9cLCK3A8VARxHZp6rHOPENwzCM7BOnMvkAKBeR\n03FKZDLwrWAFESkBdqjqF8A9wEwAVb0+UOcGnJnLFIlhGEaOEpsDXlWbgGnAG8BKYI6qLheRB0Xk\nCl9tDLBaRNbgnO0PpXHLGenIm0FMzmgxOaPF5IyOfJARIpIzNge8YRiG8f+D7YA3DMMw0saUiWEY\nhpE2eadMWhGipZOIzPbli0WkfxZkPFVEForIChFZLiJ3JKkzRkR2i0iVf9yfaTm9HBtEpMbLcMwS\nQXE87ttzqYhEHP+9VTKeGWinKhHZIyJ3JtTJSnuKyEwR2SoiywJ5PUVkvojU+uceIddO8XVqRWRK\nFuT8qYis8t/rqyLSPeTalH0kA3I+ICKbAt/thJBrU44NMcs4OyDfBhGpCrk2k22ZdByKrX+qat48\ngAJgHVAGdASqgUEJdW4HnvXpycDsLMjZBxju092ANUnkHAP8NQfadANQkqJ8AvA6IMB5uE2m2e4D\nW4DTcqE9ga8Bw4FlgbzHgOk+PR14NMl1PYH1/rmHT/fIsJyVQKFPP5pMztb0kQzI+QBwVyv6Rcqx\nIU4ZE8p/DtyfA22ZdByKq3/m28ykNSFargRm+fTLwGWSaudjDKhqvaou8em9uNVsp2RShgi5Ehc/\nTVV1EdBdRPpkUZ7LgHWq+mkWZTiMqr4DJB6LGOyDs4BvJLl0HDBfVXeo6k5gPjA+k3Kq6jx1qy4B\nFuH2gmWVkPZsDa0ZGyIhlYx+rLkG+H0c924LKcahWPpnvimTU4CNgdd1HDtIH67jfyi7gV4ZkS4J\n3sz2FWBxkuLzRaRaRF4XkcEZFewICswTkY/EhadJpDVtnkkmE/5DzYX2BChV1Xqf3oJb9p5IrrXr\nTbgZaDKO10cywTRvjpsZYpbJlfa8GGhQ1dqQ8qy0ZcI4FEv/zDdlkleI29X/CnCnqu5JKF6CM9VU\nAE8Af860fJ6LVHU4Lrrz90Xka1mS47iISEfgCuCPSYpzpT2PQp3NIKfX34vIfUAT8FJIlWz3kWeA\nAcAwoB5nRspVriP1rCTjbZlqHIqyf+abMjluiJZgHREpBE4C/pMR6QKISBHuC3xJVf+UWK6qe1R1\nn0+/BhSJiwiQUVR1k3/eCryKMxcEaU2bZ4rLgSWq2pBYkCvt6WloMQX6561J6uREu4qLMPF14Ho/\nsBxDK/pIrKhqg6o2q4uU8VzI/bPenn68mQTMDquT6bYMGYdi6Z/5pkwOh2jx/1InA3MT6swFWlYe\nXA28GfYjiQtvN30BWKmqvwipc3KLL0dERuK+i4wqPRHpKiLdWtI4h2xi5OW5wHfFcR6wOzBFzjSh\n//pyoT0DBPvgFCDZKVhvAJUi0sObbSp9XsYQkfHA3cAVqvpZSJ3W9JFYSfDRfTPk/q0ZG+JmLLBK\nVeuSFWa6LVOMQ/H0z0ysKoh4hcIE3KqEdcB9Pu9B3A8CoDPODLIWeB8oy4KMF+GmjkuBKv+YANwG\n3ObrTAOW41adLAIuyIKcZf7+1V6WlvYMyim4Q87WATW4OGnZ+N674pTDSYG8rLcnTrnVA4dwduWb\ncT66BUAt8Hegp687Ang+cO1Nvp+uBW7MgpxrcXbxlj7asgqyLy5qd2gfybCcv/V9byluIOyTKKd/\nfczYkCkZff6vW/pjoG422zJsHIqlf1o4FcMwDCNt8s3MZRiGYeQgpkwMwzCMtDFlYhiGYaSNKRPD\nMAwjbUyZGIZhGGljysQw2oCINMvREYwji04rIv2DkWgNI5+I8wx4w/hf5ICqDsu2EIaRa9jMxDAi\nwJ9T8Zg/q+J9ERno8/uLyJs+SOECEfmyzy8Vd4ZItX9c4N+qQESe8+dPzBORLln7UIbRBkyZGEbb\n6JJg5ro2ULZbVc8BngR+6fOeAGap6lBcIMXHff7jwNvqAlMOx+2IBigHnlLVwcAu4KqYP49hRILt\ngDeMNiAi+1S1OEn+BuBSVV3vg+ttUdVeIrIdF/7jkM+vV9USEdkG9FPVg4H36I87Q6Lcv/4RUKSq\nP4n/kxlGetjMxDCiQ0PSbeFgIN2M+TWNPMGUiWFEx7WB5/d8+l+4CLYA1wPv+vQCYCqAiBSIyEmZ\nEtIw4sD+9RhG2+giIlWB139T1ZblwT1EZCludnGdz/sB8KKI/BDYBtzo8+8AZojIzbgZyFRcJFrD\nyEvMZ2IYEeB9JiNUdXu2ZTGMbGBmLsMwDCNtbGZiGIZhpI3NTAzDMIy0MWViGIZhpI0pE8MwDCNt\nTJkYhmEYaWPKxDAMw0ib/wKisdNsOIOSdgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f7dbd39e400>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "train_accuracy = dropout_cost[:,1]\n",
    "test_accuracy = dropout_cost[:,2]\n",
    "plt.title('Training vs Testing Accuracy')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.axis([0, len(train_accuracy), 0.94, 1])\n",
    "plt.text(1, 0.99, 'avg_gap=' + str(100 * (np.mean(train_accuracy) - np.mean(test_accuracy))) + '%')\n",
    "plt.plot(train_accuracy)\n",
    "plt.plot(test_accuracy)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, we'll run the same network without dropout:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3.241258594519445, 0.20367272727272728, 0.2064]\n",
      "[1.0230284234925897, 0.83250909090909087, 0.84250000000000003]\n",
      "[0.47223848675841551, 0.92594545454545452, 0.92220000000000002]\n",
      "[0.33825706339903183, 0.94563636363636361, 0.94230000000000003]\n",
      "[0.24510427681969427, 0.96294545454545455, 0.95689999999999997]\n",
      "[0.20726699368012619, 0.9675636363636364, 0.96089999999999998]\n",
      "[0.1688444421297628, 0.97605454545454551, 0.96809999999999996]\n",
      "[0.13739342433534793, 0.98045454545454547, 0.97160000000000002]\n",
      "[0.14015988108310357, 0.97923636363636368, 0.9698]\n",
      "[0.10289326630193114, 0.98599999999999999, 0.97409999999999997]\n",
      "[0.098108094312724117, 0.9867818181818182, 0.97399999999999998]\n",
      "[0.083263827090352932, 0.98898181818181818, 0.97499999999999998]\n",
      "[0.078170346924485587, 0.9897636363636364, 0.97489999999999999]\n",
      "[0.063356026551615971, 0.99245454545454548, 0.97550000000000003]\n",
      "[0.059958267021666671, 0.99298181818181819, 0.97750000000000004]\n",
      "[0.05715593579868361, 0.99370909090909088, 0.97470000000000001]\n",
      "[0.046878876748961101, 0.99481818181818182, 0.97699999999999998]\n",
      "[0.051069057040222625, 0.99414545454545455, 0.97570000000000001]\n",
      "[0.038401063356539519, 0.99672727272727268, 0.97829999999999995]\n",
      "[0.039848814891493431, 0.99583636363636363, 0.97829999999999995]\n"
     ]
    }
   ],
   "source": [
    "boring_network = DropoutNetwork(784,100,30,10, activations=relu_ce_activations, cost=Cost(cross_entropy_cost, cross_entropy_cost_prime), dropout=False)\n",
    "boring_cost = boring_network.train(mnist.train.images, mnist.train.labels, learning_rate=0.15, batch_size=100, epochs=20, test_epoch=1, test_images=mnist.test.images, test_labels=mnist.test.labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And plot the results, noting the gap:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f7dbd308588>]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZMAAAEWCAYAAACjYXoKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzs3XlclWX6+PHPBaKooOCu4C5qKi64\nlVpZZllmjk2LbZbt82udlmmame8002ZTTttUU045ZdNi2apZaeZuLlhauYOigCubCogIXL8/ngc8\noiDKORwOXO/X67w4z349Bz0X9/Lct6gqxhhjTGUE+TsAY4wxgc+SiTHGmEqzZGKMMabSLJkYY4yp\nNEsmxhhjKs2SiTHGmEqzZGJ8RkSCRSRbRNp5c9+aTERGichqf8dhzKkSe87EFBORbI/FBsBhoNBd\nvkNV36v6qPxPRIYDs4oXcT6bHI9duqrqztM4bxhwEGitqrsrG+cpXHcGMAZopaqZVXVdU7NZMjEn\nJCJJwK2q+l05+9RR1YKqi8r/RKQLsEVVxQvnqvJkIiJNgF1ANvAXVf13VVzXvXat+/dSm1g1l6kw\nEXlSRKaLyAcichC4XkTOEpHlIpIlIrtE5GURCXH3ryMiKiId3OX/udu/FpGDIvKDiHQ81X3d7ReL\nyGYR2S8i/xKRpSJy0wlibisih0Sksce6gSKy171mVxFZ5J4nTUTeP83Ppqkb824R2SEifxERcbf1\ndOPbLyL7ROS/7mGL3J+JbhXfpe5ro8d500TkPhFZ7x4/rfjzdbc/5t5Lsojc4X6GrcoJdTywDfgn\ncGOpewgRkb+LyDYROSAiK0Wkubutn4gsEJFM9/d8v7t+hoj80eMcJ4r/ARFZD2S46/4uIknu7/UX\nEbnYY38RkXtEZJO7/WcR6eEe806peKeKyFMn/eWYqqGq9rLXcS8gCbig1LongXycKpIgoD4wEBgM\n1AE6AZuBu9396wAKdHCX/wekAQOAEGA68L/T2LcFzl/0Y91tDwBHgJvKuJdFwESP5ReAV9z3HwOP\nuPcTCgw9yefSxflvc9z6ue556wNtgJ+B69xtM4H7carI6hdfAwhz77mVx3kuBTZ6LKe58Td373sb\ncL277QpgBxDjnuuT0uc7QZwrgL8CHYAioJvHtr8D8e7vMQjoDzQGmgLpwJ1AXXfdQPeYGcAfTxL/\ncqA1UN9dNx5oBQQDE4H9QBN320RgK9DH/by6A1FAZ+CAxznqu8d1K+te7VW1LyuZmFO1RFVnqmqR\nqh5S1VWqukJVC1R1KzAFOLec42eoaryqHgHeA/qexr6XAmtU9Qt32ws4X1pleR+4BkBEgoCr3XXg\nJKEOOFVNeaq6tLybPxER6YyTUB92P5OdwL9wvjSLr9ERaOluP9VrPK+q+1R1L/A1Rz+Hq4A3VHWL\nqmYDT5wkzu7AIOB9VU0CfgAmeOxyK/CIqm51f7+rVXU/cDmwXlVfV9V8Vd2vqqtOMf5dqnoIQFU/\nVNXdqlqoqv8F9gH9PGJ4SlXXqmOjqqaqaiKwFviNu99YYJOqbjqFOIwPWTIxpyrZc0FEuovIV271\nzgHgcaBZOcd7tg3k4vxFfar7tvGMQ1UVSCnnPB8DZ4tIS+A8IE9Vl7nbHsQp3cS7VS43lnWScrQH\nGgJpbnVfFk41Ukt3+31AI2CNiKwVkWtO8fwV+hxKvT+RG4GVqprgLr8H3CAiQSISjFN6SDzBcW3L\nWF9Rpf/N3O5+1sWfVQeO/psp71rvANe7768H3q1ETMbLLJmYU1W6x8YbwK9AF1VthFOFUunG6ZPY\nBUQXL7htE1Fl7ayq6cD3wJXAtcAHHtt2qeqtqtoauAuY4tk2U0HJOFUukaoa4b4aqeog9xrJqjoR\n58v6AWCaiLTh+M/yVB3zOeB8EZ+QWyK7Hoh1E/9unJJMW+A8VS10z9e5jPs70XpwerU18Fg+UXtN\nyX2KSA/geeAWnKqtCJwq1eJ/M+Vd6yOcPwp6AucDH5axn/EDSyamssJxvkhzROQM4I4quOYsIE5E\nxohIHZy//Juf5Jj3cf4yv5yjVVyIyFUiUpyIsnC++AqPP7xsqroF+Al4WkTC3L/0u4rIUPca40Wk\ntVuCKrmGqubgfBl3OpXrefgIuF1EOovTM+zP5ew7Aucz6uvx6gl8wdGqrjeBSSLSwb2HOLfjwqdA\nD7dEUVdEGovIAPeYNcAYd100cPdJYg7DaavZBwSJyN04JZNibwJ/EpHebmN89+Lfj6oeAL7C+WPg\nO1Xdd7IPyFQdSyamsh7E+ZI+iFNKme7rC6rqHpx2j+dxGoY743yZHy7nsM+BHsAOVV3nsX4wsEpE\ncnC+NO9S1R2nEdZVONVam3B6LX3A0QQ3FPhRnOd4PgBuce8B4P+Az9wqn9GneM0ZwDScto+NwEJ3\n/Yk+hxuB6aq62W2v2K1Od+SXgd+6yehJYI57nizgNaCuW7IbiVOy2QdsAIa45/0PsB2nRPElHon6\nRFR1JfAWzu9rJ05V3RqPXd7GaW/6BKfBfTpOFWGxd4BYrIqr2rHnTEzAc+v7dwJXqOpif8fjLyIy\nEFigqg39HYuvuNVky3B6rOX5Ox5zlJVMTEASZ9iRCBGph/PX/RFgpZ/DqnIicrn7fEhz4CngM3/H\n5CvuHw2/B6ZZIql+fJZM3AeK9orIr2VsF3EeSktwH0yK89h2o4hscV+n07vG1HzDcJ5H2AdcBIxT\n1fKquWqq3+NU9W0AMt3lGkdEWuBUew3ESZqmmvFZNZeInIMzZMM0Ve11gu2XAPcAl+DUW7+kqoPF\nGe4hHudhNQVWA/3VxhAyxphqy2clE1VdhDt8QhnG4iQaVdXlQISItMb5K3Ouqma4CWQuMMpXcRpj\njKm8On68dhTHPsyU4q4ra/1xROR24HaAhg0b9u/evbtvIjXGmBpq9erVaap6sq71J+XPZFJpqjoF\nZ/gOBgwYoPHx8X6OyBhjAouIbPfGefzZmyuVY5/YjXbXlbXeGGNMNeXPZPIlMMHt1XUmsF9VdwHf\nAheKSKSIRAIXuuuMMcZUUz6r5hKRD4DhQDMRSQEewxlQD1V9HZiN05MrAWfwuonutgwReQIoHpX0\ncVUtryHfGGOMn/ksmahquSOjuuMU3VXGtqnAVF/EZYwxxvvsCXhjjDGVZsnEGGNMpVkyMcYYU2mW\nTIwxxlSaJRNjjDGVFtBPwBtjTCDKzS9gxuoUgkToFdWY7q3CCQ0J9ndYlWLJxBhjqkhRkfLJjylM\nnrOJPQeOzpgQHCR0aR5GzzaN6NGmEb2iGtOjTSMahYb4MdpTY8nEGGOqwLKENJ78agPrdx2gT9sI\nXrk2jlaNQlm38wDrd+7n150HWJqYxqc/HR09ql2TBvSKakTPNo3p2cb52Ty8nh/vomyWTIwxxocS\n9mYzafYG5m3cS1REfV4a35cxvdsQFCQAtG3SgFG9WpXsv+/gYdbt3M+6nQdKfs7+ZXfJ9hbh9egV\nVZxcGnFmp6ZENKhb5fdVmiUTY4zxgfTsw7z43RbeX7mDBiHBPDKqOxOHdjhp20jz8HoM79aC4d1a\nlKw7kHeE9TsPOAkm1UkwCzfvo7BIee/WwQzt0szXt3NSlkyMMcaL8o4U8vayJF79PoHcI4VcO6gd\n918QQ9Ow06+eahQawpmdmnJmp6bHXGfj7oPEtAjzRtiVZsnEGGO8QFWZ+fMu/vH1RlKzDjGiewse\nvaQ7XVqE++R6oSHB9G0b4ZNznw5LJsaYWqGgsIh1Ow+wKimD5Ixc2jZpQIemDenYvCFtIxtQt87p\nP3a3ensGT8zawJrkLHq0bsSzV/SuFlVPVcmSiTGmRso7Usia5CxWbctgZVIGP27PJCe/EICGdYNL\n3gMECURHNqBDs4Z0ataQDk2d9x2bNSQqoj51gk+caLan5/CPbzYy+5fdtGxUj+eu6M3lcdEEu43r\ntYklE2NMjXAg7wirt2eyclsGq7Zl8HPKfvILixCBbi3D+W3/aAZ2aMKgjk1o2SiUzJx8tqXnkJSW\nwzb3lZSew+qkjGMSTUiw0LZJAzo2bUiHZu6raQMWbtrHOz8kUScoiN9f0JXbzulIg7q19yu19t65\nMSagpWUfZtW2DFZsy2BVUgYbdh2gSKFOkPNU+cShHRjYoQkDOkSesOtsZMO6RDasS1y7yGPWqyr7\nsg+TlJZLUloOW9OchJOUnsOShDQOFxQBIAJX9W/Lgxd2pUWj0Cq55+rMkokxplorKlJSsw6RuC+b\nxH05bN59kFXbM9i6LweA0JAg4tpFcs/5MQzu2IS+7SIqVUIQEVqEh9IiPJRBHZscF8vuA3kkpeXQ\nolEoXapJT6rqwJKJMaZayDtSyLa0HCdp7M0hYV82iXuz2ZqWTd6RopL9IhuEENcukqsGtGVQxyb0\natO4Uo3npyIoSGgTUZ82EfWr5HqBxJKJMaZKZeTkuwkjm4S92SUljuTMXFSdfUQgOrI+nZuHcVbn\npnRpEUbn5mF0bt6wUs9rGN+xZGKM8Zr8giL2HMhjZ9Yhdu4/xM6sPHbtP8SurDx27nfW7z90pGT/\nenWC6NQ8jN7RjRnXL6okaXRs1pD6dQN7FN3axpKJMaZCVJW9Bw+zM+sQu9zEUJwsihNFWvbhktJF\nscb1Q5yqocah9G8fQYemDencIowuzcNoE1G/VnajrYksmRhjTuqHxHT+PnMdG3cfPGZ9g7rBtG4c\nSpuI+nTr1pzWjesTFVGf1hGhtG5cnzYRobW6u2xtYr9lY0yZUjJzeXr2Bmb/spuoiPr89dIetG/a\noCRRNK4fgoiVLIwlE2PMCRzKL+TfCxN5Y2EiIvDAyK7cfk6ngJ8N0PiOzQFfQy1atIi4uDjq1KnD\njBkzytxv9erVxMbG0qVLF+69917UrfD+29/+RlRUFH379qVv377Mnj0bgLlz59K/f39iY2Pp378/\n33//fcm5hg8fTrdu3UqO2bt3LwDbt29nxIgR9O7dm+HDh5OSklJyzDvvvENMTAwxMTG88847Jeun\nT59O79696dmzJ4888kjJ+sOHD3P11VfTpUsXBg8eTFJSEgD5+flMnDiR2NhY+vTpw4IFC056j2vX\nruWss84iNjaWMWPGcODAAQCWLl1K7969GTBgAFu2bAEgKyuLCy+8kKKio11UayJVZebanYz45wJe\nnreFC3u24vsHh3PviBhLJKZ8qlojXv3791dz1LZt23Tt2rV6ww036Mcff1zmfgMHDtQffvhBi4qK\ndNSoUTp79mxVVX3sscf0ueeeO27/H3/8UVNTU1VV9ZdfftE2bdqUbDv33HN11apVxx1zxRVX6Ntv\nv62qqvPmzdPrr79eVVXT09O1Y8eOmp6erhkZGdqxY0fNyMjQtLQ0bdu2re7du1dVVSdMmKDfffed\nqqq++uqrescdd6iq6gcffKBXXXWVqqq+8soretNNN6mq6p49ezQuLk4LCwvLvccBAwboggULVFX1\nrbfe0r/85S+qqjpu3DhNTk7WxYsX6wMPPKCqqg8++KDOnz+/nE888P2amqVXvr5M2z8ySy9+cZGu\n2Jru75BMFQDi1QvfwVYyOYHf/OY39O/fn549ezJlyhRef/11Hn744ZLtb7/9NnfffTcATzzxBN26\ndWPYsGFcc801TJ48uczzrlq1it69e9O3b18efvhhevXqBUBSUhJnn302cXFxxMXFsWzZMgAWLFjA\nOeecw+jRo+nWrRt33nlnhf8y7tChA7179yYoqOxf8a5duzhw4ABnnnkmIsKECRP4/PPPyz1vv379\naNOmDQA9e/bk0KFDHD58uNxj1q9fz/nnnw/AeeedxxdffAHAt99+y8iRI2nSpAmRkZGMHDmSb775\nhq1btxITE0Pz5s0BuOCCC/jkk08A+OKLL7jxxhsBuOKKK5g3bx6qesw1WrRoQUREBPHx8eXe4+bN\nmznnnHMAGDlyZMk1QkJCyM3NJTc3l5CQEBITE0lOTmb48OHl3megysjJ50+f/cKYfy0hYW82T4+L\nZeY9w457+tuY8vg0mYjIKBHZJCIJIvLHE2xvLyLzRORnEVkgItEe2/4hIr+6r6t9GWdpU6dOZfXq\n1cTHx/Pyyy8zbtw4Pvvss5Lt06dPZ/z48axatYpPPvmEtWvX8vXXXxMfH1/ueSdOnMgbb7zBmjVr\nCA4+WmXQokUL5s6dy48//sj06dO59957S7atXLmSf/3rX6xfv57ExEQ+/fRTAK6++uqS6iTP17Rp\n0yp8n6mpqURHl3zkREdHk5p6dP7pV155hd69e3PzzTeTmZl53PGffPIJcXFx1Kt39CGyiRMn0rdv\nX5544omS6qQ+ffqUxP3ZZ59x8OBB0tPTSU1NpW3btsddv0uXLmzatImkpCQKCgr4/PPPSU5OLom5\n+Jg6derQuHFj0tPT6dOnD19++SUFBQVs27aN1atXk5ycXO499uzZsySxffzxxyXXePTRR5kwYQKT\nJk3i7rvv5s9//jNPPvlkhT/XQHGksIj/Lt3G8OfmM31VMjcO6cD8B4dz7eB21l3XnDKfJRMRCQZe\nBS4GegDXiEiPUrtNBqapam/gcWCSe+xoIA7oCwwGHhKRRr6KtbSXX36ZPn36cOaZZ5KcnMy2bdvo\n1KkTy5cvJz09nY0bNzJ06FCWLl3K2LFjCQ0NJTw8nDFjxpR5zqysLA4ePMhZZ50FwLXXXluy7ciR\nI9x2223ExsZy5ZVXsn79+pJtgwYNolOnTgQHB3PNNdewZMkSwEloa9asOe41YcIEr3wGv/vd70hM\nTGTNmjW0bt2aBx988Jjt69at45FHHuGNN94oWffee+/xyy+/sHjxYhYvXsy7774LwOTJk1m4cCH9\n+vVj4cKFREVFHZNMS4uMjOTf//43V199NWeffTYdOnQod3+Am2++mejoaAYMGMD999/PkCFDTnrM\n1KlTee211+jfvz8HDx6kbl1nMMC+ffuyfPly5s+fz9atW2ndujWqytVXX83111/Pnj17yj1vIFiy\nJY1LXlrM32eup3d0BF/fdzaPjelJ4wYh/g7NBChf9uYaBCSo6lYAEfkQGAus99inB/CA+34+8LnH\n+kWqWgAUiMjPwCjgIx/GCzhVS9999x0//PADDRo0YPjw4eTl5TF+/Hg++ugjunfvzrhx47zaHfKF\nF16gZcuWrF27lqKiIkJDj45AWvo6xctXX301mzZtOu5cDzzwQIUTSlRU1DGN4SkpKURFRQHQsmXL\nkvW33XYbl1566TH7jRs3jmnTptG5c+djzgcQHh7Otddey8qVK5kwYQJt2rQpKZlkZ2fzySefEBER\nQVRU1DEN5SkpKSVVSWPGjClJzlOmTClJDFFRUSQnJxMdHU1BQQH79++nadOmiAgvvPBCybmGDBlC\n165diYyMLPMeu3fvzpw5cwCnyuurr7465vNRVZ588kk+/PBD7rnnHp599lmSkpJ4+eWXeeqppyr0\nGVc3yRm5PPnVer5dt4e2Terzxg39ubBHS+veayrNl9VcUUCyx3KKu87TWuBy9/04IFxEmrrrR4lI\nAxFpBpwHtKUK7N+/n8jISBo0aMDGjRtZvny5E9y4cXzxxRd88MEHjB8/HoChQ4cyc+ZM8vLyyM7O\nZtasWWWeNyIigvDwcFasWAHAhx9+eMw1W7duTVBQEO+++y6FhUfnUli5ciXbtm2jqKiI6dOnM2zY\nMMA7JZPWrVvTqFEjli9fjqoybdo0xo4dCzjtKcU+++yzkvadrKwsRo8ezTPPPMPQoUNL9ikoKCAt\nLQ1wSlqzZs0qOSYtLa2krWfSpEncfPPNAFx00UXMmTOHzMxMMjMzmTNnDhdddBFASU+wzMxMXnvt\nNW699VYALrvsspJeXzNmzOD8889HRMjNzSUnxxlFdu7cudSpU4cePXqUe4/F1ygqKuLJJ5/kzjvv\nPObzmTZtGpdccglNmjQhNzeXoKAggoKCyM3NrfBnXB3kFxSxec9BJn+7iRHPL2TR5jQevqgbc39/\nLhf1bGWJxHiHN1rxT/QCrgDe9Fi+AXil1D5tgE+Bn4CXcBJOhLvtz8AaYC7wHnD/Ca5xOxAPxLdr\n184rPRvy8vJ01KhR2r17dx07dqyee+65Jb14Ro8erR07djxm/8cee0xjYmJ02LBhevnll+uUKVPK\nPPfy5cs1NjZW+/Tpo/fee68OGTJEVVU3b96ssbGx2rt3b/3DH/6gDRs2VFXV+fPn69lnn62XXHKJ\ndu3aVe+4446SHkons3LlSo2KitIGDRpokyZNtEePHiXb+vTpU/J+1apV2rNnT+3UqZPeddddWlRU\npKqq119/vfbq1UtjY2N1zJgxunPnTlVVfeKJJ7RBgwbap0+fkteePXs0Oztb4+LiNDY2Vnv06KH3\n3nuvFhQUqKrqxx9/rF26dNGYmBi95ZZbNC8vr+T6b731lnbu3Fk7d+6sU6dOLVk/fvx4PeOMM/SM\nM87QDz74oGT9oUOH9IorrtDOnTvrwIEDNTExUVWd3mtdu3bV7t2764gRIzQpKemk9/jiiy9qTEyM\nxsTE6COPPFKyXlU1JydHhw8frvn5+aqqumjRIu3Vq5fGxcXpxo0bK/Q7qGpHCgo1Ye9B/fqXnfri\n3M36/95brSOfX6CdH/1K2z8yS9s/Mkvv++BH3ZV1yN+hmmoEL/XmEi09kI6XiMhZwN9U9SJ3+VE3\neU0qY/8wYKOqRp9g2/vA/1R1dlnXGzBggJ6sAdwXsrOzCQsLIzc3l3POOYcpU6YQFxdX7r4Azzzz\nDLt27eKll14q89wLFixg8uTJ5ZZ4TO1TVKQkZ+ayeU82m/ccdF/O6Lv5HhM3tY1sQNeWYcS0DKdb\ny3B6RTW2+TfMcURktaoOqOx5fNlmsgqIEZGOQCowHrjWcwe3CitDVYuAR4Gp7vpgnBJKuoj0BnoD\nc3wY62m7/fbbWb9+PXl5edx4441lJhKAr776ikmTJlFQUED79u15++23qy5QE7Ayc/L59KdU1u3c\nz5Y9zrDth44crQqNiqhPTMswzolpRkzLcLq2DKNLizAbE8tUKZ+VTABE5BLgRSAYmKqqT4nI4zjF\nqi9F5AqcHlwKLALuUtXDIhIK/Oie5gBwp6quKe9a/iqZnMhdd93F0qVLj1l33333MXHiRD9FZALR\nngN5vLl4K++t2EFufiEtG9Wja8tw9+WUOGJahBEeaj2wzOnzVsnEp8mkKlWnZGJMZWxPz+H1hVv5\nZHUKhapc1qcNvxvema4tw/0dmqmBAqGayxhzCjbuPsC/FyQyc+1O6gQFceWAaO44pzPtmjbwd2jG\nnJQlE2P87Mcdmbw2P5HvNuyhQd1gbj27E7cO60iLRqEnP9iYasKSiTF+oKosS0zn1fkJLEtMJ6JB\nCPdfEMNNQzoQ0aCuv8Mz5pRZMjGmChUVKXM37OG1BYmsTc6iRXg9/jL6DK4Z1I6G9ey/owlc9q/X\nmCpQUFjEzJ938tr8RLbszaZdkwY8PS6Wy+OibJ4QUyNYMjHGB1SVPQcOs3nPQdbtPMD7K7eTnHGI\nri3DeGl8X0bHtqZOsM0AYWoOSybGVIKqkpadz5Y9B9nkPole/P5gXkHJfn3bRvDXS3syonsLgmx4\nd1MDWTIxpoIyc/KPGb6k+H1m7pGSfRrXD6FryzAu69OGri3DiWkZRteW4TQLq1fOmY0JfJZMjCnH\n/I17eXPJVjbtziYt++iMkuH16hDTMoyLerYqGfuqa8swmofXs1F4Ta1kycSYMiRn5HL3+z8S2bAu\nw7s1p5tHSaN141BLGsZ4sGRizAkUFSkPfryWIBGm33EWURH1/R2SMdWadScx5gSmLt3Gym0Z/HVM\nD0skxlSAJRNjSknYe5Bnv93EBWe04Ir+x02vY4w5AUsmxng4UljEAx+tpWHdYJ6+PNbaRYypIGsz\nMcbDa/MT+TllP69dF0eLcBto0ZiKspKJMa5fUvbzr++3MLZvGy6Jbe3vcIwJKJZMjAHyjhTywEdr\naBpWl8cv6+XvcIwJOFbNZQzwwtzNbNmbzdsTB9K4gU2Da8ypspKJqfVWJWUwZfFWrh3cjuHdWvg7\nHGMCkiUTU6vlHC7gwY/WEh1Znz9dcoa/wzEmYFk1l6nVnp69geTMXKbffhZhNjmVMafNSiam1lq4\neR/vrdjBrcM6MqhjE3+HY0xAs2RiaqX9uUf4w4y1xLQI48ELu/k7HGMCnpXrTa30t5nrSM/O580J\nA23aXGO8wEomptb55tddfPZTKnef34XY6Mb+DseYGsGSialV9h08zJ8++5XYqMbcdV4Xf4djTI1h\nycTUGqrKnz77hezDBTx/VR9Cgu2fvzHeYv+bTK3xyY+pzF2/h4cv7EZMy3B/h2NMjeLTZCIio0Rk\nk4gkiMgfT7C9vYjME5GfRWSBiER7bHtWRNaJyAYReVlsLPBaJe9IIW8t2cbnP6Wy92Bepc+XmnWI\nv3+5jkEdmnDzsI5eiNAY48lnvblEJBh4FRgJpACrRORLVV3vsdtkYJqqviMi5wOTgBtEZAgwFOjt\n7rcEOBdY4Kt4TfWRc7iAW9+J54et6SXrurYMY2iXZgzt3IzBnZoQHlrx8bOKipRHZvxMoSqTr+xD\ncJD9XWKMt/mya/AgIEFVtwKIyIfAWMAzmfQAHnDfzwc+d98rEArUBQQIAfb4MFZTTew/dISJ/13J\n2pT9/PPKPnRrFc6ShDSWJqTxwcod/HdpEsFBQp/oxgzr0owhXZrRr10E9eqU3b33fyu2syQhjafH\nxdKuaYMqvBtjag9fJpMoINljOQUYXGqftcDlwEvAOCBcRJqq6g8iMh/YhZNMXlHVDaUvICK3A7cD\ntGvXzvt3YKpUevZhbnhrJVv2HuTVa+MY1asVAL2iGnPnuZ05XFDIj9uzWJqQxtLENF6Zn8DL3ycQ\nGhLEoI5NGdq5KUO7NKNH60YEuaWPbWk5PD17A+d2bc41g9r68/aMqdH8/dDiQ8ArInITsAhIBQpF\npAtwBlDchjJXRM5W1cWeB6vqFGAKwIABA7TKojZet+dAHte9uYLkjFz+M2HACUfvrVcnmLM6N+Ws\nzk15iG4cyDvCiq0ZTnJJSGPS1xsBiGwQwpDOzRjSpSmfrE6hbnAQ//htb5uC1xgf8mUySQU8/xSM\ndteVUNWdOCUTRCQM+K2qZokERyC7AAAgAElEQVTIbcByVc12t30NnAUck0xMzZCckct1b64gPfsw\n79w8iDM7Na3QcY1CQxjZoyUje7QEnIS0LDGNJVvSWZqQxle/7ALgpfF9adXYpuA1xpd8mUxWATEi\n0hEniYwHrvXcQUSaARmqWgQ8Ckx1N+0AbhORSTjVXOcCL/owVuMnifuyuf7NFeQcLuB/tw6mX7vI\n0z5Xy0ahjOsXzbh+0agqW9Ny2L0/jyGdK5acjDGnz2ddg1W1ALgb+BbYAHykqutE5HERuczdbTiw\nSUQ2Ay2Bp9z1M4BE4BecdpW1qjrTV7Ea/9iw6wBXv/ED+QVFfHj7WZVKJKWJCJ2bOz3ArHrLGN8T\n1ZrR1DBgwACNj4/3dximgtYmZzFh6krqhwTzv1sH06VFmL9DMqbq7N0Ii56Dg7srf66LnoQ2/U77\ncBFZraoDKhuGvxvgTS20Yms6t7wTT2TDEN6/9UzaNrHuuqaWyEmD+U/D6rehbhi0ivV3RF5jycRU\nqYWb93HHu/FERdTnvVvPtIZxUzscyYMVr8Pif0J+Dgy8Fc59BBrWnPY8Syamyny7bjf3vP8TnVuE\n8e4tg2gWVs/fIZmqUpAP6z+HNe9Dy57OF2loI39H5Xuqzn3PfQyytkPXUTDyCWje1d+ReZ0lE1Ml\nPv8plQc/XktsVGPemTiIxg0qPhyKCWDZ+2D1f2HVW5C9Gxq3ha0L4JeP4aKnoddvwR8dJFR9f92U\n1fDto5C8Alr2ghs+h87n+faafmTJxPjcByt38KfPfmFwxya8eeNAwurZP7sab9daWPGGkzQK86Hz\nCBj7ivNz10/w1YPwyS1O28Hof0LzKpo6OWkpLHwGkldCuzOhywXOq3l37yWXrGSY93fn3hu2gMv+\nBX2vg6CaPaOn9eYyPvXWkm08MWs9w7s15/Xr+9sUuTVZYQFs+gqWvw47lkFIQ+h7DQy64/hqnaJC\nJ5HM+7vThnDW3XDuH6BuQ9/ElrQEFjwDSYudL/huFzsJZZ87SlOjaOgywkksnc6F0NOYgfPwQVjy\nAvzwqrN81t0w7H6oV72nO/BWby5LJsYnVJVXvk/gn3M3M6pnK166pm+5gzGaAJabAT+9Cyv/A/uT\nIaIdDLod+t0A9SPKPzZ7H3z3GKx5z/lCHzUJzhjjvVLCtsWw8B9OEglrCUPvh/43QV23B+H+FEiY\nBwnfOdVvhw+ABEPbwUeTS6veEFTOI3lFhc79f/8U5OyF2KtgxF8hIjDGgrNkUoolk+rl+bmbeXne\nFi7vF8WzV/Smjs1qWPPs3ej0UPp5OhzJhQ5nw+A7oNslp16ls/0HmP0Q7PkVuoyES56FJp1OP7Zt\ni52SyPYlThIZ9nsniYTUL/uYwiOQsspJLAnfOVV1AA2bO9VzXS6Azucf2wMr8Xv49i+wdx20PdNp\nB4ruf/px+4Elk1IsmVQfG3cf4OKXFjOubxSTr+xTMoKvqQGKiiBhLiz/N2ydD8H1oPeVMPjOyj8z\nUVgAK6c4z2EU5jsJYNj95ScAT6pOCWTBM7B9KYS1cpPIjRU/h6fsvU6ySPjOKb0cygDEeUCwywgn\n2WyZAxHtYeTj0GOsfzoTVJIlk1IsmVQfE/+7ktXbM1n0h/OIaFDX3+GY01Vw2GlMzkqCzO2QmQQb\nv4KMRAhvDQNvgf4ToWEz7173wC6Y82f49ROI7AAXPwddLyx7f1XYtshJIjuWOUnk7AcgbsLpJZET\nKSqEXWuOVomlrHIeOjznIadNKCRwn5eqsmQiIvcA/1PVzMpezJcsmVQPyxLTuPY/K3j04u7ccW5n\nf4djylNUBAd3Oc8/ZCY5CSNr+9HEcXAXzjx1ruC60CYOBt3m/BUe7OPu3VsXwOyHIW0zdL8URj1z\nbDuEKmxb6CaRHyC8jVMSiZvg+y/3Q1kQVAfqBf4wQFU5nEpLnCl3f8QZ1fdbrSnFGeNVRUXKM19v\npE3jUG4c0sHf4QQ2VcjYCinxkBoPu39x/joOrgvBdZyfQSHOF3pwiLvsri9eFxRy7P5FBZC142iy\n2J/sVCeVEGjUxqm26TQcIts77yPbOyWEsFblN0R7W6fhcOdS+OEVZxyrVwfBOQ87vaS2L3WSSPJy\nJ4lcMtlp8K+qEsLJOhbUQhWq5hJn2NULgYnAAOAj4C1VTfRteBVnJRP/m7l2J/d88BOTr+zDFf2j\nT36AOepQJqSudh50S1nlvD+U4WwLaQite0OdUKeRuOiIkwQKC9yf+U6iKMx3tnvuo0XHXqd+pJsg\nOpRKFh2hcTTUqaajEmTtgG8ehY2znG67efuhUdTRkkh1jTsAVOlAj6qqIrIb2A0UAJHADBGZq6p/\nqGwQJvDlFxTx3Leb6N4qnHH9ovwdTvVWeAT2rHNKHCnuK32Lu1GcB+i6j4boARA1AFqccfoPvBUV\nHk0uSOBWy0S0g/HvweY5zhP1XUY4JRFLItXGSZOJiNwHTADSgDeBh1X1iIgEAVsASyaG91ZsZ0dG\nLm9PHEhwTeq9lbwSNn/jVhmVUb10siqmoDqQnng0eexcAwWHnPM3bA7RA6HPeCd5tInz7phVQcFu\nIgrcBuJjdL2w/MZ44zcVKZk0AS5X1e2eK1W1SEQu9U1YJpAczDvCv75PYEjnppzbtbm/w/Ge9V/A\njFucKiS80EwYXBda94EBEyGqv5NEItoFZHdSY0qrSDL5GsgoXhCRRsAZqrpCVTf4LDITMN5YuJWM\nnHwevfiMmjOr4Y/vwsx7nS/8a6dDaMTx7RKF+W7bhOdyGW0XjaKd5zDqWFdpUzNVJJn8G4jzWM4+\nwTpTS+05kMebS7Yypk8bYqNPYzyj6mjZv2DOX5ynnq9+9+h4UcVVWMaY41QkmYhnV2C3esuGfTUA\nvDB3M4VFysMXVtGor76kCt8/4Uxg1OM3cPl/rCRhTAVVpNP4VhG5V0RC3Nd9wFZfB2aqvy17DvJR\nfDLXn9medk0DfOrdoiJnWPTF/4S4G+GKqZZIjDkFFUkmdwJDgFQgBRgM3O7LoExg+Mc3m2hYtw73\nnB/j71Aqp/AIfHobxL8FQ++DMS/V+LknjPG2k1ZXqepeYHwVxGICyMptGXy3YQ8PX9SNJg0D+C/4\n/Fz4+EZnwL4RjzljOhljTllFnjMJBW4BeuLRWV1Vb/ZhXKYaU1Umfb2Blo3qcfPQjv4O5/Tl7Yf3\nxzvjOl36Agywf9LGnK6KVHO9C7QCLgIWAtHAQV8GZaq3b37dzU87snhgZFfq1/VRdVBBPvz8Mez6\n2Tfnz94Hb4+GlJVwxVuWSIyppIr0yuqiqleKyFhVfUdE3gcW+zowUz0dKSzi2W83EdMijN/G+Wj8\nrawd8PFNzvhUAK37OuMvxV5xetOpHnf+ZHj3N7A/Fa75EGJGVv6cxtRyFSmZHHF/ZolIL6Ax0MJ3\nIZnq7MNVyWxLy+GPF3f3zeyJm76B18+GtC0wboozl0VRAXz1AEzuBp/dCduXOd14T8e+zTD1Iqdk\ncsNnlkiM8ZKKlEymiEgk8BfgSyAM+D+fRmWqpezDBbz03WYGdWzC+d29/PdE4RHnGY+lLzlPil/5\nDjR150MZdBvs/Al+nAa/zIC1H0DTLs5Af32vhbAKxrLzJ/jfb0GC4KZZzki8xhivKPdPS3cwxwOq\nmqmqi1S1k6q2UNU3KnJyERklIptEJEFE/niC7e1FZJ6I/CwiC0Qk2l1/nois8XjlichvTusOjdf8\nZ9FW0rLzefTi7t4dNuXATnhnjJNI+k+EW747mkjAGbsqKg7GvAgPbYLf/NsZIPG7x+D5M+DD65zR\nZIsKy75G0hJ4ewyENICJ31giMcbLKjLTYvzpjHUvIsHAZmAkzvMpq4BrVHW9xz4fA7PctpjzgYmq\nekOp8zQBEoBoVc0t63o2n4lv7T2Yx/DnFjC8W3Neu66/906cMM95xuNInpMsel9V8WP3bYafpsGa\nDyA3zZkkqd910O96Z76OYpu+cbr/RrSDGz6HxjZEvjHFqnI+k+9E5CFgOpBTvFJVM8o+BIBBQIKq\nbgUQkQ+BscB6j316AMUd++cDn5/gPFcAX5eXSIzvvTxvC/kFRTx8UXfvnLCoEBb+AxY+68zfcdU0\naN711M7RvCtc+CSc/1dnmPgfp8Giyc6sfJ2GO9VgBYfhy3ucqrPrP4WGTb0TvzHmGBVJJle7P+/y\nWKdAp5McFwUkeywXPz3vaS1wOfASMA4IF5Gmqprusc944PkTXUBEbsd9Gr9du3YnCcecrsR92Xyw\nMpnrBrejY7OGlT9h9l745FZn/u4+18LoyUcHUzwddepCj8uc1/4U+Ok9+Ol/8Mktzvb2w+CaD7w7\nT4gx5hgVeQLel0+lPQS8IiI3AYtwhmwpqfgWkdZALPBtGbFNAaaAU83lwzhrtee+2URonSDuHeGF\nYVOSlsCMm50HBse+6lRJeVPjaBj+iDNX+LYFzkRUZ/4OQup79zrGmGNU5An4CSdar6rTTnJoKtDW\nYznaXed5jp04JRNEJAz4rapmeexyFfCZqh7B+MXq7Zl8s243v7+gK83CKjFFalERLH0Bvn8SmnRy\nqpxa9fJeoKUFBUHn852XMcbnKlLNNdDjfSgwAvgROFkyWQXEiEhHnCQyHrjWcwcRaQZkqGoR8Cgw\ntdQ5rnHXGz9QVSbN3kDz8HrcenYlCqi5GfDp7ZAwF3pe7gykaFVOxtQoFanmusdzWUQigA8rcFyB\niNyNU0UVDExV1XUi8jgQr6pfAsOBSSKiONVcJe0yItIBp2SzsKI3Y7xr7vo9xG/P5KlxvWhY7zSn\nsEleCR9PhJy9cMlkGHirTVNrTA10Ot8QOUCF/kxV1dnA7FLr/urxfgYwo4xjk3Aa8Y0fFBQW8Y9v\nNtKpeUOuHtD25AeUpgrLX4O5f4VGUXDLHGjTz/uBGmOqhYq0mczE6b0FzkOOPYCPfBmU8b+PV6eQ\nuC+H16/vf+rDpqg6w578/CF0v9RpaK8f4ZtAjTHVQkVKJpM93hcA21U1xUfxmGogN7+A5+dupn/7\nSC7q2fLUT/DrJ04iOfshOP8vVq1lTC1QkWSyA9ilqnkAIlJfRDq41VCmhlFVnp+zmX0HD/P69XGn\nPmxKTjp8/QdoEwfn/ckSiTG1REXqLz4GijyWC911poZRVV74bgtvLtnGdYPb0b99k1M/yZw/O8+Q\nXPYvm/rWmFqkIsmkjqrmFy+47wN4nlZzIsWJ5OV5W7hqQDRPjD2NZ0AS5jkj+g77vW+fITHGVDsV\nSSb7ROSy4gURGQuk+S4kU9VKJ5JnLu9NUNApVk8dzoZZ90PTGKetxBhTq1SkzeRO4D0RecVdTgFO\n+FS8CTxeSSQA8592Zkic+DWEhHo/UGNMtVaRhxYTgTPd4U5Q1WyfR2WqhNcSScpqWPFvZx719kO8\nH6gxpto7aTWXiDwtIhGqmq2q2SISKSJPVkVwxne8lkgKjzhDvIe1ggv+5u0wjTEBoiJtJhd7Dr6o\nqpnAJb4Lyfia1xIJOLMj7l0Ho/8JoY29G6gxJmBUJJkEi0jJcLEiUh+oxPCxxp+8mkjStjiTW/X4\nDXS3vy+Mqc0q0gD/HjBPRP4LCHAT8I4vgzK+4dVEUlQEX97rzBNy8bPeDdQYE3Aq0gD/DxFZC1yA\nM0bXt0B7XwdmvMuriQTgx7dhxzJn3K3w0xhyxRhTo1R0BL89OInkSuB8YIPPIjJe5/VEcmAnzH0M\nOp4Dfa/zXqDGmIBVZslERLriTE51Dc5DitMBUdXzqig24wVeTySq8NVDUJjvTHJlY28ZYyi/mmsj\nsBi4VFUTAETk91USlfEKrycSgPVfwKavYOTjzvS7xhhD+dVclwO7gPki8h8RGYHTAG8CgE8SyaFM\nmP0wtO4DZ9518v2NMbVGmclEVT9X1fFAd2A+cD/QQkT+LSIXVlWA5tT5JJEAzPk/yE13RgQOPs1p\nfI0xNdJJG+BVNUdV31fVMUA08BPwiM8jM6fFZ4lk60L46V0Yco9TMjHGGA+nNB+rqmaq6hRVHeGr\ngEzlfL9xr/cTSX4uzLwPIjvC8D9W/nzGmBrH6ipqmO827CWsXh2eGhfrnUQCsPAZyNwGN850HlI0\nxphSTqlkYqq/pQlpnNmpKSHBXvrV7lwDy16BuAnOcyXGGHMClkxqkB3puezIyOXsmGbeOWFhgTMi\ncMNmTldgY4wpg1Vz1SBLEpwJMId28VIy+eEV2P0zXDUN6kd655zGmBrJSiY1yNKENFo1CqVz84aV\nP1l6IiyYBN0vhTMuO/n+xphazZJJDVFYpCxNTGNYTDOkskOcqDq9t4LrwiWTbcgUY8xJWTVXDbF+\n5wGyco8wzBtVXCteh6TFcOmL0Kh15c9njKnxfFoyEZFRIrJJRBJE5LgHFESkvYjME5GfRWSBiER7\nbGsnInNEZIOIrBeRDr6MNdAVt5cM6dL09E+iCt8/Bd/8EWIugrgbvRSdMaam81kyEZFg4FXgYqAH\ncI2I9Ci122Rgmqr2Bh4HJnlsmwY8p6pnAIOAvb6KtSZYkrCP7q3CaREeenonKDgMn90Bi56FfjfA\n+PcgyGpBjTEV48tvi0FAgqpuVdV84ENgbKl9egDfu+/nF293k04dVZ0LoKrZqprrw1gDWt6RQlYl\nZZ5+L67cDHh3HPw8Hc7/P3fsrRDvBmmMqdF8mUyigGSP5RR3nae1OKMTA4wDwkWkKdAVyBKRT0Xk\nJxF5zi3pHENEbheReBGJ37dvnw9uITDEJ2WSX1B0eu0lGdvgrQshZRX89i045yFrcDfGnDJ/12M8\nBJwrIj8B5wKpQCFOx4Cz3e0DgU44c88fwx0nbICqDmjevHmVBV3dLE7YR0iwMKhjk1M7MCUe3rwA\nctNgwhcQe4VvAjTG1Hi+TCapQFuP5Wh3XQlV3amql6tqP+DP7rosnFLMGreKrAD4HIjzYawBbWlC\nGv3aRdKw3il0zlv/Bbw9GuqFwS1zof0Q3wVojKnxfJlMVgExItJRROoC44EvPXcQkWYiUhzDo8BU\nj2MjRKS4uHE+sN6HsQasjJx81u08UPEqLlVnrK2PboRWsXDrPGgW49sgjTE1ns+SiVuiuBv4FtgA\nfKSq60TkcREpfqR6OLBJRDYDLYGn3GMLcaq45onILzgzPP7HV7EGsmWJaajCsIqMx1VYALMfgjl/\nhh6XOaMAN/TS0CvGmFrNpw8tqupsYHapdX/1eD8DmFHGsXOB3r6MryZYmpBGeL069I5qXP6Oh7Nh\nxs2w5VsYci9c8Hfr+muM8Rp7Aj7ALUlI48zOTalT3pDzB3bB+1fBnl9h9D9h4K1VF6AxplawP00D\n2Pb0HJIzDpXfXrJnndNjKz0RrpluicQY4xNWMglgxUOolNlekjDPaWivFwY3f21ztxtjfMZKJgFs\naUIarRuH0qnZCYac/3EavHclRLSDW7+zRGKM8SkrmQSowiJlWWI6F5zR8tgh54uKYP6TsPif0HkE\nXPk2hDbyW5zGmNrBkkmAWrdzP1m5R46fonfdp04iiZsAo5+3MbaMMVXCkkmAKhlyvnPpZPIZhLeG\nS1+yrr/GmCpj3zYBamlCGt1bhdM8vN7Rlfm5TqN799GWSIwxVcq+cQJQ8ZDzx3UJTvweCg4587Yb\nY0wVsmQSgFYlZZBfUMTQ0u0lG2dBaAR0GOafwIwxtZYlkwC0JCGNkGBhsOeQ84VHYNPX0HWUNbob\nY6qcJZMAtGRLGnHtImlQ16P/xPalkJcFZ1gVlzGm6lkyCTBlDjm/YRbUqe88W2KMMVXMkkmAWZbo\ndAk+pr2kqAg2fgVdRkDdBn6KzBhTm1kyCTBLtqQRHlpqyPmdP8HBndaLyxjjN5ZMAoiqsnhLGmd1\nKjXk/MaZIMHQ9SL/BWeMqdUsmQSQHRm5pGYdOn6U4A2znO7ADZqc+EBjjPExSyYBZPEWt73Es/F9\n3yZI3wJnjPFTVMYYY8kkoCxNSKNN6SHnN8x0fnYf7Z+gjDEGSyYBo3jI+aFdmh075PzGWRDVHxq1\n8V9wxphaz5JJgPg1dT/7Dx05tr1kf4rTk8t6cRlj/MySSYA44ZDzG79yflp7iTHGzyyZBIgTDjm/\nYSY06wbNYvwXmDHGYMkkIBzKLyS+9JDzuRmwfZmNxWWMqRYsmQSAVUkZ5BcWHdtesulr0EJrLzHG\nVAuWTALAUnfI+UGeQ85vnAWNoqFNP/8FZowxLksmAWBJQqkh5/NznFkVu48Gz27CxhjjJz5NJiIy\nSkQ2iUiCiPzxBNvbi8g8EflZRBaISLTHtkIRWeO+vvRlnNVZevZh1u08wNmeVVwJ30FBnrWXGGOq\njTon3+X0iEgw8CowEkgBVonIl6q63mO3ycA0VX1HRM4HJgE3uNsOqWpfX8UXKJYlpgOlhlDZMAvq\nN4F2Q/wUlTHGHMuXJZNBQIKqblXVfOBDYGypfXoA37vv559ge623NMEZcj62eMj5gnzY/C10uxiC\nffa3gDHGnBJfJpMoINljOcVd52ktcLn7fhwQLiJN3eVQEYkXkeUi8hsfxlltFQ85P6Szx5DzSYvh\n8H7rxWWMqVb83QD/EHCuiPwEnAukAoXutvaqOgC4FnhRRDqXPlhEbncTTvy+ffuqLOiqsj3dHXLe\ns4pr4ywIaQidz/NfYMYYU4ovk0kq0NZjOdpdV0JVd6rq5araD/izuy7L/Znq/twKLACO6wOrqlNU\ndYCqDmjevLlPbsKfiodQKWkvKSqCjbOd6XlD6vsxMmOMOZYvk8kqIEZEOopIXWA8cEyvLBFpJiLF\nMTwKTHXXR4pIveJ9gKGAZ8N9rbBkSxpREfXpWDzkfGo8ZO+2sbiMMdWOz5KJqhYAdwPfAhuAj1R1\nnYg8LiKXubsNBzaJyGagJfCUu/4MIF5E1uI0zD9TqhdYjecMOZ/G0C5Njw45v2EmBNWBmAv9G5wx\nxpTi0+5AqjobmF1q3V893s8AZpzguGVArC9jq+5+Td3PgbyCo1Vcqk57ScdzoH6Ef4MzxphS/N0A\nb8pwXHvJ3g2QsdV6cRljqiVLJtXUki1pnNG6Ec3C3CHnN84CxKbnNcZUS5ZMqqFD+YWs3p7JsC5N\nj67cMBOiB0J4K/8FZowxZbBkUg2tdIecL6niytwOu3+2sbiMMdWWJZNqaGlCGnWDg44OOV88Pa+1\nlxhjqilLJtXQki1pxLWPODrk/MZZ0KIHND1uEABjjKkWLJlUM+nZh1m/68DRIVRy0mDHD1YqMcZU\na5ZMqpml7pDzw2Lc4WE2zQYtsvYSY0y1Zsmkmlm6pdSQ8xtmQeN20Kq3fwMzxphyWDKpRlSVJQnO\nkPPBQQKHD8LW+U6pxKbnNcZUY5ZMqpHkjEOkZh062iV4y1wozLf2EmNMtWfJpBpZlZQB4NEleBY0\naAbtzvRjVMYYc3KWTKqR+O0ZhIfWoWuLcCg4DJvnONPzBgX7OzRjjCmXJZNqZFVSJgPaRxIUJLBt\nEeQftLlLjDEBwZJJNZGRk0/C3mwGdHCruDbMhLph0PFc/wZmjDEVYMmkmli9PROAgR2aQFGh83xJ\nzEgICfVzZMYYc3KWTKqJ+KQM6gYH0Tu6MSSvhJx91ovLGBMwLJlUE6uSMugd3ZjQkGCnF1dwXZue\n1xgTMCyZVAN5Rwr5JXW/016i6rSXdDwXQhv5OzRjjKkQSybVwNrkLI4UKgM7RMKeXyFru43FZYwJ\nKJZMqoF4t/G9f/tIZywuBLpd4t+gjDHmFFgyqQZWJWXQtWUYEQ3qOu0l7c6EsBb+DssYYyrMkomf\nFRYpq7dnOu0l+1Odai4rlRhjAowlEz/bvOcgB/MKnPaSHT84KzsM829QxhhziiyZ+Fm8O7jjgPZN\nnOdLQhpAq1g/R2WMMafGkomfrUrKpFWjUKIj60PycojqD8Eh/g7LGGNOiSUTP4tPymBAh0gkPwd2\n/2rDzRtjApIlEz9KzTrEzv15znhcqfGghdDWkokxJvD4NJmIyCgR2SQiCSLyxxNsby8i80TkZxFZ\nICLRpbY3EpEUEXnFl3H6S0l7SYdI2LECEGg70L9BGWPMafBZMhGRYOBV4GKgB3CNiPQotdtkYJqq\n9gYeByaV2v4EsMhXMfrbqqQMwurVoXurRk57SYseENrY32EZY8wp82XJZBCQoKpbVTUf+BAYW2qf\nHsD37vv5nttFpD/QEpjjwxj9Kj4pk7j2kQRTBCnx0G6wv0MyxpjTUseH544Ckj2WU4DS35ZrgcuB\nl4BxQLiINAUygX8C1wMXlHUBEbkduN1dPCwiv3ondJ9qBqR5rnj3luJ3L7qvauG4OKspi9O7LE7v\nCYQYAbp54yS+TCYV8RDwiojchFOdlQoUAv8PmK2qKSJS5sGqOgWYAiAi8ao6wOcRV5LF6V0Wp3dZ\nnN4TCDGCE6c3zuPLZJIKtPVYjnbXlVDVnTglE0QkDPitqmaJyFnA2SLy/4AwoK6IZKvqcY34xhhj\n/M+XyWQVECMiHXGSyHjgWs8dRKQZkKGqRcCjwFQAVb3OY5+bgAGWSIwxpvryWQO8qhYAdwPfAhuA\nj1R1nYg8LiKXubsNBzaJyGacxvanKnHJKZWJtwpZnN5lcXqXxek9gRAjeClOUVVvnMcYY0wtZk/A\nG2OMqTRLJsYYYyot4JJJBYZoqSci093tK0Skgx9ibCsi80VkvYisE5H7TrDPcBHZLyJr3NdfqzpO\nN44kEfnFjeG4LoLieNn9PH8WkTg/xNjN43NaIyIHROT+Uvv45fMUkakistfzGScRaSIic0Vki/sz\nsoxjb3T32SIiN/ohzudEZKP7e/1MRCLKOLbcfyNVEOffRCTV43d7wtnjTvbd4OMYp3vElyQia8o4\ntio/yxN+D/ns36eq/v/27i1UqiqO4/j3jxcSEzsqmBUlli9JZQcJE5PIMJPQbqQiFOqLllEPXQQh\nInpRKEKLIrtJSEh3iSxNoYJSIVErivSIQXE8apYmhZj8e1hrbDvOHmeY2ZeJ3weGWbP2mjP/WWfN\nWrPX3rN2x9yAfkAPMHIxJ/YAAAVFSURBVAYYSPjR45VVZe4HXorpOcC6AuIcBXTH9BDgpxpx3gh8\nVII63Q+MqLN9BrABMGAisK0EbeAAcFkZ6hOYAnQD3yXyVgBLY3opsLzG84YB++J9V0x35RznNKB/\nTC+vFWcjbSSHOJ8EHmmgXdTtG7KMsWr7M8ATJajLmv1QVu2z0/ZMGlmiZRawJqbfAaZavV8+ZsDd\ne919R0z/STib7eI8Y2ijWYT109zdtwIXmNmoAuOZCvS4+88FxnCau38BHKnKTrbBNcDtNZ56C7DJ\n3Y+4++/AJmB6nnG6+0YPZ10CbCX8FqxQKfXZiEb6hraoF2Psa+4B3sritZtRpx/KpH122mBSa4mW\n6k76dJn4QTkKDM8luhriNNu1wLYam683s11mtsHMxuUa2H8c2Ghm31hYnqZaI3Wepzmkf1DLUJ8A\nI929N6YPEE57r1a2el1A2AOt5VxtJA9L4nTcaynTMmWpzxuAPnffk7K9kLqs6ocyaZ+dNph0FAu/\n6n8XeNjdj1Vt3kGYqrkGWAV8kHd80WR37yas7vyAmU0pKI5zMrOBwEzg7Rqby1KfZ/AwZ1Dq8+/N\nbBnwD7A2pUjRbeRF4HJgPNBLmEYqq7nU3yvJvS7r9UPtbJ+dNpicc4mWZBkz6w8MBX7LJboEMxtA\n+Aeudff3qre7+zF3Px7THwMDLKwIkCt3/zXeHwTeJ0wXJDVS53m5Fdjh7n3VG8pSn1FfZSow3h+s\nUaYU9WphhYnbgHmxYzlLA20kU+7e5+6nPKyUsTrl9Quvz9jf3AmsSyuTd12m9EOZtM9OG0xOL9ES\nv6XOAdZXlVkPVM48uBvYkvYhyUqcN30V+MHdn00pc2HlWI6ZXUf4X+Q66JnZYDMbUkkTDshWr7y8\nHrjXgonA0cQuct5Sv/WVoT4Tkm3wPuDDGmU+BaaZWVectpkW83JjZtOBx4CZ7v5XSplG2kimqo7R\n3ZHy+o30DVm7GfjR3X+ptTHvuqzTD2XTPvM4q6DNZyjMIJyV0AMsi3lPET4QAOcRpkH2AtuBMQXE\nOJmw67gb2BlvM4BFwKJYZgnwPeGsk63ApALiHBNff1eMpVKfyTiNcJGzHuBbwjppRfzfBxMGh6GJ\nvMLrkzC49QInCfPKCwnH6DYDe4DPgGGx7ATglcRzF8R2uheYX0Ccewnz4pU2WjkL8iLCqt2pbSTn\nON+MbW83oSMcVR1nfHxW35BXjDH/jUp7TJQtsi7T+qFM2qeWUxERkZZ12jSXiIiUkAYTERFpmQYT\nERFpmQYTERFpmQYTERFpmQYTkSaY2Sk7cwXjtq1Oa2ajkyvRinSSLK8BL/J/9Le7jy86CJGy0Z6J\nSBvE61SsiNeq2G5mV8T80Wa2JS5SuNnMLo35Iy1cQ2RXvE2Kf6qfma2O15/YaGaDCntTIk3QYCLS\nnEFV01yzE9uOuvtVwPPAczFvFbDG3a8mLKS4MuavBD73sDBlN+EX0QBjgRfcfRzwB3BXxu9HpC30\nC3iRJpjZcXc/v0b+fuAmd98XF9c74O7DzewwYfmPkzG/191HmNkh4BJ3P5H4G6MJ15AYGx8/Dgxw\n96ezf2cirdGeiUj7eEq6GScS6VPouKZ0CA0mIu0zO3H/dUx/RVjBFmAe8GVMbwYWA5hZPzMbmleQ\nIlnQtx6R5gwys52Jx5+4e+X04C4z203Yu5gb8x4EXjezR4FDwPyY/xDwspktJOyBLCasRCvSkXTM\nRKQN4jGTCe5+uOhYRIqgaS4REWmZ9kxERKRl2jMREZGWaTAREZGWaTAREZGWaTAREZGWaTAREZGW\n/QtxxCwkT/N0ngAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f7dbd3a9cc0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "train_accuracy = boring_cost[:,1]\n",
    "test_accuracy = boring_cost[:,2]\n",
    "plt.title('Training vs Testing Accuracy')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.text(1, 0.99, 'avg_gap=' + str(100 * (np.mean(train_accuracy) - np.mean(test_accuracy))) + '%')\n",
    "plt.axis([0, len(train_accuracy), 0.94, 1])\n",
    "plt.plot(train_accuracy)\n",
    "plt.plot(test_accuracy)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main takeaway from this example is that without regularization (through dropout, in this case), the accuracy of the training classification pushes toward 100% while the testing accuracy tapers off. With regularization, the gap between the two decreases and helps ensure that improvements in training classification carry over to the testing classification (i.e. the network isn't just memorizing the training set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Additional Notes\n",
    "Bear in mind, much of what's implemented in the previous notebook and this one are quick and dirty implementations. While I've tried to ensure that everything is robust, there are almost certainly some issues with the code and a million ways it could be improved (e.g. the train() method should most definitely be split into multiple methods, the test method could be a standalone function, etc). However, the basic ideas discussed should be apparent. \n",
    "\n",
    "All of this is simply intended to be a review of basic deep learning principles. If you spot anything blatantly wrong with what's presented here, please let me know! My goal is to better understand everything foundational about machine learning\n",
    "\n",
    "Next up, we'll take these basic ideas and implement them in TensorFlow. From there, we'll quickly move on to other network types, such as CNN and LSTM. Time permitting, I'll try to touch on reinforcement learning, generative adversarial networks, and more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}