Practical_0_5_Machine_Learning_Basics.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Practical 0.5: Machine Learning Basics",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [
        "RyJ1PgtEFpa7",
        "sFJBmN5Hivhv",
        "EgLuuVNRQHqy"
      ]
    },
    "kernelspec": {
      "name": "python2",
      "display_name": "Python 2"
    }
  },
  "cells": [
    {
      "metadata": {
        "id": "SB0EeXzyu_sz",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Practical 0.5: Machine Learning Basics"
      ]
    },
    {
      "metadata": {
        "id": "c9tLLZUyVwcV",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Introduction\n",
        "In this practical, we introduce the idea of classification (sorting things into categories) using a machine-learning model. We explore the relationship between a classifier's parameters and the decision boundary (a line that separates categories) and also introduce the idea of a loss function. Finally, we briefly introduce Tensorflow.\n",
        "\n",
        "## Learning Objectives \n",
        "* Understand the idea of **classification**\n",
        "* Understand the concept of (linear) **separability** of a dataset.\n",
        "* Understand what the **parameters** of a classifier are and how they relate to the **decision boundary**\n",
        "* Be able to briefly explain what **Tensorflow** is."
      ]
    },
    {
      "metadata": {
        "id": "mHlHxAdBu7Dy",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        },
        "outputId": "9b77f6e9-46c7-4004-f0bb-9d108bbffa95"
      },
      "cell_type": "code",
      "source": [
        "#@title [RUN ME!] Imports { display-mode: \"form\" }\n",
        "!pip install -q moviepy\n",
        "!pip install -q imageio\n",
        "from __future__ import absolute_import, division, print_function\n",
        "\n",
        "import tensorflow as tf\n",
        "import numpy as np                 # Numpy is an efficient linear algebra library.\n",
        "import matplotlib.pyplot as plt    # Matplotlib is used to generate plots of data.\n",
        "from matplotlib import animation, rc\n",
        "\n",
        "from IPython import display\n",
        "\n",
        "try:\n",
        "  tf.enable_eager_execution()\n",
        "  print('Running eagerly')\n",
        "except ValueError:\n",
        "  print('Already running eagerly')\n",
        "  \n",
        "tfe = tf.contrib.eager"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Imageio: 'ffmpeg-linux64-v3.3.1' was not found on your computer; downloading it now.\n",
            "Try 1. Download from https://github.com/imageio/imageio-binaries/raw/master/ffmpeg/ffmpeg-linux64-v3.3.1 (43.8 MB)\n",
            "Downloading: 8192/45929032 bytes (0.0%)\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b630784/45929032 bytes (1.4%)\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b1761280/45929032 bytes (3.8%)\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b3309568/45929032 bytes (7.2%)\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b5693440/45929032 bytes (12.4%)\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b9068544/45929032 bytes (19.7%)\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b13049856/45929032 bytes (28.4%)\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b17113088/45929032 bytes (37.3%)\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b21045248/45929032 bytes (45.8%)\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b24961024/45929032 bytes (54.3%)\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b28975104/45929032 bytes (63.1%)\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b33030144/45929032 bytes (71.9%)\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b37052416/45929032 bytes (80.7%)\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b41066496/45929032 bytes (89.4%)\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b45080576/45929032 bytes (98.2%)\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b45929032/45929032 bytes (100.0%)\n",
            "  Done\n",
            "File saved as /root/.imageio/ffmpeg/ffmpeg-linux64-v3.3.1.\n",
            "Running eagerly\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "1xhQkS8A_KrJ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Outline\n",
        "In this practical, we tackle the task of **classification** of a simple, synthetic dataset. Classification in machine learning involves learning a labelling of examples into one (or more) discrete categories. This differs from another common task in machine learning called **regression**, which involves learning a mapping from inputs to a continuous-valued output. \n",
        "\n",
        "1. We begin by introducing a synthetic dataset of red and blue points which we want to separate\n",
        "2. We introduce and explore the idea of **linear seperability**\n",
        "3. We define a **loss** as a measure of how good of a seperator a particular line is\n",
        "4. We briefly introduce TensorFlow and show how it can be used to automatically find the minimum of a loss function."
      ]
    },
    {
      "metadata": {
        "id": "H020s1EsB_9p",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#@title [RUN ME!] Helper functions { display-mode: \"form\" }\n",
        "def plot_dataset(inputs, labels):\n",
        "  # Plot the given 2D inputs and labels using Matplotlib.   \n",
        "  plt.scatter(\n",
        "      inputs[:, 0], inputs[:, 1], \n",
        "      c=['red' if label > 0 else 'blue' for label in labels])\n",
        "\n",
        "  plt.axis('equal')\n",
        "\n",
        "  plt.xlabel('x1')\n",
        "  plt.ylabel('x2')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "3l1rLP3HufZv",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## The Data"
      ]
    },
    {
      "metadata": {
        "id": "FCu4YZy-uj0v",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Run the code in the cell below, and look at the resulting plot. It should produce a simple 2-D data set consisting of 2 classes of points, the classes are represented by colours blue and red. Our task is to build a **binary classifier** that can distinguish between red and blue points (red and blue are referred to as the **classes** of the points), using only the 2-D coordinates of a point. In other words, we want a function that takes as input a 2-D vector representing the coordinates of a point and returns a value of 1 or 0 indicating whether the point is red or blue. Here we have **encoded** the colours red and blue into the numbers 1 and 0 (which make it easier to work with in maths and code!)\n",
        "\n",
        "Note: we have arbitrarily encoded red as 1 and blue as 0, you could do it the other way around too as long as you're consistent!"
      ]
    },
    {
      "metadata": {
        "id": "2SrsrFSTtrl6",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 361
        },
        "outputId": "73c152d0-097e-444b-ce34-dd483940c083"
      },
      "cell_type": "code",
      "source": [
        "#@title Generate the Dataset  {run: \"auto\"}\n",
        "# Define the centre(s) of the points\n",
        "centre = 1    #@param {type:\"slider\", min:0, max:2, step:0.1}\n",
        "\n",
        "points_in_class = 20  # How many points we want per class\n",
        "\n",
        "# A fixed random seed is a common \"trick\" used in ML that allows us to recreate\n",
        "# the same data when there is a random element involved. \n",
        "np.random.seed(0)  \n",
        "\n",
        "# Generate random points in the \"red\" class\n",
        "red_inputs = np.random.normal(loc=centre, scale=1.0, size=[points_in_class, 2])     \n",
        "# Generate random points in the \"blue\" class\n",
        "blue_inputs = np.random.normal(loc=-centre, scale=1.0, size=[points_in_class, 2])    \n",
        "# Put these together\n",
        "inputs = np.concatenate((red_inputs, blue_inputs), axis=0) \n",
        "    \n",
        "# The class (label) is 1 for red or 0 for blue\n",
        "red_labels = np.ones(points_in_class)    \n",
        "blue_labels = np.zeros(points_in_class)\n",
        "labels = np.concatenate((red_labels, blue_labels), axis=0)\n",
        "\n",
        "# num_data_points is the total data set size\n",
        "num_data_points = 2 * points_in_class\n",
        "\n",
        "plot_dataset(inputs, labels)"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAe0AAAFYCAYAAAB+s6Q9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3XlgFOXhxvFn75yQAAtyiFfrhUWk\noIIFQRARFcWLqD+vinhQFI8i4EE9qgVqlaIIcsklxFAEajmsN1YEQYWCVJFbChgkgUCOveb3R2ow\nZsO5O7Oz+/38I8xkM4/DJs/OOzPvOAzDMAQAABKe0+oAAADg8FDaAADYBKUNAIBNUNoAANgEpQ0A\ngE1Q2gAA2ITb6gCHUlhYYvo2c3MzVFRUavp2Uw372Tzsa/Owr82TrPva78+udR1H2lG43S6rI6QE\n9rN52NfmYV+bJxX3NaUNAIBNUNoAANgEpQ0AgE1Q2gAA2ASlDQCATVDaAADYBKUNAIBNUNoAANhE\nws+IBgAJrbxc6eNekeeLz2V4PFKvntIlV0oOh9XJkIQobQA4WmVlqnvTdfJ+/NGBZXNnK+vGm7Xv\nL6Osy4WkxfA4AByljFdGVS9sSYpElFYwU+6fLwdigNIGgKPk/nx51OWOigr53l5gchqkAkobAI6W\ns/YHVhiu1HuYBeKP0gaAoxT4TYeoyyOZWaroda3JaZAKKG0AOErlv+2r8iuukuH8ya/SjAyV3XWP\nwi1bWRcMSYurxwHgaLndKhn3miremifv4g9keLzKuP1mlf7yV1YnQ5KitAHgWDidCvS8SoGeV0mS\nMvzZUmGJxaGQrBgeBwDAJihtAABswtTh8bKyMg0aNEg//PCDKioqdO+996pz585mRgAAwLZMLe33\n339fZ511lu68805t27ZNv/3tbyltAAAOk6ml3aNHj6o/b9++XY0aNTJz8wAA2JolV4/n5eVpx44d\nGjNmjBWbBwDAlhyGYRhWbHjt2rUaOHCg5s2bJ8dBHmEXCoXldjMdIAAAph5pr169WvXr11fjxo11\nxhlnKBwOa/fu3apfv36trykqKjUxYSW/P1uF3GcZd+xn87CvzcO+Nk+y7mu/P7vWdabe8rV8+XJN\nnDhRkrRr1y6VlpYqNzfXzAgAANiWqaWdl5en3bt368Ybb1Tfvn31xBNPyOnkVnEAAA6HqcPjaWlp\nev75583cJAAASYPDXAAAbILSBgDAJihtAABsgtIGAMAmKG0AAGyC0gYAwCYobQAAbILSBgDAJiht\nAABsgtIGAMAmKG0AAGyC0gYAwCYobQAAbILSBgDAJihtAABswtTnaQPAMTOMyv86HNbmMFMkIt/r\nU+X96AMpElHw/PYqv/W3ksdjdTKYjNIGYAvOjRuU+adn5Fn+mWREFGzdRvsfHqTI6WdYHS2+IhFl\n39NHvjdn6cePKWnz3pT3w/e0d9J0yc2v8VTCvzaAhOfYV6K6t98k91drqpa5vtsq99o1Kp67UEaD\nBhamiy/v3+fIN+dv+vm4gm/RAqVNn1J5xI2UwTltAAkvbdyYaoX9I/e6b5T+6mgLEpnH++H7cvx4\nSuBnPEs+NjkNrEZpA0h4rg3ra1+3aaOJSSzgdNW6ynDVvg7JidIGkPCMevVqX5db+7pkUHHpZTKi\nXHBmOBwKdOpiQSJYidIGkPDKbvmtwv6GNZaHc+up7KZbLEhknuBFXVV2y+0yvN6qZYbbrfLrb1Dg\n2t4WJoMVuBANQMKLnPIL7Xt2uDL+MkKetZXntoOnnq6y/gMUbnm2xenizOHQ/mdHKHBJD/kW/kOK\nGAp0uViBbt1T67Y3SKK0AdhE4MqrFbispzzvvyuFwwp2uTh17lN2OBTsdJGCnS6yOgksRmkDsA+3\nW8GLL7E6BWAZzmkDAGATlDYAADbB8DgAmMi5aaPSJk+UY+8ehc88S+X/d6vk81kdCzZBaQOASXyz\n3lDm0MFyFRYeWDb7De2dPDOpp2JF7DA8DgBmKC9Xxp+fq1bYkuT9bJkyn3vKolCwG0obAEzgm/M3\nuWuZjtX92VKT08CuKG0AMIEjUFH7unDYxCSwM0obAExQ0etahZs1i7ou2OrXJqeBXVHaAGACI7uO\nSu+8V5GMzGrLg6efodKHBlqUCnbD1eMAYJLye36nUItfKe1v+XLu3avQyb9Q2d39ZPj9VkeDTVDa\nAGCiUMcLta/jhVbHgE0xPA4AgE1YcqQ9fPhwrVixQqFQSHfddZe6detmRQwAAGzF9NL+9NNPtW7d\nOuXn56uoqEi9evWitAEAOAyml3bbtm3VsmVLSVKdOnVUVlamcDgsl8tldhQAAGzFYRiGYdXG8/Pz\ntXz5co0YMaLWrwmFwnK7KXQAACy7evydd97RrFmzNHHixIN+XVFRqUmJDvD7s1VYWGL6dlMN+9k8\n7GvzsK/Nk6z72u/PrnWdJaW9ePFijRkzRuPHj1d2du3hAADAAaaXdklJiYYPH67XXntNOTk5Zm8e\nAADbMr2058+fr6KiIg0YMKBq2bBhw9SkSROzowAAYCuml3bv3r3Vu3dvszcLAIDtMSMaAAA2QWkD\nAGATlDYAADZBaQMAYBOUNgAANkFpAwBgE5ZNYwrEiqOwUOljX5Z743pFcnJVfsNNCrU5z+pYABBz\nlDZszfn1f1Tnjpvl+ebrqmW+ubO1//EnVX7rHRYmA4DYY3gctpb5l+HVCluSnHv3Kv2Vl6SyMotS\nAUB8UNqwNfcXK6Iv37BevgVvmZwGAOKL0oa9OWt/Cxsej4lBACD+KG3YWrDNudGXn3aGAt0vMzkN\nAMQXpQ1b2z/4cQVbtqq2LOxvqLKHBkocaQNIMlw9DlszmjZT8byFSn9tvFzrvlGkbo7Kb7tDkRNP\nsjoaAMQcpQ37y8hQ2b33WZ0CAOKO4XEAAGyC0gYAwCYYHgdgnbIypU98Ve5/r5KRka6Knlcr2Oki\nq1PZnmPvHjk3bFDkxBNl5ORaHQcxRGkDsIRj7x7Vuel6eZcuqVqWNusN7f/dAJUNHHLQ1znXf6vI\nSSdTSD8XCilzyED5Fv5Drh3bFW7YSIGLL9G+Pz0v+XxWp0MMMDwOwBLpfxlRrbAlyVFervQJY+Xc\nvKnmC4JBZQ58QLm/aat6l3RW7m/OVdYD/aWKCnMC20DmHx5Vxmvj5dqxXZLk+n6n0qdPUdbghy1O\nhlihtAFYwvP58qjLXUVF8v3tjRrLM4cOUcZrE+TasaPy677fqfTpk5U15PeH3lhZmVz/WStH0e5j\nypzQysvlXbQg6irvO4vk2LvH5ECIB0obgEUch/+lZWXyLVoYdZX3n4vkKNkb/XWGoYznnlZup3aq\n1/E85bZvo+y7+0j7So4ib2Jz/rBLrp07oq5z7dgh53//a3IixAOlDcASwbbRp6AN16uviutvqLbM\nuatQzp3bo369a8d2OXdEL6v0US8oY+Tzcm/cUPm1P+xS2uw3lH3fvceQPDFF/A0Vbnp81HXhZscr\n3Cz6OtgLpQ3AEqUPDlTgNx2rLYtkZKrs7t8p8rOCiTRspPDxtRTS8c0VbtI06jrf3+fIEYnUWO79\n8D05v113lMkTlNerip5XyoiyquKyK6SsLNMjIfa4ehyANTIztWfmbKVNm1z5iNX0DJVfe71C555f\n82t9PlVcfpVcI5+vMaheftkVUmZmzdeEw7UegTtLSuRevUqBX/zy2P8/EkjpI49Jcsj31lw5/7tN\nkeMaq6L7ZSp97A9WR0OMUNoArOP1qvy3d0q685BfWjr4ccnprCykbdsUadxYFZdertJHh0Z/gcul\nSJOmUc/zRurWVeicXx9j+ATkdKp08OMqfXiQHEVFMnJyJK/X6lSIIUobgD38vJBycw/5JLfyXtfI\n/e+VcoRC1ZYHunRT5IQT4xjWYh6PjIYNrU6BOKC0AdjLERRS+V395AgE5ZuVL/emDYo0aKBAp67a\n98dhcQ4JxAelDSD2wmE5d2yXUbeujKxs63I4HCq77wGV3dtfzu93KpKTK2VkWJcHOEZcPQ4gptLG\nj1FO146q1661cs9vrex775RjT7G1odxuRZo0pbBhexxpA4gZ38zXlfXUE3KUl0uSXOXlcs3Kl6O4\nWHtfL7A43ZFzL10i31vzJMNQxaU9FLqg46FfBMQRpQ0gZnwFM6oK+6e8H38o9/KlCrU5z4JURyfz\nicFKf21C1f9P+uQJKrvxZu3/0/OS4whmcwNiiOFxADHjqmWqTEd5udwrVpic5uh53n1H6RPHVfsA\n4qioUPqUSfLO/7uFyZDqKG0AMRM57rioyw2fT6GWZ5uc5uj55s+TIxCosdwRDsu7MPpDOQAzUNoA\nYqa817UyokzmEWh/gULtLrAg0VH62X3dP+UIBU0MAlRHaQOImYpbbtf+wY8reOrpMiSFc+up/Iqr\nVPLyeKujHZFg+wuizuEtScHz25uaBfgpS0r7m2++UdeuXTVt2jQrNg8gjsr63a/i9z5W0b+Wq+iT\nFSqZMEVGgwZWxzoiFdfmKXDpZTWXd+2m8htvtiARUMn0q8dLS0v19NNPq127dmZvGoBZvF6Ff3mq\n1SmOnsulveOnKG3iq/J+8i9JhoLntVNZn7sPOXUqEE+ml7bX69W4ceM0btw4szeN/5k926W//92j\n4mKHTjopojvvDOiMM2obDARSlMej8rv6qfyuflYnAaqYXtput1tuN7eHW+XPf/Zq5EivKioq7zP9\n17+kjz5yaezYcv361zWfOwwASBwJ3565uRlyu12mb9fvt3C+5DgpKpKmT5cqKqov37LFpfHjM9W9\nu/mZknE/Jyr2tXnY1+ZJtX2d8KVdVFRq+jb9/mwVFpaYvt14mzbNrW3b0qOuW748rMJCc/d1su7n\nRMS+Ng/72jzJuq8P9kGEW75SSE6OIdVyI0tamrlZgKRRVibXN19b/1AUpATTj7RXr16tYcOGadu2\nbXK73Vq0aJFGjRqlnJwcs6OknEsvDatFi4jWrKl5uqF9+9onkwAQhWEo449Pyjf3Tbk3b1S4YSMF\nOneRJnKRLeLHYRhGQl82bMXQR7IOuUjShx+6NHCgTxs3Vha302moQ4ewJk0qU1aWuVmSeT8nGlvu\n6/JypU17Ta6NGxRpeJzK77jT2mdz/0z6n/+kzBHPyfHzX6E33qjCF8dYEyrF2PJ9fRgONjye8Oe0\nEVsXXhjWu++WavJkj3bvdqhVq7AuvzzMQ4uQUJybNqpOn1vkWbWyalnazOkq+etohdomwJPCDEO+\nt+bVLGxJWrhQzv9uq3x+dwx4Fy2Qd+6bcpbsVegXp6rs3v4y/P6YfG/YD6WdgrKypH79mD8ZiSvz\nqcerFbYkudevU+bTQ7Vn7gLrH41ZWirnzu3R1+3eLddXq2NS2ul/GaHMF0dUPW3Mt2i+vO+9rb2T\nZyhy4knH/P1hP1yIBiCxlJbKs3Rp1FWeFZ/J9Z+1JgeKIiNDkcZNoq9r0EChXx37E80cu3YpfeKr\nNZ5P7ln7lTJeGHHM3x/2RGkDSCiOUFAK1nwspiQpGJTKzL8NtAaHQxVXXi3DGeVX6GWXyWgU/RGl\nR8I3u0Cu73dGXede+cUxf3/YE8PjAGJn3z755r0pIyNDgct6HtU83Uadugq1PFuujz6osS7U4iyF\nzz4nBkGPXdl9D0rBoNLe/JtcWzYp4m+oii4XK2PsaGlvLR86jsTB7sN0M/95qqK0AcRE2isvKWP8\nGLm2bpEkBU87XaWDHqss7yNU1n+AXOu+lnv7gfPGkZwcld3VT3KZP0NiVA6Hyh4epLL7H5Lz+52K\n5NaTMjKU4fNJOvbSLr/meqWPelHuzRtrrAuemwAX48ES3PIVRbLeRpBo2M/mife+9rz7jur0uVnO\n/furLQ8f11hFb38g47jGR/w9nWtWK2PSODm3fadIA7/Kb/w/hdr9JlaR4yaW+9r3xgxlPvmYXIWF\nVcsC7Tto79QZMrLrxGQbdpasv0O45QtAXPlmv1GjsCXJtWO70l8br9JBjx/x94y0OEv7/jwyFvFs\nq+L6GxQ8r53Spr4mR8lehX7VShV5N0o8dCll8S8P4Jg5DzKFp2PPHhOTJJ/ICSeq9LE/WB0DCYKr\nxwEcs/BJJ9e+7vQzTUwCJDdKG8AxK737dwqd8ssaywPnnqfyG2+2IBGQnChtAMfMaNpMeyZMUXmv\naxU66WQFTz1NZf93q/ZOnnlUt30BiI5z2gBiInJmC5WMnWh1DCCpcaQNAIBNUNoAANgEpQ0AgE1Q\n2gAA2ASlDQCATVDaAADYBKUNAIBNUNoAANgEpQ0AgE1Q2gAA2ASlDQCATRzV3OOGYcjhcMQ6CwAk\nPOc3X8v3/jsKN2mqQI8rJJfL6khIIbUeaa9du1a33HKLevbsqSlTplRbd+utt8Y9GAAklHBYWfff\nq9weXZT1+GDV6XOrcrpfJPcXK6xOhhRSa2k/+eSTuu222/T0009r2bJlGjJkSNU6wzBMCQcAiSL9\nL8OVPmOanHv3SpIchiHPyi+U9chDUiRicTqkilpL2+Px6KKLLtLZZ5+tl156SRUVFXrhhRfMzAYA\nCcP33jtRl7tXfiHv/LdMToNUddAL0ZYuXVr152HDhunrr7/W8OHDFQwG4x4MABKJY8+e6MsNQ85t\n35mcBqmq1tJ+7LHHNGLECO3fv1+S5Ha7NXr0aKWnp+vf//63aQEB4HA5Cgvlm5Uv1/LPYv69w788\nNerySN26CnTtFvPtAdHUWtqnnXaaZs2apWuvvVbLly+v/GKnU82bN1fjxo1NCwgAh2QYynzsEeV2\nbq86996p3KsuVd2rL5dz/bcx20Rpn7sV9jessbzi8qsUOeUXMdsOcDCHvOXr5Zdf1lNPPaXTTjtN\n27dvl8fjUX5+vhnZAOCwpP/1L0ofN0aO/10k6wgE5P34I2U/8DvtmbtAisEtqqEOHbV3zERlTBgr\n17ffKFI3R4EuF6vs/oeO+XsDh+uQpX3yySfrvvvu04ABA5SZmakxY8aofv36ZmQDgMPiWzi/qrB/\nyrN8mTwfvqdgpy4x2U6oQ0ft7dAxJt8LOBqHLO3HH39cmzZt0rRp01RcXKwHHnhAF198se655x4z\n8gHAITmKfoi+PBSS69tvY1bagNUOOY3pKaecoilTpqh58+Zq2bKlZsyYoX379pmRDQAOS/jEk6Mu\nj2RnK9DhQpPTAPFzyCPt2267rdrffT6ffv/738crDwAcsfJbbpdn+bKqiU9+FOh2qSKnnW5RqsTh\n+eci+ea9KWfpfoXOaKGyu/vJyMq2OhaOwlHNPX4snn32Wa1cuVIOh0NDhgxRy5YtzY4AIMkEelyh\nkmBQaVMmyf3tOkVychXo1EWljw21Oprl0oc/q8xRL8hRUSFJ8v19rrz/XKQ90wtkNGhgcTocKVNL\ne9myZdq8ebPy8/O1fv16DRkyhCvRAcRE4MqrFbjyaskwYnK1eDJwbt2i9AmvVhX2jzxfrFDG88O0\n/7kRFiXD0TL10ZxLlixR165dJVWeK9+zZw/nxwHEFoVdxfe3ArmKdkdd5/nyc5PTIBZMLe1du3Yp\nNze36u/16tVTYWGhmREAIHW4a39sqMEjRW3J9HPaP3U4TwvLzc2Q+yBvvHjx+7lIwwzsZ/Owr82T\nMPu6313Sq6OlHTtqrPJe2CFxch6DZPh/OBKmlnbDhg21a9euqr9///338vv9B31NUVFpvGPV4Pdn\nq7CwxPTtphr2s3nY1+ZJqH3tSFfa7x5Q5ohn5fzJA08Cv+moPfc+KCVKzqOUUPs6hg72QcTU0r7g\nggs0atQo5eXlac2aNWrYsKGysrLMjAAAKaW87z0Ktv+N0vKnS6X7FTqnjSp63yh5PFZHw1EwtbRb\nt26tFi1aKC8vTw6HQ0OHcjsGAMRb+Kxfaf9Zf7I6BmLA9HPaDz/8sNmbBAAgKZh69TgAADh6ll49\njuRgGNLbb7v0wQcuud3SNdeE1KpVxOpYAJB0KG0ck3BY6tfPp7lzPQqHKye1mDrVq3vuCeiRRwIW\npwOA5MLwOI7JlCkezZ7trSpsSSotdWjMGK9WreLtBQCxxG9VHJOPPoo+8c3+/Q7Nns1ADgDEEqWN\nYxIK1b4uGDQvBwCkAkobx+Scc6JfcOb1GuraNWxyGgBIbpQ2jsk99wR0wQU/P9w21KtXUJ06UdoA\nEEucdMQxSU+XXn+9TK++6tXnnzvl8UidO4d0ww0hnpAIADFGaeOYpadL99/P7V0AEG8MjwMAYBOU\nNgDg8BmGVFZW+V+YjtIGAByaYSh9xHPK6XyB6rVuoZyLOyr9pRcpb5NxThsAcEgZz/xBGS+9KMf/\nStr1wy6516yWgkGVPfB7a8OlEI60AQAHV1qqtLlvVhX2jxzhsHxvzmImJRNR2lBxsVRQ4NbHHzsZ\n6QJQg2vTRrm2bKpl3SY5dxWaGyiFMTye4p57zquZMz3avt0pl8tQ69ZhPfNMuc45h/YGUCnSuLHC\nDRrItWtXjXWG369ITq4FqVITR9opbOpUt156yavt2yvfBuGwQ5995tZDD6Uz2vUTK1c69dBDPt18\nc5oeecSr//yHWWOQWozcegp27hp1XUWXbpWTNcAUHGmnsLfecisYrFlAq1e7NGuWWzfccJCngaSI\nefNcGjQoTbt2Hfh8u2CBRyNHlqtzZ6ZpReooGf6CFAzK+/67cu4pVrh+fQW6XqL9Tz9ndbSUQmmn\nsKKi2gdafjz6NkMgII0c6dUnn7gUCEi/+lVYDzwQVKNG1g7RG4b00kveaoUtSTt2ODVqlFedO5dZ\nlAywQGamSl6dJOfWLXKt/UqhlmfLOK6x1alSDqWdwk48MaIvv6z5PGyv19Cvf23OUbZhSHfemaYF\nCzxVyz77zK1ly9zKzy+V329KjKjWrXNo1arozwv/8kuXCgsd8vs594/UEjm+uSLHN7c6RsrinHYK\nu/32oBo2rPlozQsvDKljx+iP3Iy1hQtdevvtmp8dV692afRorykZauPzSR5P9HUejyGPh8IGYC5K\nO4W1axfWqFHl6tIlqKZNwzr11LBuvz2gcePKTXtC15IlLoXD0Tf21VfWvj1POMFQmzbRz1u3aRNR\nnTpMBgXAXAyPp7jOncPq3Dksw5Alj9I82EWnGRnm5ajNo49WqH9/h9avPzBM3qxZWMXF0jnnZCgr\nS7rggrD+8IeKhMgLILlR2pBkTWFL0k03BTV1qqfGxV4ul6Fu3ay/er1Nm4gWLSrV+PFe7dzpUDBo\naMECjz777ECJr1vn0vbtDk2dWm5hUgCpgOFxWKp5c0ODB1eoUaMD59Czsw3ddltQeXnWl7Yk1akj\nPfhgQMOGVai42Kkffqj5Y/PBB2598gk/TgDiiyNtWO7mm0Pq0SOk11/3qKLCocsuC+qMMxLzZPHm\nzdGLuaLCoWXL3GrfPmByIgCphNJGQqhfX+rfP/GnYatXLyIp2m1ghpo1M+eKewCpi/E84AhcdllI\nbnfNUYCzzw6rV6/EGM4HkLwobeAI3HprSP37B9SkSeWtYB6PofPOC+mFF8rlij4PCwDEDMPjwBFw\nOKTBgwPq1y+gd991q2nTiNq2jZh29X0kIn33nUOZmVL9+ol53h9A/FDawFGoU0emD4e/8YZb48d7\ntGaNSxkZhs4/P6ynn67QiSdS3kCqoLQBG3j/fZcefdSnPXsqz2jt2ePQokVOFRY69NZbZXLzkwyk\nBM5pAzYwY4anqrB/6vPPKx+jCiA18NMOy+3aJb36qldbtzrVoEFEt90W1CmnxGfI99NPnZo82avN\nmx2qX9/QVVcFdc01if9c7P/+t7aT5g5t2MBnbyBVUNqw1KpVDt11V3q1ub3ffNOj4cPL1aNHbMv0\n3Xdduu++NBUWHii5Dz5wa9u2Ct13X2LfI37ccbV/iDnppCO/P9zx/fdy7tur8AknicveAfvgIzos\nNWKEr1phS9L33zv1wgs+RWI8V8nYsZ5qhS1VzmQ2bZpHpaWx3Vas3XhjUHXr1twh55wT0nXXHf4F\ncc5Nm1Tn5t6q1+4c5bZvo5yuHeWbMimWUQHEkemlvWzZMrVr107vv/++2ZtGgikrqzwnG82qVU6t\nWBG7t2cwKH31VfRtbdrk0kcfJfbR5kUXhfXMMxVq1apycpe6dSPq1i2oMWPKD/8itHBY2ff2kW/R\nAjlLSuSIRORZ829lDX1U3vl/j2t+ALFh6vD4li1bNGnSJLVu3drMzcKmYnnvs8slpadHH2L2eAz5\n/Yl/21Tv3pVH1Vu3Vt6n3aDBkWX2zn1TnhWf1Vju3L9PafkzFOhxRayiAogTU4+0/X6/XnrpJWVn\nZ5u5WSSo9HSpdevo561btoyodevYjY87nVL79tG31bp1OKbbiienUzrhBOOIC1uSXN9+I4cR/XXO\nnTuONRoAE5h6pJ2enn7Er8nNzZDbbf7Qpd/PBwszPPecR5s2SV9/fWBZ48bSk0+61KhRbP8NRo+W\nCgul99+Xwv/r77PPll5+2a2GDZP/3zvr3NaVwxdRittzYnPe8zHEvjRPqu3ruJV2QUGBCgoKqi3r\n37+/OnTocETfp6jI/CuE/P5sFRaWmL7dVOP3Z6tZsxLNmSONG+fVli2Vt3zdfntQJ51kqLAw9tuc\nPl1atMillStdatrU0PXXB+X1Ki7bSiR+f7YKO3ZT3XPPl3fpkmrrInXqaO/VeQryno8Jfn+YJ1n3\n9cE+iMSttK+77jpdd9118fr2SCL160uDBpnzHGqHQ+rePazu3RP/3uyYczq1d+xEZT36iDyffiLn\n/n0KnX6myn7bV8Fu3a1OB+AwcJ82kEKMJk1VMmmaHHuK5di3T5HGTSpPlAOwBVNL+4MPPtCECRO0\nYcMGrVmzRlOnTtXEiRPNjABAklE3R0bdHKtjADhCppZ2p06d1KlTJzM3CQBA0mB4HNq506FXX/Vo\nxw6nGjaMqE+foJo2Tfz7lgEg1VDaKW7ZMqf69UvT5s0HbqubO9etF1+sUMeOKXixFgAkMK5ASXHD\nh3urFbYkffedS88/7412Oy8AwEKUdgorLHToiy+iD7asWOHS1q0xnEcUAHDMKO0UV9vRtGHUvg4A\nYA1KO4X5/YZat47+WMfWrcNq3pzWBoBEQmmnuIceCqp58+oXnDVtGtZDDwVi+pQtAMCx4+rxFNeu\nXVjz5pVp3DiPtm93qlGjiO5jLThtAAAOKElEQVS4I5iQR9kVFdIf/+jVBx+4tW2bQy6X1LRpRBdf\nHNYDDwR0FM+jAQBbobShJk0MDR1qztzfx+Luu9P0j394qi0rLnZqzRq3vvjCpddfL5PHU8uLTbJv\nn+TzyfIcAJITw+OwhaVLnXr33do/Y374oVv5+dY15fz5LvXqla42bTJ13nkZ6tfPp6Iiy+IASFIc\nacMWlixxq7z84CfZly936oQTnFq+3K1mzSLq1Ssktwnv8MWLXXrwwTTt3l35GXj3bqmgwKUdO5ya\nNauMawMAxAylDVs47rjIIb9m8WKXZs3yKBBwSDL06qthvfhiuVq0iO/5+SlTPFWF/VOffOLS22+7\ndMklzCwHIDYYHoctXHNNSGedVXv5ud2Gtm51/a+wJcmhlSvdGjIkLe73m9c2CU047NCqVa6o6wDg\naFDasAWPRxoxovx/95VXb+Hc3Ijq1Il+JL58uUvLl8f3be731/6poHnzQ48QAMDhYngctvHrX0c0\nf36Z3nvPpZUrndq3z6HsbOmaa4K6/PKMqK8JBh3ascMpKX7l2atXUB9+WPOce8uWIV1zTfTJawDg\naFDasBWnU+raNayuXasPlZ9+elg7d9Y8om7WLKzOneNbnFdfHdb27RWaNs2j9etd8vkMtW0b1jPP\nlJtyIRyA1MGvFCQUw5DCYR1x2d15Z1CrV7v0ww8HitvjMZSXF1JWVoxDRtGvX1B33BHUsmVO+f2G\nzjgj8SanAWB/lDYSwv790h/+4NPHH7tUUuLQ6aeH1adPUN27H96V1926hTV2bLkmT/Zo82an6teP\n6PLLQ7r5ZvOGp9PSpI4dOYcNIH4obSSEvn3T9M9/Hpgc5fvvnVq92qWxY8t14YWHV9wdO4bVsSO3\nVwFIXlw9DsstXuzUhx/W/Py4e7dTkyczHygA/IjShuVWrHD/5P7q6jZv5i0KAD9ieByWq7yX2ZBU\ns7jr10++c8SRiPTll5UfRlq1isjJ5xIAh4lfF7DclVeG1KpVzXPRHo+hK65IrvucFyxwqXv3dPXo\nkaEePTLUvXu6Fixg1jQAh4fShuVcLunFF8vVrl1IXm/lrVLNmoV1//0BU6/+jrcNGxwaODBNX37p\nViTiUCTi0JdfujVwYJo2buSpIgAOjeFxJIQzzzQ0Z06ZPv/cqe3bnerUyZz7q800aZIn6gQwO3c6\nNXGiR08/nfjPNAdgLUobCcPhqJyqNJ5TjlrppxO/HMk6APgRvykAkzRrVvuHkYOtA4AfUdqASfr2\nDeiUU2pecHfKKWH17cvQOIBDo7QBkzRoII0dW6ZLLgnK74/I74/okkuCGju2TA0aWJ0OgB1wThsw\nUcuWhqZOLVdpaeXfM6I/URQAoqK0AQtQ1gCOBsPjAADYBKUNAIBNUNoAANgEpQ0AgE1Q2gAA2ISp\nV4+HQiE9+uij2rJli8LhsAYOHKg2bdqYGQEAANsytbTnzp2r9PR0zZgxQ+vWrdPgwYM1a9YsMyMA\nAGBbppZ2z549dfnll0uS6tWrp+LiYjM3DwCArZla2h6Pp+rPkydPripwAABwaA7DMIx4fOOCggIV\nFBRUW9a/f3916NBB06dP13vvvacxY8ZUK/JoQqGw3G5XPCICAGArcSvt2hQUFGjhwoUaPXq0fD7f\nIb++sLDEhFTV+f3Zlmw31bCfzcO+Ng/72jzJuq/9/uxa15k6PL5161bNnDlT06ZNO6zCBgAAB5ha\n2gUFBSouLlbfvn2rlk2YMEFer9fMGAAA2JLpw+NHiuHx5BXr/fzttw6NH+/Rtm1ONWhg6KabgmrT\nJhKz729nvKfNw742T7Lu64QZHgfi5V//cql/f5++++7ARYvz57v1zDMVuu66kIXJACB2mMYUSWHk\nSE+1wpakoiKnXnnFq3DYolAJbv16h+bOdWvbNofVUQAcJo60YXv79kmrVkW/LXD1aqe++MLJMPlP\n7N0r3Xdfmj76yK19+xzKzY2oW7eQnn++QlxeAiQ2Shu253RK7lreyW63lJ5ubp5E9+CDaZo//8D8\nCEVFTuXne5WVJT33XIWFyQAcCsPjsL2MDKlt2+hj4K1bh3XmmRxl/2jnToc++ij6qMR777kUCJgc\nCMARobSRFB59tEItWlQv7hNPDGvw4IAcnLKtsnmzQ8XF0X/sd+1yaN8+kwMBOCIMjyMp/OIXhv7x\nj1JNnuzRpk1O+f2G7rgjoNxcq5MlljPPjKhZs3CNi/Yk6YQTDOXkWBAKwGGjtJE0MjKke+4JWh0j\noWVlST17hjR6tFPSgSEIr9fQddcF5WTsDUholDaQYp54IqCsrMr72L//3qFmzSK69tqg+vThfnYg\n0VHaQIpxOqWHHw7o4YcDCoVqv/IeQOLhxxWmCIWkv/7Vq48+cqm8vPLc6tChUt26VidLbRQ2YC/8\nyMIU99zj09y5B2bu+PxzaflyacoUh048MaGnvweAhMFlJ4i7jz92asECT43l//mP9MorNZcDAKKj\ntBF3ixe7FQhEv1l67droE30AAGqitBF3mZkHW8fQOAAcLkobcXfzzQE1bVpzmlGHQ+rShUdwAcDh\norQRd7m50tChFTr++AMFnZVlqG9f6Y47mAwFAA4XV4/DFFddFVbXrqV6/XWP9u1zqHv3oC68MEuF\nhVYnAwD7oLRhmqwsqW9fjqwB4GgxPA4AgE1Q2gAA2ASlDQCATVDaAADYBKUNAIBNcPU4Es6XXzr0\n0ks+rV7tlM8nnX9+SI8/XvkMaABIZZQ2EsrXXzvUp0+6tmw5MCf52rUubdjgUn5+mZyMDQFIYfwK\nREIZN85brbB/tHixS2+9xcNFAKQ2ShsJZf366G/JSMShlSspbQCpjdJGQsnJqf2pXwdbBwCpgNJG\nQrniiqB8vprlfMIJYd12G1OgAkhtlDYSytVXh3X//QEdd1zkf0sMnXlmWMOGVSg729JoAGA5rh5H\nwnn44YD69Alo3jy3cnKkHj1CcvNOBQBKG4kpJ0e65ZaQ1TEAIKEwPA4AgE1Q2gAA2ASlDQCATZh6\nTvuHH37QI488ooqKCgWDQQ0ePFhnn322mREAALAtU4+0582bpyuvvFJTp07Vgw8+qJEjR5q5eQAA\nbM3UI+3bb7+96s/bt29Xo0aNzNw8AAC2ZvotX4WFhbr77ru1f/9+TZ482ezNAwBgWw7DMOIyoXNB\nQYEKCgqqLevfv786dOggSfrwww81efJkTZw48aDfJxQKy+3mQREAAMSttKNZtmyZTjvtNNWtW1eS\ndN5552np0qUHfU1hYYkZ0arx+7Mt2W6qYT+bh31tHva1eZJ1X/v9tc/ZbOrw+Ntvv62vvvpKt912\nm77++ms1btz4kK85WPh4smq7qYb9bB72tXnY1+ZJtX1t6pH27t27NWjQIO3fv1+BQECPPvqoWrVq\nZdbmAQCwNVNLGwAAHD1mRAMAwCYobQAAbILSBgDAJihtAABsgtKuxa5du9S2bdtD3keOoxcKhfTI\nI4/ohhtu0PXXX6/ly5dbHSkpPfvss+rdu7fy8vK0atUqq+MkteHDh6t379665ppr9Pbbb1sdJ6mV\nl5era9eumj17ttVRTGX6NKZ2MXz4cB1//PFWx0hqc+fOVXp6umbMmKF169Zp8ODBmjVrltWxksqy\nZcu0efNm5efna/369RoyZIjy8/OtjpWUPv30U61bt075+fkqKipSr1691K1bN6tjJa1XXnmlaqKu\nVEJpR7FkyRJlZmbq1FNPtTpKUuvZs6cuv/xySVK9evVUXFxscaLks2TJEnXt2lWSdMopp2jPnj3a\nt2+fsrKyLE6WfNq2bauWLVtKkurUqaOysjKFw2G5XEzDHGvr16/Xt99+q06dOlkdxXQMj/9MIBDQ\nyy+/rAceeMDqKEnP4/HI5/NJkiZPnlxV4IidXbt2KTc3t+rv9erVU2FhoYWJkpfL5VJGRoYkadas\nWerYsSOFHSfDhg3ToEGDrI5hiZQ+0o72UJOOHTvquuuuU506dSxKlZwO9gCZ6dOna82aNRozZoxF\n6VIHcynF3zvvvKNZs2Yd8mFIODpz5sxRq1atUvb0JTOi/UxeXp4ikYgkacuWLapXr55GjhypX/7y\nlxYnS04FBQVauHChRo8eXXXUjdgZNWqU/H6/8vLyJEldunTR3LlzGR6Pk8WLF2vkyJEaP368cnJy\nrI6TlAYMGKCtW7fK5XJpx44d8nq9euqpp9S+fXuro5kipY+0o5k5c2bVnwcNGqRevXpR2HGydetW\nzZw5U9OmTaOw4+SCCy7QqFGjlJeXpzVr1qhhw4YUdpyUlJRo+PDheu211yjsOHrxxRer/jxq1Cg1\nbdo0ZQpborRhoYKCAhUXF6tv375VyyZMmCCv12thquTSunVrtWjRQnl5eXI4HBo6dKjVkZLW/Pnz\nVVRUpAEDBlQtGzZsmJo0aWJhKiQbhscBALAJrh4HAMAmKG0AAGyC0gYAwCYobQAAbILSBgDAJiht\nAFHNnj1brVq10ieffGJ1FAD/Q2kDqGHOnDlavXq1Tj/9dKujAPgJShtIcZMmTdJjjz0mSdqwYYO6\nd++uLl266IknnpDH47E4HYCforSBFHfrrbdq48aNWrFihZ588kk99dRTys7OtjoWgCgobSDFOZ1O\nPfvssxowYIBOPfVUnXvuuVZHAlALShuA9uzZo4yMDG3fvt3qKAAOgtIGUlxFRYWGDh2qMWPGyOPx\naM6cOVZHAlALHhgCpLjhw4crMzNT/fr1065du9S7d2/16tVLS5cu1dq1a9WkSRPVrVtXI0eOVL16\n9ayOC6Q0ShsAAJtgeBwAAJugtAEAsAlKGwAAm6C0AQCwCUobAACboLQBALAJShsAAJugtAEAsIn/\nB/OXgJIZT2EKAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<matplotlib.figure.Figure at 0x7f06f0e943d0>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "an9GF8nZWS4K",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "###What does the data look like? \n",
        "The inputs are 2-dimensional vectors (points in a 2-D space). Here are the coordinates of 4 points, which we've deliberately chosen so that points 1 and 2 are \"red\" and points 3 and 4 are \"blue\". "
      ]
    },
    {
      "metadata": {
        "id": "5f8M-vQYWUuI",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "print('Input 1:\\t', inputs[0])\n",
        "print('Input 2:\\t', inputs[1])\n",
        "\n",
        "print('Input 3:\\t', inputs[-1])\n",
        "print('Input 4:\\t', inputs[-2])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "E14elwRYWoxB",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The labels are either 0 or 1. Here are the labels corresponding to the points above:"
      ]
    },
    {
      "metadata": {
        "id": "q4EJG8g4Wtih",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "print('Label 1:\\t', labels[0])\n",
        "print('Label 2:\\t', labels[1])\n",
        "\n",
        "print('Label 3:\\t', labels[-1])\n",
        "print('Label 4:\\t', labels[-2])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "RyJ1PgtEFpa7",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Aside: Other Examples of Binary Classification Problems"
      ]
    },
    {
      "metadata": {
        "id": "7LAhNs_GFwOZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "In this practical, we are using a synthetic dataset where we have 2 classes of 2-D points that come from different distributions, distinguised by the colours, red and blue. To make this more concrete, here are some examples of more real-world binary classification problems.\n",
        "\n",
        "* Determine whether an email message (input) is SPAM or NOT SPAM (label)\n",
        "* Determine whether an image, represented by its encoded pixel values (input) is a picture of a DOG or a CAT (label)\n",
        "* Determine whether energy usage of a building will go UP or DOWN (label) next month, using a time series of past energy usage values (input)\n"
      ]
    },
    {
      "metadata": {
        "id": "qNuMy1XwIJ-z",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Linear separability\n",
        "\n",
        "Linear separability of a D-dimensional dataset with 2 classes means that there exists a single (D-1)-dimensional (hyper-)plane that separates the classes (a hyperplane is a generalisation of a straight line to many dimensions). In this case, the dataset is 2-dimensional and is **linearly separable** if it is possible to draw a (1-D) line between the red and blue points such that all of the red points lie on one side of the line and all of the blue points on the other. \n",
        "\n",
        "### Exploratory Task\n",
        "In the code cell under the heading \"The Data\", change the slider for the ```centre``` value. This will automatically update the value in the code and will redraw the plot.\n",
        "\n",
        "* At what value of centre does the dataset become linearly separable?\n",
        "\n",
        "\n",
        "### Question for discussion\n",
        "Can you think of some 2-D, 2-class datasets, similar to the one above, that are separable (the points from the 2 classes don't overlap each other), but are not **linearly** separable? Draw some examples on paper or plot them using Matplotlib and discuss this with your neighbour and tutors. \n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "GV5plkAowLy8",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Drawing the line\n",
        "\n",
        "As you may recall from school, a line in 2 dimensions, with coordinate axes $x_1$ and $x_2$, which passes through the origin (0, 0) can be represented by the equation $w_1x_1 + w_2x_2 = 0$\n",
        "\n",
        "We can also write this in vector form as: $\\mathbf{w}^T\\mathbf{x} = 0$, where $\\mathbf{w}^T = [w_1, w_2]$ and $\\mathbf{x}^T = [x_1, x_2]$.\n",
        "\n",
        "When a line (or hyperplane) is defined this way, we call the **parameters**, $\\mathbf{w} = (w_1, w_2)$ a **normal vector** for the line. The normal vector is orthogonal (perpendicular) to the line. We want to construct such a line that separates red and blue  points, which we will call a **decision boundary**. \n",
        "\n",
        "In the following cell, we plot our dataset along with a normal vector $\\mathbf{w}$ and decision boundary. You can adjust the values of $w_1$ and $w_2$ by using the sliders on the right. Observe the effect that the values have on the normal vector drawn in *red* and decision boundary in *black*. Adjust the values so that the black line separates the blue and red points (i.e. red points on one side and blue on the other). Your line should also have the normal vector pointing in the direction of the red points. The reason that direction is significant is that we want to eventually **classify** points on one side of the line as being red and the other as being blue. \n",
        "\n",
        "Is it possible to find a line through the origin that perfectly separates the points?\n",
        "\n",
        "**Note**: Each of our inputs is a 2-D vector, made up of two coordinate values. We refer to these 2 coordinate axes as $x_1$ and $x_2$. For example, if we have an input $(1, 2)$, then we would say $x_1 = 1$ and $x_2 = 2$ for that point."
      ]
    },
    {
      "metadata": {
        "id": "TAXUNshcvPsg",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 365
        },
        "outputId": "e8a20f5e-039b-4932-83f2-033d9513899b"
      },
      "cell_type": "code",
      "source": [
        "#@title Effect of parameters {run: \"auto\"}\n",
        "\n",
        "# Define the parameters\n",
        "w1 = -1 #@param { type: \"slider\", min: -5, max: 5, step: 0.1 }\n",
        "w2 = 1 #@param { type: \"slider\", min: -5, max: 5, step: 0.1 }\n",
        "\n",
        "plot_dataset(inputs, labels)\n",
        "\n",
        "# Add the weight vector to the plot. We plot it in red, as it has to \"point\"\n",
        "# in the direction of the red points.\n",
        "ax = plt.axes()\n",
        "ax.arrow(0, 0, w1, w2, head_width=0.3, head_length=0.3, fc='r', ec='r')\n",
        "\n",
        "# Plot part of the decision boundary in black. It is orthogonal to the weight\n",
        "# vector.\n",
        "t = 2\n",
        "plt.plot([-t * w2, t * w2], [t * w1, -t * w1], 'k-')\n",
        "\n",
        "plt.xlim([-4, 4])\n",
        "plt.ylim([-4, 4])\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAe0AAAFcCAYAAADlIuYrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xd0FFX/BvBntu+mkegCBkUUaYI0\nKYI0aYogiggJvD8FUbGiiCgYVKQKeRXpIiX0EhIRBAVERSw0wZ4XFaIGREoCIYRsts/vj0hCzG6y\nSXZndnafzzmek8wkM99cd3n23rlzRxBFUQQREREFPZXcBRAREZFvGNpEREQKwdAmIiJSCIY2ERGR\nQjC0iYiIFIKhTUREpBCyhLbVakXPnj2xadMmOU5PRESkSLKE9jvvvIOYmBg5Tk1ERKRYkod2ZmYm\njh07hm7dukl9aiIiIkWTPLRnzpyJ8ePHS31aIiIixZM0tDdv3oyWLVviuuuu8/l3nE5XACsiIiJS\nDo2UJ/v8889x4sQJfP755zh9+jR0Oh1q166Njh07ev2d3FyLhBUqk9kchezsfLnLUAS2le/YVr5j\nW/mG7eQbsznK6z5JQ3v27NnFX8+bNw916tQpN7CJiIioBO/TJiIiUghJe9pXGjVqlFynJiIiUiT2\ntImIiBSCoU1ERKQQDG0iIiKFYGgTEREpBEObiIhIIRjaRERECsHQJiIiUgiGNhERkUIwtImIiBSC\noU1ERKQQDG0iIiKFYGgTEREpBEObiIhIIRjaRERECsHQJiIiUgiGNhERkUIwtImIiBSCoU1ERKQQ\nDG0iIiKF0MhdABGREqm//w66j7cDBiMw+mkAOrlLojDA0CYiqgxRRMTY52B4Lw0qS0HRtpR3oR/3\nCmxD/k/e2ijkcXiciKgSDMuXwrhmZUlgA8DffyNi2iQIZ8/KVxiFBYY2EVEl6D7bBUEUy2xXnz0D\nw+rlMlRE4YShTURUCUJhofd9FouElVA4YmgTEVWCs3ETj9tFnQ6OLt2kLYbCDkObiKgSLM+MhuPm\npmW22/rcw9CmgOPscSKiShCviUfeunSY5r0NzU8/QjQYoL+rN/KHPwEIgtzlUYhjaBMRVZIYXwcF\nb7xZ/L3ZHAVk58tYEYULDo8TEREphKQ97cLCQowfPx7nzp2DzWbDU089hTvuuEPKEoiIiBRL0tDe\nvXs3mjVrhsceewwnT57EiBEjGNpEREQ+kjS077777uKvT506hVq1akl5eiIiIkUTRNHD0j4BlpiY\niNOnT2PRokVo3LhxuT/rdLqg0aglqoyIiCh4yRLaAHDkyBG89NJL+OCDDyCUc5tENmdkVshsjmI7\n+Yht5Tu2le/YVr5hO/nGbI7yuk/S2eM///wzTp06BQBo0qQJXC4Xzp8/L2UJREREiiVpaB86dAgp\nKSkAgJycHFgsFsTGxkpZAhERkWJJGtqJiYk4f/48hg4dipEjR+K1116DSsVbxYmIiHwh6exxg8GA\nt956S8pTEhERhQx2c4mIiBSCoU1ERKQQDG0iIiKFYGgTEREpBEObiIhIIRjaRERECsHQJiIiUgiG\nNhERkUIwtImIiBSCoU1ERKQQki5jSkTkV6IIFBQARiOgVstdTbUIuedhXPwOVH+dgNtcE9ZHRsJd\n51q5y6Igw9AmIkXSr1oB4/rVUP2eCTEuDvY7eqLg9amATid3aZWm/vF7RD/xCDTHjhZvM7yfjvy3\n5sLRvaeMlVGwYWgTkeLo169B1CvjIFgLizbknocm8xhUF3KRv3CJvMVVQcTM6aUCGwDUJ/+C6a2Z\nyLujByAIMlVGwYbXtIlIcQwb1pYE9hV0u3ZA9cfvMlRUdUL+RWgOf+Nxn/bbQ1D/8j+JK6JgxtAm\nIsVRnzjucbsqLw/ar7+UuJpqEkVAdHvf5/Kyj8ISQ5uIFMd9tdnjdtFghLPZLRJXUz1idAycrdt4\n3Ods0Qqups0kroiCGUObiBTH1qcvRA/Xee0db4erZWsZKqoeywvj4Ly+Xqltrpq1YRk9ltezqRRO\nRCMixSkcPRbChTwYtrwH9d8n4Y6MhKNjZ+S/NVfu0qrE2aYdLmzZDuOSRVD/dQLumjVROOwRuBs2\nkrs0CjKCKIqi3EWUJzs7X+4Sgp7ZHMV28hHbyndKaCvhYh40h7+Bq96NcN9wo2x1KKGtggHbyTdm\nc5TXfexpE5FiidExcNzB+5gpfDC0iYgCRHPoAAwbUyFcyoezaTMUjhhZtHobURUxtImIAsCwaD4i\nkt+A6tI/w8HpqdB9+AEurkmDGBcnb3GkWJw9TkTkZ8L5czAtmFsS2P/QHfoGpv9Ol6kqCgUMbSIi\nPzNsXA/1mdMe92m9rH5G5AuGNhERkUIwtImI/Mw6eAhctWp73OfwsvoZkS8Y2kREfibGXQXLU8/C\nHVn6flv7rW1heTFJpqooFHD2OBFRAFiffAbOtm1L3/L18GOAySR3aaRgsoR2cnIyDh8+DKfTiccf\nfxy9e/eWowwiooBytmmPS23ay10GhRDJQ3v//v04evQoUlNTkZubiwEDBjC0iYiIfCB5aLdt2xbN\nmzcHAERHR6OwsBAulwtqtVrqUoiIqkUURfz44/fo1Kmd3KVQmJB8IpparYbpn2s66enp6NKlCwOb\niBRHFEVMmzYJvXp1xbJly+Quh8KEbE/5+uSTT/Duu+8iJSUFUVHen2jidLqg0TDUiSh4iKKIpKQk\nzJgxAw0bNsSePXtQu7bnW7yI/EmWiWhffvklFi1ahKVLl5Yb2ACQm2uRqCrl4uPufMe28h3byrPL\nPey5c2ehfv2bkJ6+FbVr12Zb+YCvKd8E1aM58/PzkZycjBUrVqBGjRpSn56IqMr+Hdjvv/8hate+\nRu6yKIxIHtofffQRcnNzMXr06OJtM2fORHx8vNSlEBH5jIFNwUDy0E5ISEBCQoLUpyUiqjIGNgUL\nLmNKRFQOBjYFE4Y2EZEXDGwKNgxtIiIPGNgUjBjaRL4SRahO/gXh3Dm5K6EAY2BTsGJoE/lAt2UT\navTthbgOrRF3WytE/2cwVEd/k7ssCgAGNgUzPpqTqAKafV8jctwLUJ8v6mELViv0u3ZAdeYULnz0\nKaDTyVwh+QsDm4Ide9pEFTCsXVUc2FfS/vgDDOtWy1ARBQIDm5SAoU1UAfXfJ73v+/MPCSuhQGFg\nk1IwtIkq4KrtfbU+V93rJayEAoGBTUrC0CaqgPX/HoI7NrbMdkezW2D9z0MyVET+wsAmpWFoE1XA\n2bETLr3xJhytboWo0cAdGQlbj164uHApoNfLXR5VEQOblIizx4l8YLt/EGwDHoAq60+IRhPEWrXk\nLomqgYFNSsXQJvKVIMBd7wa5qwgfbjdQWAiYTIAg+O2wDGxSMoY2EQUXlwum6ZOh2/kRVDk5cNet\nC+sDCbCOfKrah1Z6YGs//wz6DzZBKLDA2bwFCkeMBIxGucsiCTG0iSioRCS9CNPypcXfq8+fg+Z/\nGYAowvr40yU/aLXCNPu/0O7fB7hccLZsBcvzL0GMi/N4XKUHtjF5OiLmz4ZgtRZteD8duu0f4uK6\nNIjRMfIWR5LhRDQiChrC+XPQf7i17Ha7HYb3NgKiWLTB5UL0sKGImPVf6PZ+Bd2BfTC9uxDRQx+A\ncCm/zO+LoohpUyYWBfb19RQX2KqsP2Fc9m5JYP9Dd3A/jLP+K1NVJAeGNhEFDc2P30N99ozHfaqs\nLAgFlwAAuvRU6HZ/UuZndN8egnHRglLbRFHEjP8Mxtz5s9EQwOenT6PRqCegzvjJ7/UHiv69jVDn\n5nrcp/32kMTVkJwY2kQUNFz1G8AdHe1xn3i1GaLRBADQHjoIb1PTNFeEsSiKeOOJEXj7k51oCGA3\ngGttVuj27Eb00yOBf/Vcg5YfJ+KRsjG0iShouK+rC3uXbh732XrfBajVRd+UM/nKbSoK9svXsGe/\n/15xYF+5tp3mfxkwrFnpl7oDzTZ4CFxXXeVxn6Nte4mrITkxtIkoqFyavQDWfvcW97hdNWvBMmwE\nLBMmFv+MdfBQuGPKTr4StVrY+/QrNemsgdFYJrAvU5/8K0B/hX+561yLwidHwW2KKLXd3rEzLGNe\nkqkqkgNnjxNRUBGjY5Cfshqq41lQ//YrnK1uhfivXqar2S0oeGEcTHPfhjonGwDgjolB4YMPw9a3\nf6lZ4tubNkP8B5s9nst5w40B/3v8pfDZMXB06AhD+kbAYoGzZWtYHxzOR8OGGYY2EQUld93r4S7n\ngSzWJ56B7b6BMGxYBzgdsN03EK76N5W5rSvuxHG49n5dHO6XOVq0gm3I/wX6z/ArZ9vbcKntbXKX\nQTJiaBORpFR/nYBhZQqEgktwtGgF+wMJJdeqK0msfQ0KR79Q9LWX+7Cdta/BpTdnw7hoATQ//gDR\naISj/W0oeH0aoNX6808jCjiGNlGgFRYCBgNnAAPQp29ExOsTim/rEgHY09bj4or1QGRklY9b0cIp\n9rvvgb1PPwjZ2YBB79NiJMLFPGi+/grua6+D65bmVa6NyJ84EY0owEzzZ6NG766ImDAOmgP7ShYI\nCTcFBTDNnFrqPmwBgP6LPYiYMaXKh/V5pTNBgFizZsWBLYowTXkdsV1uQ41hQxDbtydiHugP1e+Z\nVa6RyF8Y2kQBVvjQCKiOH4dpyTuocX8/1LizW1gGuCE9FZqsPz3u0x7cX6VjBmJpUsPihTAtmA31\n3ycBAILVCt0XnyPquafC6v8XBSeGNlGAibVqwdn6VgCA4HBA+/13YRnggrXQ+067o9LHC9Ra4voP\nt0Jwu8ts1x46CO0nH1f7+ETVwdAmkoCjfYcy27wFuPqnH2SoMPCs9z0Al7mmx33OFi0rdaxAPvxD\nde6cx+2CywVN5lG/nIOoqmQJ7d9++w09e/bEmjVr5Dg9keSsQx6EK9bz06eAkgDXb90M1RnPa28r\nnVirFgqHPwJRry+13dG4CSyjx/p+nAA/rctVt67H7W5TBOy33e638xBVheSzxy0WC6ZMmYIOHcr2\nPIhC1eUhcvWnu7z+jKv2NcifNQ+Onr0lrExahS++DNfNTaHfugVC/kU4GzRC4ZPPQKxV26ffl+Lx\nmtYHh0P7zQGoLl4std3RoydcLVv59VxElSV5aOt0OixZsgRLliyR+tQUZDIyBCxapMOvv6oRGSmi\ne3cnnnrKAVWIXrRxtO8AvZfQDofAvszetz/sfftX+vekeh62/e57cMluh2HFMqiP/gYxOhr2rt1R\n8PpUv5+LqLIkD22NRgONhreHh7uMDAHDhxuRlVWyqMZXX6mRmanC22/bZKwscKxDHoTxnflQ554v\ntd1Vs1bYBHZVSRXYl9nuGwjbfQMBu71oARbeY09BIujTMzbWBI2maqslhROzOUruEiolJQXIyvr3\nVgEffKDD+PE6NGsWuHPL1lbmKOC29sD27SXb6tSBun9/1Mj4DhgyUJ66yhEMrytRFJGUlIS5c2eh\nYcOG2L17N+LjPT3+Q17B0FZKwHaqnqAP7dxci9wlBD2zOQrZ2flyl1Ep339vAlD2w1h+PrBhgw3P\nPWcPyHnlbitjq7aI/Ce0XbWvQf5/58DRszf0qeugfv5FWJJek622f5O7rYCyPez09K3QauWvC0DR\nSndaLaDRBEVbKQHbyTflfbAJ0auHFOxMJu/3JMfGhu79ypdnkf/7GrYtYShc9W+CafpkmSsMHlIP\niftK99FWRA/oi7jWTRF7WytEPvcUkJcnd1kUJiTvaf/888+YOXMmTp48CY1Gg507d2LevHmoUaOG\n1KWQjLp3d+HAgbIvv5tucmHw4MovtKEUYq1acLa/DYUPjShzDduWMBT61HUwTZ8cVD1uOXgM7Kuu\nhmHJO9Ae2AcAcHTsDOuwEVV+2EhVaPbsRuTzo0rmJZzLgeZ4FnDuLLA6rXrXvl2uov/4qE0qhyCK\nwb0ME4dSKqbEISeXC3j2WT0+/FALi6XoH7r69V2YOtWGHj1cATtvULSVKJb7j7s+dR3UmcdkD265\n2spjYF9tRvTD/4F+5/ZSP2vtdx/yl66AVLccRI14EIZtW8ru0GpxYVUqHD16VvqYwt8nETl5IrQH\n9wF2O5wtWsLyzGg4O4TePeFB8f5TAA6PU9BRq4EFC2zYvNmC8eNtmDnTis8+swQ0sINGBb2xcB4q\n9zYkbli3ukxgA4B+22boNqVJVp/6r+Oedzgc0PzwbeUPaLcj+pEHYdi0Eeq/TkB99gz0u3Yi+umR\nUP16pHrFUkhiaJOsWrZ0Y8wYOx5+2AGjUe5qgkc4Bnd517C1+/d6/B0BgO7LPZLV6L7qaq/7XNd5\nXkmtPIZ1q6E7fKjMdvVfJ2BaurjSx6PQx9AmClKhGNyqI/+Dcc5bMCxfClhK7gypaNKZWN51a7V0\nU3Ns9w0sswwrAKBNG9jvH1Tp46mP/up1n+qEl149hTWGNlEQC5ngdrsR+cKziO3bE5HTJiFq3BjE\n9rgd2h0f+jRL3N6jF0QP161FrRa2u+6W6q+ALWEoCsaOh7PeDUXn1+th79QFWL68ShPi3F4eoAIA\n7jjva9VT+OJEtBDAyR2+U2pbyTE5zZ9tZXhnPiInJuHfV/Md19fD2Lv7Ye4788u/rUsUETn2ORg2\nrIXgKLq7QNTpUPjgcBRM/6/0K5ZZLNAe2Au3uRZczW6pclsJF/NQo1dXaP74vdR2d1QU8pathrNb\nd39VHBSU+v6TWnkT0RjaIYBvBN8pua2kDm5/tlX0oHuh37O71DYRQBKAGYBv92GLIrS7P4N+13aI\nAGx9+sHZpZtf6quu6rSV5qsvETl1IjQ/fAfB5YKzQUMUPvYkrMMf8XOV8lPy+09KDO0QxzeC75Te\nVlIGtz/bqkafHtAe/qb4+ysD+6bYOGzasy8oFk6pqmq3lShC/c0BqC7lw9Gpa8jeq630959UeMsX\nUYhQ6jVuZ6PGxV9fGdgNBQFb5i5QdGD7hSDA1e42OLr3CtnAJv9gaBMpjBKD2/LkKDhvqF86sAFs\n79MX5jv7ylsckYIwtIkUSGnB7W7UGBeWr8FLTW7GDAANDAZse3o0opaukrs0IkVhaBMplJKCWxRF\nTNmUhjeP/A/169+E9w7+gBoTJwOaoH/QIFFQYWgTKZgSgjtYn9ZFpEQMbSKFC+bgZmAT+RdDmygE\nBGNwM7CJ/I+hTRQigim4GdhEgcHQJgohwRDcDGyiwGFoE4UYOYObgU0UWAxtohAkR3AzsIkCj6FN\nFKKkDG4GNpE0GNpEIUyK4A6HwNanrkP0/f0Q2+YW1OjTA4YFc4HgftYShSiGNlGIC2Rwh0Vgr16B\nyJfGQP/VF9Acz4L28DeInPIaTFMnyV0ahSGGNlEYCERwh0NgQxRhXLcaqkJLqc2C2w3D+2kQLvEx\nkyQthjZRmPBncIdFYAMQLuVDlXnM4z71Xyeg+f5biSuicMfQJgoj/gjuoA9siwWmmdMQPfg+RCfc\nD+OsZMBmq9KhRKMJYo1Yj/vcEZFwXXt9dSolqjSGNlGYqU5wB31g22yI+b/BiHhrJvSffwb97k8Q\nOWMqoof/B3A6K388jQb2bnd43OW4vRPc9epVr16iSmJoE4WhqgR30Ac2AGPKEui++qLMdt2nH0O/\ncX2Vjlkw+Q1Y7xsId2QkAEDU62Hr1h35b82tVq1EVcHQJgpTlQluJQQ2AK/XmAUA2oP7q3ZQgwH5\ni5cj96NPkT9zFi5s2oqLGzdDrFW76oUSVRFDmyiM+RLcSglsABB1eu879eXs84G7cRNYH34Uzra3\nVes4/qb6PRPqX47wvvEwUaXQFvniIAoZ5QW3kgIbAOx394Wo1ZbZLhqMsN47QIaKAkezfx9i7u2D\nuC7tEdutA2rceQd0W7fIXRYFmNfQPnLkCB566CH0798fq1atKrVv2LBhVT7h9OnTkZCQgMTERPz4\n449VPg4R+Y+n4FZaYAOAvU8/WB55HG5TRPE2d1QULE+PgrNjZxkr8y8h9zyinnsSun1fQ7DbIbjd\n0H7/LSLHjYH6xx/kLo8CyGtoT5o0CcOHD8eUKVNw8OBBJCUlFe+rak/74MGDyMrKQmpqKqZNm4Zp\n06ZV6ThE5H9XBrcoikhKSlJUYF9mmTwdFz7YgYLnXkDB6LHI/fATWMa9IndZfmVc+i40f/xeZrs6\nJxvG1ctlqIik4jW0tVotunfvjhYtWmD+/Pmw2Wx4++23q3Wyffv2oWfPngCA+vXrIy8vD5cuXarW\nMYnIf2wJQ+G8sT6S+/bCjBkzFBfYl7mat4BlwkRYkl6Du3ETucvxO+HMaa/7VGfOSFgJSU1T3s4D\nBw6gffv2AICZM2fimWeeQXJyMhwOR5VOlpOTg6ZNmxZ/HxcXh+zsbET+cyuFJ7GxJmg06iqdL5yY\nzVFyl6AYbCvvRFFE0t9ZeOvQQTRs2BC7d+9GfHy83GUpgqSvq8YNvO7S168X1K/xYK5NCbyG9iuv\nvIIJEyZg5cqViIiIgEajwcKFC7FgwQL89NNPfjm5L8PsubmWCn8m3JnNUcjO5hrIvmBbeffva9i7\nd++GVsv28oXUryshcRhiVq6C9tdfSm13xcfjwtCH4Q7S/2d8//mmvA82XofHGzVqhPT0dDzwwAM4\ndOhQ0Q+rVKhbty6uuaZqQ2U1a9ZETk5O8fdnz56F2Wyu0rGIyH88TTpjDzt4iZFRuPjucth63QVX\nbBzc0dGwde6Ki3PegbtBQ7nLowAqd3gcABYsWIDJkyejUaNGOHXqFLRaLVJTU6t0sttvvx3z5s1D\nYmIiMjIyULNmzXKHxoko8JQ4S5wA981NcXHtRggX8wCnE2LcVXKXRBKoMLRvvPFGPPvssxg9ejQi\nIiKwaNEiXHVV1V4crVu3RtOmTZGYmAhBEDBx4sQqHYfo31wuQKUCBEHuSpSFga18YnSM3CWQhCoM\n7VdffRV//vkn1qxZgwsXLuD5559Hr1698OSTT1bphGPHjq3S7xF5smWLGitW6JCZqUJ0tIhu3Zx4\n9VV7dRe/CgsMbCLlqTC069evj8mTJ0MQBNStWxfr16/H3LlcKJ/kt327GmPHGpCXVzQ14/Rp4Lff\n1DhzRsCSJVV7FGO4YGATKVOFy5gOHz4cwhVjjnq9Hi+++GJAiyLyxerV2uLAvtKnn2rx888cJ/eG\ngU2kXHxgCCnWn396fvleuiTg668rHEQKSwxsImVjaJNixcV5vs9frRZRv75b4mqCHwObSPkY2qRY\nffo4oVKVDe42bVzo0cMlQ0XBi4FNFBo4hkiK9dRTDpw9K+D997U4fVoFvV5Eu3YuJCdbeevXFRjY\nRKGDoU2KJQjApEl2PPecHV98ocH117vRqhWHxa/EwCYKLQxtUry4OOC++5xylxF0GNhULqsV+k1p\nEAoLYRswkCuqKQSvaROFIAY2lUe/+T3EduuA6NFPI+rlsYjt2gHGWclyl0U+YGgThRgGNpVHOH0K\nEa++DM3vmcXb1GdOI2L2m9Bt3yZjZeQLhjYFjTNnBGzbpsZvv3EWWVUxsKkixhXLoD5zusx2wWqF\nfvMmGSqiyuA1bZKd0wmMG6fH9u0a5OSoEBEholMnJ95+24qrr5a7OuVgYJMvhIsXve/L976PggN7\n2iS7adN0WL1ah5ycopdjQYGAnTu1GDPGIHNlgbN0qRb33GNE27Ym3HuvEatWVe/zMwObfOVs0QKe\nlyUCXDc1kLQWqjz2tElWogjs2uX5ZfjVVxpkZgqoX9/bPzFVd+4csGGDFg6HgPvvd6BuXf+fw5tZ\ns7R48009nM6iywBZWcC336pRWGjD4487Kn08BjZVhu2BRBg2rIPu6y9LbXc2aATLE8/IVBX5ij1t\nkpXNBpw/7/ka9qVLAo4e9f9LNCVFg27dIjBpkgHTp+vRq5cJM2bo/H4eT2w2IC1NWxzYJdsFbNig\nhauSC7kxsKnS1GpcXLUelkdGwtGsORyNGqNw8BDkLV8DMb6O3NVRBdjTJlnp9UDduu7iofEr1azp\nRps2/l2ONCMDeOMNfamng+XmqjB/vg4tW7pw112BXf70jz9UyMxUe9yXmanCmTMC4uN96/UzsKmq\nxKhoFLzxptxlUBWwp02yEgQgMdEBvb5sUN19t9PvE9FSUuDxcZ52u4CtW7X+PZkHZrOI2FjPq7YZ\njSK+/VYFhw8j5AxsovDE0CbZDR/uxNSpVrRp44TZ7EaTJi6MHm3DG2/Y/H6uggLv+ywWv5+ujKuu\nEtGli+fefG6uCiNGmNCjhwnp6d4HwRjYROGLw+MUFIYNc2LYMCccDkCjQcAe+NGuHfDuu573NWki\nzbrlyclWWCzAl19qYLUKAEQAJX/wL7+o8eqretx8sws331x6BIKBTRTe2NOmoKLVBi6wAeChh4Au\nXcquU96ihRNPPGEP3ImvEBsLrF1rxZYtFnTq5MCVgX3ZuXMqrF5deriegU1E7GlTWNFogFWrCvHm\nmzocPKiGywW0bOnCmDF2REdLW0urVm7Ex3vff/58yWdqBjYRAQxtCkMmE/Daa9L0qitSr573IfnL\n+xjYRHQZh8eJZPTYY3Y0blx2YlrDhi6MHGlnYBNRKexpE8koJgZISSnEzJl6HD6sgigCrVu78eKL\nNsTF+S+w//5bwIIFWvz6qwomE3DnnU4MHeoM6PwBIvI/hjaRzG66ScSSJVa4/xkpV6n8OySelSXg\nwQeN+OWXkkVddu3S4MgRB6ZO9f9tdUQUOBweJ/Izi6VoudLKUqn8H9gAMG+erlRgA4DLJSA1VYNj\nx9jVJlIShjaFnc8+U2PYMAO6dTPh/vuNSEnRQvTD80K++EKNwYMNuPXWCLRrZ8LIkQacPFm5UAzE\nNeyff/b8Ns/LU2HbtsCvAkdE/sPhcQormzcDTz5pQG5uSZDt26fG6dMCkpKqPqP8558FjBplwKlT\nJcfdvFmNrCwBW7cWQufD80gCNemsvHObTBV8WhFF6N5/D7pPP4bgcMDRth2swx4p/6BEFDCS97QP\nHjyIDh06YPfu3VKfmgjz5qFUYANFQ8UbN2pw8WLVj7t8ua5UYF/23XcarFtXcW82kLPEO3b0vGzq\ndde5MHRo+QudR4wdjeinHoUxbQMMm99D1IRxiH4woWrj/0RUbZKG9vHjx7F8+XK0bt1aytMSAQCc\nTuDIEc/7/v5bjS++qPrA04k/GMApAAASmElEQVQT3ofBMzPLHyIP9G1dY8bY0aePAxpNSa+6Vi03\nxo+3IzKynF/87DMYU9dCcJe+l1y/+1MYl7zjt/qIyHeSDo+bzWbMnz8fEyZMkPK0RAAAtRqIjgZO\nnSq7T6cTUadO1dceN5u976tZ0/s+Ke7D1umAFSus2LVLjX371IiIAB56yIGaNSsYGt+6FYLd8yUD\n7cEDKPRrlUTkC0lD22g0Snk6olIEAejRA/j117L72rZ1oVWrqod2QoIDO3dqcPFi6V51/fouPPyw\n5+CTcuEUQQB693ahd+9KPC+8vJu4VZx1TiSHgIV2Wloa0tLSSm0bNWoUOnfuXKnjxMaaoNGoK/7B\nMGc2R8ldgiK8+Sbw99/Azp1AYWFRLrVvDyxerKlWGw4cCJw/D8yZA2RkFK1x3r49MGOGGjfcUPa4\noigiKSkJc+fOQsOGDbF7927El7cQuRwGDSp6JJqHZ5bqe/fka+5f2B6+YTtVjyCK/rjZpXLGjx+P\nO++8E3fccUeFP5udnS9BRcpmNkexnXx0ua2++UaF/fvVqFfPjb59XVD5aXaHwwEcPKhCZCTQvLnb\nY2dVKUuTms1RKHhmNExLF0H4Z+KZKAiw9b0H+YtXFH0yIQB8D/qK7eSb8j7Y8F1HYaltWzfatvX/\n87O1WuD2270fVymBfZll4hTY7+gBw7YPAKcD9ts7wz7gAfjtUw4RVYqkof35559j2bJl+P3335GR\nkYHVq1cjJSVFyhIoyIli0X+hmAlKC+zLnF264VKXbnKXQUSQOLS7deuGbt26SXlKUoiCAmDSJD2+\n+kqNggKgSRM3Hn/cgTvuqMTEqSCm1MAmouDC4XGSnSgCjz5qwKeflixCcuqUGj/9pMbSpVZ06KDs\n4GZgE5G/hOAgJCnN5597XtgkO1uFlSuVvTY2A5uI/ImhTbI7fFgNh8Pzfb9//qnc+4EZ2ETkbwxt\nkl18vPfZ1nFxkt+R6BcMbCIKBF7TJtkNGuTEkiUuZGSUXkRHqxXRr59Tlpry84seAvLbbwIuXRLQ\nqVPRwzVMpop/l4FNRIHCnjbJTqsFZs0qRNu2zuKHWlx7rQvPPWfH0KHSh/bXX6vRs6cJU6fqsXGj\nDh99pEVSkgFdupjw4Yflr87HwCaiQGJPm4JCq1Yitm0rxP79Kpw5o0KPHk5EybDaoSgCU6fq8Mcf\nZcP5+HE1XntNjy5dLB5r81dgWyzA9Ok67N2rgdUKNGvmxjPP2NC8uTIvFRCR/zC0KWgIAtChgxuA\n/1cq89U336jw/ffee9MnTqgxYoQBdeuKiI8X8eijdsTE+C+wRREYMcKAzz4rmTV/7Jga332nwpo1\nhWjUiMFNFM4Y2kRXyM8HXK7yZ6zv2VMSqO+9p8HChRZs2+afIfHt29XYs6fs2zIrS43Fi3V46y1b\nlY5LRKGBoU10hU6d3LjpJheOHfPtyXLHjqnw2GNTkZXln2vYhw+rvX5oyMzkFBSicMd/BYiuoNcD\njz1mh8nkyzC0CCAJWVnJqFfPP5POYmO9nzcmhkPjROGOoU30Lw8/7ERKSiF69XKgdm0X4uLcaNjQ\nCUG4MjSLAhuYAUFoiGXLPvLLLPFhwxy4/vqyy7ZqtSL69pXn9jciCh4cHqewVVAAbNmigSAA993n\nhNFYsq97dxe6dy8JT1EE+vY14tAhDa4MbKAhbr11F265JdYvNUVFAW+8YcOUKXocOVI0RF+zphtD\nhzoweDBDmyjcMbQpLK1cqcH8+TpkZRUF4+zZ5d8XLgjAc8/ZMWaMgOzsV3A5sGvW/BQvvRQHwH8P\nNenZ04WuXS344AMNLlwQ0L+/E2Yzh8aJiKFNYejbb1WYOlWPvLySq0N//KHG5Ml6tGrlQpMmngOy\nd28nevd+FWvXzkJERAPcc88OjBpVAw0a+P8pZFotMHAge9ZEVBqvaVPYWb9eWyqwLzt/XoU1azw/\nVezyfdhr1xbNEt+3bxvmzr0aDRqwB0xE0mFPm8LOxYve78O+eLFsmAfr0qQHDqhw+LAajRq50b27\nC4JyH4hGRD5iaFPYuekm7yuuNWhQel8wBvalS8ATTxjwxRcaWK0CNBoR7dq5MG+eFdddx54/USjj\n8DiFnZEj7WjatOx16BYtnHjkEXvx98EY2AAwYYIeH3+shdVa1LV2OgXs3avB+PEGmSsjokBjaFPY\niYkBVqwoREKCHQ0auNCggQtDhtixcqUVERFFPxOsgW2zAV995Xm1tr171cjK4hg5USjj8DiFpeuv\nFzFvnud1vIM1sIGiJ4B5uyZfUCDg9GkB11/PIXKiUMWeNtEVgjmwAaBGDe/X5G+4wYXmzeV7QhoR\nBR5Dm+gfwR7YQNEiL8OGORARUbo3rdWKSEhwlFrVjYhCD4fHiaCMwL4sMdEJk6kQ69drceKECrVq\nuXHvvU489BAXYyEKdQxtCntKCuzL+vd3oX9//6/ERkTBjcPjFNaUGNhEFL4Y2hS2GNhEpDQMbQpL\nDGwiUiJJr2k7nU5MmDABx48fh8vlwksvvYQ2bdpIWQIRA5uIFEvS0N6yZQuMRiPWr1+Po0eP4uWX\nX0Z6erqUJVCYY2ATkZJJGtr9+/dHv379AABxcXG4cOGClKenMCeKIpKSkhjYRKRYkoa2VlvyrOKV\nK1cWBziRFBYvXogZM2YwsIlIsQIW2mlpaUhLSyu1bdSoUejcuTPWrl2LjIwMLFq0qMLjxMaaoNF4\nfkAClTCbo+QuIegZjVp06tQJqampiI+Pl7scReDryndsK9+wnapHEEVR0qcLpKWlYceOHVi4cCH0\nen2FP5+dnS9BVcpmNkexnXzEtvId28p3bCvfsJ18U94HG0mHx0+cOIENGzZgzZo1PgU2ERERlZA0\ntNPS0nDhwgWMHDmyeNuyZcug0+mkLIOoyrZuVSM9XYvsbAF16oh48EEHunThcqJEJA3Jh8cri0Mp\nFeOQk++q01aLF2sxbZoehYUlz7OOjXXjzTetuOee0Atuvq58x7byDdvJN+UNj3NFNCIf2O3AqlXa\nUoENALm5KixdqkNwf/StHqcTmDNHh0GDDLjvPiMmTtQhJ0fuqojCE5/yReSDH35Q4bffPN/F8L//\nqXDxIhATI3FREhBFYORIA7ZtK7ldc+9eDfbuVWPjxkLExspYHFEYYk+byAdxcSIMBs/daZNJRKjO\nq9y5U40dO8p+tv/hBw0WLuRcFCKpMbSJfFC/voh27Txft+7QwQWDQeKCJPLVV2o4nYLHfRkZ/OeD\nSGp81xH5aOpUK5o3dxZ/r1KJ6NDBiSlTbDJWFVjlfRgJ1dEFomDGa9pEPmrcWMSOHYVIS9Pg+HEV\nbr7Zhb59XVCF8Eff//zHgdWrtcjNLf1HqlQievZ0evktIgoUhjZRJWg0wJAh4RNWN9wg4sUXbZg1\nS4+cnKLgNplEDBrkwNCh4dMORMGCoU0hSxSBJUu02LFDg9xcATfc4MaYMUCzZnJXpiyPPupE374u\nrFunhd0O9OnjRMuWbrnLIgpLDG0KWa+/rsO77+rgdhdNpMrIUOPgQWDuXDW6dw+9xVAC6ZprRLzw\ngl3uMojCXghfjaNwlp0t4L33tMWBfdnZs8CyZVovv0VEFNwY2hSSdu5U4+xZzy/vX37hy56IlIn/\nelFIuvZaN9Rqz4uhREZKXAwRkZ8wtCkkde3qRqtWnq9bd+3KWc9EpEwMbQpJggDMmGFFixZOAEU9\nbqNRxAMPABMmcEIVESkTZ49TyGreXMT27YXYskWDkydV6NjRiT59IpCdLXdlRERVw9CmkKbRAAMH\ncjiciEIDh8eJiIgUgqFNRESkEBwep7CXmwusWaOFzSagTx8Hmjb1fKsYEZHcGNoU1tat02DGDD1O\nny4adFqwQIdBgxyYOdMGwfNjpImIZMPhcQpbp08LmD69JLABoKBAwKpVWqxezaVOiSj4MLQpbK1e\nrfW41KnbLeCTT9QyVEREVD6GNoUti8X7vsJC6eogIvIVQ5vCVqdOLmi1niedNW7M50UTUfBhaFPY\n6t7dhbvuKrvwys03u/DMMw4ZKiIiKh9nj1PYEgRg0SIrmjZ14+uv1bDZgGbNXHj2WQdq1eJtX0QU\nfBjaFNa0WmDMGDvGjJG7EiKiinF4nIiISCEY2kRERAoh6fD4uXPnMG7cONhsNjgcDrz88sto0aKF\nlCUQEREplqQ97Q8++AD33nsvVq9ejTFjxmDOnDlSnp6IiEjRJO1pP/zww8Vfnzp1CrVq1ZLy9ERE\nRIomiKIo6b0t2dnZeOKJJ1BQUICVK1dWGNxOpwsaDZeUJCIiClhop6WlIS0trdS2UaNGoXPnzgCA\nPXv2YOXKlUhJSSn3ONnZ+YEoL6SYzVFsJx+xrXzHtvId28o3bCffmM1RXvcFbHh80KBBGDRoUKlt\nBw8eRF5eHmJiYtC1a1e89NJLgTo9ERFRyJF0ItrHH3+M999/HwDw66+/4pprrpHy9ERERIom6US0\np556CuPHj8euXbtgt9vx+uuvS3l6IiIiRZM0tOPi4rB48WIpT0lERBQyuCIaERGRQjC0iYiIFIKh\nTUREpBAMbSIiIoVgaBMRESkEQ5uIiEghGNpEREQKwdAmIiJSCIY2ERGRQjC0iYiIFIKhTUREpBAM\nbSIiIoVgaBMRESkEQ5uIiEghGNpEREQKwdAmIiJSCIY2ERGRQjC0iYiIFIKhTUREpBAMbSIiIoVg\naBMRESkEQ5uIiEghGNpEREQKwdAmIiJSCIY2ERGRQjC0iYiIFIKhTUREpBAMbSIiIoWQJbRzcnLQ\ntm1bHDhwQI7TExERKZIsoZ2cnIzrrrtOjlMTEREpluShvW/fPkRERKBhw4ZSn5qIiEjRJA1tu92O\nBQsW4Pnnn5fytERERCFBE6gDp6WlIS0trdS2Ll26YNCgQYiOjvb5OGZzlL9LC0lsJ9+xrXzHtvId\n28o3bKfqEURRFKU6WWJiItxuNwDg+PHjiIuLw5w5c9CgQQOpSiAiIlIsSUP7SuPHj8eAAQPQvn17\nOU5PRESkOLxPm4iISCFk62kTERFR5bCnTUREpBAMbSIiIoVgaIcQLg9bMafTiXHjxmHIkCEYPHgw\nDh06JHdJQWf69OlISEhAYmIifvzxR7nLCWrJyclISEjAwIED8fHHH8tdTtCzWq3o2bMnNm3aJHcp\nihWw+7RJelwetmJbtmyB0WjE+vXrcfToUbz88stIT0+Xu6ygcfDgQWRlZSE1NRWZmZlISkpCamqq\n3GUFpf379+Po0aNITU1Fbm4uBgwYgN69e8tdVlB75513EBMTI3cZisbQDhFcHtY3/fv3R79+/QAA\ncXFxuHDhgswVBZd9+/ahZ8+eAID69esjLy8Ply5dQmRkpMyVBZ+2bduiefPmAIDo6GgUFhbC5XJB\nrVbLXFlwyszMxLFjx9CtWze5S1E0Do+HAC4P6zutVgu9Xg8AWLlyZXGAU5GcnBzExsYWfx8XF4fs\n7GwZKwpearUaJpMJAJCeno4uXbowsMsxc+ZMjB8/Xu4yFI89bYXx1/Kw4cBTW40aNQqdO3fG2rVr\nkZGRgUWLFslUnTLwjtCKffLJJ0hPT0dKSorcpQStzZs3o2XLlrx85we8TzsEcHnYyklLS8OOHTuw\ncOHC4l43FZk3bx7MZjMSExMBAD169MCWLVs4PO7Fl19+iTlz5mDp0qWoUaOG3OUErdGjR+PEiRNQ\nq9U4ffo0dDodJk+ejI4dO8pdmuKwpx0CNmzYUPz15eVhGdienThxAhs2bMCaNWsY2B7cfvvtmDdv\nHhITE5GRkYGaNWsysL3Iz89HcnIyVqxYwcCuwOzZs4u/njdvHurUqcPAriKGNoWVtLQ0XLhwASNH\njizetmzZMuh0OhmrCh6tW7dG06ZNkZiYCEEQMHHiRLlLClofffQRcnNzMXr06OJtM2fORHx8vIxV\nUajj8DgREZFCcPY4ERGRQjC0iYiIFIKhTUREpBAMbSIiIoVgaBMRESkEQ5uIPNq0aRNatmyJvXv3\nyl0KEf2DoU1EZWzevBk///wzGjduLHcpRHQFhjZRmFu+fDleeeUVAMDvv/+Ou+66Cz169MBrr70G\nrVYrc3VEdCWGNlGYGzZsGP744w8cPnwYkyZNwuTJkxEVFSV3WUTkAUObKMypVCpMnz4do0ePRsOG\nDdGuXTu5SyIiLxjaRIS8vDyYTCacOnVK7lKIqBwMbaIwZ7PZMHHiRCxatAharRabN2+WuyQi8oIP\nDCEKc8nJyYiIiMDTTz+NnJwcJCQkYMCAAThw4ACOHDmC+Ph4xMTEYM6cOYiLi5O7XKKwxtAmIiJS\nCA6PExERKQRDm4iISCEY2kRERArB0CYiIlIIhjYREZFCMLSJiIgUgqFNRESkEAxtIiIihfh/6Q2E\nx8PzCIMAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<matplotlib.figure.Figure at 0x7f0707754dd0>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "igAjsyMldbMr",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Classification\n",
        "Given a normal vector $\\mathbf{w}$, we can evaluate which side of the decision boundary a particular point $\\mathbf{x_i} = (x_{i,1}, x_{i, 2})$ lies by evaluating $\\mathbf{w^Tx_i}$. If $\\mathbf{w^Tx_i} > 0$, the point $\\mathbf{x_i}$ lies to one side of the boundary (in the direction of the normal vector), and we can classify that point as belonging to class 1 (in our case, \"red\"). If $\\mathbf{w^Tx_i} < 0$ the point lies on the other side and can be classified as class 0 (in our case, \"blue\"). Finally if $\\mathbf{w^Tx_i} = 0$ the point lies on the decision boundary and we can decide whether to classify it as either 0 or 1, or ignore it. "
      ]
    },
    {
      "metadata": {
        "id": "jqW7RpSTaRZH",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## How \"good\" is the line?\n",
        "\n",
        "If you've played around with the above code, you may have developed some intuition around how different settings of the parameters influence the final placement of the decision boundary. The purpose of machine learning is to *automatically* adjust the values of $w_1$ and $w_2$ to find a suitable decision boundary! But to do this, we need to mathematically specify some **loss** or **objective** function. The loss is a function of the parameters $w_1$ and $w_2$ and tells us how good a certain configuration of the parameter values are at classifying the data. This function is defined such that it reaches its optimum setting when it is minimised, i.e. the *smaller* its value, the *better* the separation between the classes. An additional property a loss function can have that is often crucial for machine learning is being *differentiable*. A differentiable loss function allows us to use *gradient-based optimisation* to find its minimum and the corresponding optimal values of $w_1$ and $w_2$. \n",
        "\n",
        "For this classification problem, we consider the **binary cross-entropy** loss function to measure how good the model's predictions are.  This loss function compares the model's prediction for each example, $\\mathbf{x_i}$ to the true **target** $y_i$ (we often refer to the true label associated with an input as the \"target\"). It then applies the non-linear log function to penalise the model for being further from the true class. The equation for the binary cross entropy loss, on a dataset with $N$ points is:\n",
        "\n",
        "\\begin{align}\n",
        "l(\\mathbf{w}) = -\\frac{1}{N}\\sum_{i=1}^N y_i log(\\hat{y}_i) + (1-y_i)log(1-\\hat{y}_i)\n",
        "\\end{align}\n",
        "\n",
        "where $\\hat{y}_i = \\operatorname{sigmoid}(\\mathbf{w}^T\\mathbf{x_i})$ and the $\\operatorname{sigmoid}$ function is defined as:\n",
        "\n",
        "$$\n",
        "\\mathrm{sigmoid}(a) = \\frac{1}{1 + e^{-a}} .\n",
        "$$\n",
        "\n",
        "The reason we use the $\\operatorname{sigmoid}$ function is because our classifier can output any real value. The binary cross entropy loss function, however, expects the predictions made by a classifier to be between $0$ and $1$. The sigmoid function \"squashes\" any real number inputs to lie in the interval $(0, 1)$.\n",
        "\n",
        "Let's now wrap this in a Python function so that we can compute the loss for any values of $w_1$ and $w_2$:\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "wKkpBZ6ZWLoF",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def compute_loss(w1, w2):\n",
        "  \n",
        "  total_log_likelihood = 0  \n",
        "  \n",
        "  # Add the contribution of each datapoint to the loss\n",
        "  for (x1, x2), target in zip(inputs, labels):\n",
        "    # As our targets are 0 or 1, our prediction function must output a value between 0 and 1.\n",
        "    # The sigmoid function 'squashes' any value to lie between 0 and 1:\n",
        "    prediction = tf.sigmoid(w1*x1 + w2*x2)  \n",
        "    \n",
        "    # Compute the local loss term\n",
        "    # We add 1e-10 to make the log operations numerically stable (i.e. avoid taking the log of 0.)\n",
        "    log_likelihood = target * tf.log(prediction + 1e-10) + (1.-target)*tf.log(1.-prediction + 1e-10)\n",
        "    total_log_likelihood += log_likelihood\n",
        "  \n",
        "  loss = -total_log_likelihood\n",
        "  average_loss = loss / len(inputs)\n",
        "  return average_loss"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "x2P-s50pgj-N",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### More on the sigmoid function"
      ]
    },
    {
      "metadata": {
        "id": "AQqQ_quqwCFF",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "The sigmoid function is defined as\n",
        "$$\n",
        "\\mathrm{sigmoid}(a) = \\frac{1}{1 + e^{-a}} .\n",
        "$$\n",
        "Can you show that\n",
        "$$\n",
        "1 - \\mathrm{sigmoid}(a) = \\frac{1}{1 + e^{a}} ,\n",
        "$$\n",
        "and draw both of these on a sheet of paper?\n",
        "\n",
        "* What is its value when $a = \\mathbf{w}^{T}\\mathbf{x}$ is positive? negative? and zero?\n",
        "* What happends to its value when  $a = \\mathbf{w}^{T}\\mathbf{x}$ becomes larger?\n",
        "* What is the value of $\\mathrm{sigmoid}(\\mathbf{w^Tx})$ when $\\mathbf{w}^T\\mathbf{x} = 0$? How does this change how we classify points on either side of the decision boundary?\n",
        "\n",
        "After working through the above questions, explain to your neighbour why the binary cross-entropy loss function makes sense. \n",
        "\n",
        "**HINT**: Remember the idea of the loss function is to return small values when the classifier makes good predictions and large values when the classifier makes bad predictions. "
      ]
    },
    {
      "metadata": {
        "id": "x46fjqTUf4Dj",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Bonus Question\n",
        "We derived the `compute_loss()` function above based on minimising the log-loss of the prediction error. This is related to a concept called 'cross-entropy'. But another way of deriving exactly the same loss function is by maximising the likelihood of the data under the model $P(y | x, w_1, w_2)$. If you are familiar with this concept (eg. from statistics), see if you can derive it this way as well.\n",
        "\n",
        "### Optional Further Reading\n",
        "More information on the [cross-entropy loss](http://ml-cheatsheet.readthedocs.io/en/latest/loss_functions.html) and another interesting connection to [information theory](https://rdipietro.github.io/friendly-intro-to-cross-entropy-loss/)."
      ]
    },
    {
      "metadata": {
        "id": "0tJJrBynf6ms",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Loss value for your chosen $w_1$ and $w_2$\n"
      ]
    },
    {
      "metadata": {
        "id": "edKlqlACgFsE",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The following line of code computes the loss value for your chosen values of $w_1$ and $w_2$. Try changing the values of $w_1$ and $w_2$ using the sliders above and rerun the line below. Can you see how a better separation results in a lower loss? \n",
        "\n",
        "Note: If you've used TensorFlow before, it might be confusing how this code cell works! We explain more about this later... "
      ]
    },
    {
      "metadata": {
        "id": "QyzwKx6ef_Vm",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "4f013900-35c4-4d70-9a61-4e1677767c07"
      },
      "cell_type": "code",
      "source": [
        "compute_loss(w1, w2).numpy()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.09962160420831814"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "metadata": {
        "id": "Z9KAMYSUgmkM",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Visualising the loss function"
      ]
    },
    {
      "metadata": {
        "id": "ukphZS4_hMgN",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We can visualise the loss function for our dataset by plotting its value at every point in a whole grid of $w_1$ and $w_2$ parameter values. We do this using a **contour plot**, which is a technique for visualising a 3-D function on a 2-D plot by letting colour represent the third dimension. All of the points with the same colour have the same loss value. "
      ]
    },
    {
      "metadata": {
        "id": "y4HZS5zZt3Pu",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 376
        },
        "outputId": "0befe58e-fc5e-42aa-aae9-49f9fe5cab8c"
      },
      "cell_type": "code",
      "source": [
        "# We define a function so we can re-use this code later\n",
        "def plot_contours():  \n",
        "  # Generate a whole bunch of (w1, w2) points in a grid\n",
        "  ind = np.linspace(-5, 5, 50)\n",
        "  w1grid, w2grid = np.meshgrid(ind, ind)\n",
        "\n",
        "  # Compute the loss for each point in the grid\n",
        "  losses = []\n",
        "  for w1s, w2s in zip(w1grid, w2grid):\n",
        "    loss = compute_loss(w1s, w2s)\n",
        "    losses.append(loss)\n",
        "\n",
        "  # Pack the loss values for every value of w1 & w2 into one (50,50) array\n",
        "  losses_array = np.concatenate(losses).reshape(50,50)\n",
        "\n",
        "  # Now plot the resulting loss function as a contour plot over the whole grid of (w1, w2) values.\n",
        "  fig = plt.figure()\n",
        "  plt.contourf(w1grid, w2grid, losses_array, 20, cmap=plt.cm.jet)\n",
        "  cbar = plt.colorbar()\n",
        "  cbar.ax.set_ylabel('Binary cross-entropy loss value')\n",
        "  plt.xlabel('w1 value')\n",
        "  plt.ylabel('w2 value')\n",
        "  plt.title('Total loss for different values of w1 and w2')\n",
        "\n",
        "plot_contours()"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeEAAAFnCAYAAACRo/HLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xd8VFX+//HXkAJIaKH3jbqgohBY\nei8hBKQYpAQxoCC6i1JE6Sgi0vITgQVRVJS6gokRcGkCCoj03uRLkRIILSEFSDAkub8/2IyEhMyd\nyZ2Ze+98no+HjweZyblzAru8+Zz7OedaFEVREEIIIYTLFXD3BIQQQghPJSEshBBCuImEsBBCCOEm\nEsJCCCGEm0gICyGEEG4iISyEEEK4iYSwzk2YMIGQkBBCQkKoWbMmrVu3tn59+/btPMd+9913Nq9/\n4cIFnnvuOdWvaykpKYnOnTvTvn17bt26pdl1a9SowdWrV1m/fj3jx48H4MCBA7Rs2ZJBgwaRnp5O\neHg4bdq04ezZs5p97qMcPHiQU6dOaXKtyMhIBgwYoMm1HJHfPzNFUfjiiy+oWbMmhw4dcsIM/zJn\nzhzGjRvn0NirV6/y+uuv06FDBzp06MCKFSs0np0Q93m7ewIibxMnTrT+uk2bNkRERFCvXj2b4+7d\nu8fHH39Mz549nTm9fPn999+5c+cOP//8s1Oun/WPFYDt27fTpEkTpk6dSkxMDAcOHODw4cN4ezv/\n/wJRUVE0adKE6tWrO/2znC2/f2bjx4/Hy8uLEiVKaDwzbY0bN466devyxRdfcPXqVTp16kSjRo2o\nVq2au6cmTEYqYYO7dOkSr776Ku3bt6dTp06sXr0agFdeeYXk5GRCQkKIjY3l7NmzhIWF0aFDB4KD\ng1m7dq3qz8jIyGDGjBnWUBs7diypqakArF27lk6dOtGhQwe6dOnCvn378nw9S0xMDKNHj+b69euE\nhISQnJzMzp07eeGFFwgJCaFnz56cOHECuF/9DRkyhPDwcGbMmJFjfr/88gtBQUF07NiRb775xvp6\nVtW4du1ali1bxqZNmxg4cCCvvvoqGRkZdOrUidOnT3Pq1Cn69OlDcHAwnTt3tn7ujh07eOmllxg8\neDAjR44E4KeffqJz5860bduW1157jcTERABmzpzJRx99xKBBg2jbti09e/YkLi6OpUuX8t///pdp\n06axaNGibPN+6623sr129OhRWrZsiaIobNq0iU6dOtG+fXu6devGyZMnc/zcvXv3Zs2aNbl+vXfv\nXrp160a7du3o1asXly5dAuDKlSv07duXjh07EhQUxOzZs3P9M8/tzyK3P7Msy5cvZ/To0dav27dv\nz7///W/g/v9+6tWrR1JSEt27d+fDDz/Ey8sr18/Nsm/fPkJDQwkJCeH5559n165dwP0VmpYtW7Jw\n4UI6depEixYt2LBhAwCpqakMGTKE1q1bEx4ezrVr13Jc948//qBNmzbWr8ePH8/LL79s/XrgwIFs\n3ryZ3r1707dvXwDKly9PhQoV+OOPP/KcsxAOUYRhtG7dWtm7d2+21/r166d8+eWXiqIoysWLF5W6\ndesqsbGxyvnz55Vnn33W+n0DBgxQvvrqK0VRFGXHjh1KYGCgkp6enuP7sjz4+sqVK5Vu3bopKSkp\nSnp6uvLGG28o8+fPVxRFUerVq6dcvXpVURRF2bVrlzJt2rQ8X3/Qb7/9prRv315RFEW5deuW0qBB\nA+XQoUOKoijKmjVrlJCQECUzM1P57rvvlMDAQOXChQs5rpGWlqY0adJE2bFjh6IoivLFF18o1atX\nV65cuaJ89913Sv/+/RVFUZRPPvlEee+993L8bOnp6Urbtm2V6OhoRVEUZc+ePUqLFi2U9PR05bff\nflOee+45Zc+ePYqiKMq5c+eUOnXqKGfOnFEURVHmzp2rDBs2zHr9pk2bKrGxsUpmZqbSv39/6+9R\nWFiY8t///jfH3FevXq307dvX+vUnn3yiTJkyRUlLS1Pq1KmjHDlyRFEURZk1a5b153jwZ3r4ullf\nJycnK/Xq1VN27typKIqi/PDDD0qPHj0URVGUyZMnK/PmzVMURVHu3LmjDB06VLlx40a2eeX1Z/Hg\nn9mD/vjjDyU4OFhRFEW5du2a0rNnT6Vfv36KoijK0aNHlW7dumX7/ubNmysHDx7McZ0sISEhyrp1\n6xRFUZTIyEjrZ54/f16pWbOmsmzZMkVRFOXHH39UQkJCFEVRlEWLFinh4eFKenq6Eh8fr7Rs2VIZ\nO3Zsjms3bdpUuXbtmqIoitKtWzelW7duSlpampKRkaE0aNBASU5Ozvb9MTExSu3ata3/exZCS1IJ\nG9iff/7Jrl276N27NwBVqlShfv367N69O8f3fvHFF7zyyisA1KtXj5SUFOLi4lR9ztatW+nWrRuF\nCxfGy8uL0NBQtm/fDoC/vz/ffvstsbGxNGzYkFGjRuX5+qMcPHiQypUrU7t2bQA6dOjA9evXuXLl\nCgBPPPEEVatWzTHu3LlzZGZm0rhxYwBCQ0NV/UxZTp8+za1bt6zj6tevT9GiRTl8+DAARYoUoX79\n+gBs27aNJk2a8MQTTwD3K89Nmzah/O/k1wYNGlChQgUsFgvPPPOMde6P0rp1a44cOWKtKDdu3EiH\nDh3w8fFh9+7d1nvy9erVs1ayauzZs4fKlSvTqFEjALp27crp06e5du0apUqVYtu2bezfv5+CBQsy\na9YsSpcunW28rT+L3AQEBJCamkpCQgL79u2jefPmxMfHk5GRwf79+61/Pmr9+OOPtG/fPtefPz09\nnRdffBEg2+/zvn37aN++PV5eXvj7+9OyZctcr92wYUMOHTpEfHw8jz32GNWrV+fEiROcOnWKatWq\nUbRoUev3JiUlMXjwYN58803KlStn188ghBpyT9jAEhIS8Pb2pkiRItbXihUrRnx8fI7v3bp1K/Pn\nzychIQGLxQJgDQ9bbt68SbFixbJ9xs2bN4H74T5v3jxCQ0OpWLEi48aNo169eo98Pa/PKF68uPVr\ni8VC0aJFrT/Lg+89KCkpCT8/v2xzs8etW7e4c+eO9d4xwO3bt0lMTKRQoULZPjc5OZldu3Zl+94i\nRYqQlJQEkG0eBQoUICMjI8/P9vPzo2HDhmzbto2aNWty9+5dAgMDAVi4cCGrV68mLS2NP//8Ex8f\nH7t+pnPnzmWbZ+HChUlISKB///7A/Ya/uLg4Xn75Zd56661s4239WTxKgwYNOHz4MPv27aNt27ac\nO3eOkydPsn//fnr16qV6/gCrV69m6dKl3Llzh4yMDDIzM63v+fj4ULBgQQC8vLysv8+JiYnZ/gyK\nFy+e65wbNWrEwYMHyczMpG7dulSsWJEDBw7g4+OT7R8L165dY+DAgbRv356BAwfaNX8h1JIQNjB/\nf3/S09O5ffu29S+fxMRESpUqle370tLSGDp0KJ9++inNmzfn7t271ipHjVKlSlnvfT78GdWqVWP6\n9OlkZGQQHR3NiBEj+OWXXx75+qOULl0622dkZmaSlJSUo0p7WLFixbJ1iSckJKj+uQDKli1L8eLF\nWb9+fY73duzYke3rcuXK0bx5c2bOnGnXZ+QlJCSEn3/+mStXrlgrv71797Jw4UIiIyOpWLEiW7du\n5aOPPsox9sEAAqwVddmyZalevfoju+PfeOMN3njjDf744w9ee+016tWrZ62aIe8/i7w6ohs2bMjB\ngwc5ePAgw4cP59y5cxw4cIBjx44RERGh+vckNjaWCRMmEBUVRY0aNTh79iydO3e2Oa548eLZ/reQ\n9Q/F3OYZHR3NvXv3aN68ORUqVGDOnDl4e3tbGxlv3brFgAED6NWrF+Hh4arnLoS9ZDnawHx9fWna\ntKl1+8T58+c5ePAgjRs3xtvbm4yMDFJSUrh9+zZpaWk8++yzKIrC4sWL8fHx4c6dO6o+p3Xr1qxa\ntYq7d++Snp5OVFQUrVq14saNG/Tv3587d+7g5eVF7dq1sVgsj3w9L7Vr1yY2NpYjR44A9yuhqlWr\nUr58+TzHBQQEkJmZaW38io6OVvUzZalatSr+/v5s2rQJuP8X9/Dhw62NZw9q3rw5e/bssS6NHjx4\nkKlTp9r8DG9v72xNTA9q06YN+/btY/PmzXTo0AGA+Ph4SpcuTYUKFUhJSWHVqlW5zqdMmTLWhq19\n+/YRExMDQGBgILGxsRw9ehS438w0atQoFEVh7Nix7Ny50/qzly5dOsefjaN/Fo0aNWL79u1YLBb8\n/PwIDAzkv//9L5UqVaJQoUI2f5+yxMfHU6RIEQICAkhPT+e7774jMzOTu3fv5jkuMDCQzZs3k5mZ\nSXx8PNu2bcv1+6pWrcrNmzfZt28fgYGBPPnkk5w9e5YTJ05Qt25dAGbMmEHz5s0lgIXTSSVscB9+\n+CHvvfcekZGR+Pj4MHXqVMqVK0dGRga1atWiZcuWLFiwgFdffZWuXbtSqlQpBg0aRJs2bRg4cCCf\nf/65zc/o2LEjp0+f5oUXXgCgSZMm9OnTB19fXxo3bky3bt3w8vLC19eXjz76iDJlyuT6el78/PyY\nNWsWH3zwASkpKZQqVYoZM2bYDG9fX18+/PBDRo4cSaFChejevbtdf+FbLBZmzpzJhAkT+Pjjj/Hy\n8qJ///4ULlw4x/eWL1+eiRMn8q9//Yv09HT8/PxU7UNt164d06dP5+LFiznujRcrVowaNWpw7tw5\natWqBUDLli1Zvnw5bdu2pXz58owZM4bDhw8zbNgwmjZtah3bv39/3nnnHX755RcaNWpkrWYfe+yx\nbL+Xvr6+DBs2DIvFQu/evfnggw+4c+cOiqIQFBREw4YNs83J0T+LKlWqkJCQYL0X+9RTT3Hq1Cle\ne+016/dkLZHHx8czfPhwfH19mTFjBjVr1rR+z7PPPkuTJk0IDg6mdOnSjB49mv379xMeHs7HH3/8\nyM/v1asX+/fvp23btlSqVIng4OBc//EC9wP7+PHj1mX3ChUqkJGRQcGCBcnMzCQyMpJy5cplW73p\n37+/rrf8CWOyKGpvDAohhBBCU7IcLYQQQriJhLAQQgjhJhLCQgghhJtICAshhBBuIiEshBBCuInu\ntyhZPtHwYts1vJZav7rhMx8lTv3Rh/q0yd0TEMLjKcorTrnuQhtb4PLyioE3+UglLIQQQriJZ4Vw\nMzd8ZnM3fKYQQghD8KwQFkIIIXTE80LYHdWwEEIIkQvPC2F30MuSdOnK7p6BEEKIB0gICyGEEG7i\nmSEsS9JCCCF0wDND2B30siQthBBCNzw3hKUaFkII4WaeG8KeSpqzhBBCNySEXUmWpIUQQjzAs0NY\nlqSFEEK4kWeHsBBCCOFGEsKurob1sCQt94WFEB7m1KlTBAUFsXTpUgCuXLnCK6+8wssvv8wrr7zC\njRs3sn3/nTt3eOuttwgPDycsLIxff3XOI/H0H8JN3T0BIYQQRpaSksKkSZNo3Lix9bVZs2bRs2dP\nli5dSrt27fjmm2+yjfnhhx8ICAhgyZIlzJ49m8mTJztlbvoPYVeQe8NCCGFavr6+fPnll5QtW9b6\n2oQJE2jfvj0AJUuWJDExMduYB19LTk6mZMmSTpmbMULYbNWwHpakhRDCQ3h7e1OoUKFsrz322GN4\neXmRkZHBf/7zHzp37pzt/eeff57Y2FjatWvHyy+/zKhRo5wzN6dc1YiaAdvdPQkXKl0Z4i65exZC\nCAFAOzd8ZkZGBiNHjqRRo0bZlqoBVq1aRcWKFVmwYAEnT55k7NixREdHaz4HY1TCYL5qWAghhFuN\nGTOGatWq8dZbb+V478CBAzRrdv9e5VNPPcX169fJyMjQfA7GCWGzkSVpIYRwm9WrV+Pj48OQIUNy\nfb9atWocPnwYgMuXL1OkSBG8vLw0n4dFURRF86tqyLL7oRd+c/IHunJJ2jkd7+oZbjl6k7snIITH\nU5RXnHLdyxaLw2Mr2YixY8eOMX36dC5fvoy3tzflypUjPj6eggUL4ufnB8ATTzzBBx98wNtvv83U\nqVPJyMhg7NixxMfHk56eztChQ3MsWWtBQvhhEsI6JiEshLsZMYT1zHjL0c6+N+zK7UruXpKWQzuE\nEMKtjBfCQgghhEkYM4SlGhZCCGECbgnhu3fvEhQU5JQ9V0IIIYRRuCWEP/vsM4oXL56/i5ipGnYn\nuS8shBBu4/IQPnv2LGfOnKFVq1au/mj9kiVpIYTwSC4P4enTpzN69GhtLibVsBBCCANzaQivXLmS\nwMBAqlSpot1F5TjL/JMlaSGEcAuXPsBhy5YtxMTEsGXLFq5evYqvry/ly5enSZMmrpyGfVz1YIfm\nuP/wDt0LQg7sEEKYiUtDeNasWdZfz5kzh0qVKmkTwE1x/klaQgghhMaMuU/YrKRBSwghPIrxzo7O\nizOrYVedKe3OJWlDnCUty9FCuJOcHa0tly5HCyGEELmpVM/dM3APcy1HO7NT2lXblWRJWgghPIa5\nQlgIIYQwEPOFsFTDjpP9wkII4VLmC2GQAzyEEEIYgjlD2JnkKEshhBAaMW8IG70aliXpRwhy9wSE\nEEIz5g1hZ5JqWAghhAbMHcJSDQshhNAxc4ewM5m5Gtb9krQQQpiD+UPY6NWwEEII0zJ/CIPzgtgV\n1bAsSQshhGl5Rgg7k5mXpYUQQjiV54SwkZel3VENy31hIYRwOs8JYWeSatjFZK+wEMI+p06dIigo\niKVLl1pfW7x4MTVr1uTOnTu5jomIiKBXr168+OKL/PTTT06Zl2c9yrApzn3msDM1x73PGhZCCINK\nSUlh0qRJNG7c2PraypUriY+Pp2zZsrmO2bVrF6dPn2bFihUkJCQQGhpKcHCw5nOTSlgrZqyGZUla\nCGECvr6+fPnll9kCNygoiLfffhuLxZLrmPr16zN79mwAihUrRmpqKhkZGZrPzfNCWO4NCyGER/H2\n9qZQoULZXvPz88tzjJeXF4899hgAUVFRtGjRAi8vL+3npvkVNVah4Tmu7A7Q9qLOWpZuBmx3wnWF\nEMLsGrl7ArnbtGkTUVFRfP311065vudVwsI+siQthPBQv/76K59//jlffvklRYsWdcpnGCKEKzQ8\np/1FjXqAhyxJ/490SAshnOfWrVtEREQwf/58SpQo4bTP0f1ydBanLEsLIYQwvWPHjjF9+nQuX76M\nt7c3GzZsoEmTJuzYsYMbN24wcOBAAgMDGTlyJG+//TZTp05l7dq1JCQkMGzYMOt1pk+fTsWKFTWd\nm0VRFEXTK2qsIn9VwU4JYWdtWXL2vWFXb1eKu+TiD1Rjk7snIITHUZRXnHPhwbl3KasyR9cxlidD\nLEdnMdSytBBCCGGDoUIYnBTEzmC2e8PSoCWEEJozXAg7hVGbtIQQQhiaIUPYMNWws3l8p7R0SAsh\njM2QIQxOCGKphm2TJWkhhNCUYUPYKYzYpOXx1bAQQhiXoUPYMMvSZqqGhRBCaMbQIQwGWpZ2JldW\nw7IkLYQQmjF8CDuFM4JYqmEhhBAPMUUIG2ZZ2pk89t6wdEgLIYzLFCEMBlmWNks1LEvSQgihCdOE\nMEgQe241LIQQxmSqEBYuJNWwEELkm6GeoqSW5k9bcsaTlpz5lCVXPWFJV09WkicqCeEKTnuK0tJ8\nPEXpZV3HWJ5MWQlLo5YQQggjMGUIa07uDedOlqSFECJfTBvCHt+kJYQQwiWSkpKYPn067777LgA/\n//wzN2/eVDXWtCEMHr4sLdWwEEK4xPjx46lQoQKXLt3vk0lLS2PUqFGqxpo6hDUn1bCOyaEdQgj3\nuHnzJn379sXHxweAkJAQ7t69q2qs6UPYEMvSziL7hoUQwiXu3buHxXK/wzsuLo6UlBRV40wfwmCA\nZWmjV8OyJC2E8GB9+vShe/funDlzhn/+85907dqVAQMGqBpryn3CufHovcOu2Desiz3DsldYCGeT\nfcK5u3r1KgcPHsTX15fnnnuOsmXLqhrnEZUwePiytCvoohqW+8JCCNeLiopi+/bt3Llzh4SEBLZt\n20ZUVJSqsd5Onlu+BXKQQ9TR5FoVGp7TviLWUjOcUw03x3WnaAkhhIfZv3+/9ddpaWkcOXKEunXr\n0r17d5tjdR/CoOMgbor2y9JGDuLSlXWyLC2EEK4zderUbF+npqYyZswYVWM9ZjnaaWRZWmdkSVoI\n4V6FCxfm4sWLqr7XEJUw6LgadgaphoUQQjOZmZlMmDCB06dP4+PjwwcffMATTzxhff/KlSsMHz6c\ne/fu8cwzz/Dhhx/adf2XXnrJuj0J4Nq1a9SoUUPVWMOEMOg4iJ2xLC2EEEITmzdv5tatWyxfvpyL\nFy8yefJk5s+fb31/2rRp9O/fn3bt2jFx4kRiY2OpWLGi6usPGzbM+muLxYKfnx9PPfWUqrFuWY6O\niIigV69evPjii/z00092jQ3koJNmlU9aL0s7a++wHOAhhPAw58+fp1atWgBUrVqV2NhYMjIygPtV\n8v79+2nTpg0AEyZMUB3AO3fuZOfOnWRkZFj/S09PJzExkV27dqm6hssr4V27dnH69GlWrFhBQkIC\noaGhBAcHu3oagBOWpbWuiJ21LO1sbl+SDkL2DAshslSvXp1FixbRr18/Lly4QExMDAkJCZQuXZqb\nN29SpEgRpk6dyvHjx6lXrx7vvPOOquvOmzfvke9ZLBYaN25s8xouD+H69etb/0VSrFgxUlNTycjI\nwMvLS/U1dLssbRSyZUkIoTeNnHfpli1bcuDAAfr06UONGjV4/PHHyTqnSlEUrl27Rt++falUqRKv\nv/46W7ZsoVWrVjavu2TJkke+t2HDBlVzc3kIe3l58dhjjwH3Nzi3aNHCrgDOotsgNko17Owgdns1\nLIQQf3n77betvw4KCqJUqVIAlCxZkooVK1K1alUAGjduzOnTp1WFcJbY2FiWLl1KQkICcH+v8O7d\nu2nfvr3NsW7borRp0yaioqJ4//33Hb6G3B8WjyZblYQQ9508edK6b3fbtm0888wzFChwP/68vb2p\nUqUK58+fB+D48eMEBNhXlI0cOZISJUpw6NAhnn32WRISEoiIiFA11i0h/Ouvv/L555/z5ZdfUrRo\nUXdMIQfdP+TBGZzdpKWLoyyFEJ6uevXqKIpC9+7dmT9/PmPGjCE6OpqNGzcCMHbsWMaMGUNYWBhF\nixa1Nmmp5eXlxeuvv07p0qXp06cPn332GcuWLVM11uXL0bdu3SIiIoKFCxdSokSJfF9PlqWFEELk\npUCBAkybNi3ba926dbP+ulq1anz77bcOX//PP//k6tWrWCwWYmJiqFixIpcvX1Y3N4c/1UFr164l\nISGBYcOGER4eTnh4OLGxsfm6ppbL0ppWxEZYljZ1NSxL0kII53vttdfYsWMHAwYMoGvXrjRq1Ig6\nddQVh7p/lGFHolV/r1YVsabd0lof4uGMatjZndJubdCSrUpCaMlpjzI8k49HGT7p3hj7448/ePzx\nxwFIT0/nzp07FC9eXNVYOTs6F1INa0zuDQshTOxf//oX3bt3Z/HixSQnJ6sOYDBZCMuydD6Y9iQt\nWZIWQjjXhg0bmDhxIteuXSMsLIw33niDtWvXqhprqhAGDwpio5FqWAhhYjVr1mTEiBEsW7aMihUr\nMnLkSFXjTBfCoOP9w1qSalgIIXTh+vXrLF26lPDwcPr160epUqVYs2aNqrGmasx6kFZNWiCNWppy\nW5OWNGgJoQVpzMqpefPmdOzYkU6dOvHcc8/ZNdZQjzK0h8fsHxZCCOFWW7dutZ7AZS9TLkdn8Yj7\nw0ZblpZ7w0IIk3E0gMHkIQxyf1iX3BLE0iUthNAf04ewlnRbDTuDNGkJIYQq9+7d4+rVq8D9h0Ws\nXLmS1NRUVWM9IoRlWdpBpluWlmpYCKG90aNHc+jQIa5du8bgwYM5deoUo0ePVjXWI0IYdBzEWjLa\nsrQQQpjAtWvXCAkJYe3atbz00kuMHDmSpKQkVWN13x1dj/3s4x+aXEvLjmnN6P1pS81x3pal0pXd\nsGUpCNmuJIT+nHuygsNjNdxE6pC0tDQURWHjxo1MnjwZgJSUFFVjDVEJ12O/u6eQg26XpZ1B7g8L\nIcQjNWjQgH/84x+UKVOGgIAAFi5cSECAun8a6P6wjvcZZ/21VhWxRxzkofUhHqY6wEMqYSEc5azD\nOs5R0eGxAeTvcbhaSE5OplixYgBcunSJcuXK4ePjY3OcISrhLFpVxHJ/2AGmqoalQUsIoZ2tW7fy\nyy+/APDOO+/Qv39/69e2GCqEtaTL/cOeuiwtB3gIIQxs3rx5NG/enK1bt5KZmckPP/zAkiVLVI01\nXAhreX9YqyDW7f1hI3VLuzyIpRoWQmijUKFC+Pv7s3XrVrp27UqRIkVUn6JluBAGadSyiyxLCyGE\nU/3555989dVXbNu2jcaNG3P+/Hlu3bqlaqwhQxjk/rBdjBLEUg0LIQxo0qRJXLt2jWnTplGwYEG2\nb9/Ou+++q2qsYUNYS3J/WAghhKP+/ve/069fP27evMnGjRtp06YNTZo0UTXW0CEs94ftINWwEEI4\nxbfffkvfvn1Zs2YNP/74I+Hh4fzwww+qxur+xCxbtDxRSyu6ff6wUU7TculJWnKClhAif1atWsW6\ndesoWLAgcP+0rFdffZXQ0FCbYw1dCWeR+8NCCCHcxdvb2xrAAI899piqgzrAJCEM+gxizciytJNJ\ng5YQwnHly5dn0qRJbN68mc2bNzNx4kQqVFB3Frahjq20Rctlaa2OtpRjLfPJpUdayrK0ELYY7djK\nO3fuMGrUKJKSkrh37x5vvvkmzZv/VT2sXr2aRYsWUaBAAXr27EmPHj3s/vzU1FSWLFnC4cOHsVgs\n1K5dm/DwcAoVKmRzrKlCGCSI7WKEIJYQFkJXjBbCS5cu5dq1a7zzzjtcu3aNfv36sX79euD+vdvQ\n0FCioqLw8fGhe/fuLF26lBIlSqj63MzMzDzfV3Ngh+Ebsx5m+kYtTyNNWkKIfChZsiT/93//B9x/\nyELJkiWt7x0+fJjnnnuOokWLAlC3bl0OHDhAmzZtVF37mWeewWKx5HhdURQsFgu///67zWuYLoRB\nuyDW8vnDmgWxJ3ZLCyGEg55//nmio6Np164dycnJzJ8/3/peXFwc/v7+1q/9/f25ceOG6mufPHky\n3/MzZQiDPoNYM54WxFINC2G+yAiNAAAgAElEQVR6Bwl0eGxe5c2qVauoWLEiCxYs4OTJk4wdO5bo\n6Ohcv9cdd2dN0x3tTHKQhw7IIR5CCAccOHCAZs3u/yX31FNPcf36dTIyMgAoW7YscXFx1u+9fv06\nZcuWden8TB3Cpn/Qg14Z/iEPsmVJCLOoVq0ahw8fBuDy5csUKVIELy8vAGrXrs3Ro0dJTk7mzp07\nHDhwgHr16tn9GcnJyQ7Pz3Td0bnRqlFLy2VpzRq1pFvaSWRJWojcOKs7OpqODo/txtpHvnfnzh3G\njh1LfHw86enpDB06lKNHj1K/fn3q1KnD+vXrWbBgARaLhZdffpkuXbrY/fnNmjWjUaNGdO/enUaN\nGtk11iNCGPQXxLJtKR8kiIVwG6OFsCvcu3eP7du3s27dOs6cOUNwcDDdunVTtbRt6uVoZ5D7w55E\nlqWFELb5+PjQunVrIiIimDFjBtu2baNdu3a8++673Lx5M8+xNkP48uXLDBkyhPDwcAC+++47zp8/\nr8nE1dAq9OSJS27ijPvD0qQlhNCR1NRUVq5cSXh4OO+88w6dO3fmt99+o23btgwZMiTPsTZD+L33\n3qNr167W1u2AgADee+89bWaukh6DWCu6bNQywvnSLgtiqYaFEHkLCgpi7969jBgxgujoaHr37o2f\nnx8dOnTItg85NzZD+N69e7Rt29Z6Kkj9+vW1mbWd9BbEunzikqctS0tFLITQgQ0bNjBy5EgAjhw5\nwu3bt63v/fvf/85zrKp7wsnJydYQPn36NH/++aejc9UFPQaxZvQaxIbetiTVsBDi0aKioggODmbK\nlCl89NFHBAUF8Z///EfVWJsnZr355pv07NmTGzdu0LlzZxISEvh//+//5XvSjtDj6VVazUnT86X1\neqKWoU/TkpO0hBC5++GHH9i0aZP1DOqkpCT69u3LSy+9ZHOszRBu1KgRK1eu5NSpU/j6+hIQEJDt\n4cWuplXoafmgB10GsV4ZOoiFECKn0qVLWwMYoHjx4lSurO52mc0Qnj17dq6vDx06VOX0tKfHINaK\nRzzowbCkGhZC5FSlShUGDRpE06ZNURSF3bt3U6JECaKiogDo3r37I8favCfs5eVl/S8zM5Pdu3dz\n69Yt7WbvIGnUUsGT7g9Lt7QQwk3+/PNPihcvzrFjxzh+/Dh+fn5kZmayf/9+9u/PO2NsVsJvvfVW\ntq8zMjIYPHhw/masM/LEJZXk/rAQQuQwdepUABITE7FYLBQvXlz1WLtPzEpPT+fixYv2DnMKPXYn\ny0EedjBsRSzVsBDiLwcOHCAoKIgOHTrQvn17QkJCOHr0qKqxNivhli1bWrcnKYpCcnIyoaGh+Zux\nhvR4f9jUjVpGeP6wS8j9YSHEfTNmzGDevHlUr14dgBMnTjB58mSWLVtmc6zNEH5wr5PFYsHPz49i\nxYrlY7ra02MQa0UatRwgy9JCCBcqUKCANYABnnnmGevjEm15ZAhndXU9Sl7dXu6gtyCW+8N2MOz9\nYamGhdBKfv6+7KbhPBxRoEABfvrpJ5o0aQLAtm3b8h/Ctjq69BbCWtJbEMtBHg6SIBZCuMDEiROZ\nNGkS48aNo0CBAtSuXZuJEyeqGvvIEM7q9srN4sWL7Z+lC+ix+jR9EGtJ7g8LIQwoJSWFBQsWODTW\nZnf077//ztChQ+nbty99+/YlLCyMr7/+2qEPcwW97R8GnXZMa0XvD3qQ/cNCCCebNm2aw2NthvDE\niRMJDg4mKSmJ/v3787e//Y2IiAiHP9AV9BjEWpGDPBwgT1sSQjhRxYoVCQ8P5+OPP2b27NnW/9Sw\nGcKFChXi+eefp2jRorRq1YrJkyc7XHYDTJkyhV69ehEWFsaRI0ccvo4tegtiOVHLDoYNYqmGhfBE\nlStXpmHDhhQqVCjbKZNq2Nyi9Oeff3Lq1CkKFizInj17ePLJJ7l8+bJDE92zZw8XLlxgxYoVnD17\nlrFjx7JixQqHruVKemvU0pQnNWq5hNwfFsLT+Pn58corr2R7zdZzhLPYDOF3332XixcvMmTIEEaO\nHEl8fDwDBw50aKI7d+4kKOh+tfDEE0+QlJTE7du38fPzc+h6tmgZenoLYmnUcoA89lAIoaFdu3ax\na9cuVq9eTVJSkvX19PR0oqOjGTJkiM1r2AzhlJQU2rZti8ViYcOGDfmacFxcHDVr1rR+7e/vz40b\nN5wWwqDP6lOXQawVOcjjfySIhTC7xx9/nBs3bgBkW3729vbmk08+UXUNmyH89ddfM378eEJCQnjh\nhRd4+umnHZxuToqiaHatvOjtIA8tmf5ELcPuHwYJYiHMrWzZsnTu3Jk6deqofn7ww2w2Zn3zzTdE\nR0dTrVo1pkyZQpcuXfjiiy8c+rCyZcsSFxdn/fr69euUKVMmzzF1OOTQZz1MGrVUkEYtJ5BmLSHM\n7tChQ7zwwgu0bt2aVq1aWf9TQ9VTlEqVKsVLL73EiBEjCAwMZP78+Q5NtGnTptYl7ePHj1O2bFmn\nLkU/zMxBrBlPCmIhhNDAnDlzGDduHEuXLmXZsmXW/9SwuRx96NAh1q9fz88//0yVKlXo3LkzI0eO\ndGiidevWpWbNmoSFhWGxWJgwYYKqcXU4xEECHfpMZ5FGLTeQRi0hhA5Vq1aN+vXrOzTWoti4Mdu9\ne3e6dOlCx44dKV26tEMfkh/nqGj9tVZBrFWjlpb3h7Wak2ZBrGUIa9mo5YxtSy574pIEsTA+RXnF\nKdd9n3EOj/2QyTa/Z/Xq1Xz11Vd4e3szZMiQXJeLZ8yYwaFDh1iyZIldn//ZZ5+RmppKgwYNsjVo\nNW7c2OZYm5WwracpuZJWFbE0aqkgjVpOIBWxEO6QkJDAp59+yvfff09KSgpz5szJEcJnzpxh7969\n+Pj42H39HTt2AHDw4F+3GC0Wi6oQtlkJu9uDlXAWs1bEWm6lkorYAVIRC2GTESvhtWvXsmfPHj74\n4INHfs9rr73GwIEDmTt3rt2VcBZFUbBYLHaNUdWYZVbSqKWCJzVqSce0EKZ06dIl7t69yz//+U9e\neukldu7cme396OhoGjRoQKVKlRy6/smTJ+nWrRsdOnQA4NNPP+Xw4cOqxuYZwvv27WPjxo2kpqZm\ne/377793aKJa0Wrbkpb0FsSaPnFJyyDWkqE7piWIhXClxMRE5s6dy7Rp0xgzZoz1nIrExESio6N5\n9dVXHb72hx9+yJQpU6xbbjt27Jjn44AflOfzhPfv30+JEiWYPn06c+fO5amnngJg1apVvPjiiw5P\nWAt6uz8M0jGtitYnahm2YxrkHrEQf3Fmf02pUqWoU6cO3t7eVK1alSJFinDz5k1KlSrFrl27uHnz\nJn369CEtLY2LFy8yZcoUxo4dq/r63t7e1nwECAgIwNvbZssVkEclfODAASIjI/nqq6+YMWMGQ4YM\n4fz584DrTrqyRW8HeWhJlxWxVuQZxA+QilgIZ2vWrBm7du0iMzOThIQEUlJSKFmyJAAhISGsXbuW\n7777jrlz51KzZk27Ahjuh3BMTIz1fvDWrVtV5+QjQ9hisVgvWLt2baZMmcKbb75JbGys3TeenUlv\nQSzPIFZJ7g8/IAgJYyGcp1y5crRv356ePXsycOBAxo8fz8qVK9m4caMm1x81ahSDBg3iwIED1K1b\nlxkzZvDee++pGvvI7uiZM2dy4MABvvzySwoVKgTcfxThhAkTuHXrFtu3u+aU/ty6o3MjHdO2Sce0\nA1y2NJ1FlqeFvjmrO7oj0Q6PXUs3DWfiuJs3b+Lr62vXSZCPrITffvttBgwYQI8ePZg+fTq//fYb\ngYGBLF++PF83sPVObxWxHpfKpSJ2JqmIhTAqf39/Ro0aZdeYPLujW7VqxTfffMOzzz7Lhg0b6NGj\nB++88062E0H0QjqmbZOOaQdJEAshVEpOTrbr+23uEy5dujTPP/88gwYNYsCAAXh7ezv8AAdn09v9\nYZAgVkXrRi0JYiGEm9SoUcOu77d5YtbYsWOJiYmhTJky/OMf/6BevXp2f0h+qL0n/CCz3h8GOWNa\nNWfcHwa5Ryw8ntwTzt3t27fx8/MjLi6O8+fPU7duXQoUsH0els3vSElJAcDPz48SJUrg7++f/9k6\nmd4qYumYVknv94dBKmIhRA6TJk1i3bp1JCYmEhYWxpIlS/I8IvNBNkN41qxZLFmyhD59+nDz5k3G\njBljPZpLz8waxFoulUsQG4VsYRJCz06cOEGPHj1Yt24doaGhzJ49mwsXLqgaazOEb9++zdatW1m9\nejVr167l9u3btGvXLt+TNhIzB7FmPCmIXV4NZ5EgFkKPsu7qbtmyhTZt2gCQlpamaqzNEO7atSub\nNm2iZs2afPbZZyxfvpzhw4fnY7quIx3TtumyUUtrEsRCCCcKCAigY8eO3Llzh6effpqVK1dSvHhx\nVWMN+ShDe+mtUQv0d5iHZo1aoF2zltbnwZjiMI8HSdOWcD1pzMopIyODU6dO8cQTT+Dr68vx48ep\nUqUKxYoVsznWIx5lqLf7w1rSZUWsFdm6ZINUxULowe+//87Vq1fx9fVl5syZREREcOrUKVVjdR/C\nAWeuaHIdvQWxdEyrJEFsgwSxEO720UcfERAQwL59+zh69Cjvvfce//73v1WN1X0Ia8msQSwd0zrg\n9iCWMBbCXQoWLMjf/vY3Nm/eTM+ePXnyySdV7REGg4SwVtWwlswcxJrRaxCbZg/xwySIhXCH1NRU\n1q1bx6ZNm2jWrBmJiYmqj6/UfWMWZ/56bOK5JytockmtGrVAf6dqmbpRC+RULVWkYUs4j7Masyri\n+CpcLBr+feWAXbt2sXjxYjp37kyHDh2YM2cO1apVo0uXLjbHGiqEQX9BLB3TKkkQu4GEsdCehHDu\nUlJSOHfuHBaLhYCAAAoXLqxqnCGWox9k1kYtLemyY9qT9hCDDpamQZanhXCNTZs2ERwczIQJExg/\nfjzt27dn69atqsYarhLOYtaKWB72oJIR9hCDTipikKpYaEUq4ZzCwsKYN2+e9dkK165dY+jQoSxf\nvtzmWMNVwlrTW0UsW5dUMsLWJdBJRQxSFQvhPD4+PtkeblSuXDl8fHxUjTVsCGvZMW3WIJatS3Zy\nZhDrIoxlK5MQzlCkSBG+/vprTp48ycmTJ/nqq68oUqSIqrGGXY7OordladDf0rTulqXBM5emQZan\nheHJcnRO8fHxzJ49myNHjmCxWAgMDGTw4MGqHv1r+BAG/QWxdEyrJEGsAxLGwj4Swjlt3bqVli1b\nOjTWsMvRDzJzx7TelqZ12zFtlKVp0MnSdBZZohYivxYuXEh6erpDY01RCWcxa0Vs6o5pkIpYN6Qq\nFrZJJZzTkCFD+L//+z+eeeaZbA1ZERERNseaKoRBglgN3QWxnkMYPCyIQcJY5EVCOKcffvgh19dD\nQ0NtjjXFcrQz6G1pWrYuqeSMhz14zNJ0FlmeFsIewcHBFCxYkNDQUEJDQ7l79y7BwcGqxpouhOVh\nD7bJ1iUHeGQQSxgLocbo0aOJi4uzfn337l1GjhypaqzpQhj016gF5g5izXh6EEsYC2FIiYmJ9O3b\n1/r1q6++qvopSqYMYdBnEGtFb0HsMR3T4NwgBp0GMUgQC6O7e/cuQUFBREdHZ3t92bJl9OrVi969\nezN58mSHrn3v3j3Onj1r/frYsWPcu3dP1VjThjDoL4j1WH1KEDvAo4NYwlgY02effUbx4sWzvXb7\n9m0WLFjAsmXL+Pbbbzl79iyHDtn/9/2YMWMYNGgQTZo0oVGjRowYMYJx48apGmvqEAbzBrGWjVoS\nxA7w2CAGCWNhNGfPnuXMmTO0atUq2+s+Pj74+PiQkpJCeno6qampOYJajdq1a7NhwwbWrFnD+vXr\nWbduHc8995yqsaYPYZAgVkOC2AEeHcQgYSyMYvr06YwePTrH6wULFuTNN98kKCiI1q1bU7t2bQIC\nHN/uVLJkSUqUKGHXGG+HP81D1eGQJnuIAzmoyX7deuzXdA+xFio0PKfdHuKmaLePuBna7yNujnP3\nEWcFsS73E2cJQvYWi/zK198ZDR/91sqVKwkMDKRKlSo53rt9+zbz589n/fr1+Pn50a9fP06ePMlT\nTz3l+FzspP9KeJc2l5GtS7bpcuuS1oxYEYNUxUI4aMuWLWzevJmePXsSGRnJvHnz2LFjB3B/mbpK\nlSr4+/vj6+tLvXr1OHbsmN2f8WBTlr30H8KguyCWrUvq6HIPMUgQO5WEsdCXWbNm8f333/Pdd9/R\no0cPawMVQKVKlTh79ix3794F7nc1/+1vf7P7M4YMGULv3r35/vvvSU1NtWusMUIYTB3EWtFjEGtG\ngvg+QwQxSBgLPYuOjmbjxo2ULl2aAQMG0LdvX3r37s3TTz9NvXr17L7emjVrmDhxIpcuXSI8PJz3\n3nuPI0eOqBqr/7Ojlz50dnQjbS4rZ0zbprszpkHbc6bBeGdNP0jX94kfJveMzcJZZ0dbdjs+Vsnj\nnrCr7du3j08++YSLFy9SrVo1Jk+enGd1bZxKWGN6q4j1tiwNHtAxDcatiMFAVTFIZSzM7PLly8yd\nO5eQkBAWLlzIP//5T3799VdGjRrFiBEj8hxrvBDWaFkaJIjV8IggdgYJ4keQMBbmEx4eToECBVi0\naBFz586lRYsWWCwWatWqRa1atfIca7zl6Cw6W5YG8y5NazUf0PHStDOWpcF1S9NgsOXpLLJMbTSy\nHJ3T/PnzeeONNxwaa7xKOIvOGrW0pLeKWJcd06D/wzzAdRUxGLAqBqmMhRmcPn2aCxcuODTWuJVw\nFp1VxFpVwyAVsWpSEWdnyIo4i1TGeieVcE6dO3fm3LlzFC9eHB8fHxRFwWKxsGXLFptjjR/CYNog\n1jL0JIjtYIYgBglj4RQSwjldvnw5x2vJyck8/fTTNse6dDk6PT2dUaNG0bt3b3r27Mm+ffu0ubDO\nlqb11qgFsjRtFzMsTYNBl6ezyDK1MI5KlSqRmppKbGwssbGxnD9/nuHDh6sa69Kzo1etWkXhwoX5\n9ttvOX36NGPGjCEqKsqVU7Ap4MwVTSpivZ0xrSU9zklzzjhnGpx/1vTDSlc2eEWcFcRSGQv9+uij\nj/jtt9+Ii4ujatWqxMTE0L9/f1VjXVoJd+nShTFjxgDg7+9PYmKidheXrUs2ydYlOzmzInZ1w5ah\nq2KQyljo2dGjR1m3bh1PPfUU33//PV9//bXq4ytdGsI+Pj4ULFgQgEWLFtGpUydtP0DDINaKBLFt\nHhnEIMvTDpEwFvrj6+sLwL1791AUhWeffZYDBw6oGuu0xqzIyEgiIyOzvTZ48GCaN2/OsmXL+Pnn\nn/n888/x8fHJ+0JqGrMeZtJGLdBfxzTI8Zb5Jg1b+SDL1K4mjVk5vf/++9SoUYMrV65w7NgxAgIC\nOHjwICtXrrQ51uXd0ZGRkaxfv5558+ZZq+I8ORLCIEGsgnRM28lMQQwSxsIhEsK5fL6ikJSURLFi\nxVizZg3x8fGEhIRQvnx5m2NdGsIxMTEMGzaMpUuXUrhwYXWDHA1hMG0Qy9YlO0gQ581UQZxFAtmZ\nnBbCnzg+VlHXiOxUJ0+eJDExkQcjtXHjxjbHuTSEP/nkE9asWUPFihWtry1YsMC6np6r/IQwSBCr\nIEFsJ7MFMUgYC9UkhHMaPHgwJ0+ezFb5WiwWFi9ebHOsOQ7ryItGIQwSxGpIEGtAglhjEsZakhDO\nqVu3bkRHRzs01rhnR6slW5dcyiMO8wDnd027unMaTLKVKTfSUS2cKyAggLS0NIfGmr8SzqKzZWnQ\nX0Wsx45p8OCKGKQqdhqpjh0llXBOI0aM4NChQ9SqVQsvLy/r6xERETbH6j+EB1tAq843nQWxmTum\nQYJYM+4KYpAwFjlICOf0ww8/5Pp6aGiozbHGCGGQIFbBzEGsaQiD9kEM5rxPDB4QxCBhrJ6E8F+u\nX79O2bJliYmJyfX9KlWq2LyGcUIYJIhVkCC2gwSxeh4RxFkkkPMiIfyXd955hxkzZtCmTRssFku2\n7UkWi4XNmzfbvIZnhjCYNoilY9pORgtikDB2GQnj3EgIa8tY3dH5OFHFWczcMW36xx+C9l3T4NzO\naXBP53QW03ZQ5yYI6awWtpw8eZL4+HgAli1bxr/+9S9mzpzJ3bt3VY03VgiDdkEsW5dUkSB2kJmD\nGDwoiLNIGIucZsyYwdChQ+nZsyfz58/n0KFDdO/enbS0NN5//31V1zDWcvSDTHp/GGRpWi1Zmsa9\nS9NZPGqJ+kGeuVwty9F/6dGjBytWrCAhIYHnn3+e7du34+3tDUBYWBjLly+3eQ3jVcJZdFYRa1UN\ngz4rYq1IRawxdx3s8SCPq4qzSHXs6QoXLkyBAgUoVaoUTz75pDWAAdtPCPwf44YwmDqItWLm5xCD\nBLGVHoLY48NYAtmTFSiQPU4tFnUHTRl3OfpBJl2alq1L6snS9P/I8rSOmHO52mjL0ampqYwePZr4\n+Hj+/PNPBg0aROvWra3v79q1i08++YQCBQoQEBDA5MmTcwTqozz33HOUKlUKgPj4eOuvFUUhISGB\nI0eO2LyGhPDDJIhtkiDOB08JYpAwzsY8gWy0EF67di2XL19m4MCBXL58mf79+7Nhwwbr+8HBwSxe\nvJjy5cszZMgQXnzxRVq2bKnqcy9fvpzn+5UqVbJ5DW+b32EEu9EuiHehSRAHnLmiSRDX4ZCmQayF\neuzXLIgDOahZEFdoeE7bIG6K9kHcDOcHcdbStLvDuHRlCWKrrKVq84SxUXTs2NH66ytXrlCuXLls\n70dHR+Pn5weAv78/CQkJqq+tJmRtMfY94QfJHmKb9Lh1CeQesdO4+z4xePi94tzI/WN3CQsL4913\n32Xs2LHZXs8K4OvXr/Pbb7+proK1Yo7l6Afp7P4w6G9pWo9bl8ADl6bBs5anQSrjRzJOhey05ehu\njo9VVD7K9/fff2fkyJGsXr06W+NUfHw8AwcOZPjw4TRr5qp/Jd9nnko4i846prVk9opYS4aoiMF1\nndN6qIpBquJHkgrZWY4dO8aVK/dXJZ9++mkyMjK4efOm9f3bt28zcOBAhg0b5vIABjOGMOguiM2+\nh1iPp2qBBHEOegpiCeM8SCBrad++fXz99dcAxMXFkZKSQsmSJa3vT5s2jX79+tGiRQu3zM98y9EP\n0tnStN6WpUGfS9NazglkaToHPS1PgyxRq6aPJWujLUffvXuXcePGceXKFe7evctbb71FYmIiRYsW\npVmzZtSvX586df76O6dTp0706tXL8cnYSUJYLQlim/R6fxgkiHOlpzCWILaT+wLZaCGsd+Zcjs6i\nZce0zpamtVqWBvOfquUURl+aBv0sT4MsUdtNlqzNwtwhDLJ1SSWzB7Hm94dBgtgZJIwdIIFsZOYP\nYdBdoxZIEKvl0UHsyoYtPYaxcIAEstHoP4S1Cj4JYpeSINaIVMXunoWBBSGhrH/6D2EwdRBrxcxb\nl0CC2GX0FsQgYawZCWQ9MkYIa0lnQSx7iNWTIHYRPS5PgwSxpqRK1gvjhLCW1acEsU0SxA4ySxCD\nfoNYwtgJJJDdxTghDLpcBtZjEGtFgthBzg5iT1+eBgljp5Iq2ZWMFcKgv/vDoLsg1mPHNEgQa0qW\np++TMHYBCWRnMl4Igz6DWCMSxOpJEDv5+g/TaxCDBLHLSBBrzZghDPoLYpNvXZIgdlBTzHefWK9h\nLFWxMCDjhjBIEKug1yDWku6DGMwVxKDfIAYJY2Eoxg5hLekwiLWixyDW+jnEEsRIVfwwCWNhAPp/\nilJ9FU9R0ugJR4BpH38I8uQle2n+5KUsznoCUxZXPokpi56eyPQo8qQmTSiKc/5hYynj+Fjlhnbz\ncDVzVMI6rD711jENcrylvQxZEYMsTz+KVMVCh8wRwqC/+8Ng6iD2hEYtkCC2i96Xp0GWqIXumCeE\nQYJYJQli+xg6iKUqzp2EsdAJc9wTfphW94h1dn8Y9HmPWMt7sXq+RwxOuk/s7HvE4J77xGCMe8Ug\n94vtIPeEtWWuSjiL3ipiHW5d0pKnVMRg0EM9wD0VMRijKgapjIXbmDOEtWTiINbj1iXw4CA2431i\nMMa94iwSxsLFzBvCJn7qEkgQ28MwQQzmvU8MxglikDAWLmPeEAYJYpUkiB1j2CAGqYrVkiAWTmbu\nEAZT7yHWkgSxYySIHWS0IJYwFk5izu7o3OitYxrkVC07eGTXNLimcxrc1z0NxumgzuLhndTSHa0t\n81fCWfTWMQ2m3kMMUhFrwhUVMUhVbA+pjIWGdF8JX7ZYqFRPwwtKRayKVMSOkYo4n4xWFYPHVcZS\nCWvLEJXw5X0aXkwqYlX0eM40eHhFbPb7xGC8qhikMjaIU6dOERQUxNKlS3O8d+XKFXr37k337t15\n//33XTovQ4QwaBzEWtEyiDWixyDW63OIwWBBDK4LYncvT0sYCw2lpKQwadIkGjdunOv706ZNo3//\n/kRFReHl5UVsbKzL5maI5egHabY0beLHH4I+l6b1uiwNBluaBlmeNgKTLlMbcTk6PT2d9PR0vvzy\nS0qWLMnLL79sfS8zM5MWLVqwdetWvLy8HJ+EgwxTCWfRrCKWPcQup9dGLZCK+JH0sDxtxKoYpDLW\nEW9vbwoVKpTrezdv3qRIkSJMnTqV3r17M2PGDJfOzXAhrCkJYlU8oWMaJIgfyd3L0yBh7AniLjn+\nXz4oisK1a9fo27cvS5cu5cSJE2zZskWbn0kFt4RwXFwc9evXZ/dux5JLl41aWpIgVk2CGM+pisG4\nQQwSxjpVsmRJKlasSNWqVfHy8qJx48acPn3aZZ/vlhCOiIigSpUq+bqGLoNYhx3TIEFsL2cFsWmq\nYnczclUMEsY64+3tTZUqVTh//jwAx48fJyDAif0cD3F5Y9bOnTv56aefSE1NJTQ0lIYN8+5qergx\n62Gyh1gdPTZqgec1a4FJGrbA/U1bYOzGrSwGa+ByWmOWxfHfB1tzOnbsGNOnT+fy5ct4e3tTrlw5\n2rRpQ+XKlWnXrh0XLgyVJR8AAArvSURBVFxg9OjRKIpC9erV+eCDDyhQwDU1qktDOC0tjf79+zNv\n3jymTJmiSQiDBLFaEsT2kyC2QQ9BDBLGLmTEENYzp4VwZGQkkZGR2V5r0aIFVapUoWvXrowePVqz\nEAaTb10CCWI7SBDj2iAGCWMt6TyMJYS15dJKOCwsjMzMTAAuXryIv78/s2fP5u9///sjx6gNYZAg\nVkuC2H6GDGKQqtjIdBrGEsLactthHVpXwllMHcQmP8wDJIidwhODGCSMnURCWFuevU84L7KHWDXp\nms4fp3ZNg+s6p0Efe4qzGLmD+kHSTW1qhju2Ug1dNmqBVMR20HNFDAY85hI89z4xmKcqBrdXxlIJ\na8uUIQwSxPaQIHaMIYMYPHd5GiSMNSAhrC3ThjDoNIh12KgFEsSOkvvEKukpjM0UxODyMJYQ1pap\n7wnLqVrq6fGBD6Dve8Qg94lV08t9YjD+iVsPk3vGhmbqEAYJYnvo8XhLrUkQP8AdQSxh7DxZYSyB\nbCimD2HQOIi1IkGsmtYhJ0H8AFc+ACKLnoIYzBXEWSSMDcMjQlhTety6BLp8GpQEsTacHsQgQWy2\nqjiLhLHumbox62G6bNQCXTZreUKjFkizVg6ubtgCfTVtZTFb81YWDZq4pDFLW7qvhDdqeC1d3h8G\nXVbEnnCYB0hFnIOrK2LQX1UM5qyKQSpjHdJ9Jbzwf5VwOw2vafqKWLYu2c1IFTFIVexSZq2KwaHK\n2HmV8EKHxyrKK5rNw9V0Xwk7g+krYh02aoFUxFqSqtiFzHq/GKSjWgcME8JaLkuDBLE9JIgdJ0Hs\nAL1tZcpi5jAGCWM3MUwIgwcFsVYkiO0mQZwLdwQx6DOIwdxBDBLGLmaoEAYPCWIdNmqBBHF+mCKI\npSr+i9mrYpClahcxXAiD9kGsSxLEdjFKEDuzc1qqYjfwhDAGCWMnMmQIg2xdspsEsd2cEcRgkqrY\nHfRaFYNnBDFIEDuBYUMYJIjtJkFsNwniR3DX8jToO4g9JYyFZgwdwlqTILaPBHH+GD6IQari3EgY\nCzsYPoQ9olELJIjtJEHsAUEM+g1ikDAWqhg+hEGC2CF63FKFBLHWXBrEUhXnToJY5MEUIQwSxA7R\n4TnToP8gNuIWJqmK3UyqYvEIpglh0PnWJb0GsUa0DmItOSPgjBbE4EHL0xLGwkBMFcKg445p0GcQ\ne8D9YZAgzuIRy9Og7yAGCWMXmzJlCr169SIsLIwjR45ke2/Hjh10796dXr168emnn7p8bqYLYa1p\nHsR6JEHsMKMd6gEuDGJwfxBLGHu8PXv2cOHCBVasWMHkyZOZPHlytvc/+ugj5syZw7fffstvv/3G\nmTNnXDo/U4aw3B92gASxw4zasOURy9Og/yAGCWIn2rlzJ0FBQQA88cQTJCUlcfv2bQBiYmIoXrw4\nFSpUoECBArRs2ZKdO3e6dH6mDGGQIHaIBLHDjBjE4CH3icE4QSxhrLm4uDhKlixp/drf358bN24A\ncOPGDfz9/XN9z1W8XfppDnhFUdw9BeEgLR87r/Uj7LtpfD1Da2iyz3mU4W7+fJEnRXnFRZ+jr0wx\nbSUshBBClC1blri4OOvX169fp0yZMrm+d+3aNcqWLevS+UkICyGEMK2mTZuyYcMGAI4fP07ZsmXx\n8/MDoHLlyty+fZtLly6Rnp7OL7/8QtOmrr13YlH0VpsLIYQQGvr444/Zt28fFouFCRMmcOLECYoW\nLUq7du3Yu3cvH3/8MQDBwcEMGDDApXOTEBZCCCHcRJajhRBCCDeREBZCCCHcRELYAXFxcdSvX5/d\nu3V4iLMbpaenM2rUKHr37k3Pnj3Zt88TjhuzLa8j8zxdREQEvXr14sUXX+Snn35y93R06e7duwQF\nBREdHe3uqQgn0P0+YT2KiIigSpUq7p6G7qxatYrChQvz7bffcvr0acaMGUNUVJS7p+VWDx6Zd/bs\nWcaOHcuKFSvcPS1d2LVrF6dPn2bFihUkJCQQGhpKcHCwu6elO5999hnFixd39zSEk0gI22nnzp0U\nKVKE6tWru3squtOlSxc6deoE3D95JjEx0c0zcr9HHZmXtUXCk9WvX59atWoBUKxYMVJTU8nIyMDL\ny8vNM9OPs2fPcubMGVq1auXuqQgnkeVoO6SlpfHpp5/y9ttvu3squuTj40PBggUBWLRokTWQPVle\nR+Z5Oi8vLx577DEAoqKiaNGihQTwQ6ZPn87o0aPdPQ3hRFIJP0JkZCSRkZHZXmvRogU9evSgWLFi\nbpqVfuT2+zN48GCaN2/OsmXLOH78OJ9//rmbZqdfsiMwp02bNhEVFcXXX3/t7qnoysqVKwkMDJRb\nXyYn+4TtEBYWRmZmJgAXL17E39+f2bNn8/e//93NM9OPyMhI1q9fz7x586xVsSebM2cOZcqUISws\nDIC2bduyatUqWY7+n19//ZXZs2fz1VdfUaJECXdPR1eGDRtGTEwMXl5eXL16FV9fXz788EOaNGni\n7qkJDUklbIfly5dbfz169GhCQ0MlgB8QExPD8uXLWbp0qQTw/zRt2pQ5c+YQFhaW48g8T3fr1i0i\nIiJYuHChBHAuZs2aZf31nDlzqFSpkgSwCUkIC81ERkaSmJjI66+/bn1twYIF+Pr6unFW7lW3bl1q\n1qxJWFiY9cg8cd/atWtJSEhg2LBh1temT59OxYoV3TgrIVxLlqOFEEIIN5HuaCGEEMJNJISFEEII\nN5EQFkIIIdxEQlgIIYRwEwlhIYQQwk0khIWw040bN+jXrx+9e/fW5Hpz5sxh5syZmlxLCGEsEsJC\n2Gn48OE0bdrU3dMQQpiAhLAQD2nTpg3JyckADB06lDFjxgD3K+BOnTrx2WefUbt27UeOHzJkCD/+\n+KP163HjxvGf//yHs2fP8vLLL9OvXz9efPFFfv311xxja9SoQXp6OgDR0dG8++67AJw8eZJXX32V\n8PBwwsLCOHHihGY/rxDCfSSEhXhI48aN2b9/P4qiEB8fT0xMDAC7d++mWbNmNo+d7NKlCxs2bADg\n3r17bN26lY4dOxIXF8fQoUNZtGgR48ePt2sJesSIEUycOJElS5bwwQcfMH78eMd/QCGEbsixlUI8\npGnTpuzdu5cKFSrw+OOPk5yczJUrV9i9e7eqh863aNGCiRMnkpKSwt69e6lVqxYlSpSgTJkyRERE\nMHPmTO7du6f6ecvx8fGcO3eOcePGWV+7ffs2mZmZFCgg/44WwsgkhIV4SOPGjVm8eDHlypWjfv36\nJCUlsWfPHg4dOpQtCB/F19eXli1bsmXLFrZu3UrXrl0BmDRpEs8//zzdu3fn1KlT/POf/8zzOvfu\n3bNez8fHhyVLluT/hxNC6Ir8M1qIh5QsWRJFUdi2bRsNGjSgXr16rFu3jrJly1KoUCFV1+jcuTMb\nN25k//79tG7dGoC4uDjrU7fWrl1LWlpajnF+fn5cuXIFuL/8DVC0aFEqV67M1q1bATh37hxz587N\n988phHA/CWEhctGgQQMuXbpEuXLlqFGjBgcPHqRp06bExsYSHh7OlClTOHXqFOHh4Xz11Vc5xtev\nX5/Dhw/TuHFj61Ok+vfvz8iRIxkwYAD/+Mc/KF68ONOmTcs27vXXX2fAgAEMHDiQSpUqWV+fPn06\n8+fPp0+fPowePVq6s4UwCXmKkhBCCOEmUgkLIYQQbiIhLIQQQriJhLAQQgjhJhLCQgghhJtICAsh\nhBBuIiEshBBCuImEsBBCCOEmEsJCCCGEm/x/MHRwGNCA3yEAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<matplotlib.figure.Figure at 0x7f06ee305350>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "676bWBwTiXXD",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Optimising the loss using TensorFlow"
      ]
    },
    {
      "metadata": {
        "id": "MMLA-wk5xB7p",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now that we have a function that gives us the loss for different values of $w_1$ and $w_2$, we want an automated method to find the values that minimise the loss function. This is where optimisation by **gradient descent** comes in. The idea is that for each (batch of) data points, we compute the loss using the current values of $w_1$ and $w_2$ on the data. We then compute the **gradient** (or derivative) of the loss function at the current values of $w_1$ and $w_2$. The negative of the gradient points in the direction of *steepest descent* along the loss function. By adjusting the values of $w_1$ and $w_2$ in the direction of the negative gradient, we move closer towards the minimum of the loss function (provided the loss function is \"well behaved\"). How big of a step we take is mediated by the **learning rate**. To do this more easily, we will use TensorFlow.\n",
        "\n",
        "Don't worry if you don't understand what's going on here, you will see this in a lot more detail during the **Mathematics for Machine Learning** lectures!\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "sFJBmN5Hivhv",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Aside: TensorFlow"
      ]
    },
    {
      "metadata": {
        "id": "DRtDggkBi0X3",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "TensorFlow (TF) is an open source software library for numerical computation using the concept of Tensors. You can think of Tensors as being a generalisation of matrices to higher dimensions, or roughly equivalent to multi-dimensional arrays. Scalars are 0-dimensional tensors, vectors are 1-dimensional, standard matrices are 2-dimensional, and higher-dimensional tensors have 3 or more dimensions. You can think of dimensions as representing groups of numbers that mean the same thing. For example, for images, we often use 3-dimensional tensors where the first dimension represents the red, green, and blue color channels of the image, and the next two are the columns and rows of pixels of the image. \n",
        "\n",
        "**Note**: Don't be confused when people say \"2-D vector\" or \"3-D vector\", which refers to a 1-dimensional tensor that has size 2 or 3.\n",
        "\n",
        "The major advantage of using TensorFlow is that it can automatically derive the gradients of many mathematical expressions involving tensors. It achieves this through a process called \"automatic differentiation\". Tensorflow also supports multiple \"kernels\", allowing you to easily run your code on normal processors (CPUs), graphics cards (GPUs) and other more exotic hardware accelerators like Google's Tensor Processing Units (TPUs)\n",
        "\n",
        "Tensorflow actually provides **two modes of operation**, the first, called \"graph mode\", builds a computation graph upfront and then feeds data into the graph. By building the graph upfront, Tensorflow can apply optimisations to the graph that allow it to extract peak performance from the hardware you're running on. You will have encountered this mode if you used Tensorflow before or attended the Indaba last year! The second mode, called [\"Eager-mode\"](https://www.tensorflow.org/guide/eager), is a lot newer and evaluates Tensor operations imperatively (in the order you write them), similar to NumPy and PyTorch. Eager-mode is slightly less performant but a lot more intuitive, especially if you've never used a \"define-and-run\" programming style (like graph mode) before, and is therefore the mode we will use in these practicals. "
      ]
    },
    {
      "metadata": {
        "id": "eLVl2PhK2VpP",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Using Tensorflow to optimise the loss\n",
        "We use TensorFlow to optimise the parameters of the model with gradient descent. We loop over the dataset multiple times (called \"epochs\") and plot the final decision boundary along with a plot showing how the parameters and loss changed over the epochs.\n",
        "\n",
        "**Note**: TensorFlow is probably overkill for this example, becaue the gradient is very easy to calculate, but we introduce it here because it will become essential to calculate the gradients of more complex models in later practicals! "
      ]
    },
    {
      "metadata": {
        "id": "cKre-CI8IG9z",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "lr = 0.25  # The learning rate\n",
        "\n",
        "# Initialise Tensorflow variables representing our parameters.\n",
        "# We need to use TensorFlow variables here rather than Numpy or Python ones so \n",
        "# that TensorFlow is able to compute gradients.\n",
        "w1 = tfe.Variable(-2.0)  \n",
        "w2 = tfe.Variable(-4.0)  \n",
        "\n",
        "plot_contours()\n",
        "\n",
        "# Loop over the dataset multiple times\n",
        "parameter_values = []\n",
        "for epoch in range(20):\n",
        "  plt.scatter(w1.numpy(), w2.numpy(), marker='o', color='black')\n",
        "  \n",
        "  with tf.GradientTape() as tape:\n",
        "    loss = compute_loss(w1, w2)\n",
        "  \n",
        "  # Now we take a step in parameter space in the direction of the gradient to move the parameters closer (hopefully!) to their optimum\n",
        "  dw1, dw2 = tape.gradient(loss, [w1, w2])\n",
        "  \n",
        "  # Step 'lr units' in the direction of the negative gradient\n",
        "  # We achieve this by subtracting lr * dw1 and lr * dw2 from the w1 and w2 variables\n",
        "  w1.assign_sub(lr*dw1)\n",
        "  w2.assign_sub(lr*dw2)\n",
        "  \n",
        "print('Finished optimisation, the final values of w1 and w2 are:')\n",
        "print(w1.numpy(), w2.numpy())\n",
        "\n",
        "# Plot the final point on the loss surface.\n",
        "plt.scatter(w1.numpy(), w2.numpy(), marker='x', color='red')\n",
        "plt.show()\n",
        "\n",
        "# Plot the final decision boundary\n",
        "plot_dataset(inputs, labels)\n",
        "ax = plt.axes()\n",
        "ax.arrow(0, 0, w1.numpy(), w2.numpy(), head_width=0.3, head_length=0.3, fc='r', ec='r')\n",
        "plt.plot([-2 * w2.numpy(), 2 * w2.numpy()], [2 * w1.numpy(), -2 * w1.numpy()], 'k-')\n",
        "\n",
        "plt.xlim([-4, 4])\n",
        "plt.ylim([-4, 4])\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "meoIDXSQjKeE",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "How did the final values of $w_1$ and $w_2$ found by Tensorflow correspond to the ones you found manually? \n",
        "\n",
        "## Optional Tasks\n",
        "If you've worked through this practical, answered all the questions and feel you have a good understanding of what's going on, try the following tasks:\n",
        "\n",
        "1. Add a **bias** parameter to the equation for the decision boundary and visualise how that changes the decision boundary, the loss and the ultimate solution found by Tensorflow.\n",
        "2. Add a **regulariser**, for example, the L2 regulariser (see the appendix below for more information) - how does it affect the contour plot of the parameters vs the loss? How does changing the strength of regularisation affect the loss? \n",
        "\n",
        "Note: The benefit of using regularisation will be discussed in the next practical! "
      ]
    },
    {
      "metadata": {
        "id": "SVFq4xvBmGQq",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Next Steps\n",
        "Have a look at [last year's practical](https://github.com/deep-learning-indaba/practicals2017/blob/master/practical1.ipynb) which takes a more \"bottom-up\" approach, covers more detail on how gradients are computed and also looks at a multi-class classification problem with a non-linear decision boundary. \n",
        "\n",
        "Note: last year's practicals use Tensorflow's \"graph mode\" as opposed to \"Eager mode\" that we use here."
      ]
    },
    {
      "metadata": {
        "id": "4jpKVuEkQF46",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Appendix"
      ]
    },
    {
      "metadata": {
        "id": "EgLuuVNRQHqy",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### L1 and L2 Regularisation"
      ]
    },
    {
      "metadata": {
        "id": "SlO75RlmQKQF",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Two of the most simple regularization methods are L1 and L2 regularization (you may have heard of them as _Lasso_ and _Ridge_ regression if you've used linear regression before). Both of these methods regularize the model by adding a term to the loss that penalizes the model if it becomes too complex.\n",
        "L1 regularization adds a term based on the L1 norm:\n",
        "\n",
        "$loss_{L1} = loss + \\lambda \\sum_i |w_i|$\n",
        "\n",
        "where $\\lambda$ is a parameter that controls the amount of regularization, and $w_i$ are the parameters of the model. L1 regularization has the effect of forcing some parameters to shrink to 0, effectively removing them from the model.\n",
        "\n",
        "L2 regularization similarly adds a term based on the L2 norm:\n",
        "\n",
        "$loss_{L2} = loss + \\lambda \\sum_i w_i^2$.\n",
        "\n",
        "L2 regularization has the effect of preventing any of the parameters from becoming too large and _overpowering_ the others. \n",
        "\n",
        "In some cases it can work well to use both L1 and L2 regularization. \n",
        "\n",
        "For more information see the articles [here](http://enhancedatascience.com/2017/07/04/machine-learning-explained-regularization/) and [here](https://towardsdatascience.com/l1-and-l2-regularization-methods-ce25e7fc831c)."
      ]
    }
  ]
}