07_iris_clustering.ipynb

{
 "metadata": {
  "name": "07_iris_clustering"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Clustering of Iris Data"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Clustering is the task of gathering samples into groups of similar\n",
      "samples according to some predefined similarity or dissimilarity\n",
      "measure (such as the Euclidean distance).\n",
      "In this section we will explore a basic clustering task on the\n",
      "iris data."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "By the end of this section you will\n",
      "\n",
      "- Know how to instantiate and train KMeans, an unsupervised clustering algorithm\n",
      "- Know several other interesting clustering algorithms within scikit-learn"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Let's re-use the results of the 2D PCA of the iris dataset in order to\n",
      "explore clustering.  First we need to repeat some of the code from the\n",
      "previous notebook"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# make sure ipython inline mode is activated\n",
      "%pylab inline"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "Welcome to pylab, a matplotlib-based Python environment [backend: module://IPython.zmq.pylab.backend_inline].\n",
        "For more information, type 'help(pylab)'.\n"
       ]
      }
     ],
     "prompt_number": 1
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# all of this is copied from the previous notebook, '06_iris_dimensionality' \n",
      "from sklearn.datasets import load_iris\n",
      "from sklearn.decomposition import PCA\n",
      "import pylab as pl\n",
      "from itertools import cycle\n",
      "\n",
      "iris = load_iris()\n",
      "X = iris.data\n",
      "y = iris.target\n",
      "\n",
      "pca = PCA(n_components=2, whiten=True).fit(X)\n",
      "X_pca = pca.transform(X)\n",
      "\n",
      "def plot_2D(data, target, target_names):\n",
      "    colors = cycle('rgbcmykw')\n",
      "    target_ids = range(len(target_names))\n",
      "    pl.figure()\n",
      "    for i, c, label in zip(target_ids, colors, target_names):\n",
      "        pl.scatter(data[target == i, 0], data[target == i, 1],\n",
      "                   c=c, label=label)\n",
      "    pl.legend()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 2
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "To remind ourselves what we're looking at, let's again plot the PCA components\n",
      "we defined in the last notebook:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "plot_2D(X_pca, iris.target, iris.target_names)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAHMAAABKCAYAAACINmuDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADwdJREFUeJztnX9QVOX+x99nXYRlZdld2dZlWV2FGRR/ZH+0pUEL2u2P\nkq46GqyJOjkpmRdMGzP8ETpYYVZ2x0gr1BHTqS43x0kdHRFsrhl1o29QJiaCIjAKKAK2sOzu+/uH\neC4rP2IR0tbzmtkZnvM85zmf57zP85xznue9C9jO66+/znsNKabeA4AySPgMkpg+hChmbGzsXQyj\na6SYvENoH28l/uIIggB5d5larRbXrl37M+OR6AUajQZXr17tMq/bnikIAqROe+/RnS6CIEgPQL6E\nJKYPIYnpQ0hi+hCSmD7EfSNmRUUFZDIZ3G733Q5lwLhvxLyFL79uDYiYJJG5cSPMOh3MOh0yN27s\n95OYmZmJsLAwqFQqjB49GsePHwdJvPXWW4iIiEBISAgSEhLEiY/HH38cAKBWqxEUFITCwkKQREZG\nBsxmM/R6PebPn4/GxkYAQEtLC+bOnYuQkBBoNBpYLBZcuXIFALBz505ERUVBpVIhPDwcH330Ub+2\nrc/0tKTSHbW1tUyYNo3hej2nWiz85ZdfPPK3Z2VxglLJEoAlAMcHBnJ7VlanegoLC/n555+ztLTU\nq+WeM2fO0GQysaamhiR54cIFlpWVccuWLZw0aRKrqqrocDi4ePFi2mw2kmRFRQUFQaDL5RLryc7O\nZkREBMvLy9nc3MyZM2cyKSmJJLlt2zbGx8fTbrfT7XazqKiIjY2NJMmDBw/y/PnzJMkTJ04wMDCQ\nRUVFXrWhr3SnCwB6Labb7eZjEycyxc+PvwL8QBBo1GpZX18vlom3WpkLkO2fXIDxVqtHPStTUjhC\nqeQMlYo6hYJ7du/udYN+++03PvDAAzx27BgdDoe4fcyYMczLyxPT1dXV9PPzo8vlYnl5eScxp0yZ\nwg8//FBMl5aW0s/Pj06nkzt27ODkyZNZXFz8h/FMnz6d77//fq/jvxP6Vcyamhpq/f3p6iDWEyoV\nv/rqK7HMc9Onc7MgiPmbBYFzZ8wQ83/44QeaAgN5rT3/Z4CqgADa7fZeN2rv3r2Mjo6mRqNhYmIi\nq6qqqFAoqFKpqFarxY9CoWB1dXWXYo4ZM4aHDh0S03a7nYIgsLq6mm1tbVy/fj2joqIYGhrKlStX\nsq2tjSR56NAhPvLII9RqtVSr1Rw8eDDXrVvX69jvhH4Vs6GhgUo/P15tF8IJcMKQIczPzxfL/Pzz\nz9QNGcJ/yOX8h1xO3ZAhHkPx/v37+bRKJYpNgHqFgpcuXfK6cY2NjbTZbJw7dy4jIyP5zTffdFmu\nq2F26tSpzOow/N/qmR3L3No3KiqK2dnZbGlpoUKhYG5uLp1OJ8mbPXPt2rVex94XehLT6weg4OBg\nJC9ejKlKJd4G8HeFArrx4xEdHS2WGTt2LL796ScMS0/HsPR0FBYXIyoqSsx/8MEHUeh04of29F4A\nAUFBGDZsWK9iOHv2LI4fP47W1lb4+/sjICAAcrkcycnJSEtLw8WLFwEAtbW1OHDgAABAp9NBJpOh\nrKxMrMdms+G9995DRUUFmpubkZaWhsTERMhkMhQUFKCkpAQulwtBQUHw8/PDoEGD4HA44HA4EBIS\nAplMhsOHD+Po0aPensaBwdsrgLx539yzZw9fXrqUW7ZsYUtLi9dX2L9zc6kODKTa359mvZ4//vhj\nr/ctLi6mxWJhUFAQtVot4+PjWVNTQ7fbzXfffZeRkZEMCgpieHg4V69eLe63bt066nQ6qtVqFhYW\n0u12c8OGDTSZTNTpdExKSmJDQwNJct++fYyMjKRSqaRer2dqaqrYYz/44APq9Xqq1WomJSXRZrPd\nEz3zri6BOZ1ONDQ0QKvVQia77155+0RPS2DSeuZfDGk98z5BEtOHkMT0ISQxfQhJTB9CEtOHkMT0\nISQxeyAoKAgVFRV3VIfZbEZeXl7/BPQHdOtolwCampruuA5BECAIQj9E40lBQQEKCgo8j3U/zwC5\nXC4MGjRoQI8xcuRIZGdnY8qUKV7t53Q6IZd37mt/+gwQSWx8cyN0Rh10Rh02vtm/tpHMzEzMnj3b\nY1tqaipSU1PR2NiIhQsXIjQ0FGFhYVi7dq1o4tq1axcee+wxLF++HCEhIVi/fj3OnTsHq9UKtVoN\nnU6HxMREsU6ZTIbz588DAOx2O1asWAGz2Qy1Wo2YmBi0tLQAAA4cOICxY8dCo9EgLi4OZ86c6TLu\n1tZWLFu2DEajEUajES+//DIcDgeAmz0tLCwMmzZtgsFgwMKFC70/Md7OzpM3bSPTZk6j3qSnJaaz\nbSRrWxaVJiWxBMQSMDAskFnb+s82cuHCBQYGBrKpqYkk6XQ6aTAYWFhYyOnTpzM5OZm///47r1y5\nQovFwu3bt5Mkd+7cSblczq1bt9LlctFutzMxMZFvvPEGSbK1tZUnT54UjyMIAsvKykiSS5YsYVxc\nHKurq+lyuXjq1Cm2traytLSUSqWSx44do9Pp5KZNmxgRESEuZJvNZtH9sHbtWk6aNIm1tbWsra3l\n5MmTxdWW/Px8yuVyrlq1ig6Ho9uF+u50QV9tIxMtE+k32Y9YCgpPC9Q+4GkbsT5pJRJApLd/EkDr\nk1aPelKWp1CpU1I1UUVFsIK7c3pvGyHJ6Oho7m63mhw9epQRERG8fPky/f39PU7E3r17GRcXR/Km\nmMOHD/eoZ968eVy0aFGXC+O3xHS5XFQoFF1aSDZs2MCEhASP82M0GnnixAmSnmKGh4fz8OHDYtkj\nR47QbDaTvCnm4MGD2dra2mO7exLT62H28uXL+PXXX9H2tzYgBODDhHOoE6dOnRLLaNQaCA3/u+kL\nDQK0Gq2YLioqwie7P8GNhTfQOL0R9ufsWJS8SBy2esOcOXOwb98+AMDevXsxZ84cXLhwAW1tbTAY\nDNBoNNBoNEhOTkZtba24n8lk8qhn06ZNIAmLxYJx48Zh586dnY5VV1eHlpYWhIeHd8qrqanB8OHD\n/9dWQYDJZEJVVVWnstXV1RgxYoSYHj58OKqrq8W0TqfD4MGDe30Obsfrp1mFQgF3mxtoAaAA4Abc\nzW4olUqxTMa6DByLOYaWxpviBJwJQMZ/MsT8yspKyA1yIKB9wwOA4Cegvr4eRqOxV3HMmjULK1as\nQFVVFfbv349vv/0WKpUK/v7+qK+v73Z99PYnS71eL1olT548iSeeeAJWqxWjRo0Sy4SEhCAgIADn\nzp3DhAkTPPYPDQ1FSUmJmCaJysrKLtsRGhqKiooKjBkzBgBw8eJFhIaGdhubt/TJNrJ48WIo9ymB\nk4DiXwqMH9nZNvLTf39C+tPpSH86HcU/dLaNOC86gVsXZQkQFNh72whw8yqOjY3FggULMGrUKERG\nRsJgMODJJ5/E8uXL0dTUBLfbjbKyMnz99dfd1vPFF1/g0qVLAG56agVB6HQhyGQyPP/881i+fDlq\namrgcrlw6tQpOBwOPPvsszh48CCOHz+OtrY2vPPOOwgICMDkyZM7HctmsyEjIwN1dXWoq6vDhg0b\nkJSU1Os2/yHejs237gt79uzh0pS+20Zyc3MZGBRI/yH+1Bu9s43cIicnh4IgcPPmzeK269ev88UX\nX2RYWBiDg4P50EMP8bPPPiNJ7tq1izExMR51rFy5kkajkUOGDGF4eDg//vhjMU8mk4kPQHa7ncuW\nLaPRaGRwcDCtVqt4b/7yyy8ZFRXF4OBgxsbG8vTp02IdHe+ZLS0tTElJocFgoMFgYGpqqniPzM/P\np8lk+sM2d6cLJNvIXw/JNuJDSLaR+wRJTB9CEtOHkMT0ISQxfQhJTB9CEtOH8Dkxn3rqKeTk5PR5\nf2+sIv1hK+lPpEmDvxjSpEE7LpfrbocwoAycbWRjJnQ6M3Q6MzZuzPzTbCNxcXHIzs4G0LVN5OrV\nq4iPj0dwcDAsFgvWrFmDmJgYsZ6OVpEFCxbgpZdewrRp06BSqfDoo4+KebeX7cpW0traCgCYPXs2\nDAYD1Go1rFYrTp8+3W/nwgNvZ+fJdtvItATq9eG0WKZ2to1kbadSOYFACYESBgaOZ1bW9k71DIRt\nJDY2ltnZ2SS7tokkJCTQZrPRbrfz9OnTNJlMHispHa0i8+fP59ChQ/n999/T6XTyueeeY2JiYpdl\nu7OV3IqjubmZDoeDy5Yt48SJE71qb0e60wV9to1MfIx+fikEfqUgfECt1uhpG7HGE8jt8JMFubRa\n4z3qSUlZSaVyBFWqGVQodNy9e49XjerKNkKyk5gdbSJOp5N+fn48e/asuG3NmjWMjo4W0x0FWrBg\nAV944QUx79ChQxw9enSnsj3ZSm7n2rVrFARB/Bkab+lJzL7bRtreAzAa5BI4nWM8bSMaFQShXEwL\nQjm02mAxXVRUhE8+2YcbN/4PjY3/ht2ej0WLltyxbaQrOtpEamtr4XQ6PbaFhYX1eBy9Xi/+rVAo\n0Nzc3KlMT7YSt9uNVatWISIiAsHBwRg5ciQEQUBdXV3PDewDXoupUCjgdrcCuN6+xQW3+4qnbSTj\nNSiVmZDLUyCXp0CpzERGxmtifmVlJeTyCQDU7VvGQhAUqK+v73Ucs2bNQkFBgWgb6U7MjlYMnU4H\nuVyOyspKj1julI62ktv59NNPceDAAeTl5eH69esoLy8HyQF5U+ijbSQZSuVUAG9Dofg7xo/XdbaN\n/PQt0tOHIT19GIqLCzvbRpyFQIffGwkKCrhj28gfMWjQIMycORPp6emw2+04c+YMcnJyuvXe9PaE\n92QraW5uhr+/P7RaLW7cuIG0tLRet9Fb+vQ0+89/vo3t21dg6dIavPnm35Cff7CT+3rUqFFYvToN\nq1enYeTIkR55ZrMZOTkfITDwCfj7a6DXr8aRI/u9dpfPmTMHeXl5PfbK24XaunUrrl+/jmHDhmH+\n/Pmw2WwejriO5bva//b8W2zevBnjx4/Hww8/jKFDh+K1114DScybNw8jRoyA0WjEuHHjMGnSpAH5\nugJwlycN7gXbyKuvvoorV650abG8F7lnJw3kcrn440h/FqWlpSguLgZJfPfdd9ixYwdmzJjxpx1/\nILnvvgXW1NQEm82G6upq6PV6vPLKK3jmmWfudlj9gjQ3+xejp2H2vuuZvoL0/Uwf4J59AJLoX7od\nZjUazYC9D0n0HY1G022e9C8XfQRpmPUxJDF9CElMH0IS04eQxPQhJDF9iP8Hu929fEY2bA0AAAAA\nSUVORK5CYII=\n"
      }
     ],
     "prompt_number": 3
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now we will use one of the simplest clustering algorithms, K-means.\n",
      "This is an iterative algorithm which searches for three cluster\n",
      "centers such that the distance from each point to its cluster is\n",
      "minimizied. First, let's step back for a second,\n",
      "look at the above plot, and think about what this will do.\n",
      "The algorithm will look for three cluster centers, and label the\n",
      "points according to which cluster center they're closest to.\n",
      "\n",
      "**Question:** what would you expect the output to look like?"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sklearn.cluster import KMeans\n",
      "from numpy.random import RandomState\n",
      "rng = RandomState(42)\n",
      "\n",
      "kmeans = KMeans(n_clusters=3, random_state=rng)\n",
      "kmeans.fit(X_pca)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 4,
       "text": [
        "KMeans(copy_x=True, init='k-means++', k=None, max_iter=300, n_clusters=3,\n",
        "    n_init=10, n_jobs=1, precompute_distances=True,\n",
        "    random_state=<mtrand.RandomState object at 0x10fb57948>, tol=0.0001,\n",
        "    verbose=0)"
       ]
      }
     ],
     "prompt_number": 4
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import numpy as np\n",
      "np.round(kmeans.cluster_centers_, decimals=2)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 5,
       "text": [
        "array([[ 1.02, -0.71],\n",
        "       [ 0.33,  0.89],\n",
        "       [-1.29, -0.44]])"
       ]
      }
     ],
     "prompt_number": 5
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The ``labels_`` attribute of the K means estimator contains the ID of the\n",
      "cluster that each point is assigned to."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "kmeans.labels_"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 6,
       "text": [
        "array([2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,\n",
        "       2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 2, 2, 2, 2,\n",
        "       2, 2, 2, 2, 0, 0, 0, 1, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1,\n",
        "       1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1,\n",
        "       1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 1, 0, 1, 0, 0, 1, 0, 1, 1,\n",
        "       0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0,\n",
        "       1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1], dtype=int32)"
       ]
      }
     ],
     "prompt_number": 6
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The K-means algorithm has been used to infer cluster labels for the\n",
      "points.  Let's call the ``plot_2D`` function again, but color the points\n",
      "based on the cluster labels rather than the iris species."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "plot_2D(X_pca, kmeans.labels_, [\"c0\", \"c1\", \"c2\"])\n",
      "plt.title('K-Means labels')\n",
      "\n",
      "plot_2D(X_pca, iris.target, iris.target_names)\n",
      "plt.title('True labels')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 7,
       "text": [
        "<matplotlib.text.Text at 0x1139eca90>"
       ]
      },
      {
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAEYAAABICAYAAABLJIP0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAB3dJREFUeJztnE9MFFkex7/V6U7bVbUt1VDdYXsRdBKzrihM4mJw3G0T\nduYgqLB7EDJ7MRuNIW2T2dGJ8aB76M5EMsl6GTIknlhgIwnEgxzGg0JUiJMR12CHqAFDIyIxQIeA\ndOw/3z0IvbTwlG5wcUh9khe63nv1q/p9u17Vqy8PJACEQQokYSKJ9SoXLlxYt2OLygKmdfxiPmoM\nYQSsqzAHDhxYz8O/E4mLB5YBJEl6c6+hAE3TiDdPrA1ZNE1bNm/MSyK8YpLKbVBE+S3UGzdfAYYw\nAgxhBBjCCNhwwrS2tiI/Px+qqqKqqgpTU1MZxdlQwgSDQZw8eRItLS0YHx+HLMuora3NKNYv9nE9\nMjKCuro63L59G4lEAjU1NbDb7RgeHkZzczMAYGhoCDt27MDk5CQURUnZf10e1yRxMRBAga6jQNdx\nMRBYU5Hj8TgqKiqwdetWDA8P4/nz56iurkYwGERRUVGy37Zt22C1WvH48eOMknjnDHA5Xr58yaMV\nFfzE5WJZSQmDwWBKe2NDA3crCvsB9gPcJctsbGhYEufu3btsa2vjo0ePhMdajp6eHuq6zng8nlJf\nVlbGxsbGlDq3283u7u4lMUT5LdSnLUwikeBnxcX0WSwcAPi9JNHtcHBiYiLZ55DHw3bgzRsHwHaA\nhzyelDjf+HzMVxRW2e3UbTY2NzWJTmUJV65c4Z49e5bUHzlyhPX19Sl1qqqyr69vxfkt1Kc9lMbH\nxzEwMIB/RqP4LYBaEjtiMfT29ib72DUNTyUpuf1UkrDZ4Uhu9/X14d+XL+M/s7PomJ7Gzbk51J44\ngUgksqJzyMvLQygUQjweT6nfuXMnHjx4kNweHBzE69evsX379nTTTH8ohcNhKhYLJ+evhhjA3arK\nmzdvJvs8fPiQuqrylNnMU2YzdVVNGW5Xr15lud2evKII0GWz8dmzZ6LTSSEej7OoqIinT5/m7Ows\n5+bmeOfOHQaDQdrtdt66dYszMzOsqalhTU1NWvkh06FEkl97vfxUUVgPsNxmY1lpKaPRaEqfwcFB\nBvx+Bvx+Dg0NpbQ9ffqUObLMn+dFaQGY73QyFosJj/k2oVCIlZWVzM7OZk5ODuvq6kiSra2t3LJl\nCxVFYWVlJaemptLKb1XCJBIJNjc38yuvl5cuXWIkEllxQgt0tLczS5aZZbWywOXi/fv3046xGt4n\nzLrOY2KxGMLhMBwOB0ym/+9c833zmF/sBG+1GH5MhhjCCDCEEWAII8AQRoAhjABDGAEbSpgXL17g\n8OHDcLvdMJlMCIVCGccyr+F5rTsmkwkHDx7EuXPnsG/fvhXt09XVha6urqUN6b5LfCyEQiFWVVVR\n13VmZ2fT6/Um26LRKCVJ4vDwsHB/UX7I1I9ZCSQR+DYA3a1Dd+sIfPthrc3R0VFUV1evWXwAmVub\nFX+uoCvPxZI/LLU2G35ooJKnELUgakH5NzIbfvjw1uYCa3HFZGRtFpcU07LPQnhBqVyiw5lqbXq+\n8BBHQfxjvhwFPV94UuL4/u6joiu0F9tp22xj079Wb20usC5DacHajH4eBXIA/p6IZadam1qWBin8\nP2tTCktwaKnW5uWmy5j92yymK6cx9+UcTpxcvbW5lqQtjM1mQyKaABZySACJmUTK72385/1QflJg\n/tEM849mKD8p8J/3J9tHRkZgzjUDm+YrnIBkkTAxMbGic9i7dy9yc3Nx9uxZvHr1CpFIBD09PQCA\nSCSSFHjx57RJ91IjSW+dl8oWhfgctP3OxtI/Lm9t+v1++gXWprxZJk7MD7W/gM5fr421KUkSJUmi\nyWRK/kwnP6yFten1ZW5ttre3U/6VTKtqpcttWJspGNbmR4hhbWaIIYwAQxgBhjACDGEEGMIIMIQR\nsKGE6ezsxP79+6FpGnJzc3H8+HHMzMxkFGtDCTM9PY3z589jbGwMAwMDGB0dxZkzZzILlu67xMfC\nu6zNBTo6Orhr165l9xflhw9ubQYuQtcLoOsFCAQurou12d3djcLCwswOIvpG3tH0xtqsOEqX6xOW\nlJQttTYbGqkouwn0E+inLO9iQ0PjkjgfytokyevXr1PTND558mTZdlF+WJW1WfwZLRYfgQFK0vd0\nONyp1qbnEIH2RUvs2unxHEqJ4/N9Q0XJp91eRZtNZ1NTszDJt3mftdnb20td13njxg1hnzUXZmxs\njFarg0A8mbjd/ideu3Yt2aey8ktK0nfJdkn6jlVVf02237t3j7KcR2Bqvs9Dbtpk59zcnDCRxfT0\n9NApWLPX19dHp9PJzs7Od8ZYc2HC4TAtFoXA5HxSMarq7iWrNlVVp9l8imbzKaqqvmTVpt1evnjR\nJm0216pXbfb399PpdLKtre29MdZcGJL0er+monxKoJ42WzlLS8sE1maAfn9geWtTziHw87wwLXQ6\n81dlbfp8Ph47dowmk4mqqiZLYWFhWvmtSpikten9ahXWZgdlOYtWaxZdrgLD2lyMYW1+hBjWZoYY\nwggQro/RNA3Sor8g2WhomgZAvD7G+N8Ob2HcY96DIYwAQxgBhjACDGEEGMII+C/6bnE+Z3nsKQAA\nAABJRU5ErkJggg==\n"
      },
      {
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAHMAAABKCAYAAACINmuDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADwdJREFUeJztnX9QVOX+x99nXYRlZdld2dZlWV2FGRR/ZH+0pUEL2u2P\nkq46GqyJOjkpmRdMGzP8ETpYYVZ2x0gr1BHTqS43x0kdHRFsrhl1o29QJiaCIjAKKAK2sOzu+/uH\neC4rP2IR0tbzmtkZnvM85zmf57zP85xznue9C9jO66+/znsNKabeA4AySPgMkpg+hChmbGzsXQyj\na6SYvENoH28l/uIIggB5d5larRbXrl37M+OR6AUajQZXr17tMq/bnikIAqROe+/RnS6CIEgPQL6E\nJKYPIYnpQ0hi+hCSmD7EfSNmRUUFZDIZ3G733Q5lwLhvxLyFL79uDYiYJJG5cSPMOh3MOh0yN27s\n95OYmZmJsLAwqFQqjB49GsePHwdJvPXWW4iIiEBISAgSEhLEiY/HH38cAKBWqxEUFITCwkKQREZG\nBsxmM/R6PebPn4/GxkYAQEtLC+bOnYuQkBBoNBpYLBZcuXIFALBz505ERUVBpVIhPDwcH330Ub+2\nrc/0tKTSHbW1tUyYNo3hej2nWiz85ZdfPPK3Z2VxglLJEoAlAMcHBnJ7VlanegoLC/n555+ztLTU\nq+WeM2fO0GQysaamhiR54cIFlpWVccuWLZw0aRKrqqrocDi4ePFi2mw2kmRFRQUFQaDL5RLryc7O\nZkREBMvLy9nc3MyZM2cyKSmJJLlt2zbGx8fTbrfT7XazqKiIjY2NJMmDBw/y/PnzJMkTJ04wMDCQ\nRUVFXrWhr3SnCwB6Labb7eZjEycyxc+PvwL8QBBo1GpZX18vlom3WpkLkO2fXIDxVqtHPStTUjhC\nqeQMlYo6hYJ7du/udYN+++03PvDAAzx27BgdDoe4fcyYMczLyxPT1dXV9PPzo8vlYnl5eScxp0yZ\nwg8//FBMl5aW0s/Pj06nkzt27ODkyZNZXFz8h/FMnz6d77//fq/jvxP6Vcyamhpq/f3p6iDWEyoV\nv/rqK7HMc9Onc7MgiPmbBYFzZ8wQ83/44QeaAgN5rT3/Z4CqgADa7fZeN2rv3r2Mjo6mRqNhYmIi\nq6qqqFAoqFKpqFarxY9CoWB1dXWXYo4ZM4aHDh0S03a7nYIgsLq6mm1tbVy/fj2joqIYGhrKlStX\nsq2tjSR56NAhPvLII9RqtVSr1Rw8eDDXrVvX69jvhH4Vs6GhgUo/P15tF8IJcMKQIczPzxfL/Pzz\nz9QNGcJ/yOX8h1xO3ZAhHkPx/v37+bRKJYpNgHqFgpcuXfK6cY2NjbTZbJw7dy4jIyP5zTffdFmu\nq2F26tSpzOow/N/qmR3L3No3KiqK2dnZbGlpoUKhYG5uLp1OJ8mbPXPt2rVex94XehLT6weg4OBg\nJC9ejKlKJd4G8HeFArrx4xEdHS2WGTt2LL796ScMS0/HsPR0FBYXIyoqSsx/8MEHUeh04of29F4A\nAUFBGDZsWK9iOHv2LI4fP47W1lb4+/sjICAAcrkcycnJSEtLw8WLFwEAtbW1OHDgAABAp9NBJpOh\nrKxMrMdms+G9995DRUUFmpubkZaWhsTERMhkMhQUFKCkpAQulwtBQUHw8/PDoEGD4HA44HA4EBIS\nAplMhsOHD+Po0aPensaBwdsrgLx539yzZw9fXrqUW7ZsYUtLi9dX2L9zc6kODKTa359mvZ4//vhj\nr/ctLi6mxWJhUFAQtVot4+PjWVNTQ7fbzXfffZeRkZEMCgpieHg4V69eLe63bt066nQ6qtVqFhYW\n0u12c8OGDTSZTNTpdExKSmJDQwNJct++fYyMjKRSqaRer2dqaqrYYz/44APq9Xqq1WomJSXRZrPd\nEz3zri6BOZ1ONDQ0QKvVQia77155+0RPS2DSeuZfDGk98z5BEtOHkMT0ISQxfQhJTB9CEtOHkMT0\nISQxeyAoKAgVFRV3VIfZbEZeXl7/BPQHdOtolwCampruuA5BECAIQj9E40lBQQEKCgo8j3U/zwC5\nXC4MGjRoQI8xcuRIZGdnY8qUKV7t53Q6IZd37mt/+gwQSWx8cyN0Rh10Rh02vtm/tpHMzEzMnj3b\nY1tqaipSU1PR2NiIhQsXIjQ0FGFhYVi7dq1o4tq1axcee+wxLF++HCEhIVi/fj3OnTsHq9UKtVoN\nnU6HxMREsU6ZTIbz588DAOx2O1asWAGz2Qy1Wo2YmBi0tLQAAA4cOICxY8dCo9EgLi4OZ86c6TLu\n1tZWLFu2DEajEUajES+//DIcDgeAmz0tLCwMmzZtgsFgwMKFC70/Md7OzpM3bSPTZk6j3qSnJaaz\nbSRrWxaVJiWxBMQSMDAskFnb+s82cuHCBQYGBrKpqYkk6XQ6aTAYWFhYyOnTpzM5OZm///47r1y5\nQovFwu3bt5Mkd+7cSblczq1bt9LlctFutzMxMZFvvPEGSbK1tZUnT54UjyMIAsvKykiSS5YsYVxc\nHKurq+lyuXjq1Cm2traytLSUSqWSx44do9Pp5KZNmxgRESEuZJvNZtH9sHbtWk6aNIm1tbWsra3l\n5MmTxdWW/Px8yuVyrlq1ig6Ho9uF+u50QV9tIxMtE+k32Y9YCgpPC9Q+4GkbsT5pJRJApLd/EkDr\nk1aPelKWp1CpU1I1UUVFsIK7c3pvGyHJ6Oho7m63mhw9epQRERG8fPky/f39PU7E3r17GRcXR/Km\nmMOHD/eoZ968eVy0aFGXC+O3xHS5XFQoFF1aSDZs2MCEhASP82M0GnnixAmSnmKGh4fz8OHDYtkj\nR47QbDaTvCnm4MGD2dra2mO7exLT62H28uXL+PXXX9H2tzYgBODDhHOoE6dOnRLLaNQaCA3/u+kL\nDQK0Gq2YLioqwie7P8GNhTfQOL0R9ufsWJS8SBy2esOcOXOwb98+AMDevXsxZ84cXLhwAW1tbTAY\nDNBoNNBoNEhOTkZtba24n8lk8qhn06ZNIAmLxYJx48Zh586dnY5VV1eHlpYWhIeHd8qrqanB8OHD\n/9dWQYDJZEJVVVWnstXV1RgxYoSYHj58OKqrq8W0TqfD4MGDe30Obsfrp1mFQgF3mxtoAaAA4Abc\nzW4olUqxTMa6DByLOYaWxpviBJwJQMZ/MsT8yspKyA1yIKB9wwOA4Cegvr4eRqOxV3HMmjULK1as\nQFVVFfbv349vv/0WKpUK/v7+qK+v73Z99PYnS71eL1olT548iSeeeAJWqxWjRo0Sy4SEhCAgIADn\nzp3DhAkTPPYPDQ1FSUmJmCaJysrKLtsRGhqKiooKjBkzBgBw8eJFhIaGdhubt/TJNrJ48WIo9ymB\nk4DiXwqMH9nZNvLTf39C+tPpSH86HcU/dLaNOC86gVsXZQkQFNh72whw8yqOjY3FggULMGrUKERG\nRsJgMODJJ5/E8uXL0dTUBLfbjbKyMnz99dfd1vPFF1/g0qVLAG56agVB6HQhyGQyPP/881i+fDlq\namrgcrlw6tQpOBwOPPvsszh48CCOHz+OtrY2vPPOOwgICMDkyZM7HctmsyEjIwN1dXWoq6vDhg0b\nkJSU1Os2/yHejs237gt79uzh0pS+20Zyc3MZGBRI/yH+1Bu9s43cIicnh4IgcPPmzeK269ev88UX\nX2RYWBiDg4P50EMP8bPPPiNJ7tq1izExMR51rFy5kkajkUOGDGF4eDg//vhjMU8mk4kPQHa7ncuW\nLaPRaGRwcDCtVqt4b/7yyy8ZFRXF4OBgxsbG8vTp02IdHe+ZLS0tTElJocFgoMFgYGpqqniPzM/P\np8lk+sM2d6cLJNvIXw/JNuJDSLaR+wRJTB9CEtOHkMT0ISQxfQhJTB9CEtOH8Dkxn3rqKeTk5PR5\nf2+sIv1hK+lPpEmDvxjSpEE7LpfrbocwoAycbWRjJnQ6M3Q6MzZuzPzTbCNxcXHIzs4G0LVN5OrV\nq4iPj0dwcDAsFgvWrFmDmJgYsZ6OVpEFCxbgpZdewrRp06BSqfDoo4+KebeX7cpW0traCgCYPXs2\nDAYD1Go1rFYrTp8+3W/nwgNvZ+fJdtvItATq9eG0WKZ2to1kbadSOYFACYESBgaOZ1bW9k71DIRt\nJDY2ltnZ2SS7tokkJCTQZrPRbrfz9OnTNJlMHispHa0i8+fP59ChQ/n999/T6XTyueeeY2JiYpdl\nu7OV3IqjubmZDoeDy5Yt48SJE71qb0e60wV9to1MfIx+fikEfqUgfECt1uhpG7HGE8jt8JMFubRa\n4z3qSUlZSaVyBFWqGVQodNy9e49XjerKNkKyk5gdbSJOp5N+fn48e/asuG3NmjWMjo4W0x0FWrBg\nAV944QUx79ChQxw9enSnsj3ZSm7n2rVrFARB/Bkab+lJzL7bRtreAzAa5BI4nWM8bSMaFQShXEwL\nQjm02mAxXVRUhE8+2YcbN/4PjY3/ht2ej0WLltyxbaQrOtpEamtr4XQ6PbaFhYX1eBy9Xi/+rVAo\n0Nzc3KlMT7YSt9uNVatWISIiAsHBwRg5ciQEQUBdXV3PDewDXoupUCjgdrcCuN6+xQW3+4qnbSTj\nNSiVmZDLUyCXp0CpzERGxmtifmVlJeTyCQDU7VvGQhAUqK+v73Ucs2bNQkFBgWgb6U7MjlYMnU4H\nuVyOyspKj1julI62ktv59NNPceDAAeTl5eH69esoLy8HyQF5U+ijbSQZSuVUAG9Dofg7xo/XdbaN\n/PQt0tOHIT19GIqLCzvbRpyFQIffGwkKCrhj28gfMWjQIMycORPp6emw2+04c+YMcnJyuvXe9PaE\n92QraW5uhr+/P7RaLW7cuIG0tLRet9Fb+vQ0+89/vo3t21dg6dIavPnm35Cff7CT+3rUqFFYvToN\nq1enYeTIkR55ZrMZOTkfITDwCfj7a6DXr8aRI/u9dpfPmTMHeXl5PfbK24XaunUrrl+/jmHDhmH+\n/Pmw2WwejriO5bva//b8W2zevBnjx4/Hww8/jKFDh+K1114DScybNw8jRoyA0WjEuHHjMGnSpAH5\nugJwlycN7gXbyKuvvoorV650abG8F7lnJw3kcrn440h/FqWlpSguLgZJfPfdd9ixYwdmzJjxpx1/\nILnvvgXW1NQEm82G6upq6PV6vPLKK3jmmWfudlj9gjQ3+xejp2H2vuuZvoL0/Uwf4J59AJLoX7od\nZjUazYC9D0n0HY1G022e9C8XfQRpmPUxJDF9CElMH0IS04eQxPQhJDF9iP8Hu929fEY2bA0AAAAA\nSUVORK5CYII=\n"
      }
     ],
     "prompt_number": 7
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Some Notable Clustering Routines"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The following are two well-known clustering algorithms. Like most unsupervised learning\n",
      "models in the scikit, they expect the data to be clustered to have the shape `(n_samples, n_features)`:\n",
      "\n",
      "- `sklearn.cluster.KMeans`: <br/>\n",
      "    The simplest, yet effective clustering algorithm. Needs to be provided with the\n",
      "    number of clusters in advance, and assumes that the data is normalized as input\n",
      "    (but use a PCA model as preprocessor).\n",
      "- `sklearn.cluster.MeanShift`: <br/>\n",
      "    Can find better looking clusters than KMeans but is not scalable to high number of samples.\n",
      "- `sklearn.cluster.DBSCAN`: <br/>\n",
      "    Can detect irregularly shaped clusters based on density, i.e. sparse regions in\n",
      "    the input space are likely to become inter-cluster boundaries. Can also detect\n",
      "    outliers (samples that are not part of a cluster).\n",
      "\n",
      "Other clustering algorithms do not work with a data array of shape (n_samples, n_features)\n",
      "but directly with a precomputed affinity matrix of shape (n_samples, n_samples):\n",
      "\n",
      "- `sklearn.cluster.AffinityPropagation`: <br/>\n",
      "    Clustering algorithm based on message passing between data points.\n",
      "- `sklearn.cluster.SpectralClustering`: <br/>\n",
      "    KMeans applied to a projection of the normalized graph Laplacian: finds\n",
      "    normalized graph cuts if the affinity matrix is interpreted as an adjacency matrix of a graph.\n",
      "- `sklearn.cluster.Ward`: <br/>\n",
      "    Ward implements hierarchical clustering based on the Ward algorithm,\n",
      "    a variance-minimizing approach. At each step, it minimizes the sum of\n",
      "    squared differences within all clusters (inertia criterion).\n",
      "- `sklearn.cluster.DBSCAN`: <br/>\n",
      "    DBSCAN can work with either an array of samples or an affinity matrix."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Some Applications of Clustering"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Here are some common applications of clustering algorithms:\n",
      "\n",
      "- Building customer profiles for market analysis\n",
      "- Grouping related web news (e.g. Google News) and web search results\n",
      "- Grouping related stock quotes for investment portfolio management\n",
      "- Can be used as a preprocessing step for recommender systems\n",
      "- Can be used to build a code book of prototype samples for unsupervised feature extraction for supervised learning algorithms\n"
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Exercise"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Perform the K-Means cluster search again, but this time learn the\n",
      "clusters using the full data matrix ``X``, rather than the projected\n",
      "matrix ``X_pca``.\n",
      "\n",
      "Does this change the results?\n",
      "\n",
      "Plot the results (you can still use X_pca for visualization, but plot\n",
      "the labels derived from the full 4-D set).\n",
      "Do the 4D K-means labels look closer to the true labels?"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}