TextModels.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Models with Keras\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dense vector word embeddings\n",
    "\n",
    "A _dense vector_ word embedding means we represent words with number-full numerical vectors-- most components are nonzero.  This is in contrast to  _sparse vector_, or bag-of-word embeddings, which have very high-dimensional vectors (the size of the vocabulary) yet with most components zero.\n",
    "\n",
    "Dense vector models also capture word meaning, such that similar words (car and automobile) have similar numerical vectors.  In a sparse vector representation, similar words probably have completely different numerical vectors.  **Dense vectors are formed as a by-product of some prediction task.**  The quality of the embedding depends on both the prediction task and the data set upon which the prediction task was trained. \n",
    "\n",
    "When we use word embeddings in our deep learning models, we refer to their birthplace as the _embedding layer_.  Sometimes, we don't actually care about the trained predictor (skip-gram and cbow models); we're just interested in the embeddings by-product for use elsewhere.  Other times, we need an embedding layer to represent words in a larger model such as a sentiment classifier; there, we may opt for pre-trained dense vectors.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we don't care about the trained model and just want to create meaningful, dense word vectors, there are two popular prediction models: **skip-gram** and **CBOW** (continuous bag of words).  Word embeddings constructed in this manner are termed **word2vec** or **w2v**.  We will also look at another more recent method, **fastText**.  In any case, we've first got to construct training data from our corpus.  The exact procedure depends on the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Keras Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at the Keras models we'll use in this section.  (I'm keeping the code as markup since we haven't defined any of the parameters yet.  We'll run this code after we develop input data and parameters.)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 341,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from IPython.display import Image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### skip-gram"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 343,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAqkAAAC/CAYAAADdNkVoAAAgAElEQVR4Xu2dCbwVZf3/P3FZLlfg\nEuKaC2qaJGoSapqa298lNdxySUxzw6VcwF8iij8VRSyBtFxT08RcEXDf0kx/mkqY/iBLK8U9BeWy\nXNaL/9d7muE3Hs65c5Y555577ud5vcjgzDzzzHuemfnM9/kuX5KbCZiACZiACZiACZiACVQZgS8V\nOJ46SeyzvMD92nrzBknNbT0IH98ETMAETMAETMAETCA/AoWI1E6S/ijpJEl/zeh+sKTjJR0oaUV+\nh67oVtdLuj0cfxoHXlvSjyRdXqXnm8Y5ug8TMAETMAETMAETaDMChYhUhOhhko7KMlp+Gy5p1yoV\nbf0kPStpM0mLSqDdVdJXJZ0paXNJu0lqKaE/72oCJmACJmACJmACJpCFQL4idTVJr0k6WNKrYT9H\nSLpD0iRJvSQtDi2pXST9l6TRkuZLOlLSQ+E+60i6LOznAknrSbpa0lutXB0E4duSTpD0iKTfSjpP\n0jlh/6eEVtJukn4p6XFJF0n6uqSDJE0J+74yFKr3ljATNpX0Rrj/3eG5VaPluIRT9K4mYAImYAIm\nYAIm0PYE8hWpWA2fiFkifxAKwwMk4aeKEIxE2xhJ35e0n6SNJD0saQ9Jz0t6UdI0SQjGayR9W9K2\n4b/looGIPTUUvXeGllwE8nclsez+B0kDJP1T0p8l9Q0tvv1DscrYPwvHhDvCEEmfxw6GdRSxnGkR\nZZt3M7aNduP8GBPnZZHa9vPYIzABEzABEzABE6gxAvmK1G0kjY+JMiyoj0q6NeSBCwCWzu9J+ke4\nDB75reK7uVcoMh8IBSVL7lhn/xWKWYRrrnZjuB3il7a9pA8lvS9prdA6erikGaFIPTn8Nyy6f4tZ\nf7eUNFHSwAxBipjN9LHlOFiBNwwFbubYLFJr7Ebw6ZiACZiACZiACVQXgXxFaiRCsRxiecRiiTU1\nWvqPRNtpkp6WxLL4vPBUD5V0uqRfx6yZWB/rw36OSbCkIlJfCd0CIpH6O0kbx1BijUWkMp5vhcKS\n/mdKQsAighnTc6EvKZbVUppFain0vK8JmIAJmIAJmIAJJBDIV6QSGIVvKCIVC+VLkn4Si5bHV5W/\ns5T+uqRNJH0SHjuypLK8f52krUJLZuTnGonIXEONi9RI2E6QxNI/aaXIOMDYkkQqAU9TY8ePjsdS\nPyIay2m8dZb0M0kLswzMItW3lgmYgAmYgAmYgAmUkUC+IhUrJMv7+H6yVE/QE9bVHSWRmupJSbPC\npXV8RJ8K/UHXDa2YuAJgzfwg3IYAqIvDACusoCzdY1El8ClTFCJSsZDyWyRssdgyHsQxgVu7SHo5\nwZKKK8LZWSLyCeYi+GpZBmd8Uq8IA8IyL4FFahknpbs2ARMwARMwARMwgXxFaiQOSTFFMBHR/ARE\nEfhEuyVcvke4kubpGUlrhr8hRvlDYBLbs+ROmx76h2J1JfgKF4JsPqDkOEWAIlZpwySNC///5FA0\n7ylp6/C48eX+uFsCwpoxXJrCZY9blh04lQJQd2ECJmACJmACJmACcQL5ilT2OUPS6qEVNeoDIYrl\nM9P6iegkyh7r5Kfhxl8O3QUInloiqTEUvAjTBZL+R9LeOQKVMq8afSEOm8IfENHZluWj/diecWA1\n/chTwARMwARMwARMwARMoLoJFCJSo2j8b4TR9YWeWbQ/SfUJfPp5aI3Fn3SD0FcUn9FyNPK20jim\nmwmYgAmYgAmYgAmYQJUTKESkcioEHxGshG9pMW2N0Fr6NUl/kcRyfSWWy0lbxfGw4LqZgAmYgAmY\ngAmYgAlUOYFCRWqVn46HZwImYAImYAImYAImUAsE8hWpDaEFtRbOuZbOpRauh8/BBEzABEzABEzA\nBFYhkI9IJSKfEqCZ5USjnKXxpP7ViphqU++FAVos+5OlIMrjWuyYyaOKPy19Ue6VilytBW8Vexzv\nZwImYAImYAImYAIdjkCSSI2E6EGS3sigQ1J/BF9SxahqgPpgmLeVVFakj9pP0lBJ5EIttpFWi0IF\niHfywG4naQtJy4vt0PuZgAmYgAmYgAmYgAn8h0CSSCVpPblPSZpPgBMpnK6UtI8kKkidI4lk/CTq\njxLr0y8WxqtDwdZN0k/DXKlUcKIYAHlUKZ+aq5H3FOsn+14m6R5JJ0maG475h2Fu1o8lHSfpobCj\n3SXdEeZo5VgXhWIUKyfb0u+boeWT8f6zyIlAcQNysBJIRr/kjaVfUmgh3N1MwARMwARMwARMwARK\nIJAkUhF8WCFvD4XYK2EZ0l+FAhCL5CBJXwlLju4WLqPfL+k3ki6RdFVouTxU0gHhfj8ORWyuobMt\nwhRrJUUDbgiT+VN1iupQF0raPxTNpK36jqS/hsdGqH4o6S5Jd4f9TJR0a5j66rNQyFKxin+LN0R4\n1yyDIh8rAjlqiFPyuvJfyqnCkTKtT8SKDpRwWbyrCZiACZiACZiACXRsAq2JVH6jfOn5oaV081CE\nUVEKa2hUheqIsNwowhQxS8MKirhEwL4maafQ0hiJuT/mIVKxgm4VVonConu0JI7Fkv3pkn4fHotK\nUj1Cy+5boTWTsqwUE8AlgQpZiG0sq4hsGsv0fWKVq6JZgLA9LMuUoJxqVOWKn7NVnKIi1utZ+uzY\nM8xnbwImYAImYAImYAJFEGhNpEYi9PBQpGLdZJn9wHDpP/JXPTlcemf5/NVwDCyHU/4UyykVpvj7\nvPA3ypwiXHEHyNVwMVg7JvgQqQRvYbl9IRSv8X2xmBLAxViw8tLIwTo6FKaIzytCgctvUX97FJmn\ndRtJWGcpxRr5oCKEfxsK+yIuhXcxARMwARMwARMwAROICLQmUhGhRK0TGIWf5XclTYgFB0WBU/iE\nsgw/XBLVpGgEELGcvksoXPvFyqMiGJMsqblE6vfCsRCshE8o498kDICaJYliAe+H/4a1dEdJuCBQ\n4SouUo+UtG4WqycinLHGA6pY/seH9snYtIl8UqPzilgcGxPCnmUmYAImYAImYAImYAJFEsjHJ5Vl\nfKyE+GtSaQqL6ZQwoOq2MHAKAbtvGFC1LAxkYll+pKTnJT0maUy4FI+FM/JJxUo7J0MAciq5RCqB\nT4+HQvVcSY2S/hCWOyV4CxeAgaFQxerKMbF64oaA7yoCGQFK8Nc7WUQq4nfjLCIVgYpQjVok4LHg\nEtiFKwLuBIjz2UVeC+9mAiZgAiZgAiZgAiYQEkgSqYhFlrSJzqex1I/IpOH3iSUR0YovKIFDCFXa\nn8L/T7DReqFoxb+UZX7+i0UUNwDELm4BWDnjLdPSiasBApJjrRWK2q+HO2DFHRb+fzIOnBjriCAq\nsgicFeYx3UHSS6ElFwGNRbfYht8t5xM1/G+x7rqZgAmYgAmYgAmYgAmUSCBJpH45XKZHGJJqidYz\njIDHAprZ1gyX4P8d/kDCe4Qr4g0rLMfDKjkiXBbnN8RmPCgp31PiWEskEXkfbwREYen8VNLi2A+k\ns2J7UmbhT5pGTlOqV8GIJP7x6P98z8HbmYAJmIAJmIAJmIAJZCGQJFLZBT/L/mFO1EIhRtH8VGVi\niR9LKEnvvxmKOvxZJ2URmoUeJ9/tGQ8WYFwFsPa6mYAJmIAJmIAJmIAJVCGBfEQq1lCWsgmiKqZC\nU/cwiT7L/ET4k5s0ivSvNJJSz6XS4/XxTMAETMAETMAETKBDEshHpHZIMD5pEzABEzABEzABEzCB\ntiNgkdp27H1kEzABEzABEzABEzCBHAQsUj01TMAETMAETMAETMAEqo6ARWrVXRIPyARMwARMwARM\nwARMwCLVc8AETMAETMAETMAETKDqCFikVt0l8YBMwARMwARMwARMwAQsUj0HTMAETMAETMAETMAE\nqo6ARWrVXRIPyARMwARMwARMwARMwCLVc8AETMAETMAETMAETKDqCLQHkUqVqDMkTZH0T0nfl/SZ\npCclrSPpGEm/lLSw6uh6QCZgAiZgAiZgAiZgAkURaA8itV7STElHSHpZ0h2Sng+F6aaSnpO0eShc\ni4LgnUzABEzABEzABEzABKqLQHsQqdVFzKMxARMwARMwARMwARMoOwGL1LIj9gFMwARMwARMwARM\nwAQKJWCRWigxb28CJmACJmACJmACJlB2AhapZUfsA5iACZiACZiACZiACRRKIF+R2iCpudDOq3T7\nWjqXKkXsYZmACZiACZiACZhAaQTyEanflnSqpBMk/UXSt7JE0h8r6SuSLs1jOFdJ+l9Jv85j27Q2\n2VLSe5IWhOewq6RPUuocPmuEKbJS6tLdmIAJmIAJmIAJmEDHJpAkUkn/9GdJB0l6R9KrkrbJYlXt\nJ6m7pNfzwHm9pBlhCqk8Nk9lkwclXRSmsBosaT9JQyV9XkLvvSQNCMX2NZKuLqEv72oCJmACJmAC\nJmACJhAjkCRSSZx/lKSDJXUNBes9kk4J+zhM0jOSviPpy6E18UxJb4eW10ck3SjpAkkjJY0LBe/4\nBFG3pySsn90kXSaJY54kaa4kxvxDSbdI+ljScZIeCseze5hHdU1JPwuFKWKU47Et/b4p6cXwnCgO\nUGw7NBwX+//YIrVYjN7PBEzABEzABEzABFYlkCRSSZyPFfJ2SZFVlf8i0FjmptLTZpK2k7SupJ+H\nYg33gNGS7pJ0digOD5e0k6TLJZ0Yitdc1yQSgBdLeljSDeH2HA+BfKGk/cOKU1NDkfzXcAkfofph\neOy7QyE5UdKtkn4XuipwXo+G/xYfAxWsEOOZrSkUyNnGiwh/xSLVt5cJmIAJmIAJmIAJpEegNZHK\nb1hCz5c0LRSpVH7Cn/Pd0KJ5p6Q/hgJu7dBSimj7l6QxklaT9JqkAyQhImkIV/ZpbXkckcry/FaS\nWsJSqEfHqk6dLun3YX9YaXtIYsn9LUl7S3pKUl9JXcKxIkqxrCImaT+S1Cccb5wmY8M6nNkQ2liB\nLVLTm3vuyQRMwARMwARMwARyEmhNpEYCEwtoJFLxSd1W0rywRwQp//appLhIjSyLUdlS/hvfJ8ny\niItB1B+Hwu0A6yy+pC+E4jV+UlhMfyDpZEm/Cn+YHFpzORbi84rQJzXe3x6SVpQ4P2xJLRGgdzcB\nEzABEzABEzCBTAKtiVSW9fHdPCaMiOfvCNJ4dH8k0PAVzSZSCajCkvrV0CeU4xM4xb+1ZknNJVK/\nF45lSOgfy/g3CQOgZoVR9u+H/4a1dEdJu4XL/HGRemTonpBpHcXXlTHHA6pY/sfv9klbUn0DmYAJ\nmIAJmIAJmEBlCOTjk3p/GIwU+aTiAnBOaFHFqokARbjGRSpiFv/RTuHS/tOhHynL99NjgUZYaedk\nEYC5RCqBT4+HQvVcSY2S/hD6wmLtxQVgoCSEKlZXXA7IRoBPLb6rWFQRoFeG2QoyRSrid+MsIhWB\nilDN1mxJrcxc9VFMwARMwARMwAQ6EIEkkYpY3FrST2OBU1+P8TkiFH5xyySW0pdjgVGbx1JTEWGP\nbyqiEXE3RdJz4VJ8HHumpRMfVQQkWQbWCkVtNA7E8LBwZ/xSCcqKGkFUCOSzwgj/HSS9FApnsg3g\nG1tqI+/r3x04VSpG728CJmACJmACJmAC/0cgSaSSVgp/U4QhAjNq/Du+nES959NYMu8d9rU8tsO+\nkhCbuYKSWuubNFNLsoyBgCisvox7cawD0lmxPT61RPtvISk+lnzOw9uYgAmYgAmYgAmYgAlUgECS\nSGUIVJPqHy7xpz0kcpxOKkDslnp8zpfIf1wF/lRqZ97fBEzABEzABEzABEygPATyEamdJQ0Kg6hK\nqdBUnjMorNdaOpfCztxbm4AJmIAJmIAJmEA7IpCPSG1Hp+OhmoAJmIAJmIAJmIAJ1AKBQkQqVtRC\ntq80n2ofX6V5+HgmYAImYAImYAIm0G4JFCI6q10EVvv42u0k8cBNwARMwARMwARMoNIELFIrTdzH\nMwETMAETMAETMAETSCRgkZqIyBuYgAmYgAmYgAmYgAlUmoBFaqWJ+3gmYAImYAImYAImYAKJBCxS\nExF5AxMwARMwARMwARMwgUoTsEitNHEfzwRMwARMwARMwARMIJGARWoiIm9gAiZgAiZgAiZgAiZQ\naQIWqZUm7uOZgAmYgAmYgAmYgAkkErBITUTkDUzABEzABEzABEzABCpNwCK10sR9PBMwARMwARMw\nARMwgUQCFqmJiLyBCZiACZiACZiACZhApQlYpFaauI9nAiZgAiZgAiZgAiaQSMAiNRGRNzABEzAB\nEzABEzABE6g0AYvUShP38UzABEzABEzABEzABBIJWKQmIvIGJmACJmACJmACJmAClSZgkVpp4j6e\nCZiACZiACZiACZhAIgGL1ERE3sAETMAETMAETMAETKDSBCxSK03cxzMBEzABEzABEzABE0gkYJGa\niMgbmIAJmIAJmIAJmIAJVJqARWqlift4JmACJmACJmACJmACiQQsUhMReQMTMAETMAETMAETMIFK\nE7BIrTRxH88ETMAETMAETMAETCCRgEVqIiJvYAImYAImYAImYAImUGkCFqmVJu7jmYAJmIAJmIAJ\nmIAJJBKwSE1E5A1MwARMwARMwARMwAQqTcAitdLEfTwTMAETMAETMAETMIFEAhapiYi8gQmYgAmY\ngAmYgAmYQKUJWKRWmriPZwImYAImYAImYAImkEjAIjURkTcwARMwARMwARMwAROoNAGL1EoT9/FM\nwARMwARMwARMwAQSCVikJiLyBiZgAiZgAiZgAiZgApUmYJFaaeI+ngmYgAmYgAmYgAmYQCIBi9RE\nRN7ABEzABEzABEzABEyg0gQsUitN3MczARMwARMwARMwARNIJGCRmojIG5iACZiACZiACZiACVSa\ngEVqpYn7eCZgAiZgAiZgAiZgAokELFITEXkDEzABEzABEzABEzCBShOwSK00cR/PBEzABEzABEzA\nBEwgkYBFaiIib2ACJmACJmACJmACJlBpAhappRP/vPQuqqaHQuZD1QzaAzEBEzABEzABEyiJwEBJ\n2zU2Nm7ftWvXAUuXLv3KkiVLGpctW1bf0tLSqa6ubkWXLl0Wd+vWralr167vL126dEZTU9OLkl6S\nNL2kI7eycyGiBDFWyPblGnOufttqfJ9//nn716lf+lJwaav5+lZ6Pvl4JmACJmACJlDLBA5ZffXV\nhyxatGjPNdZYY8VOO+1Ut/3226/2ta99TRtuuKHWXHNN9ezZU507d9by5cs1f/58ffzxx5o1a5b+\n/ve/68UXX1z43HPPtXzyySedunfv/uScOXMmSpqUJrBCRElbicB8z7etxmeRmu8V8nYmYAImYAIm\nYAJtSWCj+vr6H3fq1OmkrbfeevnRRx/de7/99tMGG2xQ9JjeeecdPfTQQ7rtttvmvvrqq51XrFhx\nw+LFi38l6a2iOw13tEgtlaBkkVo6Q/dgAiZgAiZgAiZQPgLrNzY2Xrxo0aIhp5122udDhw7tgsU0\n7YaF9frrr1929dVXf6l79+4Tm5qaLpD0brHHsUgtltz/7WeRWjpD92ACJmACJmACJlAeAiPq6uou\nHT58+IoRI0Z0/vKXv1yeo8R6/eyzzzR27Njl48aN69TS0nKepLHFHNQitRhqX9zHIrV0hu7BBEzA\nBEyg9gm0/wCOwq9RITqr8N5b32Obvn373jZo0KD1fvGLXzSWw3KaNGAsq2eeeWbTtGnT3ps9e/bR\nkl5J2if+eyHw2srnM9/zaavxWaTme4W8nQmYgAmYQEcmUBPvy3wvYBsHJB8v6cZrrrlGp5xySr5D\nLtt21157rU499VT6P0HSTfkeyCI1X1K5t6uJm66Nb6bSr4J7MAETMAETqHYCNfG+zBdyW71XO3fu\nfNm666576r333ttr2223zXe4Zd/u5Zdf1qGHHjrvgw8+uGb58uXn5nNAi9R8KLW+TU3cdG11M5WO\n3z2YgAmYgAm0EwI18b7Ml3VbvFcbGxtv7t+//0EPPfRQ7z59+uQ71Ipt9+mnn2q//fab+/rrr09u\namo6LunAFqlJhJJ/r4mbri1upmS03sIETMAETKCGCNTE+zLf61Hp9yoCdeDAgQc/9dRTjfmOsa22\n23333ZumT59+X5JQtUgt/QrVxE1XhpupIzrI5zubCrnv8u0zvp3ZF0PN++RDIHPulmWuNTQ0XNbc\n3DwynwF5m3ZFoCbel/kSL8N7NeehWeIfNGjQyS+88ELvfMfX1tvtsMMOc6dNm3Zda0v/hbws2yow\nKV+OiyU1SFqR7w4pbVcTN10Zbqaa4JLSHFnZTRk4Zxui2ad94dyfcszd7zQ0NEy55JJLep511ll1\naWD65JNPdOCBBy6YMWPG1Hnz5g1Jo0/3UTUEOtSzqULPey7u8RtssMH4V155pVc1LvHnmn0s/W+z\nzTbz3nnnnWG5gqlqSaSSLHZ7SR9U+HasiZuuDDdTTXBJey6VgbNFatoXyf1lJdDK3N2ssbHxgRNO\nOKHfFVdc0TUtfIcffnjzY4899lJTU9MBkhak1a/7aVMCHeq9UKHn/TaSpr/00kuqpiCpfGcZwVTb\nbbcdmw/Mlp6qlkTq78NksU/kCyel7WripivDzVQTXFKaI7akpg3S/VWcQMIzokfv3r0f3Guvvba9\n6667WNFKpQ0fPnzJTTfdNKupqWl/SW+m0qk7aUsCHeq9UIb36irXrm/fvjMuvvjiLaohzVSxE4v0\nVBdccMHM2bNnD8jso5ZE6mhJLDdV2o+pJm66MtxMNcGl2Jsu135l4GxLatoXyf0VaklduX2vXr0m\nDhgwYPDUqVN79O3bNxWS48ePbxk1atT85ubmwZL+mEqn7qStCHSo90IFnvcj9tlnnxGPPPJI1QdK\nJU24fffdt+nRRx+lKtUXKlPVkkglGdhdkjZOgpHy7zVx05XhZqoJLinPlVx+fWkfxuzTJur+8p67\nDQ0NY9Zcc82fIFS32mqrVMjdeeedOvLII+mL/7kzlU7dSVsQ6FDPpjK8V+PXbP26urq3Z86c2akt\nKkmlPXmoTLXFFlusaGlp6ScJ982g1ZJI5XymSnpG0vgUAd4gabik+Tn6bPOb7u2339b222+vv/3t\nb6Im7/Tp03XJJZdo6dKlOuSQQzRkyBB16dKlVSRF3ExVzyXFOZBaV0VwznbsDsd+8eLF2nrrrfWn\nP/0pmOPFtqlTp+qmm27SlClT1KlTp5Xd0P83v/lN/e53vwuOk2/L1V+++2dut2LFCo0aNUrnnnuu\nevToUWw3ZdmvkLlbV1d3an19/fhJkyZ123vvvVMZzzPPPKODDjpoUXNz86glS5aMS6VTd1JpAm3+\nvqzkCRdyzxQ6rsbGxt8MHTp0yOWXX9650H2rdftzzjln+fXXXz+xqanpR7UqUreQ9LKk70t6KKUL\nsSh0I+ChOCaLWG3zmy4uUl9//XV9+9vf1q9//WttuOGGOvnkk7XHHnvohhvQNblbETdT1XNJ6fqn\n2k0RnLMdv8Oxj0TqK6+8ooaG4l0eEZXjxo3TH/7why+I1GXLlukb3/iGbr31Vg0aNCjva56rv7w7\niG34zjvvBOL50ksvXfnBWUw/5dqniLk7uEuXLvf8+te/7nzMMccUYhDJeQpvvvmmvve97y149913\nb1y4cOFZ5TpX91s2Am3+vizbmWXpuIh7Jt/hbdS1a9c3Pvroo86lfLTne7BKbffZZ59p7bXXXr50\n6dLNJL3FcQt5cFR7CqqI436S7pF0fkoW1TNCHwnMLjCYkCFWE2+6efPm6cQTT9SVV17JBQgsOe+9\n915gMeHleMIJJ+iiiy4KXr6nn3667rnnniBK7+abb9aAAQOCbam/y9LZGWecEbzAeFiz/LX66qsH\nohQLEP9+xRVXqK6uThdffHHAg5f6PvvsE2zfq1evnHOtiJupZC6VmvjVdJwiOGcbfrtk/+yzz+rY\nY4/Vv/71L/3iF7/Qj3/8Yy1fvlw//elPtfvuu2vYsGHBfJ4wYYJ+//vf67//+7+DbZjTn3/+eWDp\n/P73vy+c7Gl33323vvOd7wT//6233tLw4cM1efJk/eQnPwn2pS9atFTMqgL3Yn19fSAG//3vfwf3\n06OPPhrUlL788stFpCkitZj+4pbZ6KIx7nPOOQdhpZ122kkvvviiLrvsMk2cODGwlHJM7nE+JFkN\nee2117TmmmvWikgFw3Y9e/a8f8SIEX1HjhyZSoqqBQsWBCmqpk2b9nhTU9Mh1XR/eyyJBBLfl4k9\ntKMNUnrer3LG9fX140455ZSfjB8/vvUl0nbEKhrqsGHDll177bW/XLx4MSvYNSlSOS8sqlg9twz9\nl56WNFPSR0XmUZ0jKaovtjRDrM7jRdRa40W84447auzYsdp11121yy67aNasWcIH4+OPPw5eTjNm\nzNCee+4ZvMh4qSFUEa5vvPGG5s+fr8022yx4eSFWN954Yw0cOFDjx48P+uUFSEOkfvTRR4FIZXsa\nL8PRo0dr5syZ6tw596pAkTdTSVza4f1T8pCL5JztuO2KPelRmOf33Xef+vXrR/1mHX/88YEwRXwi\nHhGTWDJZBeADjmXik046KRCxhx9+eLAdFtV7771X//M//xOIUe4PAnQ233zzQHDSLwL1/fff11NP\nPRUI2aOOOkoPPPCAWlpaEDc67LDDdP311wf9cS8ihNnnoYce0rRp04L7q9D+7rjjji9YZuMXjKV7\nlvERpAjUkSNH6i9/+Yu+/vWvB9bbuIvBP/7xj+CjM3LdKXnCpdhBCXN3fSL/jzzyyK9dc8013dIa\n0rHHHrto6tSpr82dO5cUVZ+k1a/7KSsBi9QU8DY0NMyfPn16j1rwRc3EgS4aOHDggubm5p61LFKj\n8yaYCgW3oyRU2xqS0npILpPEn4YkkcpgeEEhVnkh9u79n4IQvJB4ySIkEab77bdf8Pfu3bsHlqN9\n9903sJYiRHmhsj1C9fbbb9fDDz8c/DcSp1iU4i+2JUuW6Kqrrgpe8LxAjzjiiFZvjfAFlMLtEzDJ\nm0saB2xPfaTIOdtpVy37oUOHaq211lpp4ahkrCsAACAASURBVH/66aeDFYQ///nPgShjdeFb3/pW\n8OHGx9o///lPde3aVeyHpZGViC222CJYpl9//fWD+4M5jcik3/POO2/lh9gHH3ygr3zlK4HFFkHI\nSsIxxxwT8OKeufHGG3X11VcHIji63xYuXBisVNx1113CfabQ/rD8ZrOkckwENWL71VdfDY6JeOae\n5J4lP2A0BrZlxYMP1RoTqZxa5969ez+w884773zfffet1toHcyH386hRo5ZdddVVH86bNw+h+loh\n+5Z527JU4irzmHN1X8iKa9IQLVKTCCX/fsgOO+xw4/PPP99uKksln9IXt9hxxx3nvvDCCydImlTI\n5Gsvy/2F8shn+5KtViwjshyJdZQlSV6sG220UbC8x4tq0aJFwQsyHtARvaB5ycZfXPw7S5K8uLO9\n2Hgpsj2ClqXMbbYh12/rrUgrSclcksZVa78XyTkbhnbFHjeWX/7yl184D+Ynwg2x9txzz2mNNdYI\nRBoW1shnFCHL/OXfCGjiPorcVviNf1tnnXUCN4DMxn3APRK3VLJCwWoEy/u33XbbyvstCpzCkotI\nLbS/1kQqflZYZh955JFgPFht77//fg0ePDiw8Mb9xWtYpAaXp0ePHjdsuummR0ydOrUnHxtptOuu\nu+7zs846a9HixYsPlvRYGn2m0EdNiLEUn1cR0prgku/8KAM/3Jgmjx49+sD2nBc1iR8uXaNGjZoy\nZ86cgyxSk2hJ+P9dFgZPFeWTyiFYzuRliz8Vlhx8U/FDo7EUjwWJ5UpEK8v1kaUIayqWprhIZTte\n8Fh7aPjPYYXC+sILFlcALDUskeZruSviZkqFSzL+2tqiCM7ZALQ79qwI7LDDDoE4ZEWBcnjRXI1H\n7SPSWLLHlxrLZKZIjUf3R7+xMsHy/ZNPPhn4eLOsz/7cB9w3iGMsrjTcDfj72WefHbgaRG4w8cAp\n7sVC+2tNpHIvI3oJamS1hFUTlvppiGI+QqNW6yKV8+zWrdv5ffr0OXfKlCkNYaWZkm9yAtgOO+ww\nAi74cr+l5A5L76AmxFhKz6s4zZrgku/0KAM/Ylfmv/766z022GCDfIfR7rYjiLR///7Bkr9FavLl\nSyWSmhcVVhSWNVnKJCKPGrs777xzsPw3e/bswCKEheWAAw4ILEm77bZb4LvK0n1cpPIbfqgEYeA/\nx5ImLgOIVKw0LHPyIma/qOG70ppgLeJmSoVLMv7a2qIIztkAtDv21113nX7+85/rhRdeUGNjY5B1\nAgsjH1P4ZUbiszWRissLH21RgBOiFxcYPgAjayxL9lhIEaGIYI7JEv/zzz8f+IXiSkDWC77U11tv\nvUC04qfKNkcffXRgqeUjsdD+MlNaZV40rLn4xk6aNIk0SkGg2F//+tfAkswHa0cSqZxrXV3dj+rq\n6m64++67O2NRTqPh93zggQcu/PTTT8cuWbLkkjT6LKGPmhBjKT2vLFJLmEgZuw7ccMMNn3777bdz\nR0Gnd6w27alfv37zZs2atZtFavJlSC0nJQEfWFEIkiLCmBfm/vvvr7PO+k8mlQcffDAQqFHjBcoL\nLbKu8ELu2bNnYGW98MILV/r38XLFx49+zzzzTN1yyxcNCflECxfxMEqNS/IlqJ0tiuCc7eTbHXs+\nmAh0IiiKhuWf+b7aaqt9If9p5nI/bgJ8YLHcj0hF2EWNQCtWC2isTkTuL9wjfMhhSUXAfve73w38\nQmlkF2BpH1HKByH3Fw3RiKjlnsOyW0x/uXxS6Z/zIpgR/9NNN900yPSB9RXXH0Rx1DJzHlfTzE9p\n7sZPae/6+vr7JkyY0P3kk08u5F2UE8u7776LG8X8N998884FCxac1Ib8LFKzw0/kwgoJLm2Zdeh5\nrxEQGa0gtuG1zfvQZbhnTj7qqKOumDhx4mp5D6KdbjhkyJCFt99++9mFPBg6sk9qa5c58aYrZI40\nNzdr7ty5gd9dUjJvlkx5wWGZKrWV4WZKlUup51ct+5eBc7ZTq1r2zFkaqwjFNiywCMLMeU8WDApY\n8O+ZgTlk0UAQ8yfeon2idFXZfsunPwLAmpqasq5W4K5DEFh7b2Wau1v16tXrgdNPP32d0aNHp5JO\nB3eSgw8+eOGzzz77bBj5v7wN2FftPVgIizJc80QufHiyEpIpUvmAI3ajf//+hZxCm26bNj8S+I8e\nPfpYPvhrvbEaPGrUqFssUku/0ok3XemHKH8Pad9MpOnKJ+tB+c+suo5QBs7tSqRW19VIbzSsZCCe\nMxvXGz9Ui9RWWa9B5P/gwYO3uuWWW7qndVVOPfXUJXfcccff586du3+8zGJa/Sf0UxPPvzI8rxK5\n5BKpVBzjHsM9h/SLuOsQjIiFldRu//Vf/xUYbTD0kHaRlI9k06B4B5lBeB+xQkKmDRqucaSH69at\nW5C3mdR4/I5L0WmnnbbK5cUlD9/n4447rpyxHq3fKGus8fLEiRMH7bXXXmWdxvEKfDy7iBOI4mWy\nHZgVK9wYibvB9z5p+3wG//jjj1Mtc5pFaj60Wt8m8aYr/RDl76EtHkblP6vqO0IZOFukVt9lrskR\nlXvuNjY2Tho0aNDeU6ZMWS1pFSlfwGPGjGkZM2bM7IULF5KK8KV890thO78XskNM5JJLpOKiQ3o5\nxCgikiwd+LSz0oFLD/7kuAThRkcaR4qAkD0DwYqbzf/+7/8KcYfrD/v84Ac/CIInSU8X9Ud+ZlLb\nRYGN8VMoRnylfc80NjZ+8OKLL65T7vyo8UBS3JQIMCUGIFdDoBIng8shq1NJ2+dzfxHAuv32239o\nkZoPLYvUYiglPoyK6bS975P2QysHD7Nv7xOlCsdfibnbs2fPK7/yla8cd//99/fAdzeNduutt35+\n4oknLl+2bBl5yqam0WcefdTEPViGa57IpTWRSrEaUjnitxoFDXMt+DuZIgg2RlQRHU6KM1w/KKBD\nLMiWW24ZZBXBV50sOyyZk8oRgZrZX7bri0hFwE6fPv0LfuStzYUC+ZHXfRNJj0v6j19URquvr1/4\nwQcfNMRLoVKREkswuaER3ViPyU5CgCh+7wh03J1wSSLAE5GeWfkPC/SHH36YtQIfgZ2kxyR2Bjer\nqIIfQyPfNCtFfCQQL0BMDNuR6i/aHj9/qv3RGCe8ycACfz4asGizL/75WMmjhtV83XXXbbZIzeNp\nk7BJ4k1X+iHK30OBN1M+A6oJLvmcaCHblIFztsObfSEXxdvmRaBCc5fl1+ENDQ2jJ0+e3D0qe5vX\nAFvZ6LHHHuNFuWTx4sXDWlparim1vzz2r4l7sAzXPJFLPiKVlHYIsshvNUpHh+iJqi3GrxEBm6R6\nYx8EVNR+9atfrRSpLFVnW+anb7LyZLZo3xRF6vOSSGpOJOWHkl6Q9HtJf5L0vxynrq6uZfHixZ3i\nPvdRUCZuCOeff34g+hCiWDWpokf6O0QiqStxbchW+Y8iQ5x/tgp8lGuPxDmWaYJNcW9ibvABQOo/\nhDKl2BGz+P1H23N8snewPXmwsbb+6Ec/CnyOsXrjQkFVQD4AGHe8iAkfGPX19SssUvN42likFgUp\n8WFUVK/tfKcyPPSzitR2jsnDr14ChbwzSjkLSuTdkU+1vHwPgk/d4MGDF3z88ce/bG5uHpnvfkVu\nVxPPvzI8rxK5IFJHjBixShEalvsjS2qmkI1EKrnH+bAhtRvBx1j+KEG8ySabBAIO4UNaOgI3cQMg\nxzIWvWj/bCKVPOT4aLKEjUWWrCCks0N0xdPHZZsnRfC7TdJhkqJIywVhvwQVUtZ9w8xYD0QqAhxR\niptDVDwEgUiwGeIPHgjbXJX/sHaSaShbBT5cI/AxxYKMi8T/+3//L+BFI/UlVl3cI7CCwposRPHt\nEaak36PhcoHVlb4QuKQmJA0n7gUUPImyq0Qs4VfIA8fR/dmfVok3XZEPuYruVsTNlDS+muCSdJKF\n/l4GzoUOwdubQHshsEtDQ8PU0aNH9xw2bNj/5ekqYfRYbhCqM2bMmDpv3rwhJXRVdWngMtO3lXBu\nK3ct4nlVMhcEEMKFpXiyddAQXyxXtyZSSR03ZMiQQOxQEhwhyz70g8Ak2Ipc5Fj8EHTkVCaoCutq\nVL2utah5xBcWSYRYyCURcRH8CBx8FTEaE6och5LX19fV1Z2azZJKMFhUUCRePQ+Bje9u9Fuuyn8/\n+9nPghzOUb7neB+RSKUqIDmtM4Ukg4vnt8Y/FZGabfsonWZUaTDKj83xCG7DEsv1otmSmji98t6g\nJsRYETdTEqCa4JJ0koX+XgbOhQ7B25tAeyKwaWNj44MnnHBCvyuuuCK1PF6HH35482OPPfZSU1MT\niakja1UhXKquoAZLpZTKpDhMviIq6YSLeF6VzCWbkEK0UgAEv0l8UjOX+6O8qojNqCx4dG4U90C8\nRgVy+Hdyh/NvCFeKQBDVj+tAZCHMxgWRypJ6VLI5iR2/58GvXhLl8KI/O0n6m6R+aPPwGAslHSrp\n0Ww+qQg/rJhYKbEcR4IPiy++nnGRmqvyH2kv8SHNVoEvEqkEplFUiKwJ5JWmPfHEE4ELAeI/Kmcd\nidRoez4EuH40+sfqiyWVvloTqfZJzWeG5bdNTYixPG6m/Gj831Y1waXQk07avgyckw7p302gvRPo\n0bt37wf32muvbe+6666GtE5m+PDhS2666aZZoVB9o8B+KU08VlIn0u1JmiBpjKT5YT9Zn3/47yEm\nKG5B+iP8BW+44YbAUshS84QJE4KCLfwe/Z38u/w7Vqwf/vCHgVBCsLFsjX8lAguBEFmpEFpjxowJ\nAmhYTiW1Eu3ZZ59dJWAGaxXjYAkXv8S4JStPkZWJrSguBbJP3BwLbJRvnMI5USM9FblWWe7nWbxw\n4cJVcicndl7ABlme99lE6R8lxf9cIIma55SMfF8SJdlmcNhs0f2RT2pU/Af/W4Qk1mIKppBvNBLW\nuSr/IWQJvMpWgS8SqZSaZs7dfPPNQbVLgtBwo6DvtdZaKwigoqIg8zUqbX3JJZcEacIeffRRdenS\nJdiGjwGs2fFy2NksqY7uL2CiJWxaE2KsDOKpbFzwUyMVCV/UlWjkxsNxPqn0ZT5jKQPnfA7rbUyg\n3RPo1avXxAEDBgyeMmVKD/wB02gTJkxoOf/88+c3NzcTVvxMgX3OoS5FuA/r0nGxOi9bnmiqDiJM\nL7jggsA3j2VaRCXV0shmgN8sPn68tKO/46JAMBD7sTSN1ZDtsBCyLb6EWKgQAWx36qmnBn+iqmaI\nFYRxtoAZ+osquVEqmOXsONsin1cFcymQe7vZPORHUtO4tRRBylyLhOl/fBr+r02TtIWkP0six+/c\n6Kc1suRJjURqvAMCBQkiY77hN4r4xMqaq/IfxUx4v2WrwEfgVGQlZV4edthhQUAWjXRgfBAxRxGd\nffv2DdwGmNeIV7IoEESFUKVRaZD/z4dDpkhlHlI+mn+nOU9qetO8bGIsvSEm91Tkw6i1jsvGBQd6\nlgL48q9EQ6SyZFHIMk+ucZWBcyUQ+BgmUBUEunXr9vz666+//ZtvvokFM5VG1DM+jCk1fAf505BL\npBLIEiVGxxKK2CStD9HV8eXP6CWOAOAFPmvWrECU8vx7+OGHg//G81li8WIplWV/BCtWQtI0ISiw\nrmHtwoJFI9oacYzPJhXREKhEdme2tNwGQiY5uaTEvuq6CfllWkqxkLbWWiTdLenIzI2yVZyK3Dzw\nO50zZ05giY9bj7MdKFflv9Yq8MX74f3LHIvnMyatFJZ5lv8zGymwuB+Yg/k2V5zKl1TydmUTY8mH\nTm+LMoinkrlww2BxILcaPkTkUdtxxx2DBM4sXfFvF110UWBh4CHMQxq/oZ/+9KfBtiwrkCaE/Hmk\nyGBJA2d5Xkrc2CwHYY1lWxzmeXlE5TGjXHBYFqimwU1vS2p68809mUChBGrBkhqPUOf8Eak8lxCS\nrYlUfA4jYcvzD+HKMyu+TMrzL7J4Rb6JiNtbb71Vv/3tb4Nl33jj+UkACyI1c5k/2q7I94ItqSHA\nIvltL+nFHPfHyUcdddQVEydOXFnfuZgiA4Xee22x/ZAhQxbefvvtZzu6v3T6JYux0odQeg9F3kyt\nHbhkLpdffnngYI0fzPPPPx84y/NFRnk8hCnLDFgOSEyMVQAfLpYTcNImqTBfllQQwXmcqFCWwngw\n81LAaTsSrPjPIFDff//9IOiAvG2kzCAnHF+HvCBY4uDlwMO/lFYGzqUMx/uaQHsg0KOxsfGBvffe\ne7s0fVLPPvvspTfeeOPbJfikXhbmtMzbJzVJpBIRzXJ7lEaIj2kEaeTjx/Mn3kemSM0WQIMAZSWI\nikG4AWDtwpJGeiIS22dGVccnRBHPK3xSC+bSHiZhMWMsgl/SYQZuuOGGT7/99tu9og2ZK2QcIKVT\nipbvpHGU/fd+/frNmzVr1m4WqaWjLlmMlT6E0nsow81UMheiPLGmIkaJHuSrf4MNNgjEZ3zJDH8b\n6jfjf8NDnhQW+NGwlEWOtuhBj1UBSytClW3PO++8ldGM+LjiOP6vf/1LI0eODMQv5fJovBQITojS\neJRCuwycSxmO9zWBaicQRPcff/zxG44bN27VdcQiR090/+OPP/7y3Llz8fmrWHR/LpFKJDaBUawc\n8YGMryjislCRyoc4PoH4FuJHT017fFKxpvKxT2ALFlcCrxA3fHgPGDAgTUtqydH9uS5p3E83XnGp\nyCmQc7fWjoOwx7c3nnS+teOX43nf0NAw//XXX+/Bu7BWGxXD+vfvv6C5ubmnRWrpV7lkMVb6EErv\noQw3U8lcyPEWLXNxhiz7k2KFhzBiEpFJi5bMEJFYQbG4Rg1BGj3osZ5GD5cogCGTHOlLCGKIO3DH\n+7cltfS55h5MIE8CNZcnNTPoMx7YwnONFRsa0fZ8lEfPrvgyfrwPBBWiCUGK+IxXW8I3ET96rKW5\nAmaoDpQZsBK/NkW8F0rOk5prbmABZkWN1bKuXVPLRrbK4Vo7TjWI1NVXX33y6NGjD+RdWKsNH+lR\no0ZNmTNnzkEWqaVfZZZ6aqUVMh+SzjkVkUpFDx7EBBQQDUtOO6piREmd4yKVesVf+9rXAmsqD24s\nBVgJsolULBpsT0oYgg9Y1ifFBg90xCz+W1EgAZZa/t6OLKm1NCeT5pl/ryyBNJ8RrY38cEl3drSK\nU6wcUc0IwVls41lGGiAEaGYQS66AmVzHKkKkJg07r/cCbleRexcxBqycUV+epPPk8yRgh3cBJTaJ\nFyCanPycBIbx/Md6TCwB+5ELFSs1LmGffPJJEF2OHy4tV135zOMwFuIV2A8DRltaUiUdssMOO9z4\n/PPP906C3V5/33HHHee+8MILlLWaVMgDxxWn2usVb5tx5/UwyjU0IgH33XffQJiy7E9+O3yqWHbP\nJVKvvvrqwL8KK+u6664rcrSRgxCLBA//uCUVQUr5PB5o+KYSYUs9Yb6UWRZDxPLVzgtjzz331IYb\nbtieAqdKYt8208VHrXYCZRAsWU+5W7duwxsaGkZPnjy5O/doGo2UPIcccsiSxYsXD2tpabkmjT4T\n+qiJe7AM1zyRCz64+OUiVHHzonIUFmaCXKO0XESx8/95vvPMRzxGq2YYE4hZuOKKK4JnO6mY+I1n\nOu5el156aRCnQIaDXHXlo+MQp7DlllsGQpegXdzHaG0sUnFnmz99+vQeGGVqrZEfdeDAgcFSP+dm\nkVprV7h6zifxYZQ0VBIGs7QTNR5U+FYRjR/PkxotmWF14YuavHA0Hmp8NfMFTPQ/llEeUiyD0RC8\nLO3T4ktjfIEjjln6pxGYhTWXh1w7We4vmX3StfHvHY9AGQTLKhBXW221Ceuvv/4JU6dO7RFfui6F\n9i233PL5SSedtHzZsmXflzS1lL4K2Lcm7sEyXPNELhgKNtpoI/FhQWUjRCurXQjXKC0XaQGJOYgy\nFpAHFmMC4jEzZgErKGKUfJ3Rcj0R8SyX56orH1VDItVXlO4rEqd8OLW1SK2vrx93yimn/GT8+PFd\nCpiT7WLTYcOGLbv22mt/uXjx4iARukVqu7hs7XKQiQ+jfM4KXypSRXXu3Hlleqik/VjSwmeJJSEq\nj2ANzZU3LsoLx/Iax4g3MgmwZMaftFoZHvrZhpYK+7TO2f3UBoFyz93GxsZJgwYN2otk/fH8i6XQ\nGzNmTMvYsWNnz58/HxPYS6X0VeC+NXEPluGaJ3JhFY0MLFEebILAyNLSv3//lSKVLAUI2MjIEFXb\nQjxS3SgesxAvoRptR9YYjBbxOvTZ6sqPGDEiCMTNdpx8grfKwC+ahht17dr1jY8++qhzPuMocO62\n2ea46K299trLly5dupmktyxS2+xSdIgDJz6MOgSFjJMs40MrfiSz74iTq8znXMa5u0bv3r0fGDx4\n8Fa33HJLVK+85LM55ZRTltx5551/DyP43y25w8I6qIl7sAzXPJELrl34jpJtBT/T3/zmN4HrFati\nxAwQn8AyPwFOUfAs4jSycGL5jMcs4C6AKxd5syMhis8qJTqT6spfeOGFgatAdByCeVnda2tLKlOR\nxP5Dhw4dcvnll3/RulLYPK2qrc8555zl119//cSmpqYfRQOzJbWqLlFNDSbxYVRTZ5vnyZThoW9L\nap7svVlpBMo0d7fq1avXA6effvo6o0ePTmXpEvFy0EEHLXzuuef+OHfuXCyoy0s786L2ronnXxmu\neSIXBOQee+wR5MhGqFLogLSApM+KCh6w1H/wwQcHLllsQ8pAKnIhHvMRqWyHq0BSXXniGXAJwPWM\n6l2MA3eyahCpktavq6t7e+bMmZ1qwTcVX9QttthiRUtLSz9JKz8qLVKLev54pzwIJD6M8uij5jZJ\n6aFftjQv1Qocn+B4redix4kvG+l6MquH0X9rqXhyHS9Xf8WMj5cywX64qFDpbMiQIUHpwWppKc3d\n+OnsXV9ff9/48eO7n3LKKYW8i3Iieffdd/E/nP/mm2/esWDBgqFtyK4mnn9luOaJXPjIYDmfOIKo\nEURF4Gz8GRCPKcC6iZsXSe25t+MxC/Hl/ngKKbIFJNWV7927t7CmRuVkjz766CDgaubMmerVa2U+\n/ZzTrAz8Mo81Yp999hnxyCOPFJ8Oog1vkvih991336ZHH310rCT+rGyFPBgc3V8lF7OdDCPxYdRO\nziPVYab00CpbwuxUTzbFziKRSlYGCjUU2xCVLPGRPzIeBBevgY4PWr4tV3/57h9tx3ImEcq8mMkk\nQbJ1rEksa1ZLS2nuRqdzbNeuXX999913dyaoJY1GSqLBgwc3f/bZZ5ctWbLkkjT6LKGPmnj+pXzN\nwZk3F0Qn932fPn1WiSlg2Z6ld/LJMkY+8EhHxfOh0ODWfOrKMxZEbaGpwcrAb5Up2bdv3xkXX3zx\nFu05byp5US+44IKZs2fPHpB5ghapJTyFvGurBPJ+GBXCkYdTPLF1IftG27ZmNYt/aSNASEES+SMV\nc6xVbrgvBbdcIfddtsNSepCvTWq05l2SMY3xl9LHs88+G2RKIKiBwgwERmA1IQ8iQRBU2SETw4QJ\nE1ZG6LINqWQIpsDSSWoZHmg0CjdEKYp4YZHNgcwP1DQnPyJ90e68884g5yLWSTI3EESHteXf//63\nzjjjjCDQAssNZXhffvnlIFCimP6yvRwZ9znnnBMsGRIxzLIhVYBYMiQ4iGOSyxeRykswstjwsmUJ\nk/mej8WmlOuS775pvXC7det2fp8+fUZMmTJlte222y7fw7e6HffqYYcdtrylpeXElpaWW1LptLRO\nyvL8K21Ihe+d1jWPHTkVLkTnf/3rXw8qdH31q1/VD3/4w6B6FpbRampl4Jft9LaRNJ2PNPxu21vj\nmRs+BwZKesUitb1dwfY73lQeRpmnjy8QX4ws/4QPgIIJtWY1i4tUEmIvWrQoiCpNq6X40JojqU84\nrqUZYnUe4qiaGg9QCiwQTduvX7+gFjkfGwhTxCfiETFJijGsiUTz7r333kFKMUQswQ9sxwcGKcfw\nRUOMkjO3b9++2nzzzQPBSb8IVD4umCMIWcpMPvDAA0HBBiqYkcqMQg70R4QvQph98H2bNm1a4HtW\naH+8IHNZcM4999wgwwSCFIGKXxvLkrxkSTJOcnCyUSBSo7RLiNjRo0evLNtbDdcyjbnbo0eP6zfd\ndNMjp06d2nP99ddP5bSuvfbaz4cNG7Zo8eLFB0t6LJVOS++kLM+/0odVWA9pXPOMI6bGhQ+4J554\nIrjXCYIij2m1tTLwy3WKx2+wwQbjX3nllV5YnttLw0K9zTbbzHvnnXeGSbop27gLseh4uT/7la8u\nNVDa7CxkPiQdKevDiMhNXr5jx44NRAhLryRjfu+99wILGi9tvoqPO+64YMmTxPqUJcX/CFETRWdS\nPWTMmDFBH9ddd10gfGjZrHW8/D/88MOcVjMEVLaKIq+99lpQtYqiAggikkIjZkgEjcUOwUPLVfkk\nW9qrYoV1Euzw92WS+NNQbSJ16NChWmuttVZaCvHr4hoSAMEyN36i+JXhPE++Q6J6EW7sh6WRFDDM\nE5bpETecH1YTRCb9Yu3GT4w0YvijEUyBxRZBiEXymGOOCRCR65a5ROEH5g4it3v37lq4cGGQ+Jt6\n6XyoFNpfa9XIENSIbYIwOCbiGVGLFRgLQjQGxkfKtauuuioQ5tVmGSrxhdu5d+/e9++00067TJ48\nebXMdG95zu9VNhs1atSyq6666sN58+YdIOm1Yvspw35+L+R4X1bbs6kM135llyXeMwUNrXPnzpcN\nGjTo5BdeeKHdVKLaYYcd5k6bNu265cuXn5vrZAsRJRapNXzTleFmWkWk8nBCgCAKWMLFuoVg5SVN\nvlKsSCznYl1DnCASqQRFHrj9998/ECEEkrAdy7P8ufLKK4OlYSJCSeiczVqHpYyo0GxWM8oG5qoo\nQpQoggeBitWNpNJY5lhqQqxi1W2t8km2/HUpcm5XllSqhkWJt6PbiNQuCLeo8hfJujPdORCyXDs+\nUAiaYGkoWv7mN/6NqjTMm8yGOETclJvcdgAAE3ZJREFUYqlkOxofPORgZO4wt6IAqsgFBEsuIrXQ\n/loTqXzoYJklkpjxMHfuv//+IME490Dkd8p4cQmACy4InHc1tRLm7vq9e/d+8Igjjvjatdde2y2t\nczr22GMXTZ069bW5c+ciUD9Jq1/3U1YCqVlSyzrKlDov4Z4pagSNjY03Dxw48OCnnnqq6gOpdt99\n96bp06ff19TUdFxrJ2uRWtRU+MJONXHTleFmWoULAgSB+c477wTWMPwRd91118Bnj5c4QpA0Iog7\nLF5RpY/48jyO61jGEIgIVixgLM+yVMsycTZrHUvMiNxsVjMsd7kqikSpTE477bRgbFhsd95556D6\nCeOlX3wHc1U+KaNIxSf1Mkl17cUnFUs10bmIQ647yzyIQfIexiN2mSMs2UcBEJkilRyJEdfoNyJw\nWb5/8skng2vDsj770zeiD3HMBwqNa8bfyZvIx1BkfY3PMeZEof21JlL5OEP08nHDsiTznaV+GqKY\n+RyNF+sprg1ltrgX9dQr8hmxXY8ePaaee+65a4wcOZL5WnJbsGABqxgLp02b9lhTU9MhJXfoDipJ\noCbel/kCK/Keybf7rNshVPv373/QQw891Lsal/559u+3335zX3/99clJApUTtEgtaToEO9fETVeG\nmymnSM1EjrjEmsbyOdZTlud5WWO5xIqJlYulXpZiEanxwKm4Bey3v/1tVmvdz372M02aNCmr1Yxj\n56ooEhepcSEVHw/uCrkqn5RRpLa76H4EPq4b5DrkGuLKgYWR64yLRyQ+WxOpfCjgehEFOCF6KXOL\nP2tkjWXJHgspIhQRzDH54CEwCb9QLPlEzxN8td566wWilXnHNqSXwVLL/Cu0v8yUVplzHGsuvrHM\nQyroMGdwG8GSvPbaawtLM+4JCGiW/KNG/sNqEaxFPCMGd+nS5Z4bbrih87HHHlvIuybnU5kPzcGD\nBy94//33b54/fz4fa27ti0BNvC/zRV7EPZNv161ux9L/uuuue+q9997bq5qCqXi+HnroofM++OCD\na1pb4o+fXCEPDi/3Z58WNXHTleFmWoVLVBWEFzNLtgSaEECyySabBKVP4xY0RENUNSRTpCIqWNpn\n/0zBmM1aR99nnXVWVqsZgiZXRZF8RCoWtFyVT8ooUttdnlSEFx8cUe5D/E8ffPDBoORspiU1/hGC\neEOo8W+IVIRd1Ai0wupIi+dM7NmzZ+C7iiUVAfvd7343CLSikV2AOcP8YskdwUhDNCJqozKJxfTX\nWuqbaBUBkbXpppuudFPBt5n98MG+5ZYvBqUzL/NNGp7Kmy2hk0KeEXV1dafW19ePnzRpUjf8cNNo\nzzzzDNdrUXNz86glS5aMS6NP91FxAjXxvsyXWiH3TL59FrDd8TwacW+qhvRUGAZYSZN0Qq4gqWzn\nZpFawBXPsWlN3HRluJlW4RL55hEYgrhgmRwrZuRrmq9IxWUgskgRcEPEND6pLJ1ms9ZxkxJIk81q\nxtJhrooi+YhUBHCuyidlFKlJs7Zq5yRLPbRSlqGYRwi7zJyF+DWTCJ9/zwzMIRcigpg/8RbtE6Wr\nyvZbPv0xl8kGkc3qSWAYQWDtveX7jGhoaBiz5ppr/mTq1Kk9sGyn0aI0YpLIMXRXGn26jzYhULXP\npnLQyPeeKcexwz636du3722DBg1a7xe/+EVjW1Smwn3qzDPPbJo2bdp7s2fPPjpbmqnWzt8itfTZ\nURM3XRlupqxcouCQCDuWTCrrZAbMsAwcVQ3B8kVAFIIUwRCl6aGPuNUsl7UOAcJybDarGS/RXBVF\n8HNlDPikZi73x6sT5ap8QoBYZisD52wzuCbmZOm3ZuV6IFsB4jnb9cYPtaOI1F69ek0cMGDAYAQq\nqcHSaOPHj28ZNWrU/ObmZrL+/zGNPt1HmxHoUM+mCj3v87mYI+rq6i4dPnz4ihEjRnTOZkDJp5NC\ntuF5OHbs2OXjxo3r1NLScl5mJal8+7JIzZdU7u1q4qYrw82UkwvWLiyQLPlnS9OUzyUhOAbLFZYx\nIvTjLZe1rjWrWbEVRQqtfFIGzhap+UwYb1MygYS526OxsfGBvffee7u77rqr+JJgGaM8++yzl954\n441vNzU17S/pzZJPwh20NYGaeF/mC7FCz/t8h7N+Y2PjxYsWLRpy2mmnfT506NAu5bCshoGny66+\n+uovde/efWJTU9MFkt7Nd5CrfOgXsKN9UrPDqombrgw3U01wSbo/Cq18UgbOFqlJF8m/p0Kglbm7\nGQL1+OOP33DcuHGppZg6/PDDmx9//PGX586di0BdkMpJuJO2JtAh3gsR5Ao97wu9phvV19f/uFOn\nTidtvfXWy48++ujerPZssMEGhfazcnsy9rDyeNttt8199dVXO69YseKGxYsX/0rSW0V3Gu5oS2qp\nBB3dn4tgh3kYFVL5pEIPrQ7DvvTb1z3kSyDH3P1OQ0PDlEsuuaTnWWedlUqKqU8++YSMCwtmzJgx\ndd68eUPyHZ+3axcEOtSzqULP+1Iu/CGrr776kEWLFu25xhprrNhpp53qtt9++9WwsJIFheBNXOrw\n8SdAmNVI/PtJFYnF9MUXX1z43HPPtXzyySedunfv/uScOXMmSppUyoBsSU2T3n/6qombrgw3U01w\nSXu6lIFzVktq2uN2fyaQw7BRlspKDQ0NlzU3N4809Zoj0KHeCxV63qc1SQZK2q6xsXH7rl27Dli6\ndOlXlixZ0rhs2bL6lpaWTnV1dSu6dOmyuFu3bk1du3Z9f+nSpTOamppepOiipOlpDaIUkbqYcouS\nVpRrMO2035q46cpwM9UEl7TnZBk4pz1E92cCJmAC5SLQod4Lft6XPo0KWe7H8XV7SR+Uftia6qEm\nbroy3Ew1wSXtmVoGzmkP0f2ZgAmYQLkIdKj3gp/3pU+jQkTq78MUAk+Uftia6qEmbroy3Ew1wSXt\nmVoGzmkP0f2ZgAmYQLkIdKj3gp/3pU+jQkTq6LBWuP2Evsi9Jm66MtxMNcGl9Fvsiz2UgXPaQ3R/\nJmACJlAuAh3qveDnfenTqBCRum1Y6WPj0g9bUz3UxE1XhpupJrikPVPLwDntIbo/EzABEygXgbIE\n2pVrsCn1W4jOSumQtdNNofCmSnpG0vjaQZB4JlVfJ52a41Rkitf5XrFihUaNGqVzzz1XPXr0SDzJ\nIsQTRXivaaVji9QscIrgnHjtvIEJmIAJmIAJ1CKBQkXqFpJelvR9SQ/VIpAs57QodHMYJ2mMpPkZ\n27S5GMsUqSTWpQzopZde+gXh2tr1KlA8kQ8RLvMknSvp11n6bnMu1Tg/C+RcjafgMZmACZiACZhA\nRQgUKlIZ1H6S7pF0fgexqJ4RBox1IieqpAkZYjVRjM2bN08nnniirrzySq299tpBDfr33nsvsHQu\nW7ZMJ5xwgi666CI1NDTo9NNP1z333KNtt91WN998swYMGBBse80114g682eccUYgPEkgf+SRR4q6\n9N/+9rf1u9/9Lvh36sVjVX3ttdeCRLxx62qKIpWuRkj675DJvyVdJOmW2DESuVRkhlfZQSxSq+yC\neDgmYAImYAJVS6AYkcrJYFHFqrilpDslPS1ppqSPajSP6hxJfcKruDRDrM77/PPW3Wyo1LDjjjtq\n7Nix2nXXXbXLLrusrNhA9QZE5YwZM7Tnnntqp5120jnnnBMIVYTrG2+8EVR52GyzzQLRiVjdeOON\nNXDgQI0fPz7o93vf+14wtLgg/cc//hGI1zKKVMofwmW1kMtn4d8vlHR7rRQ5SPvOtUhNm6j7MwET\nMAETqFUCxYrUiAfBVCikHSVtJmkNSanVbq5y6Msk8achSaRyHpdffnlQVuzHP/6xevfuHZwaQhIR\nOnHixECYUj+Xv2MNpc999903sJYiRL/5zW8G2yNUb7/9dj388MPBfyNx+p3vfOcLghRLK4K3QJFK\noQb+tIT/zfz/mX9HuHfPuE64AfCnTz5cqvwapz48i9TUkbrD2iQwWNLxkg5sxfDRWdI3JL0aPotr\nk4TPygQ6MIFSRWpHQVeSJRVIL7/8soYPHx5YRydPnqy11lpLG220UbAsv91222nRokW66667Al/S\nTp3wLJCGDh0aLPfvs88+XxCc/PugQYMCFwJaNkFapEjF15Q/DCD609rf/yxprdgkoCrZo5J+Lul/\nLFJXvT0sUjvKI8PnWSIBROpwSbu2IlK7SGJlqzH0jy/xkN7dBEyg2ghYpCZfEXxSLwvFW1E+qRwC\nv9RtttlGCxYs0I033hj4pu6xxx7B0WfOnKm///3vgb8porWuri6wpB5xxBGBNZVl+7hVlO2wqJ53\n3nnB/m+99Za+9a1vpWFJLWQ+DJX0ixAfgvZ+SeTSfS38N/ukZplbFqnJN5y36LAEjpB0h6RJknpJ\n4qMXS+qakq4KA3YJ3D0O7yZJv5E0RNKfJO0raW6HJecTN4EaJVCIKKlRBImnlUp0P6ITyydBU//8\n5z/15S9/WX369NHOO++sp556SrNnz9Y666yj+++/XwcccID+8Ic/aLfddgt8V5csWfIFkcpv+KG+\n+OKLgX/qyJEjA5eB+NJ+kZbUfOcDorRJUn3of3q5pNczSFqkWqQm3lzewARCAj8InyUHhAaBKZLu\nlvQjSS9Ieg6vqVCoErCJe9mmkn4n6WRJj3vJ33PJBGqPQL6ipPbOPP8zSi1P6r333hv4nhIkVV9f\nHwRK7b///jrrrLOC0Tz44IOBQI3afffdp4MOOmjlcj4+qT179gysrBdeeKEuvvjiYNOjjz5aTz/9\ndGCR7dULA4SULXdqa6dchIWPl8axuNbm6Nci1SI1/7vMW3Z0AlhQcRW6NQRxlKQTJJ3JozEUpRgM\neGc9ElpcEagzJBEbQTo8NxMwgRojYJFa+gVNVYw1Nzdr7ty5gdhMSsL/6aefBq4BjY24ZJXWihCp\nSQdMlUvSwdrL72Xg3F5O3eM0gVwEWJHBvx1rKkFQNHJxUzDkJkmHZQRQXR+KU/Izs/23JJFdxM0E\nTKDGCFikln5Ba0KMlUE81QSX0qfHF3soA+e0h+j+TKDSBMgI85Kkn0j6Y3jwg8O/k+oQ3/etwqwj\nvLNIe4g1lf8iUm1JrfQV8/FMoEIELFJLB10TYqwM4qkmuJQ+PSxS02bo/mqSwAWSWOInnSE+709K\nmhX6m74fpjp8IIz2Jy/3hmFeZgI1D4oFbNYkHJ+UCXRUAhappV/5mhBjFqmlT4R8eigD53wO621M\noNoJ4Ez/sKRvhwOleh1uAAjX70pCoEYNK+tkSeRJJcBql1C0esm/2q+yx2cCBRKwSC0QWJbNLVKz\nM6wJLqVPD1tS02bo/mqaAOmmFoZ/4ifaIIkqKARILcgggLvAkpqm4pMzgQ5KwCK19AtfE2KsDBa+\n1mvFls69Pffg+649Xz2P3QRMwARMoCIE/LIsHbNFaukM3YMJmIAJmIAJmIAJfIGARWrpE8IitXSG\n7sEETMAETMAETMAELFJTngMWqSkDdXcmYAImYAImYAImYEtq6XPAIrV0hu7BBEzABJIIZEv6n7RP\n/HcyCGwi6ZVCdvK2JmACbUfAIrV09happTN0DyZgAiaQRKCLpL9IOkbStKSNs/yOQL1f0gBJDuws\nAqB3MYFKE7BILZ24RWrpDN2DCZhAbRFYT9KZkkjEP17SY5KGhgn695C0mSTSTX0zLHl6oKRJIQL2\nu1rScknrSLpS0j6SrpF0TlhhanZYkeq/JK2Q1C/j70dIuiPs7zRJD4V5WL8eVrAaHu5XW9R9NiZQ\nYwQsUku/oBappTN0DyZgArVFYFNJb0i6WdIlki4KE/VjxdxP0j2hKEWAflnSVEm7SfoktHb+RtJV\n4dI8pVJ/FfbBvoPCPKp3SRoYlkvdXFL09wPCZP/0h8X0D5L2lIRwprLV4UVaYmvrCvlsTKAdELBI\nLf0iWaSWztA9mIAJ1BYBROqj4dL6olCI/i0UqFg9Ea1bhwLzzlCY3h4i2DK0emJdpaIUVlf6WC0s\nf4rIbJJ0r6RtQosox+PviNbfSXpC0o1hf0MkUY3qr2F/3/Byf21NNp9N7RKwSC392lqkls7QPZiA\nCdQWAUTjDZJY2mc5Pgp6wp90I0mnhr91D4UnpU5fDRGw73OSfirpkNAdILOPTJH61dAyu1Potxrv\nLyIbCdlI2NYWcZ+NCdQgAYvU0i9qLTngez6UPh/cgwmYgIQgxAqKVTQSmDMlfU8SfqGRSO0k6XlJ\n+Ig+G4LbIrTC4ps6RhJ/xz81HjiFSKX/rUJr7LaSxoXL+gjcEZKeCvv7f2HZ1A8l3SRpV/ujeoqa\nQPsgYFHSPq6TR2kCJmAC7YlA5JOKRXOypBND4YhP6v5hkFMkFvET3TcMjloWBjm9LGmCpA8k0QeC\n9ChJt4WBUwskvR4GXv1TEn6rs0KrK8FUx0naXlIfSfzOsf4d9r2DpI/bE0yP1QQ6KgGL1I565X3e\nJmACJlA+ApFIjR9hb0mPSzpUEn6iiE+srOQvxS8VoUr7U/j/54aiE5FLwzKKPyv7vSbpCknDwt/I\nHrA03B7XgrtD/1d+/rmkkZL6hi4FZAaILLDlI+CeTcAESiZgkVoyQndgAiZgAiaQQYBo+2tDv9PV\nJc2XtDiBEimpeCdh8Yy3npK6SpqTZX8yAyBOF+b4DcssVteo1UnqHC7/+6KZgAlUOYH/DxLtiQfi\niCMEAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 343,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "   Image('diagrams/skip-gram.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "word1 = Input(shape=(1,), dtype='int64', name='word1')\n",
    "word2 = Input(shape=(1,), dtype='int64', name='word2')\n",
    "\n",
    "shared_embedding = Embedding(\n",
    "    input_dim=VOCAB_SIZE+1, \n",
    "    output_dim=DENSEVEC_DIM, \n",
    "    input_length=1, \n",
    "    embeddings_constraint = unit_norm(),\n",
    "    name='shared_embedding')\n",
    "\n",
    "embedded_w1 = shared_embedding(word1)\n",
    "embedded_w2 = shared_embedding(word2)\n",
    "\n",
    "w1 = Flatten()(embedded_w1)\n",
    "w2 = Flatten()(embedded_w2)\n",
    "\n",
    "dotted = Dot(axes=1, name='dot_product')([w1, w2])\n",
    "\n",
    "prediction = Dense(1, activation='sigmoid', name='output_layer')(dotted)\n",
    "\n",
    "sg_model = Model(inputs=[word1, word2], outputs=prediction)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### fastText"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "ft_model = Sequential()\n",
    "\n",
    "ft_model.add(Embedding(\n",
    "    input_dim = MAX_FEATURES,\n",
    "    output_dim = EMBEDDING_DIMS,\n",
    "    input_length= MAXLEN))\n",
    "\n",
    "ft_model.add(GlobalAveragePooling1D())\n",
    "\n",
    "ft_model.add(Dense(1, activation='sigmoid'))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Models and Training data construction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first step for **CBOW** and **skip-gram**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our training corpus is a collection of sentences, Tweets, emails, comments, or even longer documents.  It is something composed of words.  Each word takes is turn being the \"target\" word, and we collect the _n_ words behind it and _n_ words which follow it.  This _n_ is referred to as **window size**.  If our example document is the sentence \"I love deep learning\" and the window size is 1, we'd get:\n",
    "  * **I**, love\n",
    "  * I, **love**, deep\n",
    "  * love, **deep**, learning\n",
    "  * deep, **learning**\n",
    "  \n",
    "The target word is bold."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Skip-gram model training data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Skip-gram means form word pairs with a target word and all words in the window.  These become the \"positive\" (1) samples for the skip-gram algorithm.  In our \"I love deep learning\" example we'd get (eliminating repeated pairs):\n",
    "\n",
    "  * (I, love) = 1\n",
    "  * (love, deep) = 1\n",
    "  * (deep, learning) = 1\n",
    "  \n",
    "To create negative samples (0), we pair random vocabulary words with the target word.  Yes, it's possible to unluckily pick a negative sample that usually appears around the target word.\n",
    "\n",
    "For our prediction task, we'll take the dot product of the words in each pair (a small step away from the cosine similarity).  The training will keep tweaking the word vectors to make this product as close to unity as possible for our positive samples, and zero for our negative samples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Happily, Keras include a function for creating skipgrams from text.  It even does the negative sampling for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.preprocessing.sequence import skipgrams\n",
    "from keras.preprocessing.text import Tokenizer, text_to_word_sequence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "text1 = \"I love deep learning.\"\n",
    "text2 = \"Read Douglas Adams as much as possible.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = Tokenizer()\n",
    "tokenizer.fit_on_texts([text1, text2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('love', 2),\n",
       " ('adams', 3),\n",
       " ('i', 4),\n",
       " ('possible', 5),\n",
       " ('deep', 6),\n",
       " ('read', 7),\n",
       " ('as', 1),\n",
       " ('much', 8),\n",
       " ('douglas', 9),\n",
       " ('learning', 10)]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word2id = tokenizer.word_index\n",
    "word2id.items()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note word id's are numbered from 1, not zero"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{1: 'as',\n",
       " 2: 'love',\n",
       " 3: 'adams',\n",
       " 4: 'i',\n",
       " 5: 'possible',\n",
       " 6: 'deep',\n",
       " 7: 'read',\n",
       " 8: 'much',\n",
       " 9: 'douglas',\n",
       " 10: 'learning'}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "id2word = { wordid: word for word, wordid in word2id.items()}\n",
    "id2word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[4, 2, 6, 10]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoded_text = [word2id[word] for word in text_to_word_sequence(text1)]\n",
    "encoded_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([[2, 9],\n",
       "  [6, 9],\n",
       "  [6, 3],\n",
       "  [6, 2],\n",
       "  [2, 6],\n",
       "  [2, 5],\n",
       "  [10, 2],\n",
       "  [4, 6],\n",
       "  [2, 4],\n",
       "  [10, 6],\n",
       "  [6, 10],\n",
       "  [4, 2]],\n",
       " [0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sg = skipgrams(encoded_text, vocabulary_size=len(word2id.keys()), window_size=1)\n",
    "sg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(love,douglas)=0\n",
      "(deep,douglas)=0\n",
      "(deep,adams)=0\n",
      "(deep,love)=1\n",
      "(love,deep)=1\n",
      "(love,possible)=0\n",
      "(learning,love)=0\n",
      "(i,deep)=0\n",
      "(love,i)=1\n",
      "(learning,deep)=1\n",
      "(deep,learning)=1\n",
      "(i,love)=1\n"
     ]
    }
   ],
   "source": [
    "for i in range(len(sg[0])):\n",
    "    print \"({0},{1})={2}\".format(id2word[sg[0][i][0]], id2word[sg[0][i][1]], sg[1][i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Model parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "VOCAB_SIZE = len(word2id.keys())\n",
    "VOCAB_SIZE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "DENSEVEC_DIM = 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Model build"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import keras"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras.layers.embeddings import Embedding\n",
    "from keras.constraints import unit_norm\n",
    "from keras.layers.merge import Dot\n",
    "from keras.layers.core import Activation\n",
    "from keras.layers.core import Flatten\n",
    "\n",
    "from keras.layers import Input\n",
    "from keras.models import Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a dense vector for each word in the pair.  The output of `Embedding` has shape `(batch_size, sequence_length, output_dim)` which in our case is `(batch_size, 1, DENSEVEC_DIM)`.  We'll use `Flatten` to get rid of that pesky middle dimension (1), so going into the dot product we'll have shape `(batch_size, DENSEVEC_DIM)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "metadata": {},
   "outputs": [],
   "source": [
    "word1 = Input(shape=(1,), dtype='int64', name='word1')\n",
    "word2 = Input(shape=(1,), dtype='int64', name='word2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 340,
   "metadata": {},
   "outputs": [],
   "source": [
    "shared_embedding = Embedding(\n",
    "    input_dim=VOCAB_SIZE+1, \n",
    "    output_dim=DENSEVEC_DIM, \n",
    "    input_length=1, \n",
    "    embeddings_constraint = unit_norm(),\n",
    "    name='shared_embedding')\n",
    "\n",
    "embedded_w1 = shared_embedding(word1)\n",
    "embedded_w2 = shared_embedding(word2)\n",
    "\n",
    "w1 = Flatten()(embedded_w1)\n",
    "w2 = Flatten()(embedded_w2)\n",
    "\n",
    "dotted = Dot(axes=1, name='dot_product')([w1, w2])\n",
    "\n",
    "prediction = Dense(1, activation='sigmoid', name='output_layer')(dotted)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 313,
   "metadata": {},
   "outputs": [],
   "source": [
    "sg_model = Model(inputs=[word1, word2], outputs=prediction)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 314,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sg_model.compile(optimizer='adam', loss='mean_squared_error')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point you can check out how the data flows through your compiled model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 315,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<keras.engine.topology.InputLayer at 0x117598d50>,\n",
       " <keras.engine.topology.InputLayer at 0x1175989d0>,\n",
       " <keras.layers.embeddings.Embedding at 0x117f1f510>,\n",
       " <keras.layers.core.Flatten at 0x133787e10>,\n",
       " <keras.layers.core.Flatten at 0x133787e90>,\n",
       " <keras.layers.merge.Dot at 0x129c89910>,\n",
       " <keras.layers.core.Dense at 0x117bb4e90>]"
      ]
     },
     "execution_count": 315,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sg_model.layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 316,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def print_layer(model, num):\n",
    "    print model.layers[num]\n",
    "    print model.layers[num].input_shape\n",
    "    print model.layers[num].output_shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 318,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<keras.layers.core.Dense object at 0x117bb4e90>\n",
      "(None, 1)\n",
      "(None, 1)\n"
     ]
    }
   ],
   "source": [
    "print_layer(sg_model,6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try training it with our toy data set!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 319,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 320,
   "metadata": {},
   "outputs": [],
   "source": [
    "pairs = np.array(sg[0])\n",
    "targets = np.array(sg[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 321,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1])"
      ]
     },
     "execution_count": 321,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "targets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 322,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 2,  9],\n",
       "       [ 6,  9],\n",
       "       [ 6,  3],\n",
       "       [ 6,  2],\n",
       "       [ 2,  6],\n",
       "       [ 2,  5],\n",
       "       [10,  2],\n",
       "       [ 4,  6],\n",
       "       [ 2,  4],\n",
       "       [10,  6],\n",
       "       [ 6, 10],\n",
       "       [ 4,  2]])"
      ]
     },
     "execution_count": 322,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pairs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 323,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 2],\n",
       "       [ 6],\n",
       "       [ 6],\n",
       "       [ 6],\n",
       "       [ 2],\n",
       "       [ 2],\n",
       "       [10],\n",
       "       [ 4],\n",
       "       [ 2],\n",
       "       [10],\n",
       "       [ 6],\n",
       "       [ 4]])"
      ]
     },
     "execution_count": 323,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w1_list = np.reshape(pairs[:, 0], (len(pairs), 1))\n",
    "w1_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 324,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 9],\n",
       "       [ 9],\n",
       "       [ 3],\n",
       "       [ 2],\n",
       "       [ 6],\n",
       "       [ 5],\n",
       "       [ 2],\n",
       "       [ 6],\n",
       "       [ 4],\n",
       "       [ 6],\n",
       "       [10],\n",
       "       [ 2]])"
      ]
     },
     "execution_count": 324,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w2_list = np.reshape(pairs[:, 1], (len(pairs), 1))\n",
    "w2_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 325,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(12, 1)"
      ]
     },
     "execution_count": 325,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w2_list.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 326,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('int64')"
      ]
     },
     "execution_count": 326,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w2_list.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 339,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "12/12 [==============================] - 0s - loss: 0.0329\n",
      "Epoch 2/10\n",
      "12/12 [==============================] - 0s - loss: 0.0325\n",
      "Epoch 3/10\n",
      "12/12 [==============================] - 0s - loss: 0.0320\n",
      "Epoch 4/10\n",
      "12/12 [==============================] - 0s - loss: 0.0316\n",
      "Epoch 5/10\n",
      "12/12 [==============================] - 0s - loss: 0.0311\n",
      "Epoch 6/10\n",
      "12/12 [==============================] - 0s - loss: 0.0307\n",
      "Epoch 7/10\n",
      "12/12 [==============================] - 0s - loss: 0.0303\n",
      "Epoch 8/10\n",
      "12/12 [==============================] - 0s - loss: 0.0299\n",
      "Epoch 9/10\n",
      "12/12 [==============================] - 0s - loss: 0.0295\n",
      "Epoch 10/10\n",
      "12/12 [==============================] - 0s - loss: 0.0291\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x133868b90>"
      ]
     },
     "execution_count": 339,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sg_model.fit(x=[w1_list, w2_list], y=targets,  epochs=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Continuous Bag of Words (CBOW) model "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CBOW means we take _all_ the words in the window and use them to predict the target word.  Note we are trying to predict an actual word (or a probability distribution over words) with CBOW, whereas in skip-gram we are trying to predict a similarity score. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### FastText Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "FastText is creating dense document vectors using the words in the document enhanced with n-grams.  These are embedded, averaged, and fed through a hidden dense layer, with a sigmoid activation.  The prediction task is some binary classification of the documents.  As usual, after training we can extract the dense vectors from the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### FastText Model Data Prep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 291,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_FEATURES = 20000  # number of unique words in the dataset\n",
    "MAXLEN = 400  # max word (feature) length of a review  \n",
    "EMBEDDING_DIMS = 50\n",
    "NGRAM_RANGE = 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some data prep functions lifted from the example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 260,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def create_ngram_set(input_list, ngram_value=2):\n",
    "    \"\"\"\n",
    "    Extract a set of n-grams from a list of integers.\n",
    "    \"\"\"\n",
    "    return set(zip(*[input_list[i:] for i in range(ngram_value)]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 262,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{(1, 2), (2, 3), (3, 4), (4, 5)}"
      ]
     },
     "execution_count": 262,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "create_ngram_set([1, 2, 3, 4, 5], ngram_value=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 263,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{(1, 2, 3), (2, 3, 4), (3, 4, 5)}"
      ]
     },
     "execution_count": 263,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "create_ngram_set([1, 2, 3, 4, 5], ngram_value=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 264,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def add_ngram(sequences, token_indice, ngram_range=2):\n",
    "    \"\"\"\n",
    "    Augment the input list of list (sequences) by appending n-grams values.\n",
    "    \"\"\"\n",
    "    new_sequences = []\n",
    "    for input_list in sequences:\n",
    "        new_list = input_list[:]\n",
    "        for i in range(len(new_list) - ngram_range + 1):\n",
    "            for ngram_value in range(2, ngram_range + 1):\n",
    "                ngram = tuple(new_list[i:i + ngram_value])\n",
    "                if ngram in token_indice:\n",
    "                    new_list.append(token_indice[ngram])\n",
    "        new_sequences.append(new_list)\n",
    "\n",
    "    return new_sequences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 269,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sequences = [[1,2,3,4,5], [6,7,8]]\n",
    "token_indice = {(1,2): 20000, (4,5): 20001, (6,7,8): 20002}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 270,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[1, 2, 3, 4, 5, 20000, 20001], [6, 7, 8]]"
      ]
     },
     "execution_count": 270,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "add_ngram(sequences, token_indice, ngram_range=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 271,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[1, 2, 3, 4, 5, 20000], [6, 7, 8, 20002]]"
      ]
     },
     "execution_count": 271,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "add_ngram(sequences, token_indice, ngram_range=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "load canned training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 274,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras.datasets import imdb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 275,
   "metadata": {},
   "outputs": [],
   "source": [
    "(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=MAX_FEATURES)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 276,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ list([1, 14, 22, 16, 43, 530, 973, 1622, 1385, 65, 458, 4468, 66, 3941, 4, 173, 36, 256, 5, 25, 100, 43, 838, 112, 50, 670, 2, 9, 35, 480, 284, 5, 150, 4, 172, 112, 167, 2, 336, 385, 39, 4, 172, 4536, 1111, 17, 546, 38, 13, 447, 4, 192, 50, 16, 6, 147, 2025, 19, 14, 22, 4, 1920, 4613, 469, 4, 22, 71, 87, 12, 16, 43, 530, 38, 76, 15, 13, 1247, 4, 22, 17, 515, 17, 12, 16, 626, 18, 19193, 5, 62, 386, 12, 8, 316, 8, 106, 5, 4, 2223, 5244, 16, 480, 66, 3785, 33, 4, 130, 12, 16, 38, 619, 5, 25, 124, 51, 36, 135, 48, 25, 1415, 33, 6, 22, 12, 215, 28, 77, 52, 5, 14, 407, 16, 82, 10311, 8, 4, 107, 117, 5952, 15, 256, 4, 2, 7, 3766, 5, 723, 36, 71, 43, 530, 476, 26, 400, 317, 46, 7, 4, 12118, 1029, 13, 104, 88, 4, 381, 15, 297, 98, 32, 2071, 56, 26, 141, 6, 194, 7486, 18, 4, 226, 22, 21, 134, 476, 26, 480, 5, 144, 30, 5535, 18, 51, 36, 28, 224, 92, 25, 104, 4, 226, 65, 16, 38, 1334, 88, 12, 16, 283, 5, 16, 4472, 113, 103, 32, 15, 16, 5345, 19, 178, 32]),\n",
       "       list([1, 194, 1153, 194, 8255, 78, 228, 5, 6, 1463, 4369, 5012, 134, 26, 4, 715, 8, 118, 1634, 14, 394, 20, 13, 119, 954, 189, 102, 5, 207, 110, 3103, 21, 14, 69, 188, 8, 30, 23, 7, 4, 249, 126, 93, 4, 114, 9, 2300, 1523, 5, 647, 4, 116, 9, 35, 8163, 4, 229, 9, 340, 1322, 4, 118, 9, 4, 130, 4901, 19, 4, 1002, 5, 89, 29, 952, 46, 37, 4, 455, 9, 45, 43, 38, 1543, 1905, 398, 4, 1649, 26, 6853, 5, 163, 11, 3215, 10156, 4, 1153, 9, 194, 775, 7, 8255, 11596, 349, 2637, 148, 605, 15358, 8003, 15, 123, 125, 68, 2, 6853, 15, 349, 165, 4362, 98, 5, 4, 228, 9, 43, 2, 1157, 15, 299, 120, 5, 120, 174, 11, 220, 175, 136, 50, 9, 4373, 228, 8255, 5, 2, 656, 245, 2350, 5, 4, 9837, 131, 152, 491, 18, 2, 32, 7464, 1212, 14, 9, 6, 371, 78, 22, 625, 64, 1382, 9, 8, 168, 145, 23, 4, 1690, 15, 16, 4, 1355, 5, 28, 6, 52, 154, 462, 33, 89, 78, 285, 16, 145, 95])], dtype=object)"
      ]
     },
     "execution_count": 276,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train[0:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 277,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 0])"
      ]
     },
     "execution_count": 277,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train[0:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add n-gram features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 281,
   "metadata": {},
   "outputs": [],
   "source": [
    "ngram_set = set()\n",
    "for input_list in x_train:\n",
    "    for i in range(2, NGRAM_RANGE + 1):\n",
    "        set_of_ngram = create_ngram_set(input_list, ngram_value=i)\n",
    "        ngram_set.update(set_of_ngram)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 283,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1185229"
      ]
     },
     "execution_count": 283,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(ngram_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assign id's to the new features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 285,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "start_index = MAX_FEATURES + 1\n",
    "token_indice = {v: k + start_index for k, v in enumerate(ngram_set)}\n",
    "indice_token = {token_indice[k]: k for k in token_indice}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Update MAX_FEATURES"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 292,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1205230"
      ]
     },
     "execution_count": 292,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "MAX_FEATURES = np.max(list(indice_token.keys())) + 1\n",
    "MAX_FEATURES"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add n-grams to the input data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 288,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x_train = add_ngram(x_train, token_indice, NGRAM_RANGE)\n",
    "x_test = add_ngram(x_test, token_indice, NGRAM_RANGE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make all input sequences the same length by padding with zeros"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 293,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras.preprocessing import sequence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 295,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 0, 0, 0, 0, 1, 2, 3, 4, 5],\n",
       "       [0, 0, 0, 0, 0, 0, 0, 6, 7, 8]], dtype=int32)"
      ]
     },
     "execution_count": 295,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sequence.pad_sequences([[1,2,3,4,5], [6,7,8]], maxlen=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 296,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = sequence.pad_sequences(x_train, maxlen=MAXLEN)\n",
    "x_test = sequence.pad_sequences(x_test, maxlen=MAXLEN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 297,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(25000, 400)"
      ]
     },
     "execution_count": 297,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 298,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(25000, 400)"
      ]
     },
     "execution_count": 298,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### FastText Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 299,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers.embeddings import Embedding\n",
    "from keras.layers.pooling import GlobalAveragePooling1D\n",
    "from keras.layers import Dense"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 300,
   "metadata": {},
   "outputs": [],
   "source": [
    "ft_model = Sequential()\n",
    "\n",
    "ft_model.add(Embedding(\n",
    "    input_dim = MAX_FEATURES,\n",
    "    output_dim = EMBEDDING_DIMS,\n",
    "    input_length= MAXLEN))\n",
    "\n",
    "ft_model.add(GlobalAveragePooling1D())\n",
    "\n",
    "ft_model.add(Dense(1, activation='sigmoid'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 301,
   "metadata": {},
   "outputs": [],
   "source": [
    "ft_model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 302,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<keras.layers.embeddings.Embedding at 0x117f12650>,\n",
       " <keras.layers.pooling.GlobalAveragePooling1D at 0x117bc51d0>,\n",
       " <keras.layers.core.Dense at 0x1190c3dd0>]"
      ]
     },
     "execution_count": 302,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ft_model.layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 306,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<keras.layers.embeddings.Embedding object at 0x117f12650>\n",
      "(None, 400)\n",
      "(None, 400, 50)\n"
     ]
    }
   ],
   "source": [
    "print_layer(ft_model, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 307,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<keras.layers.pooling.GlobalAveragePooling1D object at 0x117bc51d0>\n",
      "(None, 400, 50)\n",
      "(None, 50)\n"
     ]
    }
   ],
   "source": [
    "print_layer(ft_model, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 308,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<keras.layers.core.Dense object at 0x1190c3dd0>\n",
      "(None, 50)\n",
      "(None, 1)\n"
     ]
    }
   ],
   "source": [
    "print_layer(ft_model, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 310,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 25000 samples, validate on 25000 samples\n",
      "Epoch 1/3\n",
      "25000/25000 [==============================] - 212s - loss: 0.6614 - acc: 0.7317 - val_loss: 0.6059 - val_acc: 0.8069\n",
      "Epoch 2/3\n",
      "25000/25000 [==============================] - 210s - loss: 0.5023 - acc: 0.8827 - val_loss: 0.4631 - val_acc: 0.8567\n",
      "Epoch 3/3\n",
      "25000/25000 [==============================] - 207s - loss: 0.3363 - acc: 0.9321 - val_loss: 0.3735 - val_acc: 0.8788\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x129d7bbd0>"
      ]
     },
     "execution_count": 310,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ft_model.fit(x_train, y_train, batch_size=100, epochs=3, validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Our own data download and preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll use the [Large Movie Review Dataset v1.0](http://ai.stanford.edu/~amaas/data/sentiment/) for our corpus.  While Keras has its own data samples you can import for modeling (including this one), I think it's very important to get and process your own data.  Otherwise, the results appear to materialize out of thin air and it's more difficult to get on with your own research."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import glob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "datapath = \"/Users/pfigliozzi/aclImdb/train/unsup\"\n",
    "files = glob.glob(datapath+\"/*.txt\")[:1000] #first 1000 (there are 50k)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.concat([pd.read_table(filename, header=None, names=['raw']) for filename in files], ignore_index=True)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x107ee6710>"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAD8CAYAAABthzNFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFNVJREFUeJzt3X3QpXV93/H3xwV5UMqD3KGbXciC3WpJGhe6EhxNSmA0\nPCQBO8aBSeMOpa5tYKKTNHUhmagzZQY7iUQyDZEIEawKCCpbwJoFaVL/EFx0eZayylp2s7AbFRC1\nKOu3f5zfjcf12r3PDXvd59z3vl8zZ87v+l0P5/uDA5/7ejjXlapCkqSdvWTcBUiSJpMBIUnqZEBI\nkjoZEJKkTgaEJKmTASFJ6mRASJI6GRCSpE4GhCSp0z7jLuDFOPzww2vZsmXjLkOS5pW77777H6tq\naqbl5nVALFu2jPXr14+7DEmaV5J8Y5TlPMQkSepkQEiSOhkQkqROvQVEkv2T3JXkniQPJHlf6/9I\nkkeTbGivFa0/SS5LsjHJvUmO76s2SdLM+jxJ/SxwclU9k2Rf4AtJPtvm/WFV3bDT8qcBy9vrl4DL\n27skaQx624OogWfa5L7ttbunE50JXNPW+yJwSJLFfdUnSdq9Xs9BJFmUZAOwDVhXVXe2WRe3w0iX\nJtmv9S0BHhtafXPr23mbq5OsT7J++/btfZYvSXu1XgOiqnZU1QpgKXBCkl8ALgReDbwWOAx49yy3\neUVVrayqlVNTM/7OQ5L0As3JVUxV9SRwB3BqVW1th5GeBf4GOKEttgU4cmi1pa1PkjQGvZ2kTjIF\n/LCqnkxyAPBG4P1JFlfV1iQBzgLub6usBS5Ici2Dk9NPVdXWvuobp2VrbhnL52665IyxfK6k+anP\nq5gWA1cnWcRgT+X6qro5yedbeATYAPyHtvytwOnARuB7wLk91iZJmkFvAVFV9wLHdfSfvIvlCzi/\nr3okSbPjL6klSZ0MCElSJwNCktTJgJAkdTIgJEmdDAhJUicDQpLUyYCQJHUyICRJnQwISVInA0KS\n1MmAkCR1MiAkSZ36vN33RBvXMxkkab5wD0KS1MmAkCR1MiAkSZ0MCElSJwNCktTJgJAkdTIgJEmd\neguIJPsnuSvJPUkeSPK+1n90kjuTbExyXZKXtv792vTGNn9ZX7VJkmbW5x7Es8DJVfUaYAVwapIT\ngfcDl1bVPwO+DZzXlj8P+Hbrv7QtJ0kak94CogaeaZP7tlcBJwM3tP6rgbNa+8w2TZt/SpL0VZ8k\nafd6PQeRZFGSDcA2YB3wNeDJqnquLbIZWNLaS4DHANr8p4BXdGxzdZL1SdZv3769z/Ilaa/Wa0BU\n1Y6qWgEsBU4AXr0HtnlFVa2sqpVTU1MvukZJUrc5uYqpqp4E7gBeBxySZPomgUuBLa29BTgSoM0/\nGPjmXNQnSfppfV7FNJXkkNY+AHgj8BCDoHhLW2wVcFNrr23TtPmfr6rqqz5J0u71ebvvxcDVSRYx\nCKLrq+rmJA8C1yb5L8BXgCvb8lcCH02yEfgWcHaPtUmSZtBbQFTVvcBxHf1fZ3A+Yuf+/wf8Vl/1\nSJJmx19SS5I6GRCSpE4GhCSpkwEhSepkQEiSOhkQkqROBoQkqZMBIUnqZEBIkjoZEJKkTgaEJKmT\nASFJ6mRASJI6GRCSpE4GhCSpkwEhSepkQEiSOhkQkqROBoQkqZMBIUnqZEBIkjr1FhBJjkxyR5IH\nkzyQ5J2t/71JtiTZ0F6nD61zYZKNSR5O8mt91SZJmtk+PW77OeAPqurLSQ4C7k6yrs27tKr+dHjh\nJMcCZwM/D/wscFuSf15VO3qsUZK0C73tQVTV1qr6cmt/B3gIWLKbVc4Erq2qZ6vqUWAjcEJf9UmS\ndm9OzkEkWQYcB9zZui5Icm+Sq5Ic2vqWAI8NrbaZjkBJsjrJ+iTrt2/f3mPVkrR36z0gkrwcuBF4\nV1U9DVwOvBJYAWwF/mw226uqK6pqZVWtnJqa2uP1SpIGeg2IJPsyCIePVdWnAKrqiaraUVU/Av6a\nHx9G2gIcObT60tYnSRqDPq9iCnAl8FBVfWCof/HQYm8G7m/ttcDZSfZLcjSwHLirr/okSbvX51VM\nrwd+B7gvyYbWdxFwTpIVQAGbgHcAVNUDSa4HHmRwBdT5XsEkSePTW0BU1ReAdMy6dTfrXAxc3FdN\nkqTR+UtqSVInA0KS1MmAkCR1MiAkSZ0MCElSJwNCktTJgJAkdTIgJEmdDAhJUicDQpLUyYCQJHUa\nKSCS/Mu+C5EkTZZR9yD+MsldSX43ycG9ViRJmggjBURV/TLw2wwe6HN3ko8neWOvlUmSxmrkcxBV\n9Qjwx8C7gX8NXJbkq0n+TV/FSZLGZ9RzEL+Y5FLgIeBk4Deq6l+09qU91idJGpNRHxj0F8CHgYuq\n6vvTnVX1D0n+uJfKJEljNWpAnAF8f/oRoEleAuxfVd+rqo/2Vp0kaWxGPQdxG3DA0PSBrU+StECN\nGhD7V9Uz0xOtfWA/JUmSJsGoAfHdJMdPTyT5V8D3d7M8SY5MckeSB5M8kOSdrf+wJOuSPNLeD239\nSXJZko1J7h3+PEnS3Bs1IN4FfDLJ/07yBeA64IIZ1nkO+IOqOhY4ETg/ybHAGuD2qloO3N6mAU4D\nlrfXauDyWY1EkrRHjXSSuqq+lOTVwKta18NV9cMZ1tkKbG3t7yR5CFgCnAmc1Ba7GvhfDH5bcSZw\nTVUV8MUkhyRZ3LYjSZpjo17FBPBaYFlb5/gkVNU1o6yYZBlwHHAncMTQ//QfB45o7SXAY0OrbW59\nBoQkjcFIAZHko8ArgQ3AjtZdwIwBkeTlwI3Au6rq6STPz6uqSlKzKTjJagaHoDjqqKNms6okaRZG\n3YNYCRzbDv+MLMm+DMLhY1X1qdb9xPShoySLgW2tfwuDez1NW9r6fkJVXQFcAbBy5cpZ1SNJGt2o\nJ6nvB/7pbDacwa7ClcBDVfWBoVlrgVWtvQq4aaj/be1qphOBpzz/IEnjM+oexOHAg0nuAp6d7qyq\n39zNOq8Hfge4L8mG1ncRcAlwfZLzgG8Ab23zbgVOBzYC3wPOHXUQkqQ9b9SAeO9sN1xVXwCyi9mn\ndCxfwPmz/RxJUj9Gvcz175L8HLC8qm5LciCwqN/SJEnjNOrtvt8O3AB8qHUtAT7TV1GSpPEb9ST1\n+QzOKTwNzz886Gf6KkqSNH6jBsSzVfWD6Ykk+zD4HYQkaYEaNSD+LslFwAHtWdSfBP5Hf2VJksZt\n1IBYA2wH7gPeweCSVJ8kJ0kL2KhXMf0I+Ov2kiTtBUa9F9OjdJxzqKpj9nhFkqSJMJt7MU3bH/gt\n4LA9X44kaVKMdA6iqr459NpSVX8OnNFzbZKkMRr1ENPw4z9fwmCPYjbPkpAkzTOj/k/+z4bazwGb\n+PFN9iRJC9CoVzH9at+FSJImy6iHmH5/d/N3et6DJGkBmM1VTK9l8FAfgN8A7gIe6aMoSdL4jRoQ\nS4Hjq+o7AEneC9xSVf+2r8IkSeM16q02jgB+MDT9g9YnSVqgRt2DuAa4K8mn2/RZwNX9lCRJmgSj\nXsV0cZLPAr/cus6tqq/0V5YkadxGPcQEcCDwdFV9ENic5OieapIkTYBRHzn6HuDdwIWta1/gv/dV\nlCRp/EY9B/Fm4DjgywBV9Q9JDtrdCkmuAn4d2FZVv9D63gu8ncGzJQAuqqpb27wLgfOAHcDvVdXn\nZjcUzWTZmlvG9tmbLvHWXdJ8M+ohph9UVdFu+Z3kZSOs8xHg1I7+S6tqRXtNh8OxwNnAz7d1/jLJ\nohFrkyT1YNSAuD7Jh4BDkrwduI0ZHh5UVX8PfGvE7Z8JXFtVz1bVo8BG4IQR15Uk9WDU233/KXAD\ncCPwKuBPquovXuBnXpDk3iRXJTm09S0BHhtaZnPrkySNyYwBkWRRkjuqal1V/WFV/aeqWvcCP+9y\n4JXACmArP3mX2JEkWZ1kfZL127dvn3kFSdILMmNAVNUO4EdJDn6xH1ZVT1TVjqFnXE8fRtoCHDm0\n6NLW17WNK6pqZVWtnJqaerElSZJ2YdSrmJ4B7kuyDvjudGdV/d5sPizJ4qra2ibfDNzf2muBjyf5\nAPCzwHIGNwOUJI3JqAHxqfYaWZJPACcBhyfZDLwHOCnJCgZXQ20C3gFQVQ8kuR54kMEDic5vey6S\npDHZbUAkOaqq/m9Vzfq+S1V1Tkf3lbtZ/mLg4tl+jiSpHzOdg/jMdCPJjT3XIkmaIDMFRIbax/RZ\niCRpsswUELWLtiRpgZvpJPVrkjzNYE/igNamTVdV/ZNeq5Mkjc1uA6KqvB+SJO2lZvM8CEnSXsSA\nkCR1MiAkSZ0MCElSJwNCktTJgJAkdTIgJEmdDAhJUicDQpLUyYCQJHUyICRJnQwISVInA0KS1MmA\nkCR1MiAkSZ0MCElSp94CIslVSbYluX+o77Ak65I80t4Pbf1JclmSjUnuTXJ8X3VJkkbT5x7ER4BT\nd+pbA9xeVcuB29s0wGnA8vZaDVzeY12SpBH0FhBV9ffAt3bqPhO4urWvBs4a6r+mBr4IHJJkcV+1\nSZJmNtfnII6oqq2t/ThwRGsvAR4bWm5z65MkjcnYTlJXVQE12/WSrE6yPsn67du391CZJAnmPiCe\nmD501N63tf4twJFDyy1tfT+lqq6oqpVVtXJqaqrXYiVpbzbXAbEWWNXaq4Cbhvrf1q5mOhF4auhQ\nlCRpDPbpa8NJPgGcBByeZDPwHuAS4Pok5wHfAN7aFr8VOB3YCHwPOLevuiRJo+ktIKrqnF3MOqVj\n2QLO76sWSdLs+UtqSVInA0KS1MmAkCR1MiAkSZ0MCElSJwNCktSpt8tcpWHL1twyls/ddMkZY/lc\naSFwD0KS1MmAkCR1MiAkSZ0MCElSJwNCktTJgJAkdTIgJEmdDAhJUicDQpLUyYCQJHUyICRJnQwI\nSVInA0KS1MmAkCR1GsvtvpNsAr4D7ACeq6qVSQ4DrgOWAZuAt1bVt8dRnyRpvHsQv1pVK6pqZZte\nA9xeVcuB29u0JGlMJukQ05nA1a19NXDWGGuRpL3euAKigL9NcneS1a3viKra2tqPA0eMpzRJEozv\nkaNvqKotSX4GWJfkq8Mzq6qSVNeKLVBWAxx11FH9VypJe6mx7EFU1Zb2vg34NHAC8ESSxQDtfdsu\n1r2iqlZW1cqpqam5KlmS9jpzHhBJXpbkoOk28CbgfmAtsKottgq4aa5rkyT92DgOMR0BfDrJ9Od/\nvKr+Z5IvAdcnOQ/4BvDWMdQmSWrmPCCq6uvAazr6vwmcMtf1SJK6TdJlrpKkCWJASJI6GRCSpE4G\nhCSpkwEhSepkQEiSOhkQkqRO47oXkzQnlq25ZWyfvemSM8b22dKe4B6EJKmTASFJ6uQhJqkn4zq8\n5aEt7SnuQUiSOrkHIS0w7rloT3EPQpLUyYCQJHUyICRJnQwISVInA0KS1MmAkCR18jJXSXuE971a\neNyDkCR1mrg9iCSnAh8EFgEfrqpLxlySJHVa6HtNE7UHkWQR8N+A04BjgXOSHDveqiRp7zRpexAn\nABur6usASa4FzgQeHGtVkibaOP+SX8gmag8CWAI8NjS9ufVJkubYpO1BzCjJamB1m3wmycMdix0O\n/OPcVdU7xzO5FtJYwPFMsp8YS97/orb1c6MsNGkBsQU4cmh6aet7XlVdAVyxu40kWV9VK/d8eePh\neCbXQhoLOJ5JNo6xTNohpi8By5McneSlwNnA2jHXJEl7pYnag6iq55JcAHyOwWWuV1XVA2MuS5L2\nShMVEABVdStw64vczG4PQc1DjmdyLaSxgOOZZHM+llTVXH+mJGkemLRzEJKkCbHgAiLJqUkeTrIx\nyZpx19MlyVVJtiW5f6jvsCTrkjzS3g9t/UlyWRvPvUmOH1pnVVv+kSSrxjGWVseRSe5I8mCSB5K8\ncz6PKcn+Se5Kck8bz/ta/9FJ7mx1X9cupCDJfm16Y5u/bGhbF7b+h5P82jjG0+pYlOQrSW5u0/N5\nLJuS3JdkQ5L1rW9eftdaHYckuSHJV5M8lOR1EzOeqlowLwYntr8GHAO8FLgHOHbcdXXU+SvA8cD9\nQ33/FVjT2muA97f26cBngQAnAne2/sOAr7f3Q1v70DGNZzFwfGsfBPwfBrdKmZdjanW9vLX3Be5s\ndV4PnN36/wr4j639u8BftfbZwHWtfWz7Du4HHN2+m4vG9O/o94GPAze36fk8lk3A4Tv1zcvvWqvl\nauDft/ZLgUMmZTxz/g+j53/QrwM+NzR9IXDhuOvaRa3L+MmAeBhY3NqLgYdb+0PAOTsvB5wDfGio\n/yeWG/PYbgLeuBDGBBwIfBn4JQY/Utpn5+8ag6vuXtfa+7TlsvP3b3i5OR7DUuB24GTg5lbbvBxL\n++xN/HRAzMvvGnAw8CjtfPCkjWehHWKaz7fqOKKqtrb248ARrb2rMU3kWNshieMY/NU9b8fUDsls\nALYB6xj8xfxkVT3XUdvzdbf5TwGvYHLG8+fAfwZ+1KZfwfwdC0ABf5vk7gzurADz97t2NLAd+Jt2\nCPDDSV7GhIxnoQXEglCDPwHm3eVlSV4O3Ai8q6qeHp4338ZUVTuqagWDv75PAF495pJekCS/Dmyr\nqrvHXcse9IaqOp7BXZ/PT/IrwzPn2XdtHwaHmy+vquOA7zI4pPS8cY5noQXEjLfqmGBPJFkM0N63\ntf5djWmixppkXwbh8LGq+lTrntdjAqiqJ4E7GByGOSTJ9G+Hhmt7vu42/2Dgm0zGeF4P/GaSTcC1\nDA4zfZD5ORYAqmpLe98GfJpBgM/X79pmYHNV3dmmb2AQGBMxnoUWEPP5Vh1rgekrD1YxOI4/3f+2\ndvXCicBTbdfzc8CbkhzarnB4U+ubc0kCXAk8VFUfGJo1L8eUZCrJIa19AIPzKQ8xCIq3tMV2Hs/0\nON8CfL791bcWOLtdGXQ0sBy4a25GMVBVF1bV0qpaxuC/h89X1W8zD8cCkORlSQ6abjP4jtzPPP2u\nVdXjwGNJXtW6TmHweIPJGM9cn5SZg5M+pzO4iuZrwB+Nu55d1PgJYCvwQwZ/QZzH4Djv7cAjwG3A\nYW3ZMHiI0teA+4CVQ9v5d8DG9jp3jON5A4Nd4HuBDe11+nwdE/CLwFfaeO4H/qT1H8Pgf4obgU8C\n+7X+/dv0xjb/mKFt/VEb58PAaWP+3p3Ej69impdjaXXf014PTP83Pl+/a62OFcD69n37DIOrkCZi\nPP6SWpLUaaEdYpIk7SEGhCSpkwEhSepkQEiSOhkQkqROBoQkqZMBIUnqZEBIkjr9f7GkbgrbMHQI\nAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x105edc890>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df.raw.map(lambda x: len(x)).plot.hist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "100.0"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "50000. * 2000. / 10**6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}