lectures/16_natural-language-processing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![](img/330-banner.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "# Lecture 16: Introduction to natural language processing \n",
    "\n",
    "UBC 2022 Summer\n",
    "\n",
    "Instructor: Mehrdad Oveisi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import re\n",
    "import string\n",
    "import sys\n",
    "import time\n",
    "from collections import Counter, defaultdict\n",
    "\n",
    "import IPython\n",
    "import nltk\n",
    "import numpy as np\n",
    "import numpy.random as npr\n",
    "import pandas as pd\n",
    "from IPython.display import HTML\n",
    "from ipywidgets import interactive\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.tokenize import sent_tokenize, word_tokenize\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.pipeline import make_pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Learning objectives\n",
    "\n",
    "- Broadly explain what is natural language processing (NLP). \n",
    "- Name some common NLP applications. \n",
    "- Explain the general idea of a vector space model.\n",
    "- Explain the difference between different word representations: term-term co-occurrence matrix representation and Word2Vec representation.\n",
    "- Describe the reasons and benefits of using pre-trained embeddings. \n",
    "- Load and use pre-trained word embeddings to find word similarities and analogies. \n",
    "- Demonstrate biases in embeddings and learn to watch out for such biases in pre-trained embeddings.\n",
    "- Use word embeddings in text classification and document clustering using `spaCy`.\n",
    "- Explain the general idea of topic modeling. \n",
    "- Describe the input and output of topic modeling. \n",
    "- Carry out basic text preprocessing using `spaCy`.   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## What is Natural Language Processing (NLP)?\n",
    "\n",
    "- What should a search engine return when asked the following question? \n",
    "\n",
    "![](img/lexical_ambiguity.png)\n",
    "\n",
    "<!-- <center> -->\n",
    "<!-- <img src=\"img/lexical_ambiguity.png\" width=\"1000\" height=\"1000\"> -->\n",
    "<!-- </center> -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### What is Natural Language Processing (NLP)?\n",
    "#### How often do you search everyday? \n",
    "\n",
    "![](img/Google_search.png)\n",
    "\n",
    "<!-- <center> -->\n",
    "<!-- <img src=\"img/Google_search.png\" width=\"900\" height=\"900\"> -->\n",
    "<!-- </center> -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### What is Natural Language Processing (NLP)?\n",
    "\n",
    "![](img/WhatisNLP.png)\n",
    "\n",
    "<!-- <center> -->\n",
    "<!-- <img src=\"img/WhatisNLP.png\" width=\"800\" height=\"800\"> -->\n",
    "<!-- </center> -->    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Everyday NLP applications\n",
    "\n",
    "![](img/annotation-image.png)\n",
    "<!-- <center> -->\n",
    "<!-- <img src=\"img/annotation-image.png\" height=\"1200\" width=\"1200\"> -->\n",
    "<!-- </center> -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### NLP in news \n",
    "\n",
    "Often you'll NLP in news. Some examples: \n",
    "- [How suicide prevention is getting a boost from artificial intelligence](https://abcnews.go.com/GMA/Wellness/suicide-prevention-boost-artificial-intelligence-exclusive/story?id=76541481)\n",
    "- [Meet GPT-3. It Has Learned to Code (and Blog and Argue).](https://www.nytimes.com/2020/11/24/science/artificial-intelligence-ai-gpt3.html)\n",
    "- [How Do You Know a Human Wrote This?](https://www.nytimes.com/2020/07/29/opinion/gpt-3-ai-automation.html)\n",
    "- ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Why is NLP hard?\n",
    "\n",
    "- Language is complex and subtle. \n",
    "- Language is ambiguous at different levels. \n",
    "- Language understanding involves common-sense knowledge and real-world reasoning.\n",
    "- All the problems related to representation and reasoning in artificial intelligence arise in this domain. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Example: Lexical ambiguity\n",
    "\n",
    "<br><br>\n",
    "\n",
    "![](img/lexical_ambiguity.png)\n",
    "\n",
    "<!-- <img src=\"img/lexical_ambiguity.png\" width=\"1000\" height=\"1000\"> -->\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Example: Referential ambiguity\n",
    "<br><br>\n",
    "\n",
    "<!-- <img src=\"img/referential_ambiguity.png\" width=\"1000\" height=\"1000\"> -->\n",
    "\n",
    "![](img/referential_ambiguity.png)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### [Ambiguous news headlines](http://www.fun-with-words.com/ambiguous_headlines.html)\n",
    "\n",
    "<blockquote>\n",
    "KICKING BABY CONSIDERED TO BE HEALTHY    \n",
    "</blockquote> \n",
    "\n",
    "- **kicking** is used as an adjective or a verb?\n",
    "\n",
    "<blockquote>\n",
    "MILK DRINKERS ARE TURNING TO POWDER\n",
    "</blockquote>\n",
    "\n",
    "- **turning** means becoming or take up?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Overall goal\n",
    "\n",
    "- Give you a quick introduction to this important field in artificial intelligence which extensively uses machine learning.\n",
    "\n",
    "![](img/NLP_in_industry.png)\n",
    "\n",
    "<!-- <center> -->\n",
    "<!-- <img src=\"img/NLP_in_industry.png\" width=\"900\" height=\"800\"> -->\n",
    "<!-- </center> -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Today's plan\n",
    "\n",
    "- Word embeddings\n",
    "- Topic modeling \n",
    "- Basic text preprocessing "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Word Embeddings "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The idea is to represent word meaning so that similar words are close together. \n",
    "\n",
    "![](img/t-SNE_word_embeddings.png)\n",
    "\n",
    "<!-- <center> -->\n",
    "<!-- <img src=\"img/t-SNE_word_embeddings.png\" width=\"900\" height=\"900\"> -->\n",
    "<!-- </center> -->    \n",
    "\n",
    "(Attribution: Jurafsky and Martin 3rd edition)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Why do we care about word representation?\n",
    "\n",
    "- So far we have been talking about sentence or document representation. \n",
    "- Now we are going one step back and talking about **word representation**. \n",
    "- Although word representation cannot be directly used in text classification tasks such as sentiment analysis using tradition ML models, it's good to know about **word embeddings** because they are so widely used. \n",
    "- They are quite useful in more advanced machine learning models such as recurrent neural networks. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Word meaning \n",
    "\n",
    "- A favourite topic of philosophers for centuries. \n",
    "- An example from legal domain: [Are hockey gloves gloves or \"articles of plastics\"?](https://www.scc-csc.ca/case-dossier/info/sum-som-eng.aspx?cas=36258)\n",
    "\n",
    "<blockquote>\n",
    "Canada (A.G.) v. Igloo Vikski Inc. was a tariff code case that made its way to the SCC (Supreme Court of Canada). The case disputed the definition of hockey gloves as either gloves or as \"articles of plastics.\"\n",
    "</blockquote>\n",
    "\n",
    "![](img/hockey_gloves_case.png)\n",
    "\n",
    "<!-- <center> -->\n",
    "<!-- <img src=\"img/hockey_gloves_case.png\" width=\"800\" height=\"800\"> -->\n",
    "<!-- </center> -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Word meaning: ML and NLP view\n",
    "- Modeling word meaning that allows us to \n",
    "    * draw useful inferences to solve meaning-related problems \n",
    "    * find relationship between words, \n",
    "        * E.g., which words are similar, which ones have positive or negative connotations\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Word representations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### How do we represent words?  \n",
    "\n",
    "- Suppose you are building a question answering system and you are given the following question and three candidate answers. \n",
    "- What kind of relationship between words would we like our representation to capture in order to arrive at the correct answer?  \n",
    "    \n",
    "<blockquote>       \n",
    "<p style=\"font-size:30px\"><b>Question:</b> How <b>tall</b> is Machu Picchu?</p>\n",
    "    <p style=\"font-size:30px\"><b>Candidate 1:</b> Machu Picchu is 13.164 degrees south of the equator.</p>    \n",
    "<p style=\"font-size:30px\"><b>Candidate 2:</b> The official height of Machu Picchu is 2,430 m.</p>\n",
    "<p style=\"font-size:30px\"><b>Candidate 3:</b> Machu Picchu is 80 kilometres (50 miles) northwest of Cusco.</p>    \n",
    "</blockquote> \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Need a representation that captures *relationships between words*.\n",
    "\n",
    "- We will be looking at two such representations.  \n",
    "    1. Sparse representation with **term-term co-occurrence matrix**\n",
    "    2. Dense representation with **Word2Vec**\n",
    "- Both are based on two ideas: **distributional hypothesis** and **vector space model**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Distributional hypothesis\n",
    "\n",
    "<blockquote> \n",
    "    <p>You shall know a word by the company it keeps.</p>\n",
    "    <footer>Firth, 1957</footer>\n",
    "</blockquote>\n",
    "\n",
    "<blockquote>\n",
    "    <p>If A and B have almost identical environments we say that they are synonyms.</p>\n",
    "    <footer>Harris, 1954</footer>\n",
    "</blockquote>\n",
    "\n",
    "Example: \n",
    "\n",
    "- Her **child** loves to play in the playground. \n",
    "- Her **kid** loves to play in the playground. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Vector space model\n",
    "\n",
    "- Model the meaning of a word by placing it into a vector space.  \n",
    "- A standard way to represent meaning in NLP\n",
    "- The idea is to create **embeddings of words** so that distances among words in the vector space indicate the relationship between them. \n",
    "\n",
    "![](img/t-SNE_word_embeddings.png)\n",
    "\n",
    "<!-- <center> -->\n",
    "<!-- <img src=\"img/t-SNE_word_embeddings.png\" width=\"900\" height=\"900\"> -->\n",
    "<!-- </center> -->    \n",
    "\n",
    "(Attribution: Jurafsky and Martin 3rd edition)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Term-term co-occurrence matrix\n",
    "\n",
    "- So far we have been talking about documents and we created document-term co-occurrence matrix (e.g., bag-of-words representation of text). \n",
    "- We can also do this with words. The idea is to go through a corpus of text, keeping a count of all of the words that appear in context of each word (within a window).\n",
    "\n",
    "- An example: \n",
    "\n",
    "![](img/term-term_comat.png)\n",
    "\n",
    "<!-- <center> -->\n",
    "<!-- <img src=\"img/term-term_comat.png\" width=\"600\" height=\"600\"> -->\n",
    "<!-- </center> -->\n",
    "(Credit: Jurafsky and Martin 3rd edition)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Visualizing word vectors and similarity \n",
    "\n",
    "![](img/word_vectors_and_angles.png)\n",
    "\n",
    "<!-- <center> -->\n",
    "<!-- <img src=\"img/word_vectors_and_angles.png\" width=\"800\" height=\"800\"> -->\n",
    "<!-- </center> -->\n",
    "    \n",
    "(Credit: Jurafsky and Martin 3rd edition)\n",
    "\n",
    "- The similarity is calculated using dot products between word vectors.\n",
    "    - Example: $\\vec{\\text{digital}}.\\vec{\\text{information}} = 0 \\times 1 + 1\\times 6 = 6$\n",
    "    - Higher the dot product more similar the words."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Visualizing word vectors and similarity\n",
    "\n",
    "![](img/word_vectors_and_angles.png)\n",
    "\n",
    "<!-- <img src=\"img/word_vectors_and_angles.png\" width=\"600\" height=\"600\"> -->\n",
    "\n",
    "(Credit: Jurafsky and Martin 3rd edition)\n",
    "\n",
    "- The similarity is calculated using dot products between word vectors.\n",
    "    - Example: $\\vec{\\text{digital}}.\\vec{\\text{information}} = 0 \\times 1 + 1\\times 6 = 6$\n",
    "    - Higher the dot product more similar the words.\n",
    "\n",
    "- We can also calculate a normalized version of dot products. \n",
    "    $$similarity_{cosine}(w_1,w_2) = \\frac{w_1.w_2}{\\left\\lVert w_1\\right\\rVert_2 \\left\\lVert w_2\\right\\rVert_2}$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Let's build term-term co-occurrence matrix for our text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You need to run the following line once\n",
    "# nltk.download('stopwords')  # You need to run this at least once"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tall</th>\n",
       "      <th>machu</th>\n",
       "      <th>picchu</th>\n",
       "      <th>13.164</th>\n",
       "      <th>degrees</th>\n",
       "      <th>south</th>\n",
       "      <th>equator</th>\n",
       "      <th>official</th>\n",
       "      <th>height</th>\n",
       "      <th>2,430</th>\n",
       "      <th>...</th>\n",
       "      <th>mean</th>\n",
       "      <th>sea</th>\n",
       "      <th>level</th>\n",
       "      <th>1,000</th>\n",
       "      <th>3,300</th>\n",
       "      <th>ft</th>\n",
       "      <th>lower</th>\n",
       "      <th>elevation</th>\n",
       "      <th>3,400</th>\n",
       "      <th>11,200</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>tall</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>machu</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>picchu</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13.164</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>degrees</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 32 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         tall  machu  picchu  13.164  degrees  south  equator  official  \\\n",
       "tall        0      1       1       0        0      0        0         0   \n",
       "machu       1      0       5       1        1      0        0         1   \n",
       "picchu      1      5       0       1        1      1        0         1   \n",
       "13.164      0      1       1       0        1      1        1         0   \n",
       "degrees     0      1       1       1        0      1        1         0   \n",
       "\n",
       "         height  2,430  ...  mean  sea  level  1,000  3,300  ft  lower  \\\n",
       "tall          0      0  ...     0    0      0      0      0   0      0   \n",
       "machu         1      2  ...     0    0      0      0      0   0      0   \n",
       "picchu        1      2  ...     0    0      0      0      0   0      0   \n",
       "13.164        0      0  ...     0    0      0      0      0   0      0   \n",
       "degrees       0      0  ...     0    0      0      0      0   0      0   \n",
       "\n",
       "         elevation  3,400  11,200  \n",
       "tall             0      0       0  \n",
       "machu            0      0       0  \n",
       "picchu           0      0       0  \n",
       "13.164           0      0       0  \n",
       "degrees          0      0       0  \n",
       "\n",
       "[5 rows x 32 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sys.path.append(\"code/.\")\n",
    "\n",
    "from comat import CooccurrenceMatrix\n",
    "from preprocessing import MyPreprocessor\n",
    "\n",
    "corpus = [\n",
    "    \"How tall is Machu Picchu?\",\n",
    "    \"Machu Picchu is 13.164 degrees south of the equator.\",\n",
    "    \"The official height of Machu Picchu is 2,430 m.\",\n",
    "    \"Machu Picchu is 80 kilometres (50 miles) northwest of Cusco.\",\n",
    "    \"It is 80 kilometres (50 miles) northwest of Cusco, on the crest of the mountain Machu Picchu, located about 2,430 metres (7,970 feet) above mean sea level, over 1,000 metres (3,300 ft) lower than Cusco, which has an elevation of 3,400 metres (11,200 ft).\",\n",
    "]\n",
    "pp = MyPreprocessor()\n",
    "pp_corpus = pp.preprocess_corpus(corpus)\n",
    "cm = CooccurrenceMatrix(pp_corpus)\n",
    "vocab, comat = cm.fit_transform()\n",
    "words = [\n",
    "    key for key, value in sorted(vocab.items(), key=lambda item: (item[1], item[0]))\n",
    "]\n",
    "df = pd.DataFrame(comat.todense(), columns=words, index=words, dtype=np.int8)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The dot product between tall and height is 2.00 and cosine similarity is 0.71\n",
      "The dot product between tall and official is 2.00 and cosine similarity is 0.82\n"
     ]
    }
   ],
   "source": [
    "import warnings\n",
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "\n",
    "\n",
    "def similarity(word1, word2):\n",
    "    \"\"\"\n",
    "    Returns similarity score between word1 and word2\n",
    "    Arguments\n",
    "    ---------\n",
    "    word1 -- (str)\n",
    "        The first word\n",
    "    word2 -- (str)\n",
    "        The second word\n",
    "\n",
    "    Returns\n",
    "    --------\n",
    "    None. Prints the similarity score between word1 and word2.\n",
    "    \"\"\"\n",
    "    vec1 = cm.get_word_vector(word1).todense().flatten()\n",
    "    vec2 = cm.get_word_vector(word2).todense().flatten()\n",
    "    v1 = np.squeeze(np.asarray(vec1))\n",
    "    v2 = np.squeeze(np.asarray(vec2))\n",
    "    warnings.simplefilter(action=\"ignore\", category=FutureWarning)\n",
    "    print(\n",
    "        \"The dot product between %s and %s is %0.2f and cosine similarity is %0.2f\"\n",
    "        % (word1, word2, np.dot(v1, v2), cosine_similarity(vec1, vec2))\n",
    "    )\n",
    "    warnings.simplefilter(action=\"default\", category=FutureWarning)\n",
    "\n",
    "\n",
    "similarity(\"tall\", \"height\")\n",
    "similarity(\"tall\", \"official\")\n",
    "\n",
    "### Not very reliable similarity scores because we used only 4 sentences."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- We are able to capture some similarities between words now. \n",
    "- That said similarities do not make much sense in the toy example above because we're using a tiny corpus. \n",
    "- To find meaningful patterns of similarities between words, we need a large corpus. \n",
    "- Let's try a bit larger corpus and check whether the similarities make sense. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of sentences in the corpus:  129\n"
     ]
    }
   ],
   "source": [
    "import wikipedia\n",
    "from nltk.tokenize import sent_tokenize, word_tokenize\n",
    "\n",
    "corpus = []\n",
    "\n",
    "queries = [\"Machu Picchu\", \"human stature\", \"mountain\"]\n",
    "\n",
    "for i in range(len(queries)):\n",
    "    sents = sent_tokenize(wikipedia.page(queries[i]).content)\n",
    "    corpus.extend(sents)\n",
    "print(\"Number of sentences in the corpus: \", len(sents))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>machu</th>\n",
       "      <th>picchu</th>\n",
       "      <th>15th-century</th>\n",
       "      <th>inca</th>\n",
       "      <th>citadel</th>\n",
       "      <th>located</th>\n",
       "      <th>eastern</th>\n",
       "      <th>cordillera</th>\n",
       "      <th>southern</th>\n",
       "      <th>peru</th>\n",
       "      <th>...</th>\n",
       "      <th>contender</th>\n",
       "      <th>6,600</th>\n",
       "      <th>ranges</th>\n",
       "      <th>lists</th>\n",
       "      <th>accessible</th>\n",
       "      <th>encyclopædia</th>\n",
       "      <th>britannica</th>\n",
       "      <th>11th</th>\n",
       "      <th>quotations</th>\n",
       "      <th>wikiquote</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>machu</th>\n",
       "      <td>0</td>\n",
       "      <td>93</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>picchu</th>\n",
       "      <td>93</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15th-century</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>inca</th>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>citadel</th>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 3622 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              machu  picchu  15th-century  inca  citadel  located  eastern  \\\n",
       "machu             0      93             1     6        2        2        1   \n",
       "picchu           93       0             1     7        4        1        1   \n",
       "15th-century      1       1             0     1        1        1        0   \n",
       "inca              6       7             1     0        1        1        1   \n",
       "citadel           2       4             1     1        0        1        1   \n",
       "\n",
       "              cordillera  southern  peru  ...  contender  6,600  ranges  \\\n",
       "machu                  0         1     5  ...          0      0       0   \n",
       "picchu                 0         1     4  ...          0      0       0   \n",
       "15th-century           0         0     0  ...          0      0       0   \n",
       "inca                   0         0     2  ...          0      0       0   \n",
       "citadel                1         0     0  ...          0      0       0   \n",
       "\n",
       "              lists  accessible  encyclopædia  britannica  11th  quotations  \\\n",
       "machu             0           0             0           0     0           0   \n",
       "picchu            0           0             0           0     0           0   \n",
       "15th-century      0           0             0           0     0           0   \n",
       "inca              0           0             0           0     0           0   \n",
       "citadel           0           0             0           0     0           0   \n",
       "\n",
       "              wikiquote  \n",
       "machu                 0  \n",
       "picchu                0  \n",
       "15th-century          0  \n",
       "inca                  0  \n",
       "citadel               0  \n",
       "\n",
       "[5 rows x 3622 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pp = MyPreprocessor()\n",
    "pp_corpus = pp.preprocess_corpus(corpus)\n",
    "cm = CooccurrenceMatrix(pp_corpus)\n",
    "vocab, comat = cm.fit_transform()\n",
    "words = [\n",
    "    key for key, value in sorted(vocab.items(), key=lambda item: (item[1], item[0]))\n",
    "]\n",
    "df = pd.DataFrame(comat.todense(), columns=words, index=words, dtype=np.int8)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The dot product between tall and height is 28.00 and cosine similarity is 0.10\n",
      "The dot product between tall and official is 0.00 and cosine similarity is 0.00\n"
     ]
    }
   ],
   "source": [
    "similarity(\"tall\", \"height\")\n",
    "similarity(\"tall\", \"official\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Sparse vs. dense word vectors\n",
    "\n",
    "- Term-term co-occurrence matrices are long and sparse. \n",
    "    - length |V| is usually large (e.g., > 50,000) \n",
    "    - most elements are zero\n",
    "- That is OK because there are efficient ways to deal with sparse matrices.\n",
    "\n",
    "\n",
    "### Alternative \n",
    "- Learn short (~100 to 1000 dimensions) and dense vectors. \n",
    "- Short vectors may be easier to train with ML models (less weights to train).\n",
    "- They may generalize better.\n",
    "- In practice they work much better! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Word2Vec \n",
    "\n",
    "- A family of algorithms to create dense word embeddings\n",
    "\n",
    "![](img/word2vec.png)\n",
    "\n",
    "<!-- <img src=\"img/word2vec.png\" width=\"1000\" height=\"1000\"> -->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Success of Word2Vec\n",
    "\n",
    "- Able to capture complex relationships between words.\n",
    "- Example: What is the word that is similar to **WOMAN** in the same sense as **KING** is similar to **MAN**?\n",
    "- Perform a simple algebraic operations with the vector representation of words.\n",
    "    $\\vec{X} = \\vec{\\text{KING}} − \\vec{\\text{MAN}} + \\vec{\\text{WOMAN}}$\n",
    "- Search in the vector space for the word closest to $\\vec{X}$ measured by cosine distance.\n",
    "\n",
    "![](img/word_analogies1.png)\n",
    "\n",
    "<!-- <center> -->\n",
    "<!-- <img src=\"img/word_analogies1.png\" width=\"500\" height=\"500\"> -->\n",
    "<!-- </center> -->\n",
    "    \n",
    "(Credit: Mikolov et al. 2013)    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- We can create a dense representation with a library called `gensim`. \n",
    "\n",
    "`conda install -n cpsc330 -c anaconda gensim`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim.models import Word2Vec\n",
    "\n",
    "sentences = [[\"cat\", \"say\", \"meow\"], [\"dog\", \"say\", \"woof\"]]\n",
    "model = Word2Vec(sentences, min_count=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the word vector of the word _cat_. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.00713902,  0.00124103, -0.00717672, -0.00224462,  0.0037193 ,\n",
       "        0.00583312,  0.00119818,  0.00210273, -0.00411039,  0.00722533,\n",
       "       -0.00630704,  0.00464721, -0.00821997,  0.00203647, -0.00497705,\n",
       "       -0.00424769, -0.00310899,  0.00565521,  0.0057984 , -0.00497465,\n",
       "        0.00077333, -0.00849578,  0.00780981,  0.00925729, -0.00274233,\n",
       "        0.00080022,  0.00074665,  0.00547788, -0.00860608,  0.00058445,\n",
       "        0.00686942,  0.00223159,  0.00112468, -0.00932216,  0.00848237,\n",
       "       -0.00626413, -0.00299237,  0.00349379, -0.00077263,  0.00141129,\n",
       "        0.00178199, -0.0068289 , -0.00972481,  0.00904058,  0.00619805,\n",
       "       -0.00691293,  0.00340348,  0.00020606,  0.00475374, -0.00711994,\n",
       "        0.00402695,  0.00434743,  0.00995737, -0.00447374, -0.00138927,\n",
       "       -0.00731732, -0.00969783, -0.00908026, -0.00102276, -0.00650329,\n",
       "        0.00484973, -0.00616403,  0.00251919,  0.00073944, -0.00339216,\n",
       "       -0.00097922,  0.00997912,  0.00914589, -0.00446183,  0.00908303,\n",
       "       -0.00564176,  0.00593092, -0.00309722,  0.00343175,  0.00301723,\n",
       "        0.00690046, -0.00237388,  0.00877504,  0.00758943, -0.00954765,\n",
       "       -0.00800821, -0.0076379 ,  0.00292326, -0.00279472, -0.00692952,\n",
       "       -0.00812826,  0.00830918,  0.00199049, -0.00932802, -0.00479272,\n",
       "        0.00313674, -0.00471321,  0.00528084, -0.00423344,  0.00264179,\n",
       "       -0.00804569,  0.00620989,  0.00481889,  0.00078719,  0.00301345],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv[\"cat\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "What's the most similar word to the word _cat_? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('dog', 0.17018885910511017),\n",
       " ('woof', 0.004503009375184774),\n",
       " ('say', -0.027750369161367416),\n",
       " ('meow', -0.04461709037423134)]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv.most_similar(\"cat\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is good. But if you want good and meaningful representations of words you need to train models on a large corpus such as the whole Wikipedia, which is computationally intensive. \n",
    "\n",
    "So instead of training our own models, we use the **pre-trained embeddings**. These are the word embeddings people have trained embeddings on huge corpora and made them available for us to use. \n",
    "\n",
    "Let's try out Google news pre-trained word vectors. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# It'll take a while to run this when you try it out for the first time.\n",
    "import gensim.downloader as api\n",
    "\n",
    "google_news_vectors = api.load(\"word2vec-google-news-300\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of vocabulary:  3000000\n"
     ]
    }
   ],
   "source": [
    "print(\"Size of vocabulary: \", len(google_news_vectors))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `google_news_vectors` above has 300 dimensional word vectors for 3,000,000 unique words from Google news. \n",
    "- What can we do with these word vectors?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Finding similar words \n",
    "\n",
    "- Given word $w$, search in the vector space for the word closest to $w$ as measured by cosine distance. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('UVic', 0.7886475920677185),\n",
       " ('SFU', 0.7588528394699097),\n",
       " ('Simon_Fraser', 0.7356574535369873),\n",
       " ('UFV', 0.6880435943603516),\n",
       " ('VIU', 0.6778583526611328),\n",
       " ('Kwantlen', 0.6771429181098938),\n",
       " ('UBCO', 0.6734487414360046),\n",
       " ('UPEI', 0.6731126308441162),\n",
       " ('UBC_Okanagan', 0.6709133982658386),\n",
       " ('Lakehead_University', 0.6622507572174072)]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "google_news_vectors.most_similar(\"UBC\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('info', 0.7363681793212891),\n",
       " ('infomation', 0.680029571056366),\n",
       " ('infor_mation', 0.673384964466095),\n",
       " ('informaiton', 0.6639009118080139),\n",
       " ('informa_tion', 0.6601256728172302),\n",
       " ('informationon', 0.6339334845542908),\n",
       " ('informationabout', 0.6320980191230774),\n",
       " ('Information', 0.6186580657958984),\n",
       " ('informaion', 0.6093292236328125),\n",
       " ('details', 0.6063088774681091)]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "google_news_vectors.most_similar(\"information\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to extract all documents containing information or similar words, you could use this information.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Finding similarity scores between words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.27610135"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "google_news_vectors.similarity(\"Canada\", \"hockey\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.060384665"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "google_news_vectors.similarity(\"China\", \"hockey\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The similarity between height and tall is 0.473\n",
      "The similarity between pineapple and mango is 0.668\n",
      "The similarity between pineapple and juice is 0.418\n",
      "The similarity between sun and robot is 0.029\n",
      "The similarity between GPU and lion is 0.002\n"
     ]
    }
   ],
   "source": [
    "word_pairs = [\n",
    "    (\"height\", \"tall\"),\n",
    "    (\"pineapple\", \"mango\"),\n",
    "    (\"pineapple\", \"juice\"),\n",
    "    (\"sun\", \"robot\"),\n",
    "    (\"GPU\", \"lion\"),\n",
    "]\n",
    "for pair in word_pairs:\n",
    "    print(\n",
    "        \"The similarity between %s and %s is %0.3f\"\n",
    "        % (pair[0], pair[1], google_news_vectors.similarity(pair[0], pair[1]))\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def analogy(word1, word2, word3, model=google_news_vectors):\n",
    "    \"\"\"\n",
    "    Returns analogy word using the given model.\n",
    "\n",
    "    Parameters\n",
    "    --------------\n",
    "    word1 : (str)\n",
    "        word1 in the analogy relation\n",
    "    word2 : (str)\n",
    "        word2 in the analogy relation\n",
    "    word3 : (str)\n",
    "        word3 in the analogy relation\n",
    "    model :\n",
    "        word embedding model\n",
    "\n",
    "    Returns\n",
    "    ---------------\n",
    "        pd.dataframe\n",
    "    \"\"\"\n",
    "    print(\"%s : %s :: %s : ?\" % (word1, word2, word3))\n",
    "    sim_words = model.most_similar(positive=[word3, word2], negative=[word1])\n",
    "    return pd.DataFrame(sim_words, columns=[\"Analogy word\", \"Score\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "man : king :: woman : ?\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Analogy word</th>\n",
       "      <th>Score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>queen</td>\n",
       "      <td>0.711819</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>monarch</td>\n",
       "      <td>0.618967</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>princess</td>\n",
       "      <td>0.590243</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>crown_prince</td>\n",
       "      <td>0.549946</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>prince</td>\n",
       "      <td>0.537732</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>kings</td>\n",
       "      <td>0.523684</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Queen_Consort</td>\n",
       "      <td>0.523595</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>queens</td>\n",
       "      <td>0.518113</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>sultan</td>\n",
       "      <td>0.509859</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>monarchy</td>\n",
       "      <td>0.508741</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Analogy word     Score\n",
       "0          queen  0.711819\n",
       "1        monarch  0.618967\n",
       "2       princess  0.590243\n",
       "3   crown_prince  0.549946\n",
       "4         prince  0.537732\n",
       "5          kings  0.523684\n",
       "6  Queen_Consort  0.523595\n",
       "7         queens  0.518113\n",
       "8         sultan  0.509859\n",
       "9       monarchy  0.508741"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "analogy(\"man\", \"king\", \"woman\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Montreal : Canadiens :: Vancouver : ?\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Analogy word</th>\n",
       "      <th>Score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Canucks</td>\n",
       "      <td>0.821327</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Vancouver_Canucks</td>\n",
       "      <td>0.750401</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Calgary_Flames</td>\n",
       "      <td>0.705470</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Leafs</td>\n",
       "      <td>0.695783</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Maple_Leafs</td>\n",
       "      <td>0.691617</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Thrashers</td>\n",
       "      <td>0.687504</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Avs</td>\n",
       "      <td>0.681716</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Sabres</td>\n",
       "      <td>0.665307</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Blackhawks</td>\n",
       "      <td>0.664625</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Habs</td>\n",
       "      <td>0.661023</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Analogy word     Score\n",
       "0            Canucks  0.821327\n",
       "1  Vancouver_Canucks  0.750401\n",
       "2     Calgary_Flames  0.705470\n",
       "3              Leafs  0.695783\n",
       "4        Maple_Leafs  0.691617\n",
       "5          Thrashers  0.687504\n",
       "6                Avs  0.681716\n",
       "7             Sabres  0.665307\n",
       "8         Blackhawks  0.664625\n",
       "9               Habs  0.661023"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "analogy(\"Montreal\", \"Canadiens\", \"Vancouver\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Toronto : UofT :: Vancouver : ?\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Analogy word</th>\n",
       "      <th>Score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>SFU</td>\n",
       "      <td>0.579245</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>UVic</td>\n",
       "      <td>0.576921</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>UBC</td>\n",
       "      <td>0.571431</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Simon_Fraser</td>\n",
       "      <td>0.543464</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Langara_College</td>\n",
       "      <td>0.541347</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>UVIC</td>\n",
       "      <td>0.520495</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Grant_MacEwan</td>\n",
       "      <td>0.517273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>UFV</td>\n",
       "      <td>0.514150</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Ubyssey</td>\n",
       "      <td>0.510421</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Kwantlen</td>\n",
       "      <td>0.503807</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Analogy word     Score\n",
       "0              SFU  0.579245\n",
       "1             UVic  0.576921\n",
       "2              UBC  0.571431\n",
       "3     Simon_Fraser  0.543464\n",
       "4  Langara_College  0.541347\n",
       "5             UVIC  0.520495\n",
       "6    Grant_MacEwan  0.517273\n",
       "7              UFV  0.514150\n",
       "8          Ubyssey  0.510421\n",
       "9         Kwantlen  0.503807"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "analogy(\"Toronto\", \"UofT\", \"Vancouver\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Examples of semantic and syntactic relationships\n",
    "\n",
    "![](img/word_analogies2.png)\n",
    "\n",
    "<!-- <img src=\"img/word_analogies2.png\" width=\"800\" height=\"800\"> -->\n",
    "\n",
    "(Credit: Mikolov 2013)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Implicit biases and stereotypes in word embeddings\n",
    "\n",
    "- Reflect gender stereotypes present in broader society.\n",
    "- They may also amplify these stereotypes because of their widespread usage. \n",
    "- See the paper [Man is to Computer Programmer as Woman is to ...](http://papers.nips.cc/paper/6228-man-is-to-computer-programmer-as-woman-is-to-homemaker-debiasing-word-embeddings.pdf)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "man : computer_programmer :: woman : ?\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Analogy word</th>\n",
       "      <th>Score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>homemaker</td>\n",
       "      <td>0.562712</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>housewife</td>\n",
       "      <td>0.510505</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>graphic_designer</td>\n",
       "      <td>0.505180</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>schoolteacher</td>\n",
       "      <td>0.497949</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>businesswoman</td>\n",
       "      <td>0.493489</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>paralegal</td>\n",
       "      <td>0.492551</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>registered_nurse</td>\n",
       "      <td>0.490797</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>saleswoman</td>\n",
       "      <td>0.488163</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>electrical_engineer</td>\n",
       "      <td>0.479773</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>mechanical_engineer</td>\n",
       "      <td>0.475540</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Analogy word     Score\n",
       "0            homemaker  0.562712\n",
       "1            housewife  0.510505\n",
       "2     graphic_designer  0.505180\n",
       "3        schoolteacher  0.497949\n",
       "4        businesswoman  0.493489\n",
       "5            paralegal  0.492551\n",
       "6     registered_nurse  0.490797\n",
       "7           saleswoman  0.488163\n",
       "8  electrical_engineer  0.479773\n",
       "9  mechanical_engineer  0.475540"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "analogy(\"man\", \"computer_programmer\", \"woman\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](img/eva-srsly.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most of the modern embeddings are de-biased. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Other pre-trained embeddings\n",
    "\n",
    "A number of pre-trained word embeddings are available. The most popular ones are:  \n",
    "\n",
    "- [Word2Vec](https://code.google.com/archive/p/word2vec/)\n",
    "    * trained on several corpora using the word2vec algorithm \n",
    "- [wikipedia2vec](https://wikipedia2vec.github.io/wikipedia2vec/pretrained/)\n",
    "    * pretrained embeddings for 12 languages \n",
    "- [GloVe](https://nlp.stanford.edu/projects/glove/)\n",
    "    * trained using [the GloVe algorithm](https://nlp.stanford.edu/pubs/glove.pdf) \n",
    "    * published by Stanford University \n",
    "- [fastText pre-trained embeddings for 294 languages](https://fasttext.cc/docs/en/pretrained-vectors.html) \n",
    "    * trained using [the fastText algorithm](http://aclweb.org/anthology/Q17-1010)\n",
    "    * published by Facebook    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Note*** \n",
    "These pre-trained word vectors are of big size (in several gigabytes)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a list of all pre-trained embeddings available with `gensim`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fasttext-wiki-news-subwords-300, conceptnet-numberbatch-17-06-300, word2vec-ruscorpora-300, word2vec-google-news-300, glove-wiki-gigaword-50, glove-wiki-gigaword-100, glove-wiki-gigaword-200, glove-wiki-gigaword-300, glove-twitter-25, glove-twitter-50, glove-twitter-100, glove-twitter-200, __testing_word2vec-matrix-synopsis\n"
     ]
    }
   ],
   "source": [
    "import gensim.downloader\n",
    "\n",
    "print(*gensim.downloader.info()[\"models\"].keys(), sep=\", \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Word vectors with spaCy\n",
    "\n",
    "- spaCy also gives you access to word vectors with bigger models: `en_core_web_md` or `en_core_web_lr`\n",
    "- `spaCy`'s pre-trained embeddings are trained on [OntoNotes corpus](https://catalog.ldc.upenn.edu/LDC2013T19).\n",
    "- This corpus has a collection of different styles of texts such as telephone conversations, newswire, newsgroups, broadcast news, broadcast conversation, weblogs, religious texts. \n",
    "- Let's try it out. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-6.3358e-01,  1.2266e-01,  4.7232e-01, -2.2974e-01, -2.6307e-01,\n",
       "        5.6499e-01, -7.2338e-01,  1.6736e-01,  4.2030e-01,  9.3788e-01,\n",
       "       -1.7248e-01,  2.9126e-01, -7.8257e-01, -1.2844e-01,  3.3886e-01,\n",
       "       -3.3947e-01, -4.2727e-01,  1.4579e+00, -3.1335e-01, -5.2318e-05,\n",
       "        6.1973e-01, -3.6482e-02, -1.0523e-01, -1.0039e-01,  2.6267e-01,\n",
       "       -5.3926e-01,  1.8011e-01, -6.1530e-02,  1.9553e-01, -1.0654e+00,\n",
       "        6.8364e-02,  1.0889e-02,  6.8862e-01, -3.0893e-02, -2.9747e-01,\n",
       "        2.7474e-01,  3.5519e-01,  6.7799e-02, -4.5287e-01,  5.9265e-01,\n",
       "       -1.7061e-01, -2.9119e-01, -1.5082e-01, -3.1661e-01, -3.8160e-01,\n",
       "        2.8878e-01, -9.4547e-02,  1.9872e-01, -4.1434e-02, -4.5629e-03,\n",
       "        2.0329e-01,  4.4544e-02, -1.0714e-01,  1.3336e-02,  7.6487e-01,\n",
       "       -9.7920e-01,  4.1987e-01,  1.2322e-01,  2.2844e-01, -3.8980e-01,\n",
       "        3.5201e-02,  5.8634e-03,  5.3757e-01, -2.7319e-01, -2.0973e-01,\n",
       "       -5.6800e-01, -1.3261e-01, -4.0366e-02, -4.3473e-01,  5.9351e-01,\n",
       "       -4.8095e-01,  4.6200e-01,  3.0850e-01,  1.5563e-02, -2.6255e-01,\n",
       "       -6.2401e-02,  2.3781e-01,  3.2069e-01,  2.8044e-02, -9.3812e-02,\n",
       "        4.6437e-01, -3.8001e-01, -3.7434e-01, -3.1303e-02, -3.3077e-01,\n",
       "       -3.9036e-01,  1.2612e+00,  1.1394e+00, -7.4785e-01,  5.8881e-01,\n",
       "       -2.9830e-02, -3.4661e-01,  5.9226e-01, -1.1268e-01, -4.8890e-01,\n",
       "       -1.1186e-01, -7.3733e-02, -4.0556e-01, -5.4455e-01, -6.4546e-03,\n",
       "        5.3944e-01, -2.0552e-01, -1.9367e-01, -3.6157e-01, -2.4745e-01,\n",
       "       -1.2224e+00,  4.7183e-01, -3.5296e-01,  8.0333e-01,  4.6336e-02,\n",
       "       -4.3909e-01, -6.4112e-01, -5.2724e-01, -1.6876e-01,  2.0895e-01,\n",
       "        1.4199e-02, -4.4900e-02, -1.8501e-01,  2.6656e-01,  5.3488e-01,\n",
       "       -5.8822e-02,  1.6143e-01, -2.5204e-01, -3.0524e-01, -4.1517e-01,\n",
       "        1.8948e-01, -1.1039e-01, -2.6072e-02,  2.1508e-01,  4.5536e-01,\n",
       "       -3.6486e-01, -3.8706e-01,  6.5790e-02,  4.3741e-01, -1.2157e-01,\n",
       "       -1.9559e-01,  8.4984e-02, -2.5018e-02, -1.0655e-01, -1.7552e-01,\n",
       "       -2.1178e+00,  6.2321e-01,  3.7638e-01,  1.8257e-01, -1.0744e-01,\n",
       "       -2.0112e-01, -4.1126e-01,  4.4622e-01,  1.7428e-01, -3.6297e-01,\n",
       "        4.5627e-01,  2.8229e-01,  8.2129e-01,  2.8097e-01, -9.1766e-01,\n",
       "       -6.8294e-01, -4.6742e-02,  9.2274e-02,  1.2806e-01,  5.2903e-01,\n",
       "        2.1093e-03,  3.3689e-01,  1.6343e-01,  6.8163e-02, -1.1069e-01,\n",
       "       -7.0630e-01, -1.4520e-01, -2.7527e-01,  3.5671e-02, -1.4704e-01,\n",
       "        5.3811e-01, -1.7595e-01,  3.4800e-01, -2.3449e-01, -7.5448e-01,\n",
       "        1.4617e-01, -1.0186e-01,  5.3314e-02,  2.8269e-01, -3.3149e-01,\n",
       "       -1.7500e-01, -3.6936e-01,  1.1284e-01, -4.5708e-02,  6.5629e-02,\n",
       "       -3.2347e-01, -2.3809e-02, -2.6306e-01, -7.6422e-01,  2.3938e-02,\n",
       "       -6.1736e-01, -1.4619e-02,  1.8673e-01,  7.3024e-02,  2.8767e-01,\n",
       "        2.0519e-01,  1.1381e-01,  1.8318e-01, -4.0793e-01, -2.1453e-01,\n",
       "       -2.3576e-01,  1.8909e-01, -3.0893e-01, -2.8578e-01, -4.0883e-01,\n",
       "       -5.6930e-02,  2.6070e-01,  4.7287e-01,  5.0606e-01,  4.1980e-01,\n",
       "       -8.1708e-01, -1.3931e-01,  5.4377e-01,  4.0280e-01, -4.5356e-01,\n",
       "       -3.4729e-01,  1.1177e-02, -5.9710e-01, -3.3182e-01, -1.3462e-01,\n",
       "       -4.2745e-01, -2.3084e-01,  4.7092e-02,  4.3972e-01,  5.6560e-01,\n",
       "       -1.2565e-01,  5.3948e-02, -1.0517e-01, -4.7234e-01,  5.2615e-01,\n",
       "       -3.2875e-01, -4.0684e-01, -1.3978e-01,  7.8114e-02, -6.4585e-01,\n",
       "        1.3611e-01,  2.7017e-01, -6.7502e-01,  3.1804e-02, -1.3092e-01,\n",
       "       -2.4324e-01, -1.6927e-02, -2.9533e-01, -2.8841e-01,  3.1248e-01,\n",
       "        2.6822e-02, -8.9684e-02, -3.2709e-01, -4.5021e-01,  6.5495e-01,\n",
       "        1.6854e-02, -4.5004e-01,  1.2894e-01, -6.1869e-02, -3.3053e-02,\n",
       "        5.5370e-01, -3.1155e-01, -1.5421e-01,  3.7474e-01, -1.1047e-01,\n",
       "        2.4952e-01, -7.5216e-01, -2.6494e-01,  2.2681e-01,  1.9609e-01,\n",
       "        1.0381e-01, -3.6300e-02,  3.2758e-01, -4.8151e-01, -6.3161e-01,\n",
       "        5.0505e-01,  1.1183e-01, -1.0353e-01,  1.7729e-01,  3.6105e-01,\n",
       "       -5.6136e-02, -1.4535e-01,  2.3805e-01, -2.3554e-02,  9.8490e-01,\n",
       "       -7.6400e-02,  2.5642e-01,  7.0182e-01,  3.2332e-01, -2.0407e-02,\n",
       "        6.4603e-01,  6.2263e-01,  8.0323e-02, -2.2618e-02,  5.7249e-01,\n",
       "        6.3276e-03, -5.4889e-01, -2.2304e-01, -3.4033e-01, -4.3941e-01,\n",
       "        1.5785e-01, -5.2109e-01, -1.1121e+00,  5.0105e-01,  1.5993e-01],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_md\")\n",
    "\n",
    "doc = nlp(\"pineapple\")\n",
    "doc.vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Representing documents using word embeddings\n",
    "\n",
    "- Assuming that we have reasonable representations of words. \n",
    "- How do we represent meaning of **paragraphs** or **documents**?\n",
    "- Two simple approaches\n",
    "    - **Averaging** embeddings\n",
    "    - **Concatenating** embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Averaging embeddings\n",
    "\n",
    "<blockquote>\n",
    "All empty promises\n",
    "</blockquote>\n",
    "    \n",
    "$(embedding(all) + embedding(empty) + embedding(promise))/3$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Average embeddings with spaCy\n",
    "\n",
    "- We can do this conveniently with [spaCy](https://spacy.io/usage/linguistic-features#vectors-similarity). \n",
    "- We need `en_core_web_md` model to access word vectors. \n",
    "- You can download the model by going to command line and in your course `conda` environment and download `en_core_web_md` as follows.   \n",
    "\n",
    "```\n",
    "conda activate cpsc330\n",
    "python -m spacy download en_core_web_md\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can access word vectors for individual words in `spaCy` as follows. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.76058 , -0.21996 ,  0.095465,  0.10395 , -0.52345 ,  0.060248,\n",
       "        0.069298, -0.61524 , -0.37134 ,  1.5214  ], dtype=float32)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nlp(\"empty\").vector[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can get **average embeddings** for a sentence or a document in `spaCy` as follows: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(300,)\n",
      "Vector for: All empty promises\n",
      "[-0.6622033   0.06859333  0.18747167  0.04704666 -0.239323    0.043686\n",
      "  0.13470568 -0.00746667 -0.03796467  1.9193001 ]\n"
     ]
    }
   ],
   "source": [
    "s = \"All empty promises\"\n",
    "doc = nlp(s)\n",
    "avg_sent_emb = doc.vector\n",
    "print(avg_sent_emb.shape)\n",
    "print(\"Vector for: {}\\n{}\".format((s), (avg_sent_emb[0:10])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Similarity between documents \n",
    "\n",
    "- We can also get similarity between documents as follows. \n",
    "- Note that this is **based on average embeddings of each sentence**. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deep learning is very popular these days. <-> Machine learning is dominated by neural networks. 0.7346255041188937\n",
      "Machine learning is dominated by neural networks. <-> A home-made fresh bread with butter and cheese. 0.4759406337841433\n"
     ]
    }
   ],
   "source": [
    "doc1 = nlp(\"Deep learning is very popular these days.\")\n",
    "doc2 = nlp(\"Machine learning is dominated by neural networks.\")\n",
    "doc3 = nlp(\"A home-made fresh bread with butter and cheese.\")\n",
    "\n",
    "# Similarity of two documents\n",
    "print(doc1, \"<->\", doc2, doc1.similarity(doc2))\n",
    "print(doc2, \"<->\", doc3, doc2.similarity(doc3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Do these scores make sense? \n",
    "- There are no common words, but we are still able to identify that doc1 and doc2 are more similar that doc2 and doc3.  \n",
    "- You can use such average embedding representation in text classification tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "###  Airline sentiment analysis using average embedding representation \n",
    "\n",
    "- Let's try average embedding representation for airline sentiment analysis.\n",
    "- We used this dataset last week so you should already have it in the data directory. If not you can download it [here](https://www.kaggle.com/jaskarancr/airline-sentiment-dataset). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"data/Airline-Sentiment-2-w-AA.csv\", encoding=\"ISO-8859-1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_validate, train_test_split\n",
    "\n",
    "train_df, test_df = train_test_split(df, test_size=0.2, random_state=123)\n",
    "X_train, y_train = train_df[\"text\"], train_df[\"airline_sentiment\"]\n",
    "X_test, y_test = test_df[\"text\"], test_df[\"airline_sentiment\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>_unit_id</th>\n",
       "      <th>_golden</th>\n",
       "      <th>_unit_state</th>\n",
       "      <th>_trusted_judgments</th>\n",
       "      <th>_last_judgment_at</th>\n",
       "      <th>airline_sentiment</th>\n",
       "      <th>airline_sentiment:confidence</th>\n",
       "      <th>negativereason</th>\n",
       "      <th>negativereason:confidence</th>\n",
       "      <th>airline</th>\n",
       "      <th>airline_sentiment_gold</th>\n",
       "      <th>name</th>\n",
       "      <th>negativereason_gold</th>\n",
       "      <th>retweet_count</th>\n",
       "      <th>text</th>\n",
       "      <th>tweet_coord</th>\n",
       "      <th>tweet_created</th>\n",
       "      <th>tweet_id</th>\n",
       "      <th>tweet_location</th>\n",
       "      <th>user_timezone</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5789</th>\n",
       "      <td>681455792</td>\n",
       "      <td>False</td>\n",
       "      <td>finalized</td>\n",
       "      <td>3</td>\n",
       "      <td>2/25/15 4:21</td>\n",
       "      <td>negative</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Can't Tell</td>\n",
       "      <td>0.6667</td>\n",
       "      <td>Southwest</td>\n",
       "      <td>NaN</td>\n",
       "      <td>mrssuperdimmock</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@SouthwestAir link doesn't work</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2/19/15 18:53</td>\n",
       "      <td>5.686040e+17</td>\n",
       "      <td>Lake Arrowhead, CA</td>\n",
       "      <td>Pacific Time (US &amp; Canada)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8918</th>\n",
       "      <td>681459957</td>\n",
       "      <td>False</td>\n",
       "      <td>finalized</td>\n",
       "      <td>3</td>\n",
       "      <td>2/25/15 9:45</td>\n",
       "      <td>neutral</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Delta</td>\n",
       "      <td>NaN</td>\n",
       "      <td>labeles</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@JetBlue okayyyy. But I had huge irons on way ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2/17/15 10:18</td>\n",
       "      <td>5.677500e+17</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11688</th>\n",
       "      <td>681462990</td>\n",
       "      <td>False</td>\n",
       "      <td>finalized</td>\n",
       "      <td>3</td>\n",
       "      <td>2/25/15 9:53</td>\n",
       "      <td>negative</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Customer Service Issue</td>\n",
       "      <td>0.6727</td>\n",
       "      <td>US Airways</td>\n",
       "      <td>NaN</td>\n",
       "      <td>DropMeAnywhere</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@USAirways They're all reservations numbers an...</td>\n",
       "      <td>[0.0, 0.0]</td>\n",
       "      <td>2/17/15 14:50</td>\n",
       "      <td>5.678190e+17</td>\n",
       "      <td>Here, There and Everywhere</td>\n",
       "      <td>Arizona</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>413</th>\n",
       "      <td>681448905</td>\n",
       "      <td>False</td>\n",
       "      <td>finalized</td>\n",
       "      <td>3</td>\n",
       "      <td>2/25/15 10:10</td>\n",
       "      <td>neutral</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Virgin America</td>\n",
       "      <td>NaN</td>\n",
       "      <td>jsamaudio</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@VirginAmerica no A's channel this year?</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2/18/15 12:25</td>\n",
       "      <td>5.681440e+17</td>\n",
       "      <td>St. Francis (Calif.)</td>\n",
       "      <td>Pacific Time (US &amp; Canada)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4135</th>\n",
       "      <td>681454122</td>\n",
       "      <td>False</td>\n",
       "      <td>finalized</td>\n",
       "      <td>3</td>\n",
       "      <td>2/25/15 10:08</td>\n",
       "      <td>negative</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Bad Flight</td>\n",
       "      <td>0.3544</td>\n",
       "      <td>United</td>\n",
       "      <td>NaN</td>\n",
       "      <td>CajunSQL</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@united missed it.  Incoming on time, then Sat...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2/17/15 14:20</td>\n",
       "      <td>5.678110e+17</td>\n",
       "      <td>Baton Rouge, LA</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        _unit_id  _golden _unit_state  _trusted_judgments _last_judgment_at  \\\n",
       "5789   681455792    False   finalized                   3      2/25/15 4:21   \n",
       "8918   681459957    False   finalized                   3      2/25/15 9:45   \n",
       "11688  681462990    False   finalized                   3      2/25/15 9:53   \n",
       "413    681448905    False   finalized                   3     2/25/15 10:10   \n",
       "4135   681454122    False   finalized                   3     2/25/15 10:08   \n",
       "\n",
       "      airline_sentiment  airline_sentiment:confidence          negativereason  \\\n",
       "5789           negative                           1.0              Can't Tell   \n",
       "8918            neutral                           1.0                     NaN   \n",
       "11688          negative                           1.0  Customer Service Issue   \n",
       "413             neutral                           1.0                     NaN   \n",
       "4135           negative                           1.0              Bad Flight   \n",
       "\n",
       "       negativereason:confidence         airline airline_sentiment_gold  \\\n",
       "5789                      0.6667       Southwest                    NaN   \n",
       "8918                         NaN           Delta                    NaN   \n",
       "11688                     0.6727      US Airways                    NaN   \n",
       "413                          NaN  Virgin America                    NaN   \n",
       "4135                      0.3544          United                    NaN   \n",
       "\n",
       "                  name negativereason_gold  retweet_count  \\\n",
       "5789   mrssuperdimmock                 NaN              0   \n",
       "8918           labeles                 NaN              0   \n",
       "11688   DropMeAnywhere                 NaN              0   \n",
       "413          jsamaudio                 NaN              0   \n",
       "4135          CajunSQL                 NaN              0   \n",
       "\n",
       "                                                    text tweet_coord  \\\n",
       "5789                     @SouthwestAir link doesn't work         NaN   \n",
       "8918   @JetBlue okayyyy. But I had huge irons on way ...         NaN   \n",
       "11688  @USAirways They're all reservations numbers an...  [0.0, 0.0]   \n",
       "413             @VirginAmerica no A's channel this year?         NaN   \n",
       "4135   @united missed it.  Incoming on time, then Sat...         NaN   \n",
       "\n",
       "       tweet_created      tweet_id              tweet_location  \\\n",
       "5789   2/19/15 18:53  5.686040e+17          Lake Arrowhead, CA   \n",
       "8918   2/17/15 10:18  5.677500e+17                         NaN   \n",
       "11688  2/17/15 14:50  5.678190e+17  Here, There and Everywhere   \n",
       "413    2/18/15 12:25  5.681440e+17        St. Francis (Calif.)   \n",
       "4135   2/17/15 14:20  5.678110e+17             Baton Rouge, LA   \n",
       "\n",
       "                    user_timezone  \n",
       "5789   Pacific Time (US & Canada)  \n",
       "8918                          NaN  \n",
       "11688                     Arizona  \n",
       "413    Pacific Time (US & Canada)  \n",
       "4135                          NaN  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Bag-of-words representation for sentiment analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data matrix shape: (11712, 13064)\n"
     ]
    }
   ],
   "source": [
    "pipe = make_pipeline(\n",
    "    CountVectorizer(stop_words=\"english\"), LogisticRegression(max_iter=1000)\n",
    ")\n",
    "pipe.named_steps[\"countvectorizer\"].fit(X_train)\n",
    "X_train_transformed = pipe.named_steps[\"countvectorizer\"].transform(X_train)\n",
    "print(\"Data matrix shape:\", X_train_transformed.shape)\n",
    "pipe.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train accuracy 0.94\n",
      "Test accuracy 0.80\n"
     ]
    }
   ],
   "source": [
    "print(\"Train accuracy {:.2f}\".format(pipe.score(X_train, y_train)))\n",
    "print(\"Test accuracy {:.2f}\".format(pipe.score(X_test, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Sentiment analysis with average embedding representation\n",
    "\n",
    "- Let's see how can we get word vectors using `spaCy`. \n",
    "- Let's create average embedding representation for each example. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "X_train_embeddings = pd.DataFrame([text.vector for text in nlp.pipe(X_train)])\n",
    "X_test_embeddings = pd.DataFrame([text.vector for text in nlp.pipe(X_test)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "We have **reduced dimensionality** from 13,064 to 300! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(11712, 300)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_embeddings.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>290</th>\n",
       "      <th>291</th>\n",
       "      <th>292</th>\n",
       "      <th>293</th>\n",
       "      <th>294</th>\n",
       "      <th>295</th>\n",
       "      <th>296</th>\n",
       "      <th>297</th>\n",
       "      <th>298</th>\n",
       "      <th>299</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.561952</td>\n",
       "      <td>0.225163</td>\n",
       "      <td>-0.293684</td>\n",
       "      <td>-0.082498</td>\n",
       "      <td>-0.192374</td>\n",
       "      <td>0.050405</td>\n",
       "      <td>0.060202</td>\n",
       "      <td>-0.254210</td>\n",
       "      <td>0.089486</td>\n",
       "      <td>1.773900</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.210224</td>\n",
       "      <td>-0.033940</td>\n",
       "      <td>-0.130713</td>\n",
       "      <td>-0.145631</td>\n",
       "      <td>0.138794</td>\n",
       "      <td>0.115038</td>\n",
       "      <td>-0.081300</td>\n",
       "      <td>-0.155296</td>\n",
       "      <td>0.053767</td>\n",
       "      <td>0.150895</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.700668</td>\n",
       "      <td>0.124276</td>\n",
       "      <td>-0.259541</td>\n",
       "      <td>-0.260308</td>\n",
       "      <td>-0.129364</td>\n",
       "      <td>0.068210</td>\n",
       "      <td>0.066224</td>\n",
       "      <td>-0.307202</td>\n",
       "      <td>-0.047832</td>\n",
       "      <td>1.612876</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.172155</td>\n",
       "      <td>0.069523</td>\n",
       "      <td>-0.077437</td>\n",
       "      <td>-0.027307</td>\n",
       "      <td>0.097390</td>\n",
       "      <td>0.174096</td>\n",
       "      <td>0.023029</td>\n",
       "      <td>0.083394</td>\n",
       "      <td>-0.034370</td>\n",
       "      <td>0.089504</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.720738</td>\n",
       "      <td>0.284142</td>\n",
       "      <td>-0.285948</td>\n",
       "      <td>-0.150022</td>\n",
       "      <td>-0.134630</td>\n",
       "      <td>-0.035959</td>\n",
       "      <td>0.124829</td>\n",
       "      <td>-0.282646</td>\n",
       "      <td>0.044680</td>\n",
       "      <td>2.146762</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.308978</td>\n",
       "      <td>0.067948</td>\n",
       "      <td>-0.141636</td>\n",
       "      <td>-0.083790</td>\n",
       "      <td>0.118324</td>\n",
       "      <td>0.212624</td>\n",
       "      <td>-0.079882</td>\n",
       "      <td>-0.146916</td>\n",
       "      <td>0.038091</td>\n",
       "      <td>0.046573</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.602323</td>\n",
       "      <td>0.310292</td>\n",
       "      <td>-0.197665</td>\n",
       "      <td>-0.003409</td>\n",
       "      <td>-0.053958</td>\n",
       "      <td>0.033286</td>\n",
       "      <td>0.190612</td>\n",
       "      <td>-0.242265</td>\n",
       "      <td>0.060668</td>\n",
       "      <td>1.795106</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.165880</td>\n",
       "      <td>0.106567</td>\n",
       "      <td>-0.159539</td>\n",
       "      <td>0.017612</td>\n",
       "      <td>0.111956</td>\n",
       "      <td>0.153079</td>\n",
       "      <td>0.005538</td>\n",
       "      <td>-0.133347</td>\n",
       "      <td>0.075873</td>\n",
       "      <td>-0.150727</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-0.657547</td>\n",
       "      <td>0.228270</td>\n",
       "      <td>-0.184336</td>\n",
       "      <td>-0.162730</td>\n",
       "      <td>-0.076056</td>\n",
       "      <td>0.042302</td>\n",
       "      <td>0.014390</td>\n",
       "      <td>-0.260486</td>\n",
       "      <td>0.062895</td>\n",
       "      <td>1.678635</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.202215</td>\n",
       "      <td>0.068234</td>\n",
       "      <td>0.006729</td>\n",
       "      <td>-0.011597</td>\n",
       "      <td>0.087860</td>\n",
       "      <td>0.185101</td>\n",
       "      <td>0.031942</td>\n",
       "      <td>-0.112022</td>\n",
       "      <td>0.029528</td>\n",
       "      <td>0.123609</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 300 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        0         1         2         3         4         5         6    \\\n",
       "0 -0.561952  0.225163 -0.293684 -0.082498 -0.192374  0.050405  0.060202   \n",
       "1 -0.700668  0.124276 -0.259541 -0.260308 -0.129364  0.068210  0.066224   \n",
       "2 -0.720738  0.284142 -0.285948 -0.150022 -0.134630 -0.035959  0.124829   \n",
       "3 -0.602323  0.310292 -0.197665 -0.003409 -0.053958  0.033286  0.190612   \n",
       "4 -0.657547  0.228270 -0.184336 -0.162730 -0.076056  0.042302  0.014390   \n",
       "\n",
       "        7         8         9    ...       290       291       292       293  \\\n",
       "0 -0.254210  0.089486  1.773900  ... -0.210224 -0.033940 -0.130713 -0.145631   \n",
       "1 -0.307202 -0.047832  1.612876  ... -0.172155  0.069523 -0.077437 -0.027307   \n",
       "2 -0.282646  0.044680  2.146762  ... -0.308978  0.067948 -0.141636 -0.083790   \n",
       "3 -0.242265  0.060668  1.795106  ... -0.165880  0.106567 -0.159539  0.017612   \n",
       "4 -0.260486  0.062895  1.678635  ... -0.202215  0.068234  0.006729 -0.011597   \n",
       "\n",
       "        294       295       296       297       298       299  \n",
       "0  0.138794  0.115038 -0.081300 -0.155296  0.053767  0.150895  \n",
       "1  0.097390  0.174096  0.023029  0.083394 -0.034370  0.089504  \n",
       "2  0.118324  0.212624 -0.079882 -0.146916  0.038091  0.046573  \n",
       "3  0.111956  0.153079  0.005538 -0.133347  0.075873 -0.150727  \n",
       "4  0.087860  0.185101  0.031942 -0.112022  0.029528  0.123609  \n",
       "\n",
       "[5 rows x 300 columns]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_embeddings.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Sentiment classification using average embeddings \n",
    "\n",
    "- What are the train and test accuracies with average word embedding representation? \n",
    "- The accuracy is a bit better with **less overfitting**. \n",
    "- Note that we are using **transfer learning** here.\n",
    "- The embeddings are trained on a completely different corpus. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train accuracy 0.74\n",
      "Test accuracy 0.75\n"
     ]
    }
   ],
   "source": [
    "lgr = LogisticRegression(max_iter=1000)\n",
    "lgr.fit(X_train_embeddings, y_train)\n",
    "print(\"Train accuracy {:.2f}\".format(lgr.score(X_train_embeddings, y_train)))\n",
    "print(\"Test accuracy {:.2f}\".format(lgr.score(X_test_embeddings, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Sentiment classification using advanced sentence representations "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Since, representing documents is so essential for text classification tasks, there are more advanced methods for document representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from sentence_transformers import SentenceTransformer\n",
    "\n",
    "embedder = SentenceTransformer(\"paraphrase-distilroberta-base-v1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(768,)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "emb_sents = embedder.encode(\"all empty promises\")\n",
    "emb_sents.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>758</th>\n",
       "      <th>759</th>\n",
       "      <th>760</th>\n",
       "      <th>761</th>\n",
       "      <th>762</th>\n",
       "      <th>763</th>\n",
       "      <th>764</th>\n",
       "      <th>765</th>\n",
       "      <th>766</th>\n",
       "      <th>767</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5789</th>\n",
       "      <td>-0.120494</td>\n",
       "      <td>0.250262</td>\n",
       "      <td>-0.022795</td>\n",
       "      <td>-0.116368</td>\n",
       "      <td>0.078650</td>\n",
       "      <td>0.037357</td>\n",
       "      <td>-0.251341</td>\n",
       "      <td>0.321429</td>\n",
       "      <td>-0.143984</td>\n",
       "      <td>-0.123487</td>\n",
       "      <td>...</td>\n",
       "      <td>0.199151</td>\n",
       "      <td>-0.150143</td>\n",
       "      <td>0.167078</td>\n",
       "      <td>-0.407671</td>\n",
       "      <td>-0.066161</td>\n",
       "      <td>0.049514</td>\n",
       "      <td>0.019384</td>\n",
       "      <td>-0.357601</td>\n",
       "      <td>0.125996</td>\n",
       "      <td>0.381073</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8918</th>\n",
       "      <td>-0.182954</td>\n",
       "      <td>0.118282</td>\n",
       "      <td>0.066341</td>\n",
       "      <td>-0.136099</td>\n",
       "      <td>0.094947</td>\n",
       "      <td>-0.121303</td>\n",
       "      <td>0.069233</td>\n",
       "      <td>-0.097500</td>\n",
       "      <td>0.025739</td>\n",
       "      <td>-0.367980</td>\n",
       "      <td>...</td>\n",
       "      <td>0.113612</td>\n",
       "      <td>0.114661</td>\n",
       "      <td>0.049926</td>\n",
       "      <td>0.256736</td>\n",
       "      <td>-0.118687</td>\n",
       "      <td>-0.190720</td>\n",
       "      <td>0.011986</td>\n",
       "      <td>-0.141883</td>\n",
       "      <td>-0.230142</td>\n",
       "      <td>0.024899</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11688</th>\n",
       "      <td>-0.032988</td>\n",
       "      <td>0.630251</td>\n",
       "      <td>-0.079516</td>\n",
       "      <td>0.148981</td>\n",
       "      <td>0.194709</td>\n",
       "      <td>-0.226264</td>\n",
       "      <td>-0.043630</td>\n",
       "      <td>0.217398</td>\n",
       "      <td>-0.010716</td>\n",
       "      <td>0.069643</td>\n",
       "      <td>...</td>\n",
       "      <td>0.676791</td>\n",
       "      <td>0.244483</td>\n",
       "      <td>0.051042</td>\n",
       "      <td>0.064099</td>\n",
       "      <td>-0.146945</td>\n",
       "      <td>0.090878</td>\n",
       "      <td>-0.090059</td>\n",
       "      <td>0.077212</td>\n",
       "      <td>-0.209226</td>\n",
       "      <td>0.308773</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>413</th>\n",
       "      <td>-0.119258</td>\n",
       "      <td>0.172168</td>\n",
       "      <td>0.098698</td>\n",
       "      <td>0.319859</td>\n",
       "      <td>0.415475</td>\n",
       "      <td>0.248360</td>\n",
       "      <td>-0.025923</td>\n",
       "      <td>0.385350</td>\n",
       "      <td>0.066414</td>\n",
       "      <td>-0.334289</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.128482</td>\n",
       "      <td>-0.232447</td>\n",
       "      <td>-0.077805</td>\n",
       "      <td>0.181328</td>\n",
       "      <td>0.123244</td>\n",
       "      <td>-0.143693</td>\n",
       "      <td>0.660456</td>\n",
       "      <td>-0.048714</td>\n",
       "      <td>0.204774</td>\n",
       "      <td>0.163496</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4135</th>\n",
       "      <td>0.094240</td>\n",
       "      <td>0.360193</td>\n",
       "      <td>0.213747</td>\n",
       "      <td>0.363690</td>\n",
       "      <td>0.275521</td>\n",
       "      <td>0.134936</td>\n",
       "      <td>-0.276319</td>\n",
       "      <td>0.009336</td>\n",
       "      <td>-0.021523</td>\n",
       "      <td>-0.258992</td>\n",
       "      <td>...</td>\n",
       "      <td>0.474885</td>\n",
       "      <td>0.242125</td>\n",
       "      <td>0.294532</td>\n",
       "      <td>0.279013</td>\n",
       "      <td>0.037831</td>\n",
       "      <td>0.089761</td>\n",
       "      <td>-0.548748</td>\n",
       "      <td>-0.049258</td>\n",
       "      <td>0.154525</td>\n",
       "      <td>0.141268</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5218</th>\n",
       "      <td>-0.204409</td>\n",
       "      <td>-0.145290</td>\n",
       "      <td>-0.064201</td>\n",
       "      <td>0.213571</td>\n",
       "      <td>-0.140225</td>\n",
       "      <td>0.338555</td>\n",
       "      <td>-0.148578</td>\n",
       "      <td>0.224516</td>\n",
       "      <td>-0.042963</td>\n",
       "      <td>0.075930</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.161949</td>\n",
       "      <td>0.040582</td>\n",
       "      <td>0.003971</td>\n",
       "      <td>-0.152549</td>\n",
       "      <td>-0.582908</td>\n",
       "      <td>-0.126527</td>\n",
       "      <td>0.060502</td>\n",
       "      <td>-0.111495</td>\n",
       "      <td>-0.097492</td>\n",
       "      <td>0.199322</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12252</th>\n",
       "      <td>0.108408</td>\n",
       "      <td>0.438293</td>\n",
       "      <td>0.216812</td>\n",
       "      <td>-0.349289</td>\n",
       "      <td>0.422689</td>\n",
       "      <td>0.377760</td>\n",
       "      <td>0.045198</td>\n",
       "      <td>-0.034096</td>\n",
       "      <td>0.427570</td>\n",
       "      <td>-0.328272</td>\n",
       "      <td>...</td>\n",
       "      <td>0.257849</td>\n",
       "      <td>-0.032362</td>\n",
       "      <td>-0.275003</td>\n",
       "      <td>0.080452</td>\n",
       "      <td>-0.078975</td>\n",
       "      <td>-0.049972</td>\n",
       "      <td>-0.009762</td>\n",
       "      <td>-0.314754</td>\n",
       "      <td>-0.020773</td>\n",
       "      <td>0.268777</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1346</th>\n",
       "      <td>0.068411</td>\n",
       "      <td>0.017591</td>\n",
       "      <td>0.236154</td>\n",
       "      <td>0.221446</td>\n",
       "      <td>-0.103567</td>\n",
       "      <td>0.055510</td>\n",
       "      <td>0.062909</td>\n",
       "      <td>0.067425</td>\n",
       "      <td>-0.003504</td>\n",
       "      <td>-0.157758</td>\n",
       "      <td>...</td>\n",
       "      <td>0.007711</td>\n",
       "      <td>0.323297</td>\n",
       "      <td>0.334638</td>\n",
       "      <td>0.367042</td>\n",
       "      <td>-0.068821</td>\n",
       "      <td>0.063667</td>\n",
       "      <td>-0.329990</td>\n",
       "      <td>0.232331</td>\n",
       "      <td>-0.184768</td>\n",
       "      <td>-0.000683</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11646</th>\n",
       "      <td>-0.091488</td>\n",
       "      <td>-0.155709</td>\n",
       "      <td>0.032391</td>\n",
       "      <td>0.018313</td>\n",
       "      <td>0.524998</td>\n",
       "      <td>0.563933</td>\n",
       "      <td>-0.080984</td>\n",
       "      <td>0.097983</td>\n",
       "      <td>-0.535285</td>\n",
       "      <td>-0.377195</td>\n",
       "      <td>...</td>\n",
       "      <td>0.428014</td>\n",
       "      <td>-0.144573</td>\n",
       "      <td>0.045296</td>\n",
       "      <td>-0.107935</td>\n",
       "      <td>-0.135673</td>\n",
       "      <td>-0.290019</td>\n",
       "      <td>-0.137200</td>\n",
       "      <td>-0.503395</td>\n",
       "      <td>-0.042567</td>\n",
       "      <td>-0.282592</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3582</th>\n",
       "      <td>0.185626</td>\n",
       "      <td>0.092904</td>\n",
       "      <td>0.097085</td>\n",
       "      <td>-0.174650</td>\n",
       "      <td>-0.193584</td>\n",
       "      <td>0.047294</td>\n",
       "      <td>0.098216</td>\n",
       "      <td>0.332670</td>\n",
       "      <td>0.163098</td>\n",
       "      <td>-0.135102</td>\n",
       "      <td>...</td>\n",
       "      <td>0.078530</td>\n",
       "      <td>-0.030177</td>\n",
       "      <td>0.391598</td>\n",
       "      <td>0.073519</td>\n",
       "      <td>-0.454038</td>\n",
       "      <td>-0.244358</td>\n",
       "      <td>-0.790682</td>\n",
       "      <td>-0.607009</td>\n",
       "      <td>-0.255162</td>\n",
       "      <td>0.029779</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>11712 rows × 768 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            0         1         2         3         4         5         6    \\\n",
       "5789  -0.120494  0.250262 -0.022795 -0.116368  0.078650  0.037357 -0.251341   \n",
       "8918  -0.182954  0.118282  0.066341 -0.136099  0.094947 -0.121303  0.069233   \n",
       "11688 -0.032988  0.630251 -0.079516  0.148981  0.194709 -0.226264 -0.043630   \n",
       "413   -0.119258  0.172168  0.098698  0.319859  0.415475  0.248360 -0.025923   \n",
       "4135   0.094240  0.360193  0.213747  0.363690  0.275521  0.134936 -0.276319   \n",
       "...         ...       ...       ...       ...       ...       ...       ...   \n",
       "5218  -0.204409 -0.145290 -0.064201  0.213571 -0.140225  0.338555 -0.148578   \n",
       "12252  0.108408  0.438293  0.216812 -0.349289  0.422689  0.377760  0.045198   \n",
       "1346   0.068411  0.017591  0.236154  0.221446 -0.103567  0.055510  0.062909   \n",
       "11646 -0.091488 -0.155709  0.032391  0.018313  0.524998  0.563933 -0.080984   \n",
       "3582   0.185626  0.092904  0.097085 -0.174650 -0.193584  0.047294  0.098216   \n",
       "\n",
       "            7         8         9    ...       758       759       760  \\\n",
       "5789   0.321429 -0.143984 -0.123487  ...  0.199151 -0.150143  0.167078   \n",
       "8918  -0.097500  0.025739 -0.367980  ...  0.113612  0.114661  0.049926   \n",
       "11688  0.217398 -0.010716  0.069643  ...  0.676791  0.244483  0.051042   \n",
       "413    0.385350  0.066414 -0.334289  ... -0.128482 -0.232447 -0.077805   \n",
       "4135   0.009336 -0.021523 -0.258992  ...  0.474885  0.242125  0.294532   \n",
       "...         ...       ...       ...  ...       ...       ...       ...   \n",
       "5218   0.224516 -0.042963  0.075930  ... -0.161949  0.040582  0.003971   \n",
       "12252 -0.034096  0.427570 -0.328272  ...  0.257849 -0.032362 -0.275003   \n",
       "1346   0.067425 -0.003504 -0.157758  ...  0.007711  0.323297  0.334638   \n",
       "11646  0.097983 -0.535285 -0.377195  ...  0.428014 -0.144573  0.045296   \n",
       "3582   0.332670  0.163098 -0.135102  ...  0.078530 -0.030177  0.391598   \n",
       "\n",
       "            761       762       763       764       765       766       767  \n",
       "5789  -0.407671 -0.066161  0.049514  0.019384 -0.357601  0.125996  0.381073  \n",
       "8918   0.256736 -0.118687 -0.190720  0.011986 -0.141883 -0.230142  0.024899  \n",
       "11688  0.064099 -0.146945  0.090878 -0.090059  0.077212 -0.209226  0.308773  \n",
       "413    0.181328  0.123244 -0.143693  0.660456 -0.048714  0.204774  0.163496  \n",
       "4135   0.279013  0.037831  0.089761 -0.548748 -0.049258  0.154525  0.141268  \n",
       "...         ...       ...       ...       ...       ...       ...       ...  \n",
       "5218  -0.152549 -0.582908 -0.126527  0.060502 -0.111495 -0.097492  0.199322  \n",
       "12252  0.080452 -0.078975 -0.049972 -0.009762 -0.314754 -0.020773  0.268777  \n",
       "1346   0.367042 -0.068821  0.063667 -0.329990  0.232331 -0.184768 -0.000683  \n",
       "11646 -0.107935 -0.135673 -0.290019 -0.137200 -0.503395 -0.042567 -0.282592  \n",
       "3582   0.073519 -0.454038 -0.244358 -0.790682 -0.607009 -0.255162  0.029779  \n",
       "\n",
       "[11712 rows x 768 columns]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "emb_train = embedder.encode(train_df[\"text\"].tolist())\n",
    "emb_train_df = pd.DataFrame(emb_train, index=train_df.index)\n",
    "emb_train_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>758</th>\n",
       "      <th>759</th>\n",
       "      <th>760</th>\n",
       "      <th>761</th>\n",
       "      <th>762</th>\n",
       "      <th>763</th>\n",
       "      <th>764</th>\n",
       "      <th>765</th>\n",
       "      <th>766</th>\n",
       "      <th>767</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1671</th>\n",
       "      <td>-0.002864</td>\n",
       "      <td>0.217326</td>\n",
       "      <td>0.124350</td>\n",
       "      <td>-0.082548</td>\n",
       "      <td>0.709688</td>\n",
       "      <td>-0.582442</td>\n",
       "      <td>0.257897</td>\n",
       "      <td>0.169356</td>\n",
       "      <td>0.248880</td>\n",
       "      <td>-0.266686</td>\n",
       "      <td>...</td>\n",
       "      <td>0.501767</td>\n",
       "      <td>0.095387</td>\n",
       "      <td>0.340172</td>\n",
       "      <td>0.087452</td>\n",
       "      <td>-0.368359</td>\n",
       "      <td>0.276195</td>\n",
       "      <td>0.238676</td>\n",
       "      <td>-0.219546</td>\n",
       "      <td>0.066603</td>\n",
       "      <td>0.256149</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10951</th>\n",
       "      <td>-0.141048</td>\n",
       "      <td>0.137934</td>\n",
       "      <td>0.131319</td>\n",
       "      <td>0.194773</td>\n",
       "      <td>0.868205</td>\n",
       "      <td>0.078791</td>\n",
       "      <td>-0.131656</td>\n",
       "      <td>0.036243</td>\n",
       "      <td>-0.215749</td>\n",
       "      <td>-0.291946</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.056256</td>\n",
       "      <td>-0.056040</td>\n",
       "      <td>0.147341</td>\n",
       "      <td>0.189665</td>\n",
       "      <td>-0.357366</td>\n",
       "      <td>0.061799</td>\n",
       "      <td>-0.161923</td>\n",
       "      <td>-0.278955</td>\n",
       "      <td>-0.173722</td>\n",
       "      <td>0.065323</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5382</th>\n",
       "      <td>-0.252943</td>\n",
       "      <td>0.527507</td>\n",
       "      <td>-0.065609</td>\n",
       "      <td>0.013467</td>\n",
       "      <td>0.207989</td>\n",
       "      <td>0.003881</td>\n",
       "      <td>-0.066282</td>\n",
       "      <td>0.253166</td>\n",
       "      <td>0.021039</td>\n",
       "      <td>0.290956</td>\n",
       "      <td>...</td>\n",
       "      <td>0.180685</td>\n",
       "      <td>-0.042605</td>\n",
       "      <td>-0.173793</td>\n",
       "      <td>-0.079129</td>\n",
       "      <td>-0.169160</td>\n",
       "      <td>0.001317</td>\n",
       "      <td>-0.142593</td>\n",
       "      <td>-0.070816</td>\n",
       "      <td>-0.208826</td>\n",
       "      <td>0.400736</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3954</th>\n",
       "      <td>0.054318</td>\n",
       "      <td>0.096738</td>\n",
       "      <td>0.113037</td>\n",
       "      <td>0.032039</td>\n",
       "      <td>0.493064</td>\n",
       "      <td>-0.641102</td>\n",
       "      <td>0.078760</td>\n",
       "      <td>0.402187</td>\n",
       "      <td>0.189743</td>\n",
       "      <td>-0.089538</td>\n",
       "      <td>...</td>\n",
       "      <td>0.123879</td>\n",
       "      <td>-0.285018</td>\n",
       "      <td>-0.297771</td>\n",
       "      <td>0.557171</td>\n",
       "      <td>0.076168</td>\n",
       "      <td>-0.029826</td>\n",
       "      <td>-0.076094</td>\n",
       "      <td>0.225454</td>\n",
       "      <td>0.002134</td>\n",
       "      <td>0.235429</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11193</th>\n",
       "      <td>-0.065858</td>\n",
       "      <td>0.223270</td>\n",
       "      <td>0.507333</td>\n",
       "      <td>0.266193</td>\n",
       "      <td>0.104696</td>\n",
       "      <td>-0.219555</td>\n",
       "      <td>0.146247</td>\n",
       "      <td>0.315649</td>\n",
       "      <td>-0.126193</td>\n",
       "      <td>-0.435462</td>\n",
       "      <td>...</td>\n",
       "      <td>0.163994</td>\n",
       "      <td>0.207813</td>\n",
       "      <td>-0.001871</td>\n",
       "      <td>0.109391</td>\n",
       "      <td>-0.166779</td>\n",
       "      <td>-0.249200</td>\n",
       "      <td>-0.525419</td>\n",
       "      <td>-0.413065</td>\n",
       "      <td>0.119939</td>\n",
       "      <td>0.064297</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5861</th>\n",
       "      <td>0.077512</td>\n",
       "      <td>0.322276</td>\n",
       "      <td>0.026697</td>\n",
       "      <td>-0.111392</td>\n",
       "      <td>0.174208</td>\n",
       "      <td>0.235202</td>\n",
       "      <td>0.053888</td>\n",
       "      <td>0.244941</td>\n",
       "      <td>0.181625</td>\n",
       "      <td>-0.226870</td>\n",
       "      <td>...</td>\n",
       "      <td>0.149843</td>\n",
       "      <td>0.311337</td>\n",
       "      <td>0.045975</td>\n",
       "      <td>-0.572319</td>\n",
       "      <td>-0.068257</td>\n",
       "      <td>0.217745</td>\n",
       "      <td>-0.056509</td>\n",
       "      <td>-0.355174</td>\n",
       "      <td>-0.028610</td>\n",
       "      <td>0.090676</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3627</th>\n",
       "      <td>-0.173311</td>\n",
       "      <td>-0.023604</td>\n",
       "      <td>0.190388</td>\n",
       "      <td>-0.136543</td>\n",
       "      <td>-0.360269</td>\n",
       "      <td>-0.444687</td>\n",
       "      <td>0.056311</td>\n",
       "      <td>0.291941</td>\n",
       "      <td>-0.399719</td>\n",
       "      <td>-0.167930</td>\n",
       "      <td>...</td>\n",
       "      <td>0.042209</td>\n",
       "      <td>-0.161905</td>\n",
       "      <td>-0.040535</td>\n",
       "      <td>-0.050516</td>\n",
       "      <td>-0.252020</td>\n",
       "      <td>-0.133980</td>\n",
       "      <td>0.155001</td>\n",
       "      <td>-0.154482</td>\n",
       "      <td>-0.060201</td>\n",
       "      <td>-0.126555</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12559</th>\n",
       "      <td>-0.124635</td>\n",
       "      <td>-0.101799</td>\n",
       "      <td>0.129061</td>\n",
       "      <td>0.636908</td>\n",
       "      <td>0.681090</td>\n",
       "      <td>0.399300</td>\n",
       "      <td>-0.078321</td>\n",
       "      <td>0.221823</td>\n",
       "      <td>-0.277218</td>\n",
       "      <td>-0.178589</td>\n",
       "      <td>...</td>\n",
       "      <td>0.022364</td>\n",
       "      <td>-0.109274</td>\n",
       "      <td>-0.073540</td>\n",
       "      <td>-0.153336</td>\n",
       "      <td>-0.123705</td>\n",
       "      <td>-0.238896</td>\n",
       "      <td>0.296447</td>\n",
       "      <td>-0.116798</td>\n",
       "      <td>0.115076</td>\n",
       "      <td>-0.345925</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8123</th>\n",
       "      <td>0.063508</td>\n",
       "      <td>0.332506</td>\n",
       "      <td>0.119605</td>\n",
       "      <td>-0.001362</td>\n",
       "      <td>-0.161801</td>\n",
       "      <td>-0.082302</td>\n",
       "      <td>-0.025883</td>\n",
       "      <td>0.048027</td>\n",
       "      <td>0.126975</td>\n",
       "      <td>-0.159802</td>\n",
       "      <td>...</td>\n",
       "      <td>0.002221</td>\n",
       "      <td>-0.093885</td>\n",
       "      <td>0.430285</td>\n",
       "      <td>-0.088562</td>\n",
       "      <td>0.321488</td>\n",
       "      <td>0.447437</td>\n",
       "      <td>0.292395</td>\n",
       "      <td>-0.188566</td>\n",
       "      <td>-0.272767</td>\n",
       "      <td>0.126173</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>210</th>\n",
       "      <td>0.015537</td>\n",
       "      <td>0.425568</td>\n",
       "      <td>0.350672</td>\n",
       "      <td>0.113120</td>\n",
       "      <td>-0.128615</td>\n",
       "      <td>0.098112</td>\n",
       "      <td>0.222081</td>\n",
       "      <td>0.101655</td>\n",
       "      <td>0.224073</td>\n",
       "      <td>-0.341075</td>\n",
       "      <td>...</td>\n",
       "      <td>0.100983</td>\n",
       "      <td>-0.008055</td>\n",
       "      <td>0.202025</td>\n",
       "      <td>0.029846</td>\n",
       "      <td>-0.019181</td>\n",
       "      <td>0.107064</td>\n",
       "      <td>0.002301</td>\n",
       "      <td>0.038213</td>\n",
       "      <td>-0.139271</td>\n",
       "      <td>-0.007586</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2928 rows × 768 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            0         1         2         3         4         5         6    \\\n",
       "1671  -0.002864  0.217326  0.124350 -0.082548  0.709688 -0.582442  0.257897   \n",
       "10951 -0.141048  0.137934  0.131319  0.194773  0.868205  0.078791 -0.131656   \n",
       "5382  -0.252943  0.527507 -0.065609  0.013467  0.207989  0.003881 -0.066282   \n",
       "3954   0.054318  0.096738  0.113037  0.032039  0.493064 -0.641102  0.078760   \n",
       "11193 -0.065858  0.223270  0.507333  0.266193  0.104696 -0.219555  0.146247   \n",
       "...         ...       ...       ...       ...       ...       ...       ...   \n",
       "5861   0.077512  0.322276  0.026697 -0.111392  0.174208  0.235202  0.053888   \n",
       "3627  -0.173311 -0.023604  0.190388 -0.136543 -0.360269 -0.444687  0.056311   \n",
       "12559 -0.124635 -0.101799  0.129061  0.636908  0.681090  0.399300 -0.078321   \n",
       "8123   0.063508  0.332506  0.119605 -0.001362 -0.161801 -0.082302 -0.025883   \n",
       "210    0.015537  0.425568  0.350672  0.113120 -0.128615  0.098112  0.222081   \n",
       "\n",
       "            7         8         9    ...       758       759       760  \\\n",
       "1671   0.169356  0.248880 -0.266686  ...  0.501767  0.095387  0.340172   \n",
       "10951  0.036243 -0.215749 -0.291946  ... -0.056256 -0.056040  0.147341   \n",
       "5382   0.253166  0.021039  0.290956  ...  0.180685 -0.042605 -0.173793   \n",
       "3954   0.402187  0.189743 -0.089538  ...  0.123879 -0.285018 -0.297771   \n",
       "11193  0.315649 -0.126193 -0.435462  ...  0.163994  0.207813 -0.001871   \n",
       "...         ...       ...       ...  ...       ...       ...       ...   \n",
       "5861   0.244941  0.181625 -0.226870  ...  0.149843  0.311337  0.045975   \n",
       "3627   0.291941 -0.399719 -0.167930  ...  0.042209 -0.161905 -0.040535   \n",
       "12559  0.221823 -0.277218 -0.178589  ...  0.022364 -0.109274 -0.073540   \n",
       "8123   0.048027  0.126975 -0.159802  ...  0.002221 -0.093885  0.430285   \n",
       "210    0.101655  0.224073 -0.341075  ...  0.100983 -0.008055  0.202025   \n",
       "\n",
       "            761       762       763       764       765       766       767  \n",
       "1671   0.087452 -0.368359  0.276195  0.238676 -0.219546  0.066603  0.256149  \n",
       "10951  0.189665 -0.357366  0.061799 -0.161923 -0.278955 -0.173722  0.065323  \n",
       "5382  -0.079129 -0.169160  0.001317 -0.142593 -0.070816 -0.208826  0.400736  \n",
       "3954   0.557171  0.076168 -0.029826 -0.076094  0.225454  0.002134  0.235429  \n",
       "11193  0.109391 -0.166779 -0.249200 -0.525419 -0.413065  0.119939  0.064297  \n",
       "...         ...       ...       ...       ...       ...       ...       ...  \n",
       "5861  -0.572319 -0.068257  0.217745 -0.056509 -0.355174 -0.028610  0.090676  \n",
       "3627  -0.050516 -0.252020 -0.133980  0.155001 -0.154482 -0.060201 -0.126555  \n",
       "12559 -0.153336 -0.123705 -0.238896  0.296447 -0.116798  0.115076 -0.345925  \n",
       "8123  -0.088562  0.321488  0.447437  0.292395 -0.188566 -0.272767  0.126173  \n",
       "210    0.029846 -0.019181  0.107064  0.002301  0.038213 -0.139271 -0.007586  \n",
       "\n",
       "[2928 rows x 768 columns]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "emb_test = embedder.encode(test_df[\"text\"].tolist())\n",
    "emb_test_df = pd.DataFrame(emb_test, index=test_df.index)\n",
    "emb_test_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train accuracy 0.87\n",
      "Test accuracy 0.83\n"
     ]
    }
   ],
   "source": [
    "lgr = LogisticRegression(max_iter=1000)\n",
    "lgr.fit(emb_train, y_train)\n",
    "print(\"Train accuracy {:.2f}\".format(lgr.score(emb_train, y_train)))\n",
    "print(\"Test accuracy {:.2f}\".format(lgr.score(emb_test, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Some improvement over bag of words and average embedding representations! \n",
    "- But much slower ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Break (5 min)\n",
    "\n",
    "![](img/eva-coffee.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Topic modeling "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Topic modeling motivation\n",
    "\n",
    "- Suppose you have a large collection of documents on a variety of topics. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Example: A corpus of news articles \n",
    "\n",
    "![](img/TM_NYT_articles.png)\n",
    "\n",
    "<!-- <center> -->\n",
    "<!-- <img src=\"img/TM_NYT_articles.png\" height=\"2000\" width=\"2000\">  -->\n",
    "<!-- </center> -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Example: A corpus of food magazines \n",
    "\n",
    "![](img/TM_food_magazines.png)\n",
    "\n",
    "<!-- <center> -->\n",
    "<!-- <img src=\"img/TM_food_magazines.png\" height=\"2000\" width=\"2000\">  -->\n",
    "<!-- </center> -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### A corpus of scientific articles\n",
    "\n",
    "![](img/TM_science_articles.png)\n",
    "\n",
    "<!-- <center> -->\n",
    "<!-- <img src=\"img/TM_science_articles.png\" height=\"2000\" width=\"2000\">  -->\n",
    "<!-- </center> -->\n",
    "\n",
    "(Credit: [Dave Blei's presentation](http://www.cs.columbia.edu/~blei/talks/Blei_Science_2008.pdf))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Topic modeling motivation\n",
    "\n",
    "- Humans are pretty good at reading and understanding a document and answering questions such as \n",
    "    - What is it about?  \n",
    "    - Which documents is it related to?     \n",
    "- But for a large collection of documents it would take years to read all documents and organize and categorize them so that they are easy to search.\n",
    "- You need an automated way\n",
    "    - to get an idea of what's going on in the data or \n",
    "    - to pull documents related to a certain topic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Topic modeling\n",
    "\n",
    "- Topic modeling gives you an ability to summarize the major themes in a large collection of documents (corpus). \n",
    "    - Example: The major themes in a collection of news articles could be \n",
    "        - **politics**\n",
    "        - **entertainment**\n",
    "        - **sports**\n",
    "        - **technology**\n",
    "        - ...\n",
    "- A common tool to solve such problems is unsupervised ML methods.\n",
    "- Given the hyperparameter $K$, the idea of topic modeling is to describe the data using $K$ \"topics\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Topic modeling: Input and output\n",
    "\n",
    "- Input\n",
    "    - A large collection of documents\n",
    "    - A value for the hyperparameter $K$ (e.g., $K = 3$)\n",
    "- Output\n",
    "    1. Topic-words association \n",
    "        - For each topic, what words describe that topic? \n",
    "    2. Document-topics association\n",
    "        - For each document, what topics are expressed by the document? \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Topic modeling: Example\n",
    "\n",
    "- Topic-words association \n",
    "    - For each topic, what words describe that topic?  \n",
    "    - A topic is a mixture of words. \n",
    "\n",
    "![](img/topic_modeling_word_topics.png)\n",
    "\n",
    "<!-- <center> -->\n",
    "<!-- <img src=\"img/topic_modeling_word_topics.png\" height=\"1000\" width=\"1000\">  -->\n",
    "<!-- </center> -->    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Topic modeling: Example\n",
    "\n",
    "- Document-topics association \n",
    "    - For each document, what topics are expressed by the document?\n",
    "    - A document is a mixture of topics. \n",
    "    \n",
    "![](img/topic_modeling_doc_topics.png)\n",
    "\n",
    "<!-- <center> -->    \n",
    "<!-- <img src=\"img/topic_modeling_doc_topics.png\" height=\"800\" width=\"800\">  -->\n",
    "<!-- </center> -->    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Topic modeling: Input and output\n",
    "\n",
    "- Input\n",
    "    - A large collection of documents\n",
    "    - A value for the hyperparameter $K$ (e.g., $K = 3$)\n",
    "- Output\n",
    "    - For each topic, what words describe that topic?  \n",
    "    - For each document, what topics are expressed by the document?\n",
    "\n",
    "![](img/topic_modeling_output.png)\n",
    "\n",
    "<!-- <center> -->\n",
    "<!-- <img src=\"img/topic_modeling_output.png\" height=\"800\" width=\"800\">  -->\n",
    "<!-- </center> -->    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Topic modeling: Some applications\n",
    "\n",
    "- Topic modeling is a great EDA tool to get a sense of what's going on in a large corpus. \n",
    "- Some examples\n",
    "    - If you want to pull documents related to a particular lawsuit. \n",
    "    - You want to examine people's sentiment towards a particular candidate and/or political party and so you want to pull tweets or Facebook posts related to election.   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Topic modeling toy example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>doc_id</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>famous fashion model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>fashion model pattern</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>fashion model probabilistic topic model confer...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>famous fashion model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>fresh fashion model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>famous fashion model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7</td>\n",
       "      <td>famous fashion model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>famous fashion model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>9</td>\n",
       "      <td>famous fashion model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>creative fashion model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>11</td>\n",
       "      <td>famous fashion model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>12</td>\n",
       "      <td>famous fashion model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>13</td>\n",
       "      <td>fashion model probabilistic topic model confer...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>14</td>\n",
       "      <td>probabilistic topic model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>15</td>\n",
       "      <td>probabilistic model pattern</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>16</td>\n",
       "      <td>probabilistic topic model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>17</td>\n",
       "      <td>probabilistic topic model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>18</td>\n",
       "      <td>probabilistic topic model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>19</td>\n",
       "      <td>probabilistic topic model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>20</td>\n",
       "      <td>probabilistic topic model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>21</td>\n",
       "      <td>probabilistic topic model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>22</td>\n",
       "      <td>fashion model probabilistic topic model confer...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>23</td>\n",
       "      <td>apple kiwi nutrition</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>24</td>\n",
       "      <td>kiwi health nutrition</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>25</td>\n",
       "      <td>fresh apple health</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>26</td>\n",
       "      <td>probabilistic topic model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>27</td>\n",
       "      <td>creative health nutrition</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>28</td>\n",
       "      <td>probabilistic topic model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>29</td>\n",
       "      <td>probabilistic topic model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>30</td>\n",
       "      <td>hidden markov model probabilistic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>31</td>\n",
       "      <td>probabilistic topic model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>32</td>\n",
       "      <td>probabilistic topic model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>33</td>\n",
       "      <td>apple kiwi nutrition</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>34</td>\n",
       "      <td>apple kiwi health</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>35</td>\n",
       "      <td>apple kiwi nutrition</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>36</td>\n",
       "      <td>fresh kiwi health</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>37</td>\n",
       "      <td>apple kiwi nutrition</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>38</td>\n",
       "      <td>apple kiwi nutrition</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>39</td>\n",
       "      <td>apple kiwi nutrition</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    doc_id                                               text\n",
       "0        1                               famous fashion model\n",
       "1        2                             fashion model pattern \n",
       "2        3  fashion model probabilistic topic model confer...\n",
       "3        4                               famous fashion model\n",
       "4        5                                fresh fashion model\n",
       "5        6                               famous fashion model\n",
       "6        7                               famous fashion model\n",
       "7        8                               famous fashion model\n",
       "8        9                               famous fashion model\n",
       "9       10                             creative fashion model\n",
       "10      11                               famous fashion model\n",
       "11      12                               famous fashion model\n",
       "12      13  fashion model probabilistic topic model confer...\n",
       "13      14                          probabilistic topic model\n",
       "14      15                        probabilistic model pattern\n",
       "15      16                          probabilistic topic model\n",
       "16      17                          probabilistic topic model\n",
       "17      18                          probabilistic topic model\n",
       "18      19                          probabilistic topic model\n",
       "19      20                          probabilistic topic model\n",
       "20      21                          probabilistic topic model\n",
       "21      22  fashion model probabilistic topic model confer...\n",
       "22      23                               apple kiwi nutrition\n",
       "23      24                              kiwi health nutrition\n",
       "24      25                                 fresh apple health\n",
       "25      26                          probabilistic topic model\n",
       "26      27                          creative health nutrition\n",
       "27      28                          probabilistic topic model\n",
       "28      29                          probabilistic topic model\n",
       "29      30                  hidden markov model probabilistic\n",
       "30      31                          probabilistic topic model\n",
       "31      32                          probabilistic topic model\n",
       "32      33                               apple kiwi nutrition\n",
       "33      34                                  apple kiwi health\n",
       "34      35                               apple kiwi nutrition\n",
       "35      36                                  fresh kiwi health\n",
       "36      37                               apple kiwi nutrition\n",
       "37      38                               apple kiwi nutrition\n",
       "38      39                               apple kiwi nutrition"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "toy_df = pd.read_csv(\"data/toy_lda_data.csv\")\n",
    "toy_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from gensim import corpora, matutils, models\n",
    "\n",
    "corpus = [doc.split() for doc in toy_df[\"text\"].tolist()]\n",
    "# Create a vocabulary for the lda model\n",
    "dictionary = corpora.Dictionary(corpus)\n",
    "# Convert our corpus into document-term matrix for Lda\n",
    "doc_term_matrix = [dictionary.doc2bow(doc) for doc in corpus]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from gensim.models import LdaModel\n",
    "\n",
    "# Train an lda model\n",
    "lda = models.LdaModel(\n",
    "    corpus=doc_term_matrix,\n",
    "    id2word=dictionary,\n",
    "    num_topics=3,\n",
    "    random_state=123,\n",
    "    passes=10,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  '0.303*\"model\" + 0.296*\"probabilistic\" + 0.261*\"topic\" + 0.040*\"pattern\"'),\n",
       " (1, '0.245*\"kiwi\" + 0.219*\"apple\" + 0.218*\"nutrition\" + 0.140*\"health\"'),\n",
       " (2, '0.308*\"fashion\" + 0.307*\"model\" + 0.180*\"famous\" + 0.071*\"conference\"')]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### Examine the topics in our LDA model\n",
    "lda.print_topics(num_words=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Document:  ['famous', 'fashion', 'model']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>topic id</th>\n",
       "      <th>probability</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>0.828760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0.087849</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>0.083391</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   topic id  probability\n",
       "2         2     0.828760\n",
       "0         0     0.087849\n",
       "1         1     0.083391"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### Examine the topic distribution for a document\n",
    "print(\"Document: \", corpus[0])\n",
    "df = pd.DataFrame(lda[doc_term_matrix[0]], columns=[\"topic id\", \"probability\"])\n",
    "df.sort_values(\"probability\", ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "You can also visualize the topics using `pyLDAvis`.\n",
    "\n",
    "`conda install -n cpsc330 -c anaconda pyLDAvis`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/past/builtins/misc.py:45: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses\n",
      "  from imp import reload\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n",
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n",
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n",
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n",
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n",
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n",
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n",
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n",
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/past/builtins/misc.py:45: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses\n",
      "  from imp import reload\n",
      "/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/past/builtins/misc.py:45: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses\n",
      "  from imp import reload\n",
      "/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/past/builtins/misc.py:45: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses\n",
      "  from imp import reload\n",
      "/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/past/builtins/misc.py:45: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses\n",
      "  from imp import reload\n",
      "/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/past/builtins/misc.py:45: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses\n",
      "  from imp import reload\n",
      "/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/past/builtins/misc.py:45: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses\n",
      "  from imp import reload\n",
      "/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/past/builtins/misc.py:45: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses\n",
      "  from imp import reload\n",
      "/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/past/builtins/misc.py:45: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses\n",
      "  from imp import reload\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<link rel=\"stylesheet\" type=\"text/css\" href=\"https://cdn.jsdelivr.net/gh/bmabey/pyLDAvis@3.3.1/pyLDAvis/js/ldavis.v1.0.0.css\">\n",
       "\n",
       "\n",
       "<div id=\"ldavis_el222011398051100636647169749180\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "\n",
       "var ldavis_el222011398051100636647169749180_data = {\"mdsDat\": {\"x\": [-0.18796276816150773, 0.31219693667882936, -0.12423416851732189], \"y\": [0.14888014707464084, 0.021739796212790614, -0.1706199432874313], \"topics\": [1, 2, 3], \"cluster\": [1, 1, 1], \"Freq\": [40.125871892588755, 27.322216035712504, 32.55191207169874]}, \"tinfo\": {\"Term\": [\"fashion\", \"probabilistic\", \"kiwi\", \"topic\", \"apple\", \"nutrition\", \"model\", \"famous\", \"health\", \"conference\", \"fresh\", \"creative\", \"pattern\", \"markov\", \"hidden\", \"probabilistic\", \"topic\", \"pattern\", \"markov\", \"hidden\", \"model\", \"creative\", \"conference\", \"fresh\", \"health\", \"famous\", \"nutrition\", \"apple\", \"kiwi\", \"fashion\", \"kiwi\", \"apple\", \"nutrition\", \"health\", \"fresh\", \"creative\", \"hidden\", \"markov\", \"pattern\", \"conference\", \"famous\", \"fashion\", \"topic\", \"probabilistic\", \"model\", \"fashion\", \"famous\", \"conference\", \"model\", \"creative\", \"fresh\", \"markov\", \"hidden\", \"pattern\", \"health\", \"topic\", \"nutrition\", \"apple\", \"probabilistic\", \"kiwi\"], \"Freq\": [13.0, 15.0, 9.0, 14.0, 8.0, 8.0, 28.0, 8.0, 5.0, 3.0, 3.0, 2.0, 2.0, 1.0, 1.0, 15.092422404825797, 13.299502618730482, 2.0264060340990646, 1.1789434383263668, 1.178905185513376, 15.43498384384919, 0.2967537191968916, 0.33843495241705623, 0.29789734067112716, 0.2963123880455245, 0.2973333015197752, 0.29742748602024704, 0.29688620397042004, 0.2980852588014463, 0.3295632185972238, 8.491083928378536, 7.582725740102208, 7.581678179400556, 4.852260322422571, 2.2112535380183216, 1.2235201345432414, 0.304395493793989, 0.30434055631889, 0.3054068280782373, 0.30445747619236535, 0.30536223177492167, 0.30638717116329744, 0.30871571220344796, 0.30785196583024493, 0.30977493903951786, 12.75157176825618, 7.431464169270434, 2.929606637362001, 12.695807696117145, 1.1785066131082638, 1.1014765668986084, 0.2982582721503829, 0.2982430447122111, 0.33553011257220455, 0.2989865325625685, 0.5724222709111533, 0.2993747071052941, 0.29889414763742767, 0.5511730065956952, 0.2996084704197316], \"Total\": [13.0, 15.0, 9.0, 14.0, 8.0, 8.0, 28.0, 8.0, 5.0, 3.0, 3.0, 2.0, 2.0, 1.0, 1.0, 15.951447377251737, 14.180640601845083, 2.6673429747495065, 1.7815422667956398, 1.7815437240195762, 28.440566479005852, 2.6987804668483966, 3.5724990659714226, 3.610627445588057, 5.447559243030663, 8.03415970256513, 8.178480372526097, 8.178506091710055, 9.088777657599714, 13.387522158016703, 9.088777657599714, 8.178506091710055, 8.178480372526097, 5.447559243030663, 3.610627445588057, 2.6987804668483966, 1.7815437240195762, 1.7815422667956398, 2.6673429747495065, 3.5724990659714226, 8.03415970256513, 13.387522158016703, 14.180640601845083, 15.951447377251737, 28.440566479005852, 13.387522158016703, 8.03415970256513, 3.5724990659714226, 28.440566479005852, 2.6987804668483966, 3.610627445588057, 1.7815422667956398, 1.7815437240195762, 2.6673429747495065, 5.447559243030663, 14.180640601845083, 8.178480372526097, 8.178506091710055, 15.951447377251737, 9.088777657599714], \"Category\": [\"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\"], \"logprob\": [15.0, 14.0, 13.0, 12.0, 11.0, 10.0, 9.0, 8.0, 7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0, -1.2168, -1.3433, -3.2248, -3.7664, -3.7665, -1.1944, -5.1459, -5.0145, -5.142, -5.1474, -5.1439, -5.1436, -5.1454, -5.1414, -5.041, -1.4077, -1.5208, -1.521, -1.9673, -2.7532, -3.345, -4.7361, -4.7363, -4.7328, -4.7359, -4.733, -4.7296, -4.7221, -4.7249, -4.7186, -1.1762, -1.7161, -2.647, -1.1806, -3.5576, -3.6252, -4.9316, -4.9317, -4.8139, -4.9292, -4.2797, -4.9279, -4.9295, -4.3176, -4.9271], \"loglift\": [15.0, 14.0, 13.0, 12.0, 11.0, 10.0, 9.0, 8.0, 7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0, 0.8578, 0.849, 0.6383, 0.5003, 0.5003, 0.302, -1.2945, -1.4435, -1.5817, -1.9984, -2.3835, -2.4009, -2.4028, -2.5043, -2.7912, 1.2294, 1.2218, 1.2217, 1.1817, 0.8071, 0.5064, -0.4694, -0.4696, -0.8697, -1.165, -1.9725, -2.4798, -2.5297, -2.6502, -3.2223, 1.0737, 1.0444, 0.9239, 0.3158, 0.2938, -0.0649, -0.6649, -0.665, -0.9508, -1.7802, -2.0874, -2.1852, -2.1868, -2.2429, -2.29]}, \"token.table\": {\"Topic\": [2, 3, 2, 3, 3, 3, 2, 3, 2, 1, 2, 1, 1, 3, 2, 1, 1, 3, 1, 3], \"Freq\": [0.9781737532859462, 0.8397482951291548, 0.3705377344633697, 0.3705377344633697, 0.8712796682103611, 0.9710534814850225, 0.5539203449095433, 0.27696017245477167, 0.917842244009875, 0.5613109498900021, 0.8802063711296407, 0.5613114090178977, 0.5274156550669497, 0.4570935677246897, 0.9781768293868301, 0.749809836580098, 0.9403535394155774, 0.0626902359610385, 0.9167427879321992, 0.07051867599478455], \"Term\": [\"apple\", \"conference\", \"creative\", \"creative\", \"famous\", \"fashion\", \"fresh\", \"fresh\", \"health\", \"hidden\", \"kiwi\", \"markov\", \"model\", \"model\", \"nutrition\", \"pattern\", \"probabilistic\", \"probabilistic\", \"topic\", \"topic\"]}, \"R\": 15, \"lambda.step\": 0.01, \"plot.opts\": {\"xlab\": \"PC1\", \"ylab\": \"PC2\"}, \"topic.order\": [1, 2, 3]};\n",
       "\n",
       "function LDAvis_load_lib(url, callback){\n",
       "  var s = document.createElement('script');\n",
       "  s.src = url;\n",
       "  s.async = true;\n",
       "  s.onreadystatechange = s.onload = callback;\n",
       "  s.onerror = function(){console.warn(\"failed to load library \" + url);};\n",
       "  document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "}\n",
       "\n",
       "if(typeof(LDAvis) !== \"undefined\"){\n",
       "   // already loaded: just create the visualization\n",
       "   !function(LDAvis){\n",
       "       new LDAvis(\"#\" + \"ldavis_el222011398051100636647169749180\", ldavis_el222011398051100636647169749180_data);\n",
       "   }(LDAvis);\n",
       "}else if(typeof define === \"function\" && define.amd){\n",
       "   // require.js is available: use it to load d3/LDAvis\n",
       "   require.config({paths: {d3: \"https://d3js.org/d3.v5\"}});\n",
       "   require([\"d3\"], function(d3){\n",
       "      window.d3 = d3;\n",
       "      LDAvis_load_lib(\"https://cdn.jsdelivr.net/gh/bmabey/pyLDAvis@3.3.1/pyLDAvis/js/ldavis.v3.0.0.js\", function(){\n",
       "        new LDAvis(\"#\" + \"ldavis_el222011398051100636647169749180\", ldavis_el222011398051100636647169749180_data);\n",
       "      });\n",
       "    });\n",
       "}else{\n",
       "    // require.js not available: dynamically load d3 & LDAvis\n",
       "    LDAvis_load_lib(\"https://d3js.org/d3.v5.js\", function(){\n",
       "         LDAvis_load_lib(\"https://cdn.jsdelivr.net/gh/bmabey/pyLDAvis@3.3.1/pyLDAvis/js/ldavis.v3.0.0.js\", function(){\n",
       "                 new LDAvis(\"#\" + \"ldavis_el222011398051100636647169749180\", ldavis_el222011398051100636647169749180_data);\n",
       "            })\n",
       "         });\n",
       "}\n",
       "</script>"
      ],
      "text/plain": [
       "PreparedData(topic_coordinates=              x        y  topics  cluster       Freq\n",
       "topic                                               \n",
       "0     -0.187963  0.14888       1        1  40.125872\n",
       "1      0.312197  0.02174       2        1  27.322216\n",
       "2     -0.124234 -0.17062       3        1  32.551912, topic_info=             Term       Freq      Total Category  logprob  loglift\n",
       "1         fashion  13.000000  13.000000  Default  15.0000  15.0000\n",
       "5   probabilistic  15.000000  15.000000  Default  14.0000  14.0000\n",
       "10           kiwi   9.000000   9.000000  Default  13.0000  13.0000\n",
       "6           topic  14.000000  14.000000  Default  12.0000  12.0000\n",
       "9           apple   8.000000   8.000000  Default  11.0000  11.0000\n",
       "11      nutrition   8.000000   8.000000  Default  10.0000  10.0000\n",
       "2           model  28.000000  28.000000  Default   9.0000   9.0000\n",
       "0          famous   8.000000   8.000000  Default   8.0000   8.0000\n",
       "12         health   5.000000   5.000000  Default   7.0000   7.0000\n",
       "4      conference   3.000000   3.000000  Default   6.0000   6.0000\n",
       "7           fresh   3.000000   3.000000  Default   5.0000   5.0000\n",
       "8        creative   2.000000   2.000000  Default   4.0000   4.0000\n",
       "3         pattern   2.000000   2.000000  Default   3.0000   3.0000\n",
       "14         markov   1.000000   1.000000  Default   2.0000   2.0000\n",
       "13         hidden   1.000000   1.000000  Default   1.0000   1.0000\n",
       "5   probabilistic  15.092422  15.951447   Topic1  -1.2168   0.8578\n",
       "6           topic  13.299503  14.180641   Topic1  -1.3433   0.8490\n",
       "3         pattern   2.026406   2.667343   Topic1  -3.2248   0.6383\n",
       "14         markov   1.178943   1.781542   Topic1  -3.7664   0.5003\n",
       "13         hidden   1.178905   1.781544   Topic1  -3.7665   0.5003\n",
       "2           model  15.434984  28.440566   Topic1  -1.1944   0.3020\n",
       "8        creative   0.296754   2.698780   Topic1  -5.1459  -1.2945\n",
       "4      conference   0.338435   3.572499   Topic1  -5.0145  -1.4435\n",
       "7           fresh   0.297897   3.610627   Topic1  -5.1420  -1.5817\n",
       "12         health   0.296312   5.447559   Topic1  -5.1474  -1.9984\n",
       "0          famous   0.297333   8.034160   Topic1  -5.1439  -2.3835\n",
       "11      nutrition   0.297427   8.178480   Topic1  -5.1436  -2.4009\n",
       "9           apple   0.296886   8.178506   Topic1  -5.1454  -2.4028\n",
       "10           kiwi   0.298085   9.088778   Topic1  -5.1414  -2.5043\n",
       "1         fashion   0.329563  13.387522   Topic1  -5.0410  -2.7912\n",
       "10           kiwi   8.491084   9.088778   Topic2  -1.4077   1.2294\n",
       "9           apple   7.582726   8.178506   Topic2  -1.5208   1.2218\n",
       "11      nutrition   7.581678   8.178480   Topic2  -1.5210   1.2217\n",
       "12         health   4.852260   5.447559   Topic2  -1.9673   1.1817\n",
       "7           fresh   2.211254   3.610627   Topic2  -2.7532   0.8071\n",
       "8        creative   1.223520   2.698780   Topic2  -3.3450   0.5064\n",
       "13         hidden   0.304395   1.781544   Topic2  -4.7361  -0.4694\n",
       "14         markov   0.304341   1.781542   Topic2  -4.7363  -0.4696\n",
       "3         pattern   0.305407   2.667343   Topic2  -4.7328  -0.8697\n",
       "4      conference   0.304457   3.572499   Topic2  -4.7359  -1.1650\n",
       "0          famous   0.305362   8.034160   Topic2  -4.7330  -1.9725\n",
       "1         fashion   0.306387  13.387522   Topic2  -4.7296  -2.4798\n",
       "6           topic   0.308716  14.180641   Topic2  -4.7221  -2.5297\n",
       "5   probabilistic   0.307852  15.951447   Topic2  -4.7249  -2.6502\n",
       "2           model   0.309775  28.440566   Topic2  -4.7186  -3.2223\n",
       "1         fashion  12.751572  13.387522   Topic3  -1.1762   1.0737\n",
       "0          famous   7.431464   8.034160   Topic3  -1.7161   1.0444\n",
       "4      conference   2.929607   3.572499   Topic3  -2.6470   0.9239\n",
       "2           model  12.695808  28.440566   Topic3  -1.1806   0.3158\n",
       "8        creative   1.178507   2.698780   Topic3  -3.5576   0.2938\n",
       "7           fresh   1.101477   3.610627   Topic3  -3.6252  -0.0649\n",
       "14         markov   0.298258   1.781542   Topic3  -4.9316  -0.6649\n",
       "13         hidden   0.298243   1.781544   Topic3  -4.9317  -0.6650\n",
       "3         pattern   0.335530   2.667343   Topic3  -4.8139  -0.9508\n",
       "12         health   0.298987   5.447559   Topic3  -4.9292  -1.7802\n",
       "6           topic   0.572422  14.180641   Topic3  -4.2797  -2.0874\n",
       "11      nutrition   0.299375   8.178480   Topic3  -4.9279  -2.1852\n",
       "9           apple   0.298894   8.178506   Topic3  -4.9295  -2.1868\n",
       "5   probabilistic   0.551173  15.951447   Topic3  -4.3176  -2.2429\n",
       "10           kiwi   0.299608   9.088778   Topic3  -4.9271  -2.2900, token_table=      Topic      Freq           Term\n",
       "term                                \n",
       "9         2  0.978174          apple\n",
       "4         3  0.839748     conference\n",
       "8         2  0.370538       creative\n",
       "8         3  0.370538       creative\n",
       "0         3  0.871280         famous\n",
       "1         3  0.971053        fashion\n",
       "7         2  0.553920          fresh\n",
       "7         3  0.276960          fresh\n",
       "12        2  0.917842         health\n",
       "13        1  0.561311         hidden\n",
       "10        2  0.880206           kiwi\n",
       "14        1  0.561311         markov\n",
       "2         1  0.527416          model\n",
       "2         3  0.457094          model\n",
       "11        2  0.978177      nutrition\n",
       "3         1  0.749810        pattern\n",
       "5         1  0.940354  probabilistic\n",
       "5         3  0.062690  probabilistic\n",
       "6         1  0.916743          topic\n",
       "6         3  0.070519          topic, R=15, lambda_step=0.01, plot_opts={'xlab': 'PC1', 'ylab': 'PC2'}, topic_order=[1, 2, 3])"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Visualize the topics\n",
    "import pyLDAvis\n",
    "\n",
    "pyLDAvis.enable_notebook()\n",
    "import pyLDAvis.gensim_models as gensimvis\n",
    "\n",
    "warnings.simplefilter(action=\"ignore\", category=FutureWarning)\n",
    "\n",
    "vis = gensimvis.prepare(lda, doc_term_matrix, dictionary, sort_topics=False)\n",
    "\n",
    "warnings.simplefilter(action=\"default\", category=FutureWarning)\n",
    "\n",
    "vis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "## Topic modeling pipeline \n",
    "\n",
    "- Preprocess your corpus. \n",
    "- Train LDA using `Gensim`.\n",
    "- Interpret your topics.     "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>wiki query</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Artificial Intelligence</td>\n",
       "      <td>Artificial intelligence (AI) is intelligence d...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>unsupervised learning</td>\n",
       "      <td>Supervised learning (SL) is the machine learni...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Supreme Court of Canada</td>\n",
       "      <td>The Supreme Court of Canada (SCC; French: Cour...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Peace, Order, and Good Government</td>\n",
       "      <td>In many Commonwealth jurisdictions, the phrase...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Canadian constitutional law</td>\n",
       "      <td>Canadian constitutional law (French: droit con...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>ice hockey</td>\n",
       "      <td>Ice hockey (or simply hockey) is a winter team...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          wiki query  \\\n",
       "0            Artificial Intelligence   \n",
       "1              unsupervised learning   \n",
       "2            Supreme Court of Canada   \n",
       "3  Peace, Order, and Good Government   \n",
       "4        Canadian constitutional law   \n",
       "5                         ice hockey   \n",
       "\n",
       "                                                text  \n",
       "0  Artificial intelligence (AI) is intelligence d...  \n",
       "1  Supervised learning (SL) is the machine learni...  \n",
       "2  The Supreme Court of Canada (SCC; French: Cour...  \n",
       "3  In many Commonwealth jurisdictions, the phrase...  \n",
       "4  Canadian constitutional law (French: droit con...  \n",
       "5  Ice hockey (or simply hockey) is a winter team...  "
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import wikipedia\n",
    "\n",
    "queries = [\n",
    "    \"Artificial Intelligence\",\n",
    "    \"unsupervised learning\",\n",
    "    \"Supreme Court of Canada\",\n",
    "    \"Peace, Order, and Good Government\",\n",
    "    \"Canadian constitutional law\",\n",
    "    \"ice hockey\",\n",
    "]\n",
    "wiki_dict = {\"wiki query\": [], \"text\": []}\n",
    "for i in range(len(queries)):\n",
    "    wiki_dict[\"text\"].append(wikipedia.page(queries[i]).content)\n",
    "    wiki_dict[\"wiki query\"].append(queries[i])\n",
    "\n",
    "wiki_df = pd.DataFrame(wiki_dict)\n",
    "wiki_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Preprocessing the corpus \n",
    "\n",
    "- **Preprocessing is crucial!**\n",
    "- Tokenization, converting text to lower case\n",
    "- Removing punctuation and stopwords\n",
    "- Discarding words with length < threshold or word frequency < threshold        \n",
    "- Possibly lemmatization: Consider the lemmas instead of inflected forms. \n",
    "- Depending upon your application, restrict to specific part of speech;\n",
    "    * For example, only consider nouns, verbs, and adjectives\n",
    "    \n",
    "We'll use [`spaCy`](https://spacy.io/) for preprocessing. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import spacy\n",
    "\n",
    "warnings.simplefilter(action=\"ignore\", category=DeprecationWarning)\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_md\", disable=[\"parser\", \"ner\"])\n",
    "\n",
    "warnings.simplefilter(action=\"default\", category=DeprecationWarning)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def preprocess(\n",
    "    doc,\n",
    "    min_token_len=2,\n",
    "    irrelevant_pos=[\"ADV\", \"PRON\", \"CCONJ\", \"PUNCT\", \"PART\", \"DET\", \"ADP\", \"SPACE\"],\n",
    "):\n",
    "    \"\"\"\n",
    "    Given text, min_token_len, and irrelevant_pos carry out preprocessing of the text\n",
    "    and return a preprocessed string.\n",
    "\n",
    "    Parameters\n",
    "    -------------\n",
    "    doc : (spaCy doc object)\n",
    "        the spacy doc object of the text\n",
    "    min_token_len : (int)\n",
    "        min_token_length required\n",
    "    irrelevant_pos : (list)\n",
    "        a list of irrelevant pos tags\n",
    "\n",
    "    Returns\n",
    "    -------------\n",
    "    (str) the preprocessed text\n",
    "    \"\"\"\n",
    "\n",
    "    clean_text = []\n",
    "\n",
    "    for token in doc:\n",
    "        if (\n",
    "            token.is_stop == False  # Check if it's not a stopword\n",
    "            and len(token) > min_token_len  # Check if the word meets minimum threshold\n",
    "            and token.pos_ not in irrelevant_pos\n",
    "        ):  # Check if the POS is in the acceptable POS tags\n",
    "            lemma = token.lemma_  # Take the lemma of the word\n",
    "            clean_text.append(lemma.lower())\n",
    "    return \" \".join(clean_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "wiki_df[\"text_pp\"] = [preprocess(text) for text in nlp.pipe(wiki_df[\"text\"])]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>wiki query</th>\n",
       "      <th>text</th>\n",
       "      <th>text_pp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Artificial Intelligence</td>\n",
       "      <td>Artificial intelligence (AI) is intelligence d...</td>\n",
       "      <td>artificial intelligence intelligence demonstra...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>unsupervised learning</td>\n",
       "      <td>Supervised learning (SL) is the machine learni...</td>\n",
       "      <td>supervised learning machine learn task learn f...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Supreme Court of Canada</td>\n",
       "      <td>The Supreme Court of Canada (SCC; French: Cour...</td>\n",
       "      <td>supreme court canada scc french cour suprême c...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Peace, Order, and Good Government</td>\n",
       "      <td>In many Commonwealth jurisdictions, the phrase...</td>\n",
       "      <td>commonwealth jurisdiction phrase peace order g...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Canadian constitutional law</td>\n",
       "      <td>Canadian constitutional law (French: droit con...</td>\n",
       "      <td>canadian constitutional law french droit const...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>ice hockey</td>\n",
       "      <td>Ice hockey (or simply hockey) is a winter team...</td>\n",
       "      <td>ice hockey hockey winter team sport play ice s...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          wiki query  \\\n",
       "0            Artificial Intelligence   \n",
       "1              unsupervised learning   \n",
       "2            Supreme Court of Canada   \n",
       "3  Peace, Order, and Good Government   \n",
       "4        Canadian constitutional law   \n",
       "5                         ice hockey   \n",
       "\n",
       "                                                text  \\\n",
       "0  Artificial intelligence (AI) is intelligence d...   \n",
       "1  Supervised learning (SL) is the machine learni...   \n",
       "2  The Supreme Court of Canada (SCC; French: Cour...   \n",
       "3  In many Commonwealth jurisdictions, the phrase...   \n",
       "4  Canadian constitutional law (French: droit con...   \n",
       "5  Ice hockey (or simply hockey) is a winter team...   \n",
       "\n",
       "                                             text_pp  \n",
       "0  artificial intelligence intelligence demonstra...  \n",
       "1  supervised learning machine learn task learn f...  \n",
       "2  supreme court canada scc french cour suprême c...  \n",
       "3  commonwealth jurisdiction phrase peace order g...  \n",
       "4  canadian constitutional law french droit const...  \n",
       "5  ice hockey hockey winter team sport play ice s...  "
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wiki_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Training LDA with [gensim](https://radimrehurek.com/gensim/models/ldamodel.html)\n",
    "\n",
    "To train an LDA model with [gensim](https://radimrehurek.com/gensim/models/ldamodel.html), you need\n",
    "\n",
    "- Document-term matrix \n",
    "- Dictionary (vocabulary)\n",
    "- The number of topics ($K$): `num_topics`\n",
    "- The number of passes: `passes`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### `Gensim`'s `doc2bow`\n",
    "- Let's first create a dictionary using [`corpora.Dictionary`](https://radimrehurek.com/gensim/corpora/dictionary.html). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Word</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0026</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0030</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0036</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0040</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4976</th>\n",
       "      <td>zhenskaya</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4977</th>\n",
       "      <td>мячом</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4978</th>\n",
       "      <td>русский</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4979</th>\n",
       "      <td>хоккей</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4980</th>\n",
       "      <td>шайбой</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4981 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           Word\n",
       "0          0026\n",
       "1          0030\n",
       "2          0036\n",
       "3          0040\n",
       "4           007\n",
       "...         ...\n",
       "4976  zhenskaya\n",
       "4977      мячом\n",
       "4978    русский\n",
       "4979     хоккей\n",
       "4980     шайбой\n",
       "\n",
       "[4981 rows x 1 columns]"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus = [doc.split() for doc in wiki_df[\"text_pp\"].tolist()]\n",
    "dictionary = corpora.Dictionary(corpus)  # Create a vocabulary for the lda model\n",
    "pd.DataFrame(\n",
    "    dictionary.token2id.keys(), index=dictionary.token2id.values(), columns=[\"Word\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### `Gensim`'s `doc2bow`\n",
    "- Now let's convert our corpus into document-term matrix for LDA using [`dictionary.doc2bow`](https://radimrehurek.com/gensim/corpora/dictionary.html#gensim.corpora.dictionary.Dictionary.doc2bow).\n",
    "- For each document, it stores the **frequency of each token** in the document in the format (**token_id**, **frequency**). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(292, 4),\n",
       " (305, 3),\n",
       " (312, 1),\n",
       " (318, 3),\n",
       " (319, 1),\n",
       " (327, 1),\n",
       " (329, 3),\n",
       " (337, 1),\n",
       " (376, 52),\n",
       " (386, 1),\n",
       " (400, 4),\n",
       " (402, 1),\n",
       " (403, 1),\n",
       " (410, 1),\n",
       " (426, 2),\n",
       " (428, 4),\n",
       " (430, 3),\n",
       " (455, 1),\n",
       " (466, 1),\n",
       " (478, 2)]"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc_term_matrix = [dictionary.doc2bow(doc) for doc in corpus]\n",
    "doc_term_matrix[1][:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now we are ready to train an LDA model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from gensim.models import LdaModel\n",
    "\n",
    "num_topics = 3\n",
    "\n",
    "lda = models.LdaModel(\n",
    "    corpus=doc_term_matrix,\n",
    "    id2word=dictionary,\n",
    "    num_topics=num_topics,\n",
    "    random_state=42,\n",
    "    passes=10,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Examine the topics and topic distribution for a document in our LDA model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0, '0.034*\"hockey\" + 0.021*\"ice\" + 0.017*\"team\" + 0.016*\"player\"'),\n",
       " (1, '0.027*\"court\" + 0.015*\"law\" + 0.011*\"provincial\" + 0.010*\"canada\"'),\n",
       " (2,\n",
       "  '0.013*\"intelligence\" + 0.011*\"artificial\" + 0.008*\"learning\" + 0.007*\"algorithm\"')]"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda.print_topics(num_words=4)  # Topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Document:  Artificial Intelligence\n",
      "Topic assignment for document:  [(2, 0.9999198)]\n"
     ]
    }
   ],
   "source": [
    "print(\"Document: \", wiki_df.iloc[0][0])\n",
    "print(\"Topic assignment for document: \", lda[doc_term_matrix[0]])  # Topic distribution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Visualize topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<link rel=\"stylesheet\" type=\"text/css\" href=\"https://cdn.jsdelivr.net/gh/bmabey/pyLDAvis@3.3.1/pyLDAvis/js/ldavis.v1.0.0.css\">\n",
       "\n",
       "\n",
       "<div id=\"ldavis_el222011398002304130249566965779\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "\n",
       "var ldavis_el222011398002304130249566965779_data = {\"mdsDat\": {\"x\": [-0.18189962159915687, 0.05478601034019517, 0.12711361125896165], \"y\": [0.03530676412606142, -0.15084500502936543, 0.11553824090330402], \"topics\": [1, 2, 3], \"cluster\": [1, 1, 1], \"Freq\": [29.857844551969038, 20.420416479273687, 49.72173896875728]}, \"tinfo\": {\"Term\": [\"hockey\", \"court\", \"ice\", \"team\", \"player\", \"league\", \"play\", \"game\", \"law\", \"intelligence\", \"provincial\", \"puck\", \"federal\", \"government\", \"canada\", \"power\", \"penalty\", \"artificial\", \"justice\", \"supreme\", \"nhl\", \"learning\", \"algorithm\", \"matter\", \"case\", \"retrieve\", \"sport\", \"stick\", \"archive\", \"act\", \"hockey\", \"ice\", \"league\", \"team\", \"puck\", \"nhl\", \"player\", \"sport\", \"stick\", \"minute\", \"minor\", \"championship\", \"tournament\", \"iihf\", \"amateur\", \"cup\", \"season\", \"rink\", \"club\", \"opponent\", \"goaltender\", \"skate\", \"woman\", \"award\", \"junior\", \"ball\", \"montreal\", \"women\", \"overtime\", \"check\", \"play\", \"professional\", \"game\", \"penalty\", \"score\", \"line\", \"rule\", \"body\", \"major\", \"canada\", \"goal\", \"national\", \"world\", \"united\", \"states\", \"provincial\", \"justice\", \"parliament\", \"court\", \"constitution\", \"constitutional\", \"appeal\", \"doctrine\", \"province\", \"charter\", \"peace\", \"legislation\", \"branch\", \"supreme\", \"chief\", \"legislative\", \"hear\", \"pogg\", \"appointment\", \"legislature\", \"pith\", \"substance\", \"clause\", \"residual\", \"phrase\", \"civil\", \"enact\", \"section\", \"purpose\", \"immunity\", \"federal\", \"law\", \"matter\", \"judge\", \"government\", \"power\", \"act\", \"case\", \"jurisdiction\", \"order\", \"canada\", \"canadian\", \"good\", \"concern\", \"reference\", \"right\", \"year\", \"algorithm\", \"\\\\displaystyle\", \"intelligence\", \"retrieve\", \"datum\", \"training\", \"archive\", \"learning\", \"input\", \"network\", \"2015\", \"machine\", \"computer\", \"neural\", \"bias\", \"artificial\", \"science\", \"search\", \"logic\", \"output\", \"agent\", \"method\", \"supervised\", \"mind\", \"s2cid\", \"intelligent\", \"model\", \"problem\", \"original\", \"variance\", \"human\", \"2020\", \"learn\", \"2018\", \"function\", \"example\", \"research\", \"october\", \"2016\", \"2019\", \"pdf\", \"risk\", \"use\", \"system\", \"include\"], \"Freq\": [219.0, 119.0, 132.0, 106.0, 106.0, 101.0, 103.0, 101.0, 70.0, 141.0, 48.0, 61.0, 49.0, 53.0, 90.0, 54.0, 67.0, 118.0, 38.0, 37.0, 44.0, 88.0, 78.0, 31.0, 53.0, 72.0, 35.0, 35.0, 67.0, 39.0, 218.34595662575936, 131.80050021880837, 100.28426519559885, 104.89583369817159, 60.84594348327314, 43.57654757090638, 104.20002823678135, 34.898180463492395, 34.88678299927923, 26.2443653690032, 22.30949550796412, 21.52359377003833, 21.5231869283256, 20.74178993052338, 19.958713973624064, 19.939122186653396, 17.59235010687359, 17.58592023893543, 17.57595114290935, 16.796277326129534, 16.002622010635037, 15.996442733606896, 34.915559680134045, 14.436309423841154, 13.641224266095394, 12.861114126319315, 12.081273740287545, 12.080776980660044, 12.077856859526278, 12.067214853203598, 97.8626902551153, 28.603098005511622, 94.7377390596307, 60.92136839905939, 19.945740734806154, 22.30690262907755, 30.185715216162443, 23.89509917263665, 22.315990234726044, 42.957750170228024, 31.85389890413425, 27.877660906044174, 27.902209970840612, 21.581363819181917, 20.7495744634398, 47.90488593421719, 37.906340375450185, 20.860899071163765, 115.41562249410345, 20.15521445856433, 18.02795215544799, 18.018818371547002, 16.61141720081108, 15.188431133240124, 15.184579015327797, 15.18162716993463, 15.169156429663632, 14.472312231084924, 35.799819625311656, 12.32418840586448, 10.2002933034357, 10.200584657028127, 9.486625199146207, 9.477603319220162, 8.770509321422761, 8.0688789595913, 8.065250145556906, 8.06253470991261, 7.354207248031008, 7.352838188589778, 7.352884563210026, 7.349832407496974, 17.3213783599125, 6.643134610259517, 6.639245695094166, 45.06713823716184, 62.85531747562573, 28.712709018886002, 13.74682968821533, 45.12433629051798, 45.152540124780025, 28.773721888829485, 34.406915564122784, 13.070669578411085, 18.757515664899397, 45.19853971549022, 20.91930488093418, 18.7805376394207, 12.35218052990365, 13.76540574805635, 15.262022614853878, 13.826239168382351, 77.26933689968031, 57.4988098814116, 139.56192598557854, 70.90026720428057, 42.667946845598934, 37.53333779285487, 66.51385583882431, 86.8580048479375, 32.28955449309434, 37.34791463159837, 35.601499798740086, 70.18545388482691, 27.878809360113095, 27.84039278977453, 21.926857702937667, 115.55452993188717, 25.271436228141297, 26.95269753959854, 24.40863442459811, 20.199217894886704, 24.394041037335313, 19.330668334190307, 18.482097844802894, 22.677210645737123, 23.490097909902392, 21.812901636175543, 18.45187026351541, 49.39242750285424, 76.90823095742951, 15.882293881817606, 58.0436280360674, 36.4513700692134, 43.532808170069536, 33.88421773628296, 45.290352762027815, 36.586179092207225, 45.99295623989272, 35.589211405697604, 34.74091216429885, 34.67160032531611, 26.13701433462716, 28.743778692832574, 40.8024578278216, 32.168305414800216, 29.620989778668882], \"Total\": [219.0, 119.0, 132.0, 106.0, 106.0, 101.0, 103.0, 101.0, 70.0, 141.0, 48.0, 61.0, 49.0, 53.0, 90.0, 54.0, 67.0, 118.0, 38.0, 37.0, 44.0, 88.0, 78.0, 31.0, 53.0, 72.0, 35.0, 35.0, 67.0, 39.0, 219.9647549967664, 132.90688014011405, 101.26834449105603, 106.02871738420734, 61.712477176227026, 44.30602485808333, 106.09412933380392, 35.60463269671583, 35.606212391166565, 26.89879757029977, 22.944442859152502, 22.152485280712572, 22.154156959750644, 21.361721651323805, 20.5731108212211, 20.572082407941153, 18.198017609529394, 18.19735809192561, 18.19721603827684, 17.40610631432233, 16.614722462157644, 16.614280689741513, 36.322160387476416, 15.033355309768897, 14.242522881154684, 13.450528753389467, 12.660075760890766, 12.660456763216779, 12.658403507265236, 12.658884082806196, 103.08793592593076, 30.14300697511337, 101.81147701650903, 67.7981530509545, 21.44170160372487, 24.682065362156322, 36.32107617950113, 29.58845719402961, 26.979183847987354, 90.2921861461199, 56.76492089038361, 50.34613341827904, 52.50101185076804, 35.5931497654114, 32.19173885541939, 48.69678625406161, 38.69534787189961, 21.528968480123485, 119.17265764291496, 20.812815754746993, 18.666544444134946, 18.66846642306547, 17.23523853877466, 15.80554940713279, 15.806007351542666, 15.806270119505996, 15.808365778186989, 15.090704477389549, 37.338301567765846, 12.948687301656046, 10.801388718061109, 10.801817748726933, 10.086603941157158, 10.088888369371205, 9.372214256845174, 8.655876072257323, 8.656091327593584, 8.656870057693677, 7.941120035754978, 7.941577710882141, 7.941815819425999, 7.9420531513413595, 18.818428291283425, 7.2263986124019866, 7.226943824338176, 49.23503903211343, 70.67012422600048, 31.917564186486803, 15.169644048127987, 53.495253554972734, 54.202799546405245, 39.42305963373492, 53.461283605817485, 16.26559683369975, 26.25632369074085, 90.2921861461199, 35.844493488716324, 31.07948206446209, 16.420795793404324, 20.93987848775911, 32.713090609600506, 36.59470033679587, 78.16346244489674, 58.1902350015363, 141.50302635275276, 72.0337203981493, 43.390675956666286, 38.1827985642091, 67.67343214221111, 88.46060640730543, 32.95929075074888, 38.15512439110885, 36.41333520432189, 71.90293458563603, 28.597925743541158, 28.592512085752457, 22.527541475562217, 118.77229473696742, 25.987052609089368, 27.717701021791736, 25.12040328415967, 20.789553333588074, 25.114470429012414, 19.920272040723813, 19.05194226242539, 23.378689607245057, 24.24081216244732, 22.512758540358977, 19.047296887197067, 51.029980395151696, 79.61349107009632, 16.443163031672377, 60.44057174428159, 38.06995471043545, 45.765533579646394, 35.46477576589281, 48.22133077199998, 38.72957332799979, 49.78197717280105, 38.552328315141565, 38.553808127448654, 38.69427253121965, 27.647697116022186, 31.83695907496336, 58.495676765513814, 44.574691552934055, 52.81873127056808], \"Category\": [\"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\"], \"logprob\": [30.0, 29.0, 28.0, 27.0, 26.0, 25.0, 24.0, 23.0, 22.0, 21.0, 20.0, 19.0, 18.0, 17.0, 16.0, 15.0, 14.0, 13.0, 12.0, 11.0, 10.0, 9.0, 8.0, 7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0, -3.3671, -3.8719, -4.1452, -4.1003, -4.6449, -4.9787, -4.1069, -5.2008, -5.2011, -5.4858, -5.6482, -5.6841, -5.6841, -5.7211, -5.7596, -5.7605, -5.8858, -5.8861, -5.8867, -5.9321, -5.9805, -5.9809, -5.2003, -6.0835, -6.1401, -6.199, -6.2616, -6.2616, -6.2618, -6.2627, -4.1697, -5.3997, -4.2021, -4.6436, -5.7602, -5.6483, -5.3459, -5.5795, -5.6479, -4.993, -5.2921, -5.4254, -5.4245, -5.6814, -5.7207, -4.5041, -4.7382, -5.3354, -3.6248, -5.3698, -5.4814, -5.4819, -5.5632, -5.6528, -5.653, -5.6532, -5.654, -5.7011, -4.7954, -5.8617, -6.0509, -6.0509, -6.1234, -6.1244, -6.2019, -6.2853, -6.2857, -6.2861, -6.378, -6.3782, -6.3782, -6.3786, -5.5214, -6.4797, -6.4803, -4.5652, -4.2325, -5.016, -5.7525, -4.5639, -4.5633, -5.0138, -4.8351, -5.8029, -5.4417, -4.5622, -5.3326, -5.4405, -5.8595, -5.7512, -5.6479, -5.7467, -4.9159, -5.2115, -4.3247, -5.0019, -5.5098, -5.638, -5.0658, -4.7989, -5.7885, -5.6429, -5.6908, -5.0121, -5.9354, -5.9367, -6.1755, -4.5135, -6.0335, -5.9691, -6.0683, -6.2576, -6.0689, -6.3015, -6.3464, -6.1419, -6.1066, -6.1807, -6.3481, -5.3634, -4.9206, -6.498, -5.202, -5.6672, -5.4897, -5.7403, -5.4501, -5.6635, -5.4347, -5.6912, -5.7153, -5.7173, -5.9999, -5.9048, -5.5545, -5.7922, -5.8747], \"loglift\": [30.0, 29.0, 28.0, 27.0, 26.0, 25.0, 24.0, 23.0, 22.0, 21.0, 20.0, 19.0, 18.0, 17.0, 16.0, 15.0, 14.0, 13.0, 12.0, 11.0, 10.0, 9.0, 8.0, 7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0, 1.2013, 1.2004, 1.199, 1.198, 1.1946, 1.1921, 1.1907, 1.1887, 1.1883, 1.1841, 1.1807, 1.1799, 1.1798, 1.1793, 1.1784, 1.1775, 1.1749, 1.1745, 1.174, 1.1731, 1.1712, 1.1708, 1.1692, 1.1682, 1.1656, 1.1639, 1.1619, 1.1619, 1.1618, 1.1609, 1.1567, 1.1563, 1.1367, 1.1018, 1.1364, 1.1075, 1.0237, 0.995, 1.019, 0.4659, 0.631, 0.6176, 0.5766, 0.7084, 0.7695, 1.5722, 1.568, 1.5571, 1.5566, 1.5565, 1.5538, 1.5532, 1.5518, 1.5488, 1.5485, 1.5483, 1.5474, 1.5468, 1.5466, 1.5392, 1.5314, 1.5314, 1.5273, 1.5261, 1.5223, 1.5184, 1.5179, 1.5175, 1.5119, 1.5116, 1.5116, 1.5111, 1.5057, 1.5045, 1.5038, 1.5002, 1.4714, 1.4828, 1.4901, 1.4185, 1.4059, 1.2737, 1.1479, 1.37, 1.2523, 0.8966, 1.0501, 1.0849, 1.3039, 1.1691, 0.8262, 0.6153, 0.6872, 0.6868, 0.6849, 0.6829, 0.6819, 0.6816, 0.6814, 0.6804, 0.6782, 0.6773, 0.6762, 0.6746, 0.6733, 0.6721, 0.6717, 0.6713, 0.6708, 0.6707, 0.67, 0.6699, 0.6696, 0.6687, 0.6684, 0.6683, 0.6673, 0.6671, 0.667, 0.6661, 0.6642, 0.664, 0.6583, 0.6553, 0.6487, 0.6531, 0.636, 0.6418, 0.6196, 0.6188, 0.5946, 0.589, 0.6425, 0.5965, 0.3385, 0.3725, 0.1203]}, \"token.table\": {\"Topic\": [3, 1, 2, 3, 1, 3, 1, 2, 3, 1, 3, 3, 1, 2, 3, 3, 2, 3, 1, 2, 2, 2, 3, 1, 2, 3, 1, 1, 3, 1, 2, 3, 2, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 1, 2, 2, 2, 1, 3, 2, 3, 2, 2, 1, 2, 3, 1, 3, 2, 2, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 3, 1, 1, 2, 3, 1, 2, 3, 2, 1, 2, 3, 2, 3, 1, 2, 1, 2, 1, 2, 3, 3, 1, 2, 3, 3, 1, 2, 1, 2, 3, 2, 2, 3, 1, 2, 1, 2, 3, 2, 3, 2, 2, 2, 1, 3, 3, 2, 3, 1, 2, 3, 1, 2, 3, 3, 3, 1, 1, 3, 1, 1, 2, 3, 3, 3, 1, 2, 3, 1, 1, 2, 3, 1, 2, 3, 3, 1, 2, 1, 3, 2, 1, 3, 2, 2, 1, 2, 3, 1, 2, 3, 2, 1, 2, 3, 2, 3, 1, 3, 2, 2, 1, 2, 1, 2, 3, 1, 2, 3, 2, 2, 3, 1, 2, 3, 1, 1, 3, 1, 2, 3, 3, 3, 1, 3, 3, 1, 2, 3, 1, 1, 1, 2, 3, 1, 2, 3, 1, 2, 1, 2, 3, 1, 2, 3, 1, 3, 1, 2, 3, 1, 2, 3, 3, 1, 2, 1, 1, 2, 3, 1, 2, 3], \"Freq\": [0.9886487957776295, 0.05187554996872244, 0.05187554996872244, 0.9078221244526427, 0.028196992040810255, 0.9586977293875487, 0.10337447219798963, 0.02584361804949741, 0.9045266317324093, 0.02626743340269558, 0.9456276024970409, 0.9795457949000399, 0.1014634591318513, 0.7356100787059219, 0.15219518869777696, 0.9556243707323022, 0.012793701413943563, 0.9851150088736543, 0.9721427242481027, 0.9641927511389173, 0.8920705305178166, 0.014776847698496628, 0.9900487957992741, 0.016838943833064696, 0.008419471916532348, 0.9766587423177523, 0.931262496729695, 0.9665047551921752, 0.9765823769036452, 0.8111271176667754, 0.03379696323611564, 0.1689848161805782, 0.9277234221222902, 0.4762316855460014, 0.4983819965016294, 0.022150310955627972, 0.39057603099921423, 0.5858640464988214, 0.027898287928515304, 0.1496410011212201, 0.6359742547651854, 0.20575637654167764, 0.9931165610187613, 0.9490062649209131, 0.9479508558182377, 0.9267348666660045, 0.881410518596732, 0.9241215296849821, 0.9891622961522243, 0.979091989086789, 0.7307806607533597, 0.2435935535844532, 0.9609463820597361, 0.9642920281185533, 0.008391186533712852, 0.964986451376978, 0.016782373067425704, 0.9721913223660663, 0.9909963154974482, 0.9863513035665021, 0.8813841794571404, 0.05164012479719438, 0.9553423087480961, 0.02031073844275319, 0.9139832299238936, 0.06093221532825957, 0.020737710552373563, 0.041475421104747126, 0.9331969748568103, 0.9330971594154898, 0.009822075362268314, 0.0687545275358782, 0.5637284347104768, 0.42279632603285766, 0.9630013403138235, 0.16087784183885298, 0.6113357989876413, 0.22522897857439417, 0.03738649444748892, 0.8411961250685007, 0.11215948334246675, 0.9257701094964843, 0.9910678645004047, 0.004546182864680755, 0.004546182864680755, 0.03309035540665983, 0.9596203067931353, 0.9931765749135184, 0.007524064961466049, 0.9830668306034505, 0.968597538620143, 0.340788193260334, 0.09466338701675944, 0.5679803221005566, 0.9708946785899184, 0.007066986662936105, 0.007066986662936105, 0.9893781328110548, 0.977223646785011, 0.06592112490097651, 0.9228957486136712, 0.9829719156375318, 0.7992328921534593, 0.18443835972772135, 0.9820301945804559, 0.8914658165666767, 0.09905175739629742, 0.987475410036272, 0.009874754100362721, 0.02185050455622212, 0.02185050455622212, 0.9614222004737734, 0.011304466932948991, 0.9834886231665623, 0.948864684083765, 0.9258068810428887, 0.9602853448881282, 0.8913354566239587, 0.08103049605672352, 0.9553986744764495, 0.013907638203681452, 0.9735346742577016, 0.8154434961397544, 0.07413122692179586, 0.11119684038269378, 0.03133071164695513, 0.9085906377616988, 0.06266142329391026, 0.9538022352886314, 0.9838019318615829, 0.9588378386457196, 0.9665859573108884, 0.9450159834542704, 0.9478616263158662, 0.5561499582773146, 0.2383499821188491, 0.21848748360894502, 0.9697255765891822, 0.9792773687049449, 0.9930929290302264, 0.07781631177958555, 0.9337957413550265, 0.9766687444630755, 0.03808606306726195, 0.7236351982779771, 0.2285163784035717, 0.012560685212504275, 0.012560685212504275, 0.9671727613628293, 0.9620216307239052, 0.9479868447164488, 0.9754299198955187, 0.03616937771719466, 0.9404038206470612, 0.9489904883688528, 0.8997295244039278, 0.08849798600694372, 0.881436945508709, 0.9242276498898302, 0.9506447007573565, 0.009700456130177108, 0.03880182452070843, 0.980261590844342, 0.009425592219657134, 0.009425592219657134, 0.8922725679033155, 0.07379692623764637, 0.8302154201735217, 0.09224615779705796, 0.019596323421182598, 0.9602198476379473, 0.9620805258062987, 0.03317519054504478, 0.9490337610934765, 0.9856913298050857, 0.9884548926112224, 0.9686706166452761, 0.19102307600964794, 0.6685807660337677, 0.14326730700723594, 0.020087591067924907, 0.060262773203774725, 0.9240291891245458, 0.8814877458699056, 0.013882387227436501, 0.9856494931479916, 0.15284401158148658, 0.45853203474445975, 0.3668256277955678, 0.9891545744756662, 0.09423010510947967, 0.9108910160583035, 0.8259667156264325, 0.08259667156264325, 0.08259667156264325, 0.9488130944568959, 0.9620175237285612, 0.9327617914674078, 0.04663808957337039, 0.9741067622734123, 0.9891187263482094, 0.903369810531642, 0.05313940061950836, 0.963026946443682, 0.9830181453670325, 0.6523412759502026, 0.06212774056668597, 0.27957483255008686, 0.9829745330812845, 0.9242046666603297, 0.9447855631758842, 0.026782150178552844, 0.9641574064279024, 0.2243425507078302, 0.06730276521234907, 0.7178961622650567, 0.9902977475387195, 0.009431407119416375, 0.009431407119416375, 0.9930416237444416, 0.9952125414824766, 0.6180964636453471, 0.05619058760412246, 0.33714352562473476, 0.20514336551918674, 0.08547640229966115, 0.7009064988572214, 0.9730487965837978, 0.9635990708324639, 0.027531402023784683, 0.9478331014773775, 0.5333230544125291, 0.038094503886609225, 0.4380867946960061, 0.3552426958099323, 0.3825690570260809, 0.27326361216148637], \"Term\": [\"2015\", \"2016\", \"2016\", \"2016\", \"2018\", \"2018\", \"2019\", \"2019\", \"2019\", \"2020\", \"2020\", \"\\\\displaystyle\", \"act\", \"act\", \"act\", \"agent\", \"algorithm\", \"algorithm\", \"amateur\", \"appeal\", \"appointment\", \"archive\", \"archive\", \"artificial\", \"artificial\", \"artificial\", \"award\", \"ball\", \"bias\", \"body\", \"body\", \"body\", \"branch\", \"canada\", \"canada\", \"canada\", \"canadian\", \"canadian\", \"canadian\", \"case\", \"case\", \"case\", \"championship\", \"charter\", \"check\", \"chief\", \"civil\", \"clause\", \"club\", \"computer\", \"concern\", \"concern\", \"constitution\", \"constitutional\", \"court\", \"court\", \"court\", \"cup\", \"datum\", \"doctrine\", \"enact\", \"example\", \"example\", \"federal\", \"federal\", \"federal\", \"function\", \"function\", \"function\", \"game\", \"game\", \"game\", \"goal\", \"goal\", \"goaltender\", \"good\", \"good\", \"good\", \"government\", \"government\", \"government\", \"hear\", \"hockey\", \"hockey\", \"hockey\", \"human\", \"human\", \"ice\", \"ice\", \"iihf\", \"immunity\", \"include\", \"include\", \"include\", \"input\", \"intelligence\", \"intelligence\", \"intelligence\", \"intelligent\", \"judge\", \"judge\", \"junior\", \"jurisdiction\", \"jurisdiction\", \"justice\", \"law\", \"law\", \"league\", \"league\", \"learn\", \"learn\", \"learn\", \"learning\", \"learning\", \"legislation\", \"legislative\", \"legislature\", \"line\", \"line\", \"logic\", \"machine\", \"machine\", \"major\", \"major\", \"major\", \"matter\", \"matter\", \"matter\", \"method\", \"mind\", \"minor\", \"minute\", \"model\", \"montreal\", \"national\", \"national\", \"national\", \"network\", \"neural\", \"nhl\", \"october\", \"october\", \"opponent\", \"order\", \"order\", \"order\", \"original\", \"original\", \"original\", \"output\", \"overtime\", \"parliament\", \"pdf\", \"pdf\", \"peace\", \"penalty\", \"penalty\", \"phrase\", \"pith\", \"play\", \"play\", \"play\", \"player\", \"player\", \"player\", \"pogg\", \"power\", \"power\", \"power\", \"problem\", \"problem\", \"professional\", \"professional\", \"province\", \"provincial\", \"puck\", \"purpose\", \"reference\", \"reference\", \"reference\", \"research\", \"research\", \"research\", \"residual\", \"retrieve\", \"retrieve\", \"right\", \"right\", \"right\", \"rink\", \"risk\", \"risk\", \"rule\", \"rule\", \"rule\", \"s2cid\", \"science\", \"score\", \"score\", \"search\", \"season\", \"section\", \"section\", \"skate\", \"sport\", \"states\", \"states\", \"states\", \"stick\", \"substance\", \"supervised\", \"supreme\", \"supreme\", \"system\", \"system\", \"system\", \"team\", \"team\", \"team\", \"tournament\", \"training\", \"united\", \"united\", \"united\", \"use\", \"use\", \"use\", \"variance\", \"woman\", \"woman\", \"women\", \"world\", \"world\", \"world\", \"year\", \"year\", \"year\"]}, \"R\": 30, \"lambda.step\": 0.01, \"plot.opts\": {\"xlab\": \"PC1\", \"ylab\": \"PC2\"}, \"topic.order\": [1, 2, 3]};\n",
       "\n",
       "function LDAvis_load_lib(url, callback){\n",
       "  var s = document.createElement('script');\n",
       "  s.src = url;\n",
       "  s.async = true;\n",
       "  s.onreadystatechange = s.onload = callback;\n",
       "  s.onerror = function(){console.warn(\"failed to load library \" + url);};\n",
       "  document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "}\n",
       "\n",
       "if(typeof(LDAvis) !== \"undefined\"){\n",
       "   // already loaded: just create the visualization\n",
       "   !function(LDAvis){\n",
       "       new LDAvis(\"#\" + \"ldavis_el222011398002304130249566965779\", ldavis_el222011398002304130249566965779_data);\n",
       "   }(LDAvis);\n",
       "}else if(typeof define === \"function\" && define.amd){\n",
       "   // require.js is available: use it to load d3/LDAvis\n",
       "   require.config({paths: {d3: \"https://d3js.org/d3.v5\"}});\n",
       "   require([\"d3\"], function(d3){\n",
       "      window.d3 = d3;\n",
       "      LDAvis_load_lib(\"https://cdn.jsdelivr.net/gh/bmabey/pyLDAvis@3.3.1/pyLDAvis/js/ldavis.v3.0.0.js\", function(){\n",
       "        new LDAvis(\"#\" + \"ldavis_el222011398002304130249566965779\", ldavis_el222011398002304130249566965779_data);\n",
       "      });\n",
       "    });\n",
       "}else{\n",
       "    // require.js not available: dynamically load d3 & LDAvis\n",
       "    LDAvis_load_lib(\"https://d3js.org/d3.v5.js\", function(){\n",
       "         LDAvis_load_lib(\"https://cdn.jsdelivr.net/gh/bmabey/pyLDAvis@3.3.1/pyLDAvis/js/ldavis.v3.0.0.js\", function(){\n",
       "                 new LDAvis(\"#\" + \"ldavis_el222011398002304130249566965779\", ldavis_el222011398002304130249566965779_data);\n",
       "            })\n",
       "         });\n",
       "}\n",
       "</script>"
      ],
      "text/plain": [
       "PreparedData(topic_coordinates=              x         y  topics  cluster       Freq\n",
       "topic                                                \n",
       "0     -0.181900  0.035307       1        1  29.857845\n",
       "1      0.054786 -0.150845       2        1  20.420416\n",
       "2      0.127114  0.115538       3        1  49.721739, topic_info=         Term        Freq       Total Category  logprob  loglift\n",
       "4465   hockey  219.000000  219.000000  Default  30.0000  30.0000\n",
       "855     court  119.000000  119.000000  Default  29.0000  29.0000\n",
       "4480      ice  132.000000  132.000000  Default  28.0000  28.0000\n",
       "4885     team  106.000000  106.000000  Default  27.0000  27.0000\n",
       "2134   player  106.000000  106.000000  Default  26.0000  26.0000\n",
       "...       ...         ...         ...      ...      ...      ...\n",
       "2086      pdf   26.137014   27.647697   Topic3  -5.9999   0.6425\n",
       "2388     risk   28.743779   31.836959   Topic3  -5.9048   0.5965\n",
       "2879      use   40.802458   58.495677   Topic3  -5.5545   0.3385\n",
       "2695   system   32.168305   44.574692   Topic3  -5.7922   0.3725\n",
       "1485  include   29.620990   52.818731   Topic3  -5.8747   0.1203\n",
       "\n",
       "[167 rows x 6 columns], token_table=      Topic      Freq   Term\n",
       "term                        \n",
       "139       3  0.988649   2015\n",
       "143       1  0.051876   2016\n",
       "143       2  0.051876   2016\n",
       "143       3  0.907822   2016\n",
       "145       1  0.028197   2018\n",
       "...     ...       ...    ...\n",
       "2983      2  0.038095  world\n",
       "2983      3  0.438087  world\n",
       "2993      1  0.355243   year\n",
       "2993      2  0.382569   year\n",
       "2993      3  0.273264   year\n",
       "\n",
       "[237 rows x 3 columns], R=30, lambda_step=0.01, plot_opts={'xlab': 'PC1', 'ylab': 'PC2'}, topic_order=[1, 2, 3])"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "warnings.simplefilter(action=\"ignore\", category=FutureWarning)\n",
    "\n",
    "vis = gensimvis.prepare(lda, doc_term_matrix, dictionary, sort_topics=False)\n",
    "\n",
    "warnings.simplefilter(action=\"default\", category=FutureWarning)\n",
    "\n",
    "vis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### (Optional) Topic modeling with `sklearn`\n",
    "\n",
    "- We are using `Gensim` LDA so that we'll be able to use `CoherenceModel` to evaluate topic model later. \n",
    "- Bit we can also train an LDA model with `sklearn`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "vec = CountVectorizer()\n",
    "X = vec.fit_transform(wiki_df[\"text_pp\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import LatentDirichletAllocation\n",
    "\n",
    "n_topics = 3\n",
    "lda = LatentDirichletAllocation(\n",
    "    n_components=n_topics, learning_method=\"batch\", max_iter=10, random_state=0\n",
    ")\n",
    "document_topics = lda.fit_transform(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lda.components_.shape: (3, 4957)\n"
     ]
    }
   ],
   "source": [
    "print(\"lda.components_.shape: {}\".format(lda.components_.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "sorting = np.argsort(lda.components_, axis=1)[:, ::-1]\n",
    "feature_names = np.array(vec.get_feature_names_out())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "topic 0       topic 1       topic 2       \n",
      "--------      --------      --------      \n",
      "court         hockey        intelligence  \n",
      "law           ice           artificial    \n",
      "provincial    team          learning      \n",
      "federal       player        algorithm     \n",
      "government    league        original      \n",
      "power         play          retrieve      \n",
      "canada        game          machine       \n",
      "justice       penalty       archive       \n",
      "supreme       puck          human         \n",
      "case          nhl           displaystyle  \n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/mglearn/plot_pca.py:7: DeprecationWarning: The 'cachedir' parameter has been deprecated in version 0.12 and will be removed in version 0.14.\n",
      "You provided \"cachedir='cache'\", use \"location='cache'\" instead.\n",
      "  memory = Memory(cachedir=\"cache\")\n",
      "/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/mglearn/plot_nmf.py:7: DeprecationWarning: The 'cachedir' parameter has been deprecated in version 0.12 and will be removed in version 0.14.\n",
      "You provided \"cachedir='cache'\", use \"location='cache'\" instead.\n",
      "  memory = Memory(cachedir=\"cache\")\n"
     ]
    }
   ],
   "source": [
    "import mglearn\n",
    "\n",
    "warnings.simplefilter(action=\"ignore\", category=DeprecationWarning)\n",
    "\n",
    "mglearn.tools.print_topics(\n",
    "    topics=range(3),\n",
    "    feature_names=feature_names,\n",
    "    sorting=sorting,\n",
    "    topics_per_chunk=5,\n",
    "    n_words=10,\n",
    ")\n",
    "\n",
    "warnings.simplefilter(action=\"default\", category=DeprecationWarning)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Basic text preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Introduction \n",
    "- Why do we need preprocessing?\n",
    "    - Text data is unstructured and messy. \n",
    "    - We need to \"normalize\" it before we do anything interesting with it. \n",
    "- Example:     \n",
    "    - **Lemma**: Same stem, same part-of-speech, roughly the same meaning\n",
    "        - Vancouver's &rarr; Vancouver\n",
    "        - computers &rarr; computer \n",
    "        - rising &rarr; rise, rose, rises    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Tokenization\n",
    "\n",
    "- Sentence segmentation\n",
    "    - Split text into sentences\n",
    "- Word tokenization \n",
    "    - Split sentences into words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Tokenization: sentence segmentation\n",
    "\n",
    "<blockquote>\n",
    "MDS is a Master's program at UBC in British Columbia. MDS teaching team is truly multicultural!! Dr. George did his Ph.D. in Scotland. Dr. Timbers, Dr. Ostblom, Dr. Rodríguez-Arelis, and Dr. Kolhatkar did theirs in Canada. Dr. Gelbart did his PhD in the U.S.\n",
    "</blockquote>\n",
    "\n",
    "- How many sentences are there in this text? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\"MDS is a Master's program at UBC in British Columbia\", ' MDS teaching team is truly multicultural!! Dr', ' George did his Ph', 'D', ' in Scotland', ' Dr', ' Timbers, Dr', ' Ostblom, Dr', ' Rodríguez-Arelis, and Dr', ' Kolhatkar did theirs in Canada', ' Dr', ' Gelbart did his PhD in the U', 'S', '']\n"
     ]
    }
   ],
   "source": [
    "### Let's do sentence segmentation on \".\"\n",
    "text = (\n",
    "    \"MDS is a Master's program at UBC in British Columbia. \"\n",
    "    \"MDS teaching team is truly multicultural!! \"\n",
    "    \"Dr. George did his Ph.D. in Scotland. \"\n",
    "    \"Dr. Timbers, Dr. Ostblom, Dr. Rodríguez-Arelis, and Dr. Kolhatkar did theirs in Canada. \"\n",
    "    \"Dr. Gelbart did his PhD in the U.S.\"\n",
    ")\n",
    "\n",
    "print(text.split(\".\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Sentence segmentation\n",
    "\n",
    "- In English, period (.) is quite ambiguous. (In Chinese, it is unambiguous.)\n",
    "    - Abbreviations like Dr., U.S., Inc.  \n",
    "    - Numbers like 60.44%, 0.98\n",
    "- ! and ? are relatively ambiguous.\n",
    "- How about writing regular expressions? \n",
    "- A common way is using off-the-shelf models for sentence segmentation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\"MDS is a Master's program at UBC in British Columbia.\", 'MDS teaching team is truly multicultural!!', 'Dr. George did his Ph.D. in Scotland.', 'Dr. Timbers, Dr. Ostblom, Dr. Rodríguez-Arelis, and Dr. Kolhatkar did theirs in Canada.', 'Dr. Gelbart did his PhD in the U.S.']\n"
     ]
    }
   ],
   "source": [
    "### Let's try to do sentence segmentation using nltk\n",
    "from nltk.tokenize import sent_tokenize\n",
    "\n",
    "sent_tokenized = sent_tokenize(text)\n",
    "print(sent_tokenized)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Word tokenization\n",
    "\n",
    "<blockquote>\n",
    "MDS is a Master's program at UBC in British Columbia. \n",
    "</blockquote>\n",
    "\n",
    "- How many words are there in this sentence?  \n",
    "- Is whitespace a sufficient condition for a word boundary?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Word tokenization \n",
    "\n",
    "<blockquote>\n",
    "MDS is a Master's program at UBC in British Columbia. \n",
    "</blockquote>\n",
    "\n",
    "- What's our definition of a word?\n",
    "    - Should British Columbia be one word or two words? \n",
    "    - Should punctuation be considered a separate word?\n",
    "    - What about the punctuations in `U.S.`?\n",
    "    - What do we do with words like `Master's`?\n",
    "- This process of identifying word boundaries is referred to as **tokenization**.\n",
    "- You can use regex but better to do it with off-the-shelf ML models.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Splitting on whitespace:  [['MDS', 'is', 'a', \"Master's\", 'program', 'at', 'UBC', 'in', 'British', 'Columbia.'], ['MDS', 'teaching', 'team', 'is', 'truly', 'multicultural!!'], ['Dr.', 'George', 'did', 'his', 'Ph.D.', 'in', 'Scotland.'], ['Dr.', 'Timbers,', 'Dr.', 'Ostblom,', 'Dr.', 'Rodríguez-Arelis,', 'and', 'Dr.', 'Kolhatkar', 'did', 'theirs', 'in', 'Canada.'], ['Dr.', 'Gelbart', 'did', 'his', 'PhD', 'in', 'the', 'U.S.']]\n",
      "\n",
      "\n",
      "\n",
      "Tokenized:  [['MDS', 'is', 'a', 'Master', \"'s\", 'program', 'at', 'UBC', 'in', 'British', 'Columbia', '.'], ['MDS', 'teaching', 'team', 'is', 'truly', 'multicultural', '!', '!'], ['Dr.', 'George', 'did', 'his', 'Ph.D.', 'in', 'Scotland', '.'], ['Dr.', 'Timbers', ',', 'Dr.', 'Ostblom', ',', 'Dr.', 'Rodríguez-Arelis', ',', 'and', 'Dr.', 'Kolhatkar', 'did', 'theirs', 'in', 'Canada', '.'], ['Dr.', 'Gelbart', 'did', 'his', 'PhD', 'in', 'the', 'U.S', '.']]\n"
     ]
    }
   ],
   "source": [
    "### Let's do word segmentation on white spaces\n",
    "print(\"Splitting on whitespace: \", [sent.split() for sent in sent_tokenized])\n",
    "\n",
    "### Let's try to do word segmentation using nltk\n",
    "from nltk.tokenize import word_tokenize\n",
    "\n",
    "word_tokenized = [word_tokenize(sent) for sent in sent_tokenized]\n",
    "# This is similar to the input format of word2vec algorithm\n",
    "print(\"\\n\\n\\nTokenized: \", word_tokenized)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Word segmentation \n",
    "\n",
    "For some languages you need much more sophisticated tokenizers. \n",
    "- For languages such as Chinese, there are no spaces between words.\n",
    "    - [jieba](https://github.com/fxsjy/jieba) is a popular tokenizer for Chinese. \n",
    "- German doesn't separate compound words.\n",
    "    * Example: _rindfleischetikettierungsüberwachungsaufgabenübertragungsgesetz_\n",
    "    * (the law for the delegation of monitoring beef labeling)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Types and tokens\n",
    "- Usually in NLP, we talk about \n",
    "    - **Type** an element in the vocabulary\n",
    "    - **Token** an instance of that type in running text \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Exercise for you \n",
    "\n",
    "<blockquote>    \n",
    "UBC is located in the beautiful province of British Columbia. It's very close \n",
    "to the U.S. border. You'll get to the USA border in about 45 mins by car.     \n",
    "</blockquote>  \n",
    "\n",
    "- Consider the example above. \n",
    "    - How many types? (task dependent)\n",
    "    - How many tokens? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Other commonly used preprocessing steps\n",
    "\n",
    "- Punctuation and stopword removal\n",
    "- Stemming and lemmatization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Punctuation and stopword removal\n",
    "\n",
    "- The most frequently occurring words in English are not very useful in many NLP tasks.\n",
    "    - Example: _the_ , _is_ , _a_ , and punctuation\n",
    "- Probably not very informative in many tasks "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\"haven't\", 'mustn', 'his', 'had', 'should', 'd', 'have', 'on', 'has', 'me', 'that', 'theirs', 'ourselves', \"you'll\", 'our', \"that'll\", 'ma', 'shouldn', 'didn', 'm', 'weren', \"should've\", 'shan', 'then', 'such', 'so', 'i', 'just', \"you'd\", 'been', 'does', 'because', 'if', 'those', 'were', 'him', 'y', 'into', 'than', \"she's\", 'its', 'them', 'through', 'can', 'hadn', \"shouldn't\", 'yourself', 'couldn', 'of', \"don't\", 'for', 'about', \"hadn't\", 'both', 'here', 'hers', 'in', 'above', 'only', 'some', 'any', 'now', 'after', \"aren't\", \"mightn't\", 'until', \"it's\", 've', 'hasn', 'ain', 'same', 'her', \"wouldn't\", 'up', 'isn', 'they', 'during', 'before', 'there', 'what', \"you're\", 'or', 'few', 'to', 'will', \"won't\", \"weren't\", 'do', 'down', 'am', 'off', 'once', 'your', 'himself', 'my', 'whom', \"isn't\", 'is', 'out', 'no', 'who', 'each', 'this', 't', 'from', 'again', 'wasn', 'herself', 'by', 'll', 'with', 'most', 'myself', 'be', 'at', 'further', 'mightn', 'o', 'it', 'between', 'themselves', 'how', 'itself', 'doing', \"you've\", \"mustn't\", 'which', 'having', \"shan't\", 'too', 'aren', 'their', \"needn't\", 'an', \"doesn't\", 'haven', 'more', 'against', 'are', 'and', 'a', 'yours', 'as', 'very', 're', 'being', 'these', 'all', 'nor', 'own', 'needn', \"couldn't\", 'don', 'below', 'yourselves', \"didn't\", 'why', \"wasn't\", 'the', 'where', 'wouldn', 'while', 'under', 'we', 'was', 'she', 'over', 'doesn', 's', \"hasn't\", 'other', 'not', 'when', 'ours', 'he', 'won', 'did', 'you', 'but', '!', '\"', '#', '$', '%', '&', \"'\", '(', ')', '*', '+', ',', '-', '.', '/', ':', ';', '<', '=', '>', '?', '@', '[', '\\\\', ']', '^', '_', '`', '{', '|', '}', '~']\n"
     ]
    }
   ],
   "source": [
    "# Let's use `nltk.stopwords`.\n",
    "# Add punctuations to the list.\n",
    "stop_words = list(set(stopwords.words(\"english\")))\n",
    "import string\n",
    "\n",
    "punctuation = string.punctuation\n",
    "stop_words += list(punctuation)\n",
    "# stop_words.extend(['``','`','br','\"',\"”\", \"''\", \"'s\"])\n",
    "print(stop_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['mds', 'master', \"'s\", 'program', 'ubc', 'british', 'columbia', 'mds', 'teaching', 'team', 'truly', 'multicultural', 'dr.', 'george', 'ph.d.', 'scotland', 'dr.', 'timbers', 'dr.', 'ostblom', 'dr.', 'rodríguez-arelis', 'dr.', 'kolhatkar', 'canada', 'dr.', 'gelbart', 'phd', 'u.s']\n"
     ]
    }
   ],
   "source": [
    "### Get rid of stop words\n",
    "preprocessed = []\n",
    "for sent in word_tokenized:\n",
    "    for token in sent:\n",
    "        token = token.lower()\n",
    "        if token not in stop_words:\n",
    "            preprocessed.append(token)\n",
    "print(preprocessed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Lemmatization \n",
    "\n",
    "- For many NLP tasks (e.g., web search) we want to ignore morphological differences between words\n",
    "    - Example: If your search term is \"studying for ML quiz\" you might want to include pages containing \"tips to study for an ML quiz\" or \"here is how I studied for my ML quiz\"\n",
    "- Lemmatization converts inflected forms into the base form. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "\n",
    "# nltk.download(\"wordnet\")  # You need to run this at least once\n",
    "# nltk.download('omw-1.4')  # You need to run this at least once"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Lemma of studying:  study\n",
      "Lemma of studied:  study\n"
     ]
    }
   ],
   "source": [
    "# nltk has a lemmatizer\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "\n",
    "lemmatizer = WordNetLemmatizer()\n",
    "print(\"Lemma of studying: \", lemmatizer.lemmatize(\"studying\", \"v\"))\n",
    "print(\"Lemma of studied: \", lemmatizer.lemmatize(\"studied\", \"v\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Stemming\n",
    "\n",
    "- Has a similar purpose but it is a crude chopping of affixes \n",
    "    * _automates, automatic, automation_ all reduced to _automat_.\n",
    "- Usually these reduced forms (stems) are not actual words themselves.  \n",
    "- A popular stemming algorithm for English is PorterStemmer. \n",
    "- Beware that it can be aggressive sometimes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before stemming:  UBC is located in the beautiful province of British Columbia... It's very close to the U.S. border.\n",
      "\n",
      "\n",
      "After stemming:  ubc is locat in the beauti provinc of british columbia ... it 's veri close to the u.s. border .\n"
     ]
    }
   ],
   "source": [
    "from nltk.stem.porter import PorterStemmer\n",
    "\n",
    "text = (\n",
    "    \"UBC is located in the beautiful province of British Columbia... \"\n",
    "    \"It's very close to the U.S. border.\"\n",
    ")\n",
    "ps = PorterStemmer()\n",
    "tokenized = word_tokenize(text)\n",
    "stemmed = [ps.stem(token) for token in tokenized]\n",
    "print(\"Before stemming: \", text)\n",
    "print(\"\\n\\nAfter stemming: \", \" \".join(stemmed))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Other tools for preprocessing \n",
    "\n",
    "- We used [Natural Language Processing Toolkit (nltk)](https://www.nltk.org/) above\n",
    "- Many available tools    \n",
    "- [spaCy](https://spacy.io/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### [spaCy](https://spacy.io/)\n",
    "\n",
    "- Industrial strength NLP library. \n",
    "- Lightweight, fast, and convenient to use. \n",
    "- spaCy does many things that we did above in one line of code! \n",
    "- Also has [multi-lingual](https://spacy.io/models/xx) support. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import spacy\n",
    "\n",
    "\n",
    "warnings.simplefilter(action=\"ignore\", category=DeprecationWarning)\n",
    "\n",
    "# Load the model\n",
    "nlp = spacy.load(\"en_core_web_md\")\n",
    "\n",
    "warnings.simplefilter(action=\"default\", category=DeprecationWarning)\n",
    "\n",
    "text = (\n",
    "    \"MDS is a Master's program at UBC in British Columbia. \"\n",
    "    \"MDS teaching team is truly multicultural!! \"\n",
    "    \"Dr. George did his Ph.D. in Scotland. \"\n",
    "    \"Dr. Timbers, Dr. Ostblom, Dr. Rodríguez-Arelis, and Dr. Kolhatkar did theirs in Canada. \"\n",
    "    \"Dr. Gelbart did his PhD in the U.S.\"\n",
    ")\n",
    "\n",
    "doc = nlp(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Tokens:  [MDS, is, a, Master, 's, program, at, UBC, in, British, Columbia, ., MDS, teaching, team, is, truly, multicultural, !, !, Dr., George, did, his, Ph.D., in, Scotland, ., Dr., Timbers, ,, Dr., Ostblom, ,, Dr., Rodríguez, -, Arelis, ,, and, Dr., Kolhatkar, did, theirs, in, Canada, ., Dr., Gelbart, did, his, PhD, in, the, U.S.]\n",
      "\n",
      "Lemmas:  ['MDS', 'be', 'a', 'Master', \"'s\", 'program', 'at', 'UBC', 'in', 'British', 'Columbia', '.', 'MDS', 'teaching', 'team', 'be', 'truly', 'multicultural', '!', '!', 'Dr.', 'George', 'do', 'his', 'ph.d.', 'in', 'Scotland', '.', 'Dr.', 'Timbers', ',', 'Dr.', 'Ostblom', ',', 'Dr.', 'Rodríguez', '-', 'Arelis', ',', 'and', 'Dr.', 'Kolhatkar', 'do', 'theirs', 'in', 'Canada', '.', 'Dr.', 'Gelbart', 'do', 'his', 'phd', 'in', 'the', 'U.S.']\n",
      "\n",
      "POS:  ['PROPN', 'AUX', 'DET', 'PROPN', 'PART', 'NOUN', 'ADP', 'PROPN', 'ADP', 'PROPN', 'PROPN', 'PUNCT', 'PROPN', 'NOUN', 'NOUN', 'AUX', 'ADV', 'ADJ', 'PUNCT', 'PUNCT', 'PROPN', 'PROPN', 'VERB', 'PRON', 'NOUN', 'ADP', 'PROPN', 'PUNCT', 'PROPN', 'PROPN', 'PUNCT', 'PROPN', 'PROPN', 'PUNCT', 'PROPN', 'PROPN', 'PUNCT', 'PROPN', 'PUNCT', 'CCONJ', 'PROPN', 'PROPN', 'VERB', 'PRON', 'ADP', 'PROPN', 'PUNCT', 'PROPN', 'PROPN', 'VERB', 'PRON', 'NOUN', 'ADP', 'DET', 'PROPN']\n"
     ]
    }
   ],
   "source": [
    "# Accessing tokens\n",
    "tokens = [token for token in doc]\n",
    "print(\"\\nTokens: \", tokens)\n",
    "\n",
    "# Accessing lemma\n",
    "lemmas = [token.lemma_ for token in doc]\n",
    "print(\"\\nLemmas: \", lemmas)\n",
    "\n",
    "# Accessing pos\n",
    "pos = [token.pos_ for token in doc]\n",
    "print(\"\\nPOS: \", pos)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Other typical NLP tasks \n",
    "In order to understand text, we usually are interested in extracting information from text. Some common tasks in NLP pipeline are: \n",
    "- Part of speech tagging\n",
    "    - Assigning part-of-speech tags to all words in a sentence.\n",
    "- Named entity recognition\n",
    "    - Labelling named “real-world” objects, like persons, companies or locations.    \n",
    "- Coreference resolution\n",
    "    - Deciding whether two strings (e.g., UBC vs University of British Columbia) refer to the same entity\n",
    "- Dependency parsing\n",
    "    - Representing grammatical structure of a sentence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Extracting named-entities using spaCy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">\n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    University of British Columbia\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " is located in the beautiful province of \n",
       "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    British Columbia\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
       "</mark>\n",
       ".</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Named entities:\n",
      " [('University of British Columbia', 'ORG'), ('British Columbia', 'GPE')]\n",
      "\n",
      "ORG means:  Companies, agencies, institutions, etc.\n",
      "GPE means:  Countries, cities, states\n"
     ]
    }
   ],
   "source": [
    "from spacy import displacy\n",
    "\n",
    "\n",
    "warnings.simplefilter(action=\"ignore\", category=DeprecationWarning)\n",
    "\n",
    "doc = nlp(\n",
    "    \"University of British Columbia \"\n",
    "    \"is located in the beautiful \"\n",
    "    \"province of British Columbia.\"\n",
    ")\n",
    "displacy.render(doc, style=\"ent\")\n",
    "\n",
    "warnings.simplefilter(action=\"default\", category=DeprecationWarning)\n",
    "\n",
    "# Text and label of named entity span\n",
    "print(\"Named entities:\\n\", [(ent.text, ent.label_) for ent in doc.ents])\n",
    "print(\"\\nORG means: \", spacy.explain(\"ORG\"))\n",
    "print(\"GPE means: \", spacy.explain(\"GPE\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Dependency parsing using spaCy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" xml:lang=\"en\" id=\"42e2cd65a0e74d3486f329f03affc02a-0\" class=\"displacy\" width=\"575\" height=\"224.5\" direction=\"ltr\" style=\"max-width: none; height: 224.5px; color: #000000; background: #ffffff; font-family: Arial; direction: ltr\">\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"134.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"50\">I</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"50\">PRON</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"134.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"225\">like</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"225\">VERB</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"134.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"400\">cats</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"400\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-42e2cd65a0e74d3486f329f03affc02a-0-0\" stroke-width=\"2px\" d=\"M70,89.5 C70,2.0 225.0,2.0 225.0,89.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-42e2cd65a0e74d3486f329f03affc02a-0-0\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">nsubj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M70,91.5 L62,79.5 78,79.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-42e2cd65a0e74d3486f329f03affc02a-0-1\" stroke-width=\"2px\" d=\"M245,89.5 C245,2.0 400.0,2.0 400.0,89.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-42e2cd65a0e74d3486f329f03affc02a-0-1\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">dobj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M400.0,91.5 L408.0,79.5 392.0,79.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "</svg></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "warnings.simplefilter(action=\"ignore\", category=DeprecationWarning)\n",
    "\n",
    "doc = nlp(\"I like cats\")\n",
    "displacy.render(doc, style=\"dep\")\n",
    "\n",
    "warnings.simplefilter(action=\"default\", category=DeprecationWarning)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Many other things possible\n",
    "\n",
    "- A powerful tool \n",
    "- All my Capstone groups last year used this tool. \n",
    "- You can build your own rule-based searches. \n",
    "- You can also access word vectors using spaCy with bigger models. (Currently we are using `en_core_web_md` model.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## Bug workaround!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style> .jp-Button path { fill: #616161;} text.terms { fill: #616161;} .jp-icon-warn0 path {fill: var(--jp-warn-color0);} .bp3-button-text path { fill: var(--jp-inverse-layout-color3);} .jp-icon-brand0 path { fill: var(--jp-brand-color0);} text.terms { fill: #616161;} </style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# \"Do you have pyLDAvis installed in your virtual \n",
    "#  environment? Apparently there is a strong evidence \n",
    "#  that it is an interaction with pyLDAvis.\"\n",
    "# https://stackoverflow.com/a/69171847/14881731\n",
    "\n",
    "from IPython.display import HTML\n",
    "css_str = '<style> \\\n",
    ".jp-Button path { fill: #616161;} \\\n",
    "text.terms { fill: #616161;} \\\n",
    ".jp-icon-warn0 path {fill: var(--jp-warn-color0);} \\\n",
    ".bp3-button-text path { fill: var(--jp-inverse-layout-color3);} \\\n",
    ".jp-icon-brand0 path { fill: var(--jp-brand-color0);} \\\n",
    "text.terms { fill: #616161;} \\\n",
    "</style>'\n",
    "display(HTML(css_str ))"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python [conda env:cpsc330]",
   "language": "python",
   "name": "conda-env-cpsc330-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}