run_doc2vec_lee.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# Doc2Vec Model\n",
    "\n",
    "Introduces Gensim's Doc2Vec model and demonstrates its use on the\n",
    "[Lee Corpus](https://hekyll.services.adelaide.edu.au/dspace/bitstream/2440/28910/1/hdl_28910.pdf)_.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Doc2Vec is a `core_concepts_model` that represents each\n",
    "`core_concepts_document` as a `core_concepts_vector`.  This\n",
    "tutorial introduces the model and demonstrates how to train and assess it.\n",
    "\n",
    "Here's a list of what we'll be doing:\n",
    "\n",
    "0. Review the relevant models: bag-of-words, Word2Vec, Doc2Vec\n",
    "1. Load and preprocess the training and test corpora (see `core_concepts_corpus`)\n",
    "2. Train a Doc2Vec `core_concepts_model` model using the training corpus\n",
    "3. Demonstrate how the trained model can be used to infer a `core_concepts_vector`\n",
    "4. Assess the model\n",
    "5. Test the model on the test corpus\n",
    "\n",
    "## Review: Bag-of-words\n",
    "\n",
    ".. Note:: Feel free to skip these review sections if you're already familiar with the models.\n",
    "\n",
    "You may be familiar with the [bag-of-words model](https://en.wikipedia.org/wiki/Bag-of-words_model) from the\n",
    "`core_concepts_vector` section.\n",
    "This model transforms each document to a fixed-length vector of integers.\n",
    "For example, given the sentences:\n",
    "\n",
    "- ``John likes to watch movies. Mary likes movies too.``\n",
    "- ``John also likes to watch football games. Mary hates football.``\n",
    "\n",
    "The model outputs the vectors:\n",
    "\n",
    "- ``[1, 2, 1, 1, 2, 1, 1, 0, 0, 0, 0]``\n",
    "- ``[1, 1, 1, 1, 0, 1, 0, 1, 2, 1, 1]``\n",
    "\n",
    "Each vector has 10 elements, where each element counts the number of times a\n",
    "particular word occurred in the document.\n",
    "The order of elements is arbitrary.\n",
    "In the example above, the order of the elements corresponds to the words:\n",
    "``[\"John\", \"likes\", \"to\", \"watch\", \"movies\", \"Mary\", \"too\", \"also\", \"football\", \"games\", \"hates\"]``.\n",
    "\n",
    "Bag-of-words models are surprisingly effective, but have several weaknesses.\n",
    "\n",
    "First, they lose all information about word order: \"John likes Mary\" and\n",
    "\"Mary likes John\" correspond to identical vectors. There is a solution: bag\n",
    "of [n-grams](https://en.wikipedia.org/wiki/N-gram)_\n",
    "models consider word phrases of length n to represent documents as\n",
    "fixed-length vectors to capture local word order but suffer from data\n",
    "sparsity and high dimensionality.\n",
    "\n",
    "Second, the model does not attempt to learn the meaning of the underlying\n",
    "words, and as a consequence, the distance between vectors doesn't always\n",
    "reflect the difference in meaning.  The ``Word2Vec`` model addresses this\n",
    "second problem.\n",
    "\n",
    "## Review: ``Word2Vec`` Model\n",
    "\n",
    "``Word2Vec`` is a more recent model that embeds words in a lower-dimensional\n",
    "vector space using a shallow neural network. The result is a set of\n",
    "word-vectors where vectors close together in vector space have similar\n",
    "meanings based on context, and word-vectors distant to each other have\n",
    "differing meanings. For example, ``strong`` and ``powerful`` would be close\n",
    "together and ``strong`` and ``Paris`` would be relatively far.\n",
    "\n",
    "Gensim's :py:class:`~gensim.models.word2vec.Word2Vec` class implements this model.\n",
    "\n",
    "With the ``Word2Vec`` model, we can calculate the vectors for each **word** in a document.\n",
    "But what if we want to calculate a vector for the **entire document**\\ ?\n",
    "We could average the vectors for each word in the document - while this is quick and crude, it can often be useful.\n",
    "However, there is a better way...\n",
    "\n",
    "## Introducing: Paragraph Vector\n",
    "\n",
    ".. Important:: In Gensim, we refer to the Paragraph Vector model as ``Doc2Vec``.\n",
    "\n",
    "Le and Mikolov in 2014 introduced the [Doc2Vec algorithm](https://cs.stanford.edu/~quocle/paragraph_vector.pdf)_,\n",
    "which usually outperforms such simple-averaging of ``Word2Vec`` vectors.\n",
    "\n",
    "The basic idea is: act as if a document has another floating word-like\n",
    "vector, which contributes to all training predictions, and is updated like\n",
    "other word-vectors, but we will call it a doc-vector. Gensim's\n",
    ":py:class:`~gensim.models.doc2vec.Doc2Vec` class implements this algorithm.\n",
    "\n",
    "There are two implementations:\n",
    "\n",
    "1. Paragraph Vector - Distributed Memory (PV-DM)\n",
    "2. Paragraph Vector - Distributed Bag of Words (PV-DBOW)\n",
    "\n",
    ".. Important::\n",
    "  Don't let the implementation details below scare you.\n",
    "  They're advanced material: if it's too much, then move on to the next section.\n",
    "\n",
    "PV-DM is analogous to Word2Vec CBOW. The doc-vectors are obtained by training\n",
    "a neural network on the synthetic task of predicting a center word based an\n",
    "average of both context word-vectors and the full document's doc-vector.\n",
    "\n",
    "PV-DBOW is analogous to Word2Vec SG. The doc-vectors are obtained by training\n",
    "a neural network on the synthetic task of predicting a target word just from\n",
    "the full document's doc-vector. (It is also common to combine this with\n",
    "skip-gram testing, using both the doc-vector and nearby word-vectors to\n",
    "predict a single target word, but only one at a time.)\n",
    "\n",
    "## Prepare the Training and Test Data\n",
    "\n",
    "For this tutorial, we'll be training our model using the [Lee Background\n",
    "Corpus](https://hekyll.services.adelaide.edu.au/dspace/bitstream/2440/28910/1/hdl_28910.pdf)\n",
    "included in gensim. This corpus contains 314 documents selected from the\n",
    "Australian Broadcasting Corporation’s news mail service, which provides text\n",
    "e-mails of headline stories and covers a number of broad topics.\n",
    "\n",
    "And we'll test our model by eye using the much shorter [Lee Corpus](https://hekyll.services.adelaide.edu.au/dspace/bitstream/2440/28910/1/hdl_28910.pdf)\n",
    "which contains 50 documents.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import gensim\n",
    "# Set file names for train and test data\n",
    "test_data_dir = os.path.join(gensim.__path__[0], 'test', 'test_data')\n",
    "lee_train_file = os.path.join(test_data_dir, 'lee_background.cor')\n",
    "lee_test_file = os.path.join(test_data_dir, 'lee.cor')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define a Function to Read and Preprocess Text\n",
    "\n",
    "Below, we define a function to:\n",
    "\n",
    "- open the train/test file (with latin encoding)\n",
    "- read the file line-by-line\n",
    "- pre-process each line (tokenize text into individual words, remove punctuation, set to lowercase, etc)\n",
    "\n",
    "The file we're reading is a **corpus**.\n",
    "Each line of the file is a **document**.\n",
    "\n",
    ".. Important::\n",
    "  To train the model, we'll need to associate a tag/number with each document\n",
    "  of the training corpus. In our case, the tag is simply the zero-based line\n",
    "  number.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import smart_open\n",
    "\n",
    "def read_corpus(fname, tokens_only=False):\n",
    "    with smart_open.open(fname, encoding=\"iso-8859-1\") as f:\n",
    "        for i, line in enumerate(f):\n",
    "            tokens = gensim.utils.simple_preprocess(line)\n",
    "            if tokens_only:\n",
    "                yield tokens\n",
    "            else:\n",
    "                # For training data, add tags\n",
    "                yield gensim.models.doc2vec.TaggedDocument(tokens, [i])\n",
    "\n",
    "train_corpus = list(read_corpus(lee_train_file))\n",
    "test_corpus = list(read_corpus(lee_test_file, tokens_only=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the training corpus\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(train_corpus[:2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the testing corpus looks like this:\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(test_corpus[:2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the testing corpus is just a list of lists and does not contain\n",
    "any tags.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the Model\n",
    "\n",
    "Now, we'll instantiate a Doc2Vec model with a vector size with 50 dimensions and\n",
    "iterating over the training corpus 40 times. We set the minimum word count to\n",
    "2 in order to discard words with very few occurrences. (Without a variety of\n",
    "representative examples, retaining such infrequent words can often make a\n",
    "model worse!) Typical iteration counts in the published [Paragraph Vector paper](https://cs.stanford.edu/~quocle/paragraph_vector.pdf)_\n",
    "results, using 10s-of-thousands to millions of docs, are 10-20. More\n",
    "iterations take more time and eventually reach a point of diminishing\n",
    "returns.\n",
    "\n",
    "However, this is a very very small dataset (300 documents) with shortish\n",
    "documents (a few hundred words). Adding training passes can sometimes help\n",
    "with such small datasets.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model = gensim.models.doc2vec.Doc2Vec(vector_size=50, min_count=2, epochs=40)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Build a vocabulary\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model.build_vocab(train_corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Essentially, the vocabulary is a list (accessible via\n",
    "``model.wv.index_to_key``) of all of the unique words extracted from the training corpus.\n",
    "Additional attributes for each word are available using the ``model.wv.get_vecattr()`` method,\n",
    "For example, to see how many times ``penalty`` appeared in the training corpus:\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(f\"Word 'penalty' appeared {model.wv.get_vecattr('penalty', 'count')} times in the training corpus.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, train the model on the corpus.\n",
    "In the usual case, where Gensim installation found a BLAS library for optimized\n",
    "bulk vector operations, this training on this tiny 300 document, ~60k word corpus \n",
    "should take just a few seconds. (More realistic datasets of tens-of-millions\n",
    "of words or more take proportionately longer.) If for some reason a BLAS library \n",
    "isn't available, training uses a fallback approach that takes 60x-120x longer, \n",
    "so even this tiny training will take minutes rather than seconds. (And, in that \n",
    "case, you should also notice a warning in the logging letting you know there's \n",
    "something worth fixing.) So, be sure your installation uses the BLAS-optimized \n",
    "Gensim if you value your time.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model.train(train_corpus, total_examples=model.corpus_count, epochs=model.epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can use the trained model to infer a vector for any piece of text\n",
    "by passing a list of words to the ``model.infer_vector`` function. This\n",
    "vector can then be compared with other vectors via cosine similarity.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "vector = model.infer_vector(['only', 'you', 'can', 'prevent', 'forest', 'fires'])\n",
    "print(vector)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that ``infer_vector()`` does *not* take a string, but rather a list of\n",
    "string tokens, which should have already been tokenized the same way as the\n",
    "``words`` property of original training document objects.\n",
    "\n",
    "Also note that because the underlying training/inference algorithms are an\n",
    "iterative approximation problem that makes use of internal randomization,\n",
    "repeated inferences of the same text will return slightly different vectors.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assessing the Model\n",
    "\n",
    "To assess our new model, we'll first infer new vectors for each document of\n",
    "the training corpus, compare the inferred vectors with the training corpus,\n",
    "and then returning the rank of the document based on self-similarity.\n",
    "Basically, we're pretending as if the training corpus is some new unseen data\n",
    "and then seeing how they compare with the trained model. The expectation is\n",
    "that we've likely overfit our model (i.e., all of the ranks will be less than\n",
    "2) and so we should be able to find similar documents very easily.\n",
    "Additionally, we'll keep track of the second ranks for a comparison of less\n",
    "similar documents.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ranks = []\n",
    "second_ranks = []\n",
    "for doc_id in range(len(train_corpus)):\n",
    "    inferred_vector = model.infer_vector(train_corpus[doc_id].words)\n",
    "    sims = model.dv.most_similar([inferred_vector], topn=len(model.dv))\n",
    "    rank = [docid for docid, sim in sims].index(doc_id)\n",
    "    ranks.append(rank)\n",
    "\n",
    "    second_ranks.append(sims[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's count how each document ranks with respect to the training corpus\n",
    "\n",
    "NB. Results vary between runs due to random seeding and very small corpus\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import collections\n",
    "\n",
    "counter = collections.Counter(ranks)\n",
    "print(counter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Basically, greater than 95% of the inferred documents are found to be most\n",
    "similar to itself and about 5% of the time it is mistakenly most similar to\n",
    "another document. Checking the inferred-vector against a\n",
    "training-vector is a sort of 'sanity check' as to whether the model is\n",
    "behaving in a usefully consistent manner, though not a real 'accuracy' value.\n",
    "\n",
    "This is great and not entirely surprising. We can take a look at an example:\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print('Document ({}): «{}»\\n'.format(doc_id, ' '.join(train_corpus[doc_id].words)))\n",
    "print(u'SIMILAR/DISSIMILAR DOCS PER MODEL %s:\\n' % model)\n",
    "for label, index in [('MOST', 0), ('SECOND-MOST', 1), ('MEDIAN', len(sims)//2), ('LEAST', len(sims) - 1)]:\n",
    "    print(u'%s %s: «%s»\\n' % (label, sims[index], ' '.join(train_corpus[sims[index][0]].words)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice above that the most similar document (usually the same text) is has a\n",
    "similarity score approaching 1.0. However, the similarity score for the\n",
    "second-ranked documents should be significantly lower (assuming the documents\n",
    "are in fact different) and the reasoning becomes obvious when we examine the\n",
    "text itself.\n",
    "\n",
    "We can run the next cell repeatedly to see a sampling other target-document\n",
    "comparisons.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Pick a random document from the corpus and infer a vector from the model\n",
    "import random\n",
    "doc_id = random.randint(0, len(train_corpus) - 1)\n",
    "\n",
    "# Compare and print the second-most-similar document\n",
    "print('Train Document ({}): «{}»\\n'.format(doc_id, ' '.join(train_corpus[doc_id].words)))\n",
    "sim_id = second_ranks[doc_id]\n",
    "print('Similar Document {}: «{}»\\n'.format(sim_id, ' '.join(train_corpus[sim_id[0]].words)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing the Model\n",
    "\n",
    "Using the same approach above, we'll infer the vector for a randomly chosen\n",
    "test document, and compare the document to our model by eye.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Pick a random document from the test corpus and infer a vector from the model\n",
    "doc_id = random.randint(0, len(test_corpus) - 1)\n",
    "inferred_vector = model.infer_vector(test_corpus[doc_id])\n",
    "sims = model.dv.most_similar([inferred_vector], topn=len(model.dv))\n",
    "\n",
    "# Compare and print the most/median/least similar documents from the train corpus\n",
    "print('Test Document ({}): «{}»\\n'.format(doc_id, ' '.join(test_corpus[doc_id])))\n",
    "print(u'SIMILAR/DISSIMILAR DOCS PER MODEL %s:\\n' % model)\n",
    "for label, index in [('MOST', 0), ('MEDIAN', len(sims)//2), ('LEAST', len(sims) - 1)]:\n",
    "    print(u'%s %s: «%s»\\n' % (label, sims[index], ' '.join(train_corpus[sims[index][0]].words)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "Let's review what we've seen in this tutorial:\n",
    "\n",
    "0. Review the relevant models: bag-of-words, Word2Vec, Doc2Vec\n",
    "1. Load and preprocess the training and test corpora (see `core_concepts_corpus`)\n",
    "2. Train a Doc2Vec `core_concepts_model` model using the training corpus\n",
    "3. Demonstrate how the trained model can be used to infer a `core_concepts_vector`\n",
    "4. Assess the model\n",
    "5. Test the model on the test corpus\n",
    "\n",
    "That's it! Doc2Vec is a great way to explore relationships between documents.\n",
    "\n",
    "## Additional Resources\n",
    "\n",
    "If you'd like to know more about the subject matter of this tutorial, check out the links below.\n",
    "\n",
    "* [Word2Vec Paper](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf)\n",
    "* [Doc2Vec Paper](https://cs.stanford.edu/~quocle/paragraph_vector.pdf)\n",
    "* [Dr. Michael D. Lee's Website](http://faculty.sites.uci.edu/mdlee)\n",
    "* [Lee Corpus](http://faculty.sites.uci.edu/mdlee/similarity-data/)_\n",
    "* [IMDB Doc2Vec Tutorial](doc2vec-IMDB.ipynb)\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}