assignments/ASSIGNMENT_2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 2 \n",
    "## Chapters 6-12: containers, loops, functions\n",
    "\n",
    "**Questions? Drop em in the Slack under #questions**\n",
    "\n",
    "By now, we have acquired a toolkit of Python objects and methods that\n",
    "enable us to do really powerful things with texts. With containers, loops,\n",
    "and functions, you have learned some of the most key parts of Python. \n",
    "\n",
    "These new skills means that we can also begin to work more directly with\n",
    "the text you've placed in the `/BYOT` folder. It also means that we can begin\n",
    "to think more about how to operationalize linguistic questions into Pythonic\n",
    "procedures. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Warmup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "    \n",
    "Assign a list to a variable and populate it with five of your favorite authors.\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "    \n",
    "Assign another list and populate it with books written by the same authors of the above list. Make sure the order matches.\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "    \n",
    "Run the code below and consider what it does.\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "items = ['gameboy', 'N64', 'PC'] \n",
    "prices = [65, 100, 700]\n",
    "\n",
    "list(zip(items, prices))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "help(zip)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "    \n",
    "Use `zip` to combine your authors list with your books list. Assign it to a variable.\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "    \n",
    "Select the second author/book from the new list.\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "    \n",
    "Print the object type of the last author/book pairing from the author/book list.\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "    \n",
    "Isolate the first author from the author/book list and assign to variable. Print it.\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "    \n",
    "Write a function that takes a single argument, `author_book` which is a list of author/book two-tuples. The function should iterate through the author/book list and prints each pairing in the following format: <br>\n",
    "\n",
    "   `The author {author} wrote {book_name}`\n",
    "\n",
    "Execute the function by feeding it your author/book list as the argument.\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "    \n",
    "Assign a variable to an empty set.\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "    \n",
    "Assign a variable to an empty dictionary. Use both methods for initializing a new dictionary. How does this differ from the empty set you made above?\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Observe the small function below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9500"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def i_can_do_math(itemA, itemB):\n",
    "    \"\"\"Add two things together and multiply by 100.\"\"\"\n",
    "    ab_sum = itemA + itemB\n",
    "    product = ab_sum * 100\n",
    "    return product\n",
    "\n",
    "i_can_do_math(21, 74)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "    \n",
    "Write a small function that takes 2 arguments:\n",
    "    1. `name`\n",
    "    2. `greeting`\n",
    "The function should print any given greeting to a any given name.\n",
    "For example, \"What's up Charles\" or \"Bonjour Cindy\"\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# BYOT \n",
    "\n",
    "Now that we have some more sophisticated tools for storing, looping, and writing code, we can begin to do some interesting real analysis on the text you've placed in the `BYOT` folder. \n",
    "\n",
    "Note that some of the following exercises will require a bit of **creativity**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-warning\">\n",
    "    \n",
    "Tips for completing the exercises:\n",
    "* Use the Chapter material for reference! Learning to code is just as much learning when to look up how to do something!\n",
    "* Use the Jupyter notebook code cells to experiment. Don't feel like you should only write perfect code in a code cell. A strength of Jupyter notebooks is that you can add arbitrary cells to experiment with code. Don't remember how exactly to write a `dict`? Add a cell and experiment with what you do remember. You can then copy/past and clean up your \"scratch\" code when you're done.\n",
    "* Break the problem up into steps. Write them out if you need to.\n",
    "* Add notes (`#`) and write your code with extra spacing so it is more readable. Think of your code segments as paragraphs. You can even give them notes as headings that simply tells what a chunk does.\n",
    "* [Toggle line numbers](https://stackoverflow.com/questions/10979667/showing-line-numbers-in-ipython-jupyter-notebooks) to help you interpret error messages.\n",
    "* Practice the 15 minute rule! Don't stay stuck longer than 15 minutes. Ask for help.\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from get_byot import your_text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text-Setup\n",
    "\n",
    "Remember that `your_text` is a large string which contains all of the text from the `BYOT` text.\n",
    "Most texts contains front-matter, and some contain back-matter. Use string slices to isolate\n",
    "the main content of `your_text` from the rest. Look back on Assignment 1 if you need to, where \n",
    "we did the same thing. Assign this portion to a new, descriptive variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Word Counts Revisited \n",
    "*concepts: string methods, lists, built-in functions*\n",
    "\n",
    "\n",
    "In ASSIGNMENT 1 we made approximate word counts using spaces. Now we will use a slightly more \n",
    "sophisticated approach with `str.split`. This process is known as [\"tokenization\"](https://nlp.stanford.edu/IR-book/html/htmledition/tokenization-1.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "   \n",
    "Split your text into tokens with `.split()` (NB don't add an argument). Assign it to a variable `tokens`.\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "   \n",
    "How many tokens are in `tokens`? How does this compare with the word count from Assignment 1? What might explain the discrepancy? (Hint: use `help` to look at the default behavior of `split`.\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Rough Lexicon\n",
    "*concepts: sets, built-in functions, lists*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that `tokens` probably contains a lot of repeated items. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "\n",
    "Remove all duplicate items from `tokens` using a single line of code. Assign the result to `lexicon`.\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "\n",
    "How long is `lexicon`?\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "\n",
    "Arrange `lexicon` into an ordered object and sort it.\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "\n",
    "Show the first 50 items at the top of the sorted object.\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lexicon Frequencies\n",
    "\n",
    "*concepts: loops, dictionaries, conditionals*\n",
    "\n",
    "The lexicon tells us all the unique items in `tokens`, but it doesn't give us any idea\n",
    "how often each token occurs in the text. We want to know how often each item occurs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "\n",
    "Count how often each token occurs in `tokens`. Assign it to `lex_freq`. Do not use additional tools like `collections`.\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "\n",
    "Write a small function that takes two parameters: (1) a frequency dict, (2) an integer.\n",
    "The function should use the frequency dict to **return** the [N-th most](http://mathcentral.uregina.ca/QQ/database/QQ.09.04/alex1.html)\n",
    "item (i.e. the integer). Thus if the `integer` argument is 10, the function will return\n",
    "the top 10 items in the dict. Be sure to use a descriptive docstring. Run your function \n",
    "on your frequency dict above and show the top 25 most frequent items.\n",
    "    \n",
    "*hint: think about how you can use `list.sort()` or `sorted()` to accomplish this task. \n",
    "If you need to, do some experiments about how either of these work with lists of tuples.*\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scrub-a-dub-dub: cleaning text data\n",
    "\n",
    "*concepts: lists/sets, loops, string methods, conditionals & booleans*\n",
    "\n",
    "You've *probably* noticed at this point that `tokens` and `lex_freq` contain strings that are \n",
    "not technically words like punctuation or numbers. Or perhaps you've noticed that some word\n",
    "strings contain punctuation or other marks. This kind of \"noise\" is very common in text-mining\n",
    "tasks. \n",
    "\n",
    "We can use the Python skills we've learned so far to clean out some of this extraneous data. Note\n",
    "that some methods are more efficient than others. For now, we will use only what we've learned up\n",
    "till now. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "\n",
    "Isolate the top 100 most-frequent tokens using your function from above.\n",
    "Put them in a list. Examine them and see which ones are punctuation, which\n",
    "ones are mixed text with punctuation? Think about these marks as a set.\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "\n",
    "Write a function that takes a single parameter, a string. Call\n",
    "it `clean_token`. The function should remove any punctuation marks from a string \n",
    "but leave behind valid material. Return the result. If a string \n",
    "is completely punctuation, remove it anyways and return an empty\n",
    "string. You can use a set to manually construct a group of these \n",
    "items and use the set to test the string's membership (see `in`).\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the truth value of an empty string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bool('')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if '':\n",
    "    print('string 1 is True')\n",
    "elif 'a string':\n",
    "    print('string 2 is True')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use the `False` truth value of an empty string to filter spurious strings using `clean_token`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "\n",
    "Write code that uses `clean_token` to filter the top 100 most-common items (from above) and \n",
    "add the filtered strings to a new list `filtered_tokens`. Do not add empty strings to the list.\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "\n",
    "How many items from the original 100 are left in `filtered_tokens`?\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Baby Verb Parser\n",
    "\n",
    "*concepts: lists, dicts, loops, string methods, conditionals, functions*\n",
    "\n",
    "Let's write a simple verb parser. Keep in mind that this parser will\n",
    "be very basic and only work on a few types of patterns. It will also\n",
    "probably not be perfect with some false positives/negatives. That's ok!\n",
    "\n",
    "In most languages, verbs can be recognized with distinctive morphology at\n",
    "either the beginning or the end of a word. For instance, in English the\n",
    "ending \"-ing\" is indicative of either an infinitive or a present tense verb:\n",
    "\n",
    "    running\n",
    "    eating\n",
    "    getting\n",
    "    \n",
    "Similarly the ending \"-ed\" indicates simple past tense verbs. \n",
    "\n",
    "We can store data like this in a dictionary where the keys are\n",
    "distinctive endings/beginnings and the value is a parsing value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parser = {\n",
    "    'ly': 'adverb',\n",
    "    'able': 'adjective',\n",
    "}\n",
    "\n",
    "test = 'capable'\n",
    "\n",
    "for pattern, parse in parser.items():\n",
    "    if test.endswith(pattern):\n",
    "        print(f'It\\'s a(n) {parse}!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "\n",
    "Write a dictionary where the keys are strings that \n",
    "indicate a beginning or ending verb morphology and\n",
    "the values are a given parsing for the language contained\n",
    "in your text. Write at least 5 patterns. But feel free\n",
    "to be more detailed if you'd like.\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "\n",
    "Write a function `verb_parser` that takes two parameters: (1) a string, (2) a\n",
    "dictionary with `pattern:parsing`. The function should use the dictionary \n",
    "to parse the string. If the pattern is not found, the function should return\n",
    "`None`. Otherwise, it should return the parsing value.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "\n",
    "Run in a loop `verb_parser` using `tokens`. Store \n",
    "postive matches as a tuple of `(string, parsing)`\n",
    "in a list: `parsed_tokens`. \n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text Similarity\n",
    "\n",
    "*concepts: string methods, sets, built-in functions, functions*\n",
    "\n",
    "Most texts contain some kind of major section headings (i.e. chapters). \n",
    "We want to measure the similarity of two given setions in the text. We will\n",
    "use sets to do this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "\n",
    "Look at the body of `your_text` and consider where section markers occur. Isolate 3 sections / chapters in your text into three separate variables.\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we can loop twice over an item to compare things pairwise:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "values = [1, 10, 9]\n",
    "\n",
    "for v1 in values:\n",
    "    for v2 in values:\n",
    "        print(f'{v1}-{v2} = {v1-v2}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "\n",
    "In a double loop, tokenize each chapter and compare the overlap of each one with every other chapter \n",
    "in the dataset. Hint: see `.intersection` on sets. Store the comparisons in a list. Can you find\n",
    "which two chapters are most similar? Ignore comparisons of a chapter with itself.\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}