PipelineForClassification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 7: Using a Pipeline for Text Transformation and Classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import os \n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import plot_roc_curve, accuracy_score, roc_auc_score\n",
    "from sklearn.model_selection import train_test_split, GridSearchCV\n",
    "from sklearn.pipeline import Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this assignment, you will practice text vectorization to transform text into numerical feature vectors that can be used to train a classifier. You will then see how to use scikit-learn pipelines to chain together these processes into one step. You will:\n",
    "\n",
    "1. Load the book reviews data set.\n",
    "2. Use a single text column as a feature. \n",
    "3. Transform features using a TF-IDF vectorizer. \n",
    "4. Fit a logistic regression model to the transformed features. \n",
    "5. Evaluate the performance of the model using AUC.\n",
    "6. Set up a scikit-learn pipeline to perform the same tasks above. \n",
    "7. Execute the pipeline and verify that the performance is the same.\n",
    "8. Add a grid search to the pipeline to find the optimal hyperparameter configuration.\n",
    "9. Evaluate the performance of the optimal configuration using ROC-AUC.\n",
    "\n",
    "**<font color='red'>Note: some of the code cells in this notebook may take a while to run</font>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1: Load the Data Set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will work with the book review dataset that you worked with in the sentiment analysis demo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = os.path.join(os.getcwd(), \"data\", \"bookReviews.csv\")\n",
    "df = pd.read_csv(filename, header=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Review</th>\n",
       "      <th>Positive Review</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>This was perhaps the best of Johannes Steinhof...</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>This very fascinating book is a story written ...</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>The four tales in this collection are beautifu...</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>The book contained more profanity than I expec...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>We have now entered a second time of deep conc...</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              Review  Positive Review\n",
       "0  This was perhaps the best of Johannes Steinhof...             True\n",
       "1  This very fascinating book is a story written ...             True\n",
       "2  The four tales in this collection are beautifu...             True\n",
       "3  The book contained more profanity than I expec...            False\n",
       "4  We have now entered a second time of deep conc...             True"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2: Create Training and Test Data Sets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Labeled Examples \n",
    "\n",
    "<b>Task</b>: Create labeled examples from DataFrame `df`. We will have one text feature and one label.  \n",
    "\n",
    "In the code cell below carry out the following steps:\n",
    "\n",
    "* Get the `Positive Review` column from DataFrame `df` and assign it to the variable `y`. This will be our label.\n",
    "* Get the column `Review` from DataFrame `df` and assign it to the variable `X`. This will be our feature.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "y=df[\"Positive Review\"]\n",
    "X=df[\"Review\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<bound method NDFrame.head of 0       This was perhaps the best of Johannes Steinhof...\n",
       "1       This very fascinating book is a story written ...\n",
       "2       The four tales in this collection are beautifu...\n",
       "3       The book contained more profanity than I expec...\n",
       "4       We have now entered a second time of deep conc...\n",
       "                              ...                        \n",
       "1968    I purchased the book with the intention of tea...\n",
       "1969    There are so many design books, but the Graphi...\n",
       "1970    I am thilled to see this book being available ...\n",
       "1971    As many have stated before me the book starts ...\n",
       "1972    I love this book! It is a terrific blend of ha...\n",
       "Name: Review, Length: 1973, dtype: object>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.head"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1973,)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split Labeled Examples into Training and Test Sets\n",
    "\n",
    "<b>Task</b>: In the code cell below create training and test sets out of the labeled examples. \n",
    "\n",
    "1. Use scikit-learn's `train_test_split()` function to create the data sets.\n",
    "\n",
    "2. Specify:\n",
    "    * A test set that is 20 percent (.20) of the size of the data set.\n",
    "    * A seed value of '1234'. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "X_train, X_test, y_train, y_test=train_test_split(X,y, test_size=0.20,random_state=1234)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 3: Implement TF-IDF Vectorizer to Transform Text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Task</b>: Complete the code in the cell below to implement a TF-IDF transformation on the training and test data.\n",
    "Use the \"Transforming Text For a Classifier\" demo as a guide. Follow the following steps:\n",
    "\n",
    "1. Create a `TfidfVectorizer` object and save it to the variable `tfidf_vectorizer`.\n",
    "2. Call `tfidf_vectorizer.fit()` to fit the vectorizer to the training data `X_train`.\n",
    "3. Call the `tfidf_vectorizer.transform()` method to use the fitted vectorizer to transform the training data `X_train`. Save the result to `X_train_tfidf`.\n",
    "4. Call the `tfidf_vectorizer.transform()` method to use the fitted vectorizer to transform the test data `X_test`. Save the result to `X_test_tfidf`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1. Create a TfidfVectorizer object and save it to the variable 'tfidf_vectorizer'\n",
    "# YOUR CODE HERE\n",
    "tfidf_vectorizer=TfidfVectorizer()\n",
    "\n",
    "# 2. Fit the vectorizer to X_train\n",
    "# YOUR CODE HERE\n",
    "tfidf_vectorizer.fit(X_train)\n",
    "\n",
    "\n",
    "# 3. Using the fitted vectorizer, transform the training data and save the data to \n",
    "# variable 'X_train_tfidf'\n",
    "# YOUR CODE HERE\n",
    "X_train_tfidf=tfidf_vectorizer.transform(X_train)\n",
    "\n",
    "\n",
    "# 4. Using the fitted vectorizer, transform the test data and save the data to \n",
    "# variable 'X_test_tfidf'\n",
    "# YOUR CODE HERE\n",
    "\n",
    "X_test_tfidf=tfidf_vectorizer.transform(X_test)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (0, 18965)\t0.059491023406618646\n",
      "  (0, 18727)\t0.08752131471965732\n",
      "  (0, 18642)\t0.03533743581074492\n",
      "  (0, 18593)\t0.03466402255636781\n",
      "  (0, 18539)\t0.10400005525124341\n",
      "  (0, 18496)\t0.09274785194457173\n",
      "  (0, 18455)\t0.0276659188493222\n",
      "  (0, 18126)\t0.051656463721148134\n",
      "  (0, 17733)\t0.1213832032593689\n",
      "  (0, 17680)\t0.1213832032593689\n",
      "  (0, 17618)\t0.06952952155086067\n",
      "  (0, 17302)\t0.047594430972735094\n",
      "  (0, 17259)\t0.0725088662135299\n",
      "  (0, 17226)\t0.04030762713391491\n",
      "  (0, 17133)\t0.1295180018700078\n",
      "  (0, 17117)\t0.049162328805929446\n",
      "  (0, 17104)\t0.07672093808666652\n",
      "  (0, 17066)\t0.047152835152653096\n",
      "  (0, 17061)\t0.04603216704495873\n",
      "  (0, 17053)\t0.043249080127104864\n",
      "  (0, 17044)\t0.03390923842467748\n",
      "  (0, 17040)\t0.02324158038379852\n",
      "  (0, 16805)\t0.05557793152499216\n",
      "  (0, 16288)\t0.1104156579703061\n",
      "  (0, 16266)\t0.05275501870340633\n",
      "  :\t:\n",
      "  (394, 16266)\t0.11591490667341263\n",
      "  (394, 16183)\t0.16526647998794597\n",
      "  (394, 14669)\t0.21441548095429677\n",
      "  (394, 11847)\t0.041179983410956134\n",
      "  (394, 11711)\t0.11266334411684766\n",
      "  (394, 10947)\t0.45702407068616013\n",
      "  (394, 10044)\t0.21075247713547102\n",
      "  (394, 9429)\t0.15769702241845307\n",
      "  (394, 8834)\t0.18665425823869614\n",
      "  (394, 8715)\t0.046490734449862806\n",
      "  (394, 8146)\t0.12026637617557144\n",
      "  (394, 8045)\t0.16211290791224467\n",
      "  (394, 7953)\t0.12465219981184637\n",
      "  (394, 7346)\t0.21441548095429677\n",
      "  (394, 7117)\t0.1884375404488169\n",
      "  (394, 6962)\t0.050473977587676314\n",
      "  (394, 4766)\t0.2667068086286382\n",
      "  (394, 3090)\t0.18031559754953025\n",
      "  (394, 2641)\t0.16924411383550905\n",
      "  (394, 2587)\t0.05758922117953014\n",
      "  (394, 1914)\t0.39331184549337556\n",
      "  (394, 1898)\t0.11328978234234187\n",
      "  (394, 1344)\t0.060280163335188666\n",
      "  (394, 1248)\t0.23485069603224593\n",
      "  (394, 1240)\t0.06240793416993048\n"
     ]
    }
   ],
   "source": [
    "print(X_test_tfidf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 4: Fit a Logistic Regression Model to the Transformed Training Data and Evaluate the Model\n",
    "<b>Task</b>: Complete the code cell below to train a logistic regression model using the TF-IDF features, and compute the AUC on the test set.\n",
    "\n",
    "Follow the following steps:\n",
    "\n",
    "1. Create the `LogisticRegression` model object below and assign to variable `model`. Supply `LogisticRegression()` the following argument: `max_iter=200`.\n",
    "2. Fit the logistic regression model to the transformed training data (`X_train_tfidf` and `y_train`).\n",
    "3. Use the predict_proba() method to make predictions on the test data (`X_test_tfidf`). Save the second column to the variable `probability_predictions`. \n",
    "4. Use the `roc_auc_score()` function to compute the area under the ROC curve for the test data. Call the\n",
    "function with the arguments `y_test` and `probability_predictions`. Save the result to the variable `auc`.\n",
    "5. The 'vocabulary_' attribute of the vectorizer (`tfidf_vectorizer.vocabulary_`) returns the feature space. It returns a dictionary; find the length of the dictionary to get the size of the feature space. Save the result to `len_feature_space`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AUC on the test data: 0.9161\n",
      "The size of the feature space: 19029\n"
     ]
    }
   ],
   "source": [
    "# 1. Create the LogisticRegression model object \n",
    "# YOUR CODE HERE\n",
    "model=LogisticRegression(max_iter=200)\n",
    "\n",
    "# 2. Fit the model to the transformed training data\n",
    "# YOUR CODE HERE\n",
    "model.fit(X_train_tfidf, y_train)\n",
    "\n",
    "# 3. Use the predict_proba() method to make predictions on the test data \n",
    "# YOUR CODE HERE\n",
    "probability_predictions=model.predict_proba(X_test_tfidf)[:,1]\n",
    "\n",
    "\n",
    "# 4. Compute the area under the ROC curve for the test data. \n",
    "# YOUR CODE HERE\n",
    "auc=roc_auc_score(y_test, probability_predictions)\n",
    "print('AUC on the test data: {:.4f}'.format(auc))\n",
    "\n",
    "\n",
    "# 5. Compute the size of the resulting feature space \n",
    "# YOUR CODE HERE\n",
    "len_feature_space=len(tfidf_vectorizer.vocabulary_)\n",
    "\n",
    "print('The size of the feature space: {0}'.format(len_feature_space))\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 5: Experiment with Different Document Frequency Values and Analyze the Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Task</b>: The cell below will loop over a range of 'document frequency' values. For each value, it will fit a vectorizer specifying `ngram_range=(1,2)`. It will then fit a logistic regression model to the transformed data and evaluate the results.   \n",
    "\n",
    "Complete the loop in the cell below by \n",
    "\n",
    "1. adding a list containing four document frequency values that you would like to use (e.g. `[1, 10, 100, 1000]`)\n",
    "2. adding the code you wrote above inside the loop. \n",
    "\n",
    "Note: This may take a short while to run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Document Frequency Value: 1\n",
      "AUC on the test data: 0.9310\n",
      "The size of the feature space: 19029\n",
      "\n",
      "Document Frequency Value: 10\n",
      "AUC on the test data: 0.9254\n",
      "The size of the feature space: 19029\n",
      "\n",
      "Document Frequency Value: 100\n",
      "AUC on the test data: 0.8625\n",
      "The size of the feature space: 19029\n",
      "\n",
      "Document Frequency Value: 1000\n",
      "AUC on the test data: 0.6557\n",
      "The size of the feature space: 19029\n"
     ]
    }
   ],
   "source": [
    "document_frequency_values=[1,10,100,1000]\n",
    "for min_df in document_frequency_values:\n",
    "    \n",
    "    print('\\nDocument Frequency Value: {0}'.format(min_df))\n",
    "\n",
    "    # 1. Create a TfidfVectorizer object and save it to the variable 'tfidf_vectorizer'\n",
    "    # Use the arguments: 'ngram_range=(1,2)'' and 'min_df=min_df'   \n",
    "    # YOUR CODE HERE\n",
    "    tfidf_vectorizer=TfidfVectorizer(ngram_range=(1,2), min_df=min_df)\n",
    "\n",
    "    # 2. Fit the vectorizer to X_train  \n",
    "    # YOUR CODE HERE\n",
    "    tfidf_vectorizer.fit(X_train)\n",
    "    \n",
    "\n",
    "    # 3. Using the fitted vectorizer, transform the training data.\n",
    "    # Save the transformed training data to variable 'X_train_tfidf'\n",
    "    # YOUR CODE HERE\n",
    "    X_train_tfidf=tfidf_vectorizer.transform(X_train)\n",
    "\n",
    "    # 4. Using the fitted vectorizer, transform the test data.\n",
    "    # Save the transformed test data to variable 'X_test_tfidf'\n",
    "    # YOUR CODE HERE\n",
    "    X_test_tfidf=tfidf_vectorizer.transform(X_test)\n",
    "    \n",
    "    # 5. Create the LogisticRegression model object and save it to variable 'model'.\n",
    "    # Call LogisticRegression() with the argument 'max_iter=200'\n",
    "    # YOUR CODE HERE\n",
    "    model=LogisticRegression(max_iter=200)\n",
    "       \n",
    "    \n",
    "    # 6. Fit the model to the transformed training data\n",
    "    # YOUR CODE HERE\n",
    "    model.fit(X_train_tfidf,y_train)\n",
    "\n",
    "\n",
    "    # 7. Use the predict_proba() method to make predictions on the transformed test data. \n",
    "    # Save the second column to the variable 'probability_predictions' \n",
    "    # YOUR CODE HERE\n",
    "    probability_predictions=model.predict_proba(X_test_tfidf)[:,1]\n",
    "    \n",
    "\n",
    "    # 8. Using roc_auc_score() function to compute the AUC. \n",
    "    ## Save the result to the variable 'auc'\n",
    "    # YOUR CODE HERE\n",
    "    auc=roc_auc_score(y_test,probability_predictions)\n",
    "    \n",
    "    \n",
    "    print('AUC on the test data: {:.4f}'.format(auc))\n",
    "    \n",
    "\n",
    "    # 9. Compute the size of the resulting feature space. \n",
    "    # Save the result to the variable 'len_feature_space'\n",
    "    # YOUR CODE HERE\n",
    "\n",
    "\n",
    "    print('The size of the feature space: {0}'.format(len_feature_space))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Task</b>: Which document frequency value and feature space produced the best performing model? Do you notice any patterns regarding the number of document frequency values, the feature space and the AUC? Record your findings in the cell below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As the document frequency value decreases, the feature space size remains constant buthte AUC performance improved.The highest AUC of 0.9310 happened with a document frequency value of 1 and the lowest AUC of 0.6557 happened with a document frequency of 1000. This suggests that considering a smaller set of more informative words can lead to an improvement in model performance in the sentiment analysis tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 6: Set up a TF-IDF + Logistic Regression Pipeline\n",
    "\n",
    "We will look at a new way to chain together various methods to automate the machine learning workflow. We will use  the scikit-learn `Pipeline` utility. For more information, consult the online [documentation](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html). First, let's import `Pipeline`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code cell below will use a scikit-learn pipeline to perform TF-IDF vectorization and the fitting of a logistic regression model to the transformed data.\n",
    "\n",
    "This will be implemented in the following steps:\n",
    "\n",
    "1. First we will create a list containing the steps to perform in the pipeline. Items in the list will be executed in the order in which they appear.\n",
    "\n",
    "    Each item in the list is a tuple consisting of two items: \n",
    "    1. A descriptive name of what is being performed. You can create any name you'd like.\n",
    "    2. The code to run.\n",
    "    \n",
    "    \n",
    "2. Next we will create a Pipeline object and supply it the list of steps using the `step` parameter\n",
    "\n",
    "\n",
    "3. We will use this pipeline as we would any model object and fit this pipeline to the original training data. Note that when calling the `fit()` method on the pipeline object, all of the steps in the pipeline are performed on the data.\n",
    "\n",
    "\n",
    "4. Finally, we will use pipeline object to make predictions on the original test data. When calling the `predict_proba()` method on the pipeline object, all of the steps in the pipeline are performed on the data. \n",
    "\n",
    "\n",
    "<b>Task:</b> In the code cell below, complete step 3 and 4 using the pipeline object  `model_pipeline`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Begin ML pipeline...\n",
      "End pipeline\n"
     ]
    }
   ],
   "source": [
    "print('Begin ML pipeline...')\n",
    "\n",
    "# 1. Define the list of steps:\n",
    "s = [\n",
    "        (\"vectorizer\", TfidfVectorizer(ngram_range=(1,2), min_df=10)),\n",
    "        (\"model\", LogisticRegression(max_iter=200))\n",
    "    ]\n",
    "\n",
    "# 2. Define the pipeline:\n",
    "model_pipeline = Pipeline(steps=s)\n",
    "\n",
    "# We can use the pipeline the way would would use a model object \n",
    "# when fitting the model on the training data and testing on the test data:\n",
    "\n",
    "# 3. Fit the pipeline to the training data\n",
    "# YOUR CODE HERE\n",
    "model_pipeline.fit(X_train, y_train)\n",
    "\n",
    "# 4. Make predictions on the test data\n",
    "# Save the second column to the variable 'probability_predictions'\n",
    "# YOUR CODE HERE\n",
    "probability_predictions=model_pipeline.predict_proba(X_test)[:,1]\n",
    "\n",
    "print('End pipeline')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's compare the performance of our model. \n",
    "\n",
    "<b>Task</b>: In the code cell below, call the function `roc_auc_score()` with arguments `y_test` and `probability_predictions`. Save the results to the variable `auc_score`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AUC on the test data: 0.9254\n"
     ]
    }
   ],
   "source": [
    "# Evaluate the performance by computing the AUC\n",
    "\n",
    "auc_score = roc_auc_score(y_test, probability_predictions)\n",
    "\n",
    "print('AUC on the test data: {:.4f}'.format(auc_score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In some case, scikit-learn gives you the ability to provide a pipeline object as an argument to a function. One such function is `plot_roc_curve()`. You'll see in the online [documentation](https://scikit-learn.org/0.23/modules/generated/sklearn.metrics.plot_roc_curve.html) that this function can take a pipeline (estimator) as an argument. Calling `plot_roc_curve()` with the pipeline and the test data will accomplish the same tasks as steps 3 and 4 in the code cell above.\n",
    "\n",
    "Let's import the function and try it out.\n",
    "\n",
    "<b>Task:</b> Call `plot_roc_curve()` with the following three arguments:\n",
    "1. The pipeline object `model_pipeline`\n",
    "2.  `X_test`\n",
    "3. `y_test`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAhA0lEQVR4nO3deZxU1Z338c83LEJco2BGWQJGdFgFbQXGlWCUGAf0EXGdGUefmCeRxDwuiY4ZJYwTk4DOy0xMDDGm4wQR4hI7BpdJImPcQFCC0ERFRWiWKGgUgoDE3/xxb3eKppdqum9Vd9X3/XrVq+9y6t7fpZv61Tnn3nMUEZiZWfn6SLEDMDOz4nIiMDMrc04EZmZlzonAzKzMORGYmZW5zsUOoKV69OgR/fr1K3YYZmYdyqJFizZERM+G9nW4RNCvXz8WLlxY7DDMzDoUSW80ts9NQ2ZmZc6JwMyszDkRmJmVOScCM7My50RgZlbmMksEku6U9KakpY3sl6TvSlohaYmkI7OKxczMGpdljaASGNfE/s8AA9LXpcAPMozFzMwakdlzBBHxhKR+TRSZANwVyTjYz0raT9JBEbEuq5jMzPJ19/xVPLh4TbHD2Mmgg/fhhr8f3ObHLeYDZb2A1TnrNem2XRKBpEtJag307du3IMGZWcfTlh/e819/G4CR/fdvk+O1Zx3iyeKImAHMAKioqPBMOmbNaI/fZguhLT+8R/bfnwnDe3H+yNL/8lnMRLAG6JOz3jvdZmYtVP+Dv5y+zeYqpw/vtlTMRFAFTJZ0DzASeNf9A2b5ae6D3x+I1hKZJQJJs4CTgB6SaoAbgC4AEXE7MBc4DVgBbAH+OatYzIohy+YZf/BbW8ryrqHzmtkfwGVZnd+sLbTmwzzL5hl/8Ftb6hCdxWbF8uDiNVSve49BB+3T4vf6w9o6CicCKxu78+2+NgnM/vzojKIyKz4nAmsXCnG74+401Qw6aB8mDO+VVUhm7YITgRXd3fNX8S8PvAhke7ujm2rMGuZEYJlr7tt+7Tf1b5451B/SZkXgRGCZqU0AzTXJ+Ju6WXE5EVibyv32n5sA/EFv1n45EViTWtqJm/vh7wRg1jE4EViD8m3Wqc8f/mYdjxOBAU2PXeMPdrPS5kRQZhpr6vHYNWbly4mgxOU7PLE/+M3KlxNBCWvoQS1/4JtZfU4EJay2JuAHtcysKU4EJai2Oah63XuM7L+/k4CZNekjxQ7A2l7u0MkeMM3MmuMaQYm5e/4q5r/+NiP77++hk80sL04EJaL+A2CuCZhZvpwISkD9u4N8V5CZtYQTQQnw3UFm1hpOBB2Y7w4ys7bgRNBO5TPqZ/3xgMzMdocTQTuVewtoY9wfYGZtwYmgHRt00D6+BdTMMucHyszMypxrBO1MbgdwU81CZmZtxYmgyJqbEMbMLGtOBEVW/9u/O4DNrNCcCNoBdwqbWTE5ERRY/aYg9wWYWbH5rqECqh0TqLYfAPBQ0WZWdJnWCCSNA24FOgF3RMS36u3vC/wU2C8tc01EzM0ypmKoPzKoxwQys/Yks0QgqRNwG/BpoAZ4TlJVRFTnFPs6MCcifiBpEDAX6JdVTMWSOx6QO4LNrL3JskZwDLAiIl4DkHQPMAHITQQB1DaQ7wuszTCeonKHsJm1V1n2EfQCVues16Tbck0BLpRUQ1Ib+FJDB5J0qaSFkha+9dZbWcRqZla2it1ZfB5QGRG9gdOA/5K0S0wRMSMiKiKiomfPngUP0syslGWZCNYAfXLWe6fbcl0CzAGIiGeAbkCPDGMyM7N6skwEzwEDJPWX1BU4F6iqV2YVMBZA0kCSROC2HzOzAsosEUTEDmAy8CiwnOTuoGWSpkoanxa7EvicpN8Ds4CLIiKyisnMzHaV6XME6TMBc+ttuz5nuRo4NssYzMysacXuLDYzsyJzIjAzK3NOBGZmZc6JwMyszDkRZOzu+at2Gm3UzKy98XwEGak/4qiHmjaz9sqJYDfUn1ymIfXnHvaIo2bWXjkR7Ib68ww3xAnAzDoKJ4IWqK0J1CYBDyttZqXAncUtkJsE3OZvZqUi7xqBpI9GxJYsg+kIXBMws1LTbI1A0t9Jqgb+kK4fIen7mUdmZmYFkU/T0H8ApwIbASLi98AJWQZlZmaFk1cfQUSsrrfpLxnEYmZmRZBPH8FqSX8HhKQuwOUk8wuYmVkJyKdG8P+Ay0gmnl8DDAe+mGFMZmZWQPnUCA6PiAtyN0g6Fngqm5DMzKyQ8qkR/Gee28zMrANqtEYgaTTwd0BPSVfk7NoH6JR1YO1N7SiiI/vvX+xQzMzaVFNNQ12BvdIye+dsfw+YmGVQ7YlHETWzUtdoIoiI/wH+R1JlRLxRwJjaldphJTyInJmVqnw6i7dImgYMBrrVboyIT2UWVTuR2xzkYSXMrFTl01k8k2R4if7AN4CVwHMZxtRu1M454OYgMytl+dQIDoiIH0u6PKe5qGQTQe6kM7VNQm4OMrNSlk+N4IP05zpJn5U0AijZW2dyO4Y93LSZlYN8agQ3StoXuJLk+YF9gK9kGVSxuU/AzMpJs4kgIh5KF98FxkDdk8VmZlYCmnqgrBMwiWSMoUciYqmk04F/AboDIwoTopmZZampGsGPgT7AAuC7ktYCFcA1EfGLAsRmZmYF0FQiqACGRcSHkroB64FPRsTGwoRmZmaF0NRdQ9sj4kOAiNgKvNbSJCBpnKSXJK2QdE0jZSZJqpa0TNLdLTl+W7p7/irO+eEzVK97r1ghmJkVRVM1gr+VtCRdFvDJdF1ARMSwpg6c9jHcBnwaqAGek1QVEdU5ZQYA1wLHRsQ7kg5sxbW0Su1QEr5l1MzKTVOJYGArj30MsCIiXgOQdA8wAajOKfM54LaIeAcgIt5s5TlbZdBB+/i2UTMrO00NOtfageZ6AblzHdcAI+uVOQxA0lMkQ1tPiYhH6h9I0qXApQB9+/opXzOztpTX5PUZ6gwMAE4CzgN+JGm/+oUiYkZEVERERc+ePQsboZlZicsyEawhuf20Vu90W64aoCoiPoiI14GXSRKDmZkVSF6JQFJ3SYe38NjPAQMk9ZfUFTgXqKpX5hcktQEk9SBpKnqthecxM7NWaDYRSPp7YDHwSLo+XFL9D/RdRMQOYDLwKLAcmBMRyyRNlTQ+LfYosFFSNfA4cLWfUzAzK6x8Bp2bQnIH0DyAiFgsqX8+B4+IucDcetuuz1kO4Ir0ZWZmRZDXMNQR8W69bZFFMGZmVnj51AiWSTof6JQ+APZl4OlswzIzs0LJp0bwJZL5ircBd5MMR/2VDGMquNq5ic3MylE+NYK/jYjrgOuyDqZYPDexmZWzfBLBzZL+BrgXmB0RSzOOqWBq5yf23MRmVs6abRqKiDEkM5O9BfxQ0ouSvp55ZAXggebMzPKrERAR60kmp3kc+CpwPXBjloEVigeaM7Nyl88DZQMlTZH0Isnk9U+TDBfRYXnuATOzv8qnRnAnMBs4NSLWZhxPQbhJyMzsr5pNBBFRku0mbhIyM0s0mggkzYmISWmTUO6TxHnNUGZmZh1DUzWCy9OfpxciEDMzK45GO4sjYl26+MWIeCP3BXyxMOGZmVnW8hli4tMNbPtMWwdiZmbF0VQfwRdIvvkfImlJzq69gaeyDszMzAqjqT6Cu4GHgZuAa3K2b4oIj9BmZlYimkoEERErJV1Wf4ek/Z0MzMxKQ3M1gtOBRSS3jypnXwCHZBiXmZkVSKOJICJOT3/mNS2lmZl1TPmMNXSspD3T5Qsl3SLJ4zWbmZWIfG4f/QGwRdIRwJXAq8B/ZRqVmZkVTD6JYEdEBDAB+F5E3EZyC6mZmZWAfEYf3STpWuAfgOMlfQTokm1YZmZWKPnUCM4hmbj+4nSCmt7AtEyjMjOzgslnqsr1wExgX0mnA1sj4q7MIzMzs4LI566hScAC4GxgEjBf0sSsAzMzs8LIp4/gOuDoiHgTQFJP4NfAvVkGZmZmhZFPH8FHapNAamOe7zMzsw4gnxrBI5IeBWal6+cAc7MLyczMCimfOYuvlvR/gOPSTTMi4oFswzIzs0Jpaj6CAcB04JPAi8BVEbGmUIGZmVlhNNXWfyfwEHAWyQik/9nSg0saJ+klSSskXdNEubMkhaSKlp7DzMxap6mmob0j4kfp8kuSnm/JgSV1Am4jmeqyBnhOUlVEVNcrtzdwOTC/Jcc3M7O20VQi6CZpBH+dh6B77npENJcYjgFWRMRrAJLuIRmvqLpeuX8Dvg1c3cLYzcysDTSVCNYBt+Ssr89ZD+BTzRy7F7A6Z70GGJlbQNKRQJ+I+JWkRhOBpEuBSwH69vUI2GZmbampiWnGZHnidPC6W4CLmisbETOAGQAVFRWRZVxmZuUmywfD1gB9ctZ7p9tq7Q0MAeZJWgmMAqrcYWxmVlhZJoLngAGS+kvqCpwLVNXujIh3I6JHRPSLiH7As8D4iFiYYUxmZlZPZokgInYAk4FHgeXAnIhYJmmqpPFZndfMzFqm2SeLJQm4ADgkIqam8xX/TUQsaO69ETGXesNRRMT1jZQ9Ka+IzcysTeVTI/g+MBo4L13fRPJ8gJmZlYB8Bp0bGRFHSnoBICLeSdv8zcysBORTI/ggfUo4oG4+gg8zjcrMzAomn0TwXeAB4EBJ/w48CXwz06jMzKxg8hmGeqakRcBYkuElzoiI5ZlHZmZmBZHPXUN9gS3AL3O3RcSqLAMzM7PCyKez+Fck/QMCugH9gZeAwRnGZWZmBZJP09DQ3PV0oLgvZhaRmZkVVIufLE6Hnx7ZbEEzM+sQ8ukjuCJn9SPAkcDazCIyM7OCyqePYO+c5R0kfQb3ZROOmZkVWpOJIH2QbO+IuKpA8ZiZWYE12kcgqXNE/AU4toDxmJlZgTVVI1hA0h+wWFIV8HPgz7U7I+L+jGPLxN3zVzH/9bcZ2X//YodiZtYu5NNH0A3YSDJHce3zBAF0yETw4OJkkrQJw3sVORIzs/ahqURwYHrH0FL+mgBqdeh5g0f235/zR/YtdhhmZu1CU4mgE7AXOyeAWh06EZiZ2V81lQjWRcTUgkViZmZF0dSTxQ3VBMzMrMQ0lQjGFiwKMzMrmkYTQUS8XchAzMysOFo86JyZmZUWJwIzszLnRGBmVuacCMzMypwTgZlZmXMiMDMrc04EZmZlzonAzKzMORGYmZW5TBOBpHGSXpK0QtI1Dey/QlK1pCWSfiPpE1nGY2Zmu8osEaTzHd8GfAYYBJwnaVC9Yi8AFRExDLgX+E5W8ZiZWcOyrBEcA6yIiNciYjtwDzAht0BEPB4RW9LVZ4HeGcZjZmYNyDIR9AJW56zXpNsacwnwcEM7JF0qaaGkhW+99VYbhmhmZu2is1jShUAFMK2h/RExIyIqIqKiZ8+ehQ3OzKzE5TN5/e5aA/TJWe+dbtuJpJOB64ATI2JbhvGYmVkDsqwRPAcMkNRfUlfgXKAqt4CkEcAPgfER8WaGsZiZWSMySwQRsQOYDDwKLAfmRMQySVMljU+LTQP2An4uabGkqkYOZ2ZmGcmyaYiImAvMrbft+pzlk7M8v5mZNa9ddBabmVnxOBGYmZU5JwIzszLnRGBmVuacCMzMypwTgZlZmXMiMDMrc04EZmZlzonAzKzMORGYmZU5JwIzszLnRGBmVuacCMzMypwTgZlZmXMiMDMrc04EZmZlzonAzKzMORGYmZU5JwIzszLnRGBmVuacCMzMylznYgdgZrv64IMPqKmpYevWrcUOxTqYbt260bt3b7p06ZL3e5wIzNqhmpoa9t57b/r164ekYodjHUREsHHjRmpqaujfv3/e7yubpqG756/inB8+Q/W694odilmztm7dygEHHOAkYC0iiQMOOKDFNcmySQQPLl5D9br3GHTQPkwY3qvY4Zg1y0nAdsfu/N2UVdPQoIP2YfbnRxc7DDOzdqVsagRm1jKdOnVi+PDhDBkyhLPPPpstW7awcOFCvvzlL+/2Mffaay8A1q5dy8SJE9sqVL7yla/wxBNP1K1v2LCBLl26cPvttzd4/lqVlZVMnjy5bv2uu+5iyJAhDB06lBEjRjB9+vRWx/bII49w+OGHc+ihh/Ktb32rwTJvvPEGY8eOZdiwYZx00knU1NTUbT/yyCMZPnw4gwcP3ul6Tj75ZN55551WxwcknQsd6XXUUUfF7ph0+9Mx6fand+u9ZoVWXV1d7BBizz33rFs+//zz4+abb27TY7aVDRs2xMiRI3fa9v3vfz+OO+64OOGEE5o8/09+8pO47LLLIiJi7ty5MWLEiFizZk1ERGzdujVmzJjRqth27NgRhxxySLz66quxbdu2GDZsWCxbtmyXchMnTozKysqIiPjNb34TF154YUREbNu2LbZu3RoREZs2bYpPfOITdfFVVlbGjTfe2OB5G/r7ARZGI5+rZdU0ZNYRfeOXy6he27Y3OQw6eB9u+PvBeZc//vjjWbJkCfPmzWP69Ok89NBDTJkyhVdffZUVK1awYcMGvvrVr/K5z30OgGnTpjFnzhy2bdvGmWeeyTe+8Y2djrdy5UpOP/10li5dSmVlJVVVVWzZsoVXX32VM888k+985zsAPPbYY9xwww1s27aNT37yk/zkJz/Z5Vv9fffdx7hx43baNmvWLG6++WbOP/98ampq6N27d7PXeNNNNzF9+nQOPvhgAPbYY4+669ldCxYs4NBDD+WQQw4B4Nxzz+XBBx9k0KBBO5Wrrq7mlltuAWDMmDGcccYZAHTt2rWuzLZt2/jwww/r1sePH8/xxx/Pdddd16oYwU1DZtaMHTt28PDDDzN06NBd9i1ZsoTf/va3PPPMM0ydOpW1a9fy2GOP8corr7BgwQIWL17MokWLdmq2acjixYuZPXs2L774IrNnz2b16tVs2LCBG2+8kV//+tc8//zzVFRU1H1Y5nrqqac46qij6tZXr17NunXrOOaYY5g0aRKzZ8/O6zqXLl2603EaM3PmTIYPH77Lq6GmrjVr1tCnT5+69d69e7NmzZpdyh1xxBHcf//9ADzwwANs2rSJjRs31l3PsGHD6NOnD1/72tfqEtXHPvYxtm3bVleuNVwjMGvnWvLNvS29//77DB8+HEhqBJdccglPP/30TmUmTJhA9+7d6d69O2PGjGHBggU8+eSTPPbYY4wYMQKAzZs388orr3DCCSc0eq6xY8ey7777AjBo0CDeeOMN/vSnP1FdXc2xxx4LwPbt2xk9etebPdatW0fPnj3r1mfPns2kSZOA5Bv4xRdfzJVXXtnouVt6l80FF1zABRdc0KL3NGf69OlMnjyZyspKTjjhBHr16kWnTp0A6NOnD0uWLGHt2rWcccYZTJw4kY9//OMAHHjggaxdu5YDDjigVefPNBFIGgfcCnQC7oiIb9XbvwdwF3AUsBE4JyJWZhmTmeWne/fuLF68uMky9T9EJRERXHvttXz+85/P+1x77LFH3XKnTp3YsWMHEcGnP/1pZs2a1WycuffNz5o1i/Xr1zNz5kwg6Zh+5ZVXGDBgAN27d2f79u11TS5vv/02PXr0AGDw4MEsWrSIT33qU02eb+bMmUybNm2X7Yceeij33nvvTtt69erF6tWr69Zramro1WvX29cPPvjguhrB5s2bue+++9hvv/12KTNkyBB+97vf1dU+tm7dSvfu3ZuMNx+ZNQ1J6gTcBnwGGAScJ2lQvWKXAO9ExKHAfwDfzioeM2t7Dz74IFu3bmXjxo3MmzePo48+mlNPPZU777yTzZs3A0nzyJtvvtniY48aNYqnnnqKFStWAPDnP/+Zl19+eZdyAwcOrCvz8ssvs3nzZtasWcPKlStZuXIl1157bV0yOfHEE/nZz34GJDWeOXPmMGbMGACuvfZarr76atavXw8kNZA77rhjl/NdcMEFLF68eJdX/SQAcPTRR/PKK6/w+uuvs337du655x7Gjx+/S7kNGzbUtf/fdNNNXHzxxUCSON5//30A3nnnHZ588kkOP/xwILnRZ/369fTr1y/Pf9HGZdlHcAywIiJei4jtwD3AhHplJgA/TZfvBcbKT9GYdRjDhg1jzJgxjBo1in/913/l4IMP5pRTTuH8889n9OjRDB06lIkTJ7Jp06YWH7tnz55UVlZy3nnnMWzYMEaPHs0f/vCHXcp99rOfZd68eUBSGzjzzDN32n/WWWfVJYJbb72V+++/n+HDhzNq1CjOPvvsuiar0047jcmTJ3PyySczePBgjjzySN57r3Wd9J07d+Z73/sep556KgMHDmTSpEkMHpw09V1//fVUVVUBMG/ePA4//HAOO+ww/vjHP9Z1AC9fvpyRI0dyxBFHcOKJJ3LVVVfV9dUsWrSIUaNG0blz6xt2lNxV1PYkTQTGRcT/Tdf/ARgZEZNzyixNy9Sk66+mZTbUO9alwKUAffv2PeqNN95ocTzf+OUyoHjtrWYtsXz5cgYOHFjsMJo0ZcoU9tprL6666qpih8Jxxx3HQw89tEtzSim7/PLLGT9+PGPHjt1lX0N/P5IWRURFQ8fqEJ3FETEDmAFQUVGxW5nLCcCsdN18882sWrWqrBLBkCFDGkwCuyPLRLAG6JOz3jvd1lCZGkmdgX1JOo3NrJ2bMmVKsUOoM3LkyGKHUHCtfcYhV5Z9BM8BAyT1l9QVOBeoqlemCvindHki8NvIqq3KrIPxfwXbHbvzd5NZIoiIHcBk4FFgOTAnIpZJmiqpttv8x8ABklYAVwDXZBWPWUfSrVs3Nm7c6GRgLRLpfATdunVr0fsy6yzOSkVFRSxcuLDYYZhlyjOU2e5qbIayDt9ZbFZuunTp0qIZpsxaw2MNmZmVOScCM7My50RgZlbmOlxnsaS3gJY/WpzoAWxotlRp8TWXB19zeWjNNX8iIno2tKPDJYLWkLSwsV7zUuVrLg++5vKQ1TW7acjMrMw5EZiZlblySwQzih1AEfiay4OvuTxkcs1l1UdgZma7KrcagZmZ1eNEYGZW5koyEUgaJ+klSSsk7TKiqaQ9JM1O98+X1K8IYbapPK75CknVkpZI+o2kTxQjzrbU3DXnlDtLUkjq8Lca5nPNkialv+tlku4udIxtLY+/7b6SHpf0Qvr3fVox4mwrku6U9GY6g2ND+yXpu+m/xxJJR7b6pBFRUi+gE/AqcAjQFfg9MKhemS8Ct6fL5wKzix13Aa55DPDRdPkL5XDNabm9gSeAZ4GKYsddgN/zAOAF4GPp+oHFjrsA1zwD+EK6PAhYWey4W3nNJwBHAksb2X8a8DAgYBQwv7XnLMUawTHAioh4LSK2A/cAE+qVmQD8NF2+FxgrSQWMsa01e80R8XhEbElXnyWZMa4jy+f3DPBvwLeBUhjPOZ9r/hxwW0S8AxARbxY4xraWzzUHsE+6vC+wtoDxtbmIeAJ4u4kiE4C7IvEssJ+kg1pzzlJMBL2A1TnrNem2BstEMoHOu8ABBYkuG/lcc65LSL5RdGTNXnNaZe4TEb8qZGAZyuf3fBhwmKSnJD0raVzBostGPtc8BbhQUg0wF/hSYUIrmpb+f2+W5yMoM5IuBCqAE4sdS5YkfQS4BbioyKEUWmeS5qGTSGp9T0gaGhF/KmZQGTsPqIyImyWNBv5L0pCI+LDYgXUUpVgjWAP0yVnvnW5rsIykziTVyY0FiS4b+Vwzkk4GrgPGR8S2AsWWleaueW9gCDBP0kqSttSqDt5hnM/vuQaoiogPIuJ14GWSxNBR5XPNlwBzACLiGaAbyeBspSqv/+8tUYqJ4DlggKT+krqSdAZX1StTBfxTujwR+G2kvTAdVLPXLGkE8EOSJNDR242hmWuOiHcjokdE9IuIfiT9IuMjoiPPc5rP3/YvSGoDSOpB0lT0WgFjbGv5XPMqYCyApIEkieCtgkZZWFXAP6Z3D40C3o2Ida05YMk1DUXEDkmTgUdJ7ji4MyKWSZoKLIyIKuDHJNXHFSSdMucWL+LWy/OapwF7AT9P+8VXRcT4ogXdSnlec0nJ85ofBU6RVA38Bbg6IjpsbTfPa74S+JGk/0/ScXxRR/5iJ2kWSTLvkfZ73AB0AYiI20n6QU4DVgBbgH9u9Tk78L+XmZm1gVJsGjIzsxZwIjAzK3NOBGZmZc6JwMyszDkRmJmVOScCa5ck/UXS4pxXvybKbm6D81VKej091/PpE6otPcYdkgaly/9Sb9/TrY0xPU7tv8tSSb+UtF8z5Yd39NE4LXu+fdTaJUmbI2Kvti7bxDEqgYci4l5JpwDTI2JYK47X6piaO66knwIvR8S/N1H+IpJRVye3dSxWOlwjsA5B0l7pPArPS3pR0i4jjUo6SNITOd+Yj0+3nyLpmfS9P5fU3Af0E8Ch6XuvSI+1VNJX0m17SvqVpN+n289Jt8+TVCHpW0D3NI6Z6b7N6c97JH02J+ZKSRMldZI0TdJz6Rjzn8/jn+UZ0sHGJB2TXuMLkp6WdHj6JO5U4Jw0lnPS2O+UtCAt29CIrVZuij32tl9+NfQieSp2cfp6gOQp+H3SfT1InqqsrdFuTn9eCVyXLnciGW+oB8kH+57p9q8B1zdwvkpgYrp8NjAfOAp4EdiT5KnsZcAI4CzgRznv3Tf9OY90zoPamHLK1MZ4JvDTdLkrySiS3YFLga+n2/cAFgL9G4hzc871/RwYl67vA3ROl08G7kuXLwK+l/P+bwIXpsv7kYxFtGexf99+FfdVckNMWMl4PyKG165I6gJ8U9IJwIck34Q/DqzPec9zwJ1p2V9ExGJJJ5JMVvJUOrRGV5Jv0g2ZJunrJOPUXEIyfs0DEfHnNIb7geOBR4CbJX2bpDnpdy24roeBWyXtAYwDnoiI99PmqGGSJqbl9iUZLO71eu/vLmlxev3Lgf/OKf9TSQNIhlno0sj5TwHGS7oqXe8G9E2PZWXKicA6iguAnsBREfGBkhFFu+UWiIgn0kTxWaBS0i3AO8B/R8R5eZzj6oi4t3ZF0tiGCkXEy0rmOjgNuFHSbyJiaj4XERFbJc0DTgXOIZloBZLZpr4UEY82c4j3I2K4pI+SjL9zGfBdkgl4Ho+IM9OO9XmNvF/AWRHxUj7xWnlwH4F1FPsCb6ZJYAywy5zLSuZh/mNE/Ai4g2S6v2eBYyXVtvnvKemwPM/5O+AMSR+VtCdJs87vJB0MbImIn5EM5tfQnLEfpDWThswmGSistnYByYf6F2rfI+mw9JwNimS2uS8DV+qvQ6nXDkV8UU7RTSRNZLUeBb6ktHqkZFRaK3NOBNZRzAQqJL0I/CPwhwbKnAT8XtILJN+2b42It0g+GGdJWkLSLPS3+ZwwIp4n6TtYQNJncEdEvAAMBRakTTQ3ADc28PYZwJLazuJ6HiOZGOjXkUy/CEniqgaeVzJp+Q9ppsaexrKEZGKW7wA3pdee+77HgUG1ncUkNYcuaWzL0nUrc7591MyszLlGYGZW5pwIzMzKnBOBmVmZcyIwMytzTgRmZmXOicDMrMw5EZiZlbn/BXaMIgt2eDCJAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.metrics import plot_roc_curve\n",
    "\n",
    "plot_roc_curve(model_pipeline, X_test, y_test)\n",
    "plt.show()\n",
    "\n",
    "# YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that in newer versions of scikit-learn, this function has been replaced by [RocCurveDisplay](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.RocCurveDisplay.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 7: Perform a GridSearchCV on the Pipeline to Find the Best Hyperparameters \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will perform a grid search on the pipeline object `model_pipeline` to find the hyperparameter configuration for hyperparameter $C$ (for the logistic regression) and for the $ngram\\_range$ (for the TF-IDF vectorizer) that result in the best cross-validation score.\n",
    "\n",
    "<b>Task:</b> Define a parameter grid to pass to `GridSearchCV()`. Recall that the parameter grid is a dictionary. Name the dictionary `param_grid`.\n",
    "\n",
    "The dictionary should contain two key value pairs:\n",
    "\n",
    "1. a key specifying the  $C$ hyperparameter name, and a value containing the list `[0.1, 1, 10]`.\n",
    "2. a key specifying the $ngram\\_range$ hyperparameter name, and a value containing the list `[(1,1), (1,2)]`.\n",
    "\n",
    "Note that following:\n",
    "\n",
    "When running a grid search on a pipelines, the hyperparameter names you specify in the parameter grid are the names of the pipeline items (the descriptive names you provided to the items in the pipeline) followed by two underscores, followed by the actual hyperparameter names. \n",
    "\n",
    "For example, note what we named the pipeline items above:\n",
    "\n",
    "```\n",
    "s = [\n",
    "        (\"vectorizer\", TfidfVectorizer(ngram_range=(1,2), min_df=10)),\n",
    "        (\"model\", LogisticRegression(max_iter=200))\n",
    "    ]\n",
    "```\n",
    "\n",
    "We named the the classifier `model` and the vectorizer `vectorizer`. \n",
    "\n",
    "Since we named our classifier `model`, the hyperparameter name for $C$ that you would specify as they key in `param_grid` is `model__C`. You can find a list containing possible pipeline hyperparameter names you can use by running the code the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_pipeline.get_params().keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'model__C': [0.1, 1, 10], 'vectorizer__ngram_range': [(1, 1), (1, 2)]}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# YOUR CODE HERE\n",
    "param_grid={\n",
    "    'model__C': [0.1,1,10],\n",
    "    \"vectorizer__ngram_range\":[(1,1),(1,2)]\n",
    "}\n",
    "\n",
    "\n",
    "param_grid"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Task:</b> Run a grid search on the pipeline.\n",
    "\n",
    "1. Call `GridSearchCV()` with the following arguments:\n",
    "\n",
    "    1. Pipeline object `model_pipeline`.\n",
    "    2. Parameter grid `param_grid`.\n",
    "    3. Specify 3 cross validation folds using the `cv` parameter.\n",
    "    4. Specify that the scoring method is `roc_auc` using the `scoring` parameter.\n",
    "    5. To monitor the progress of the grid search, supply the argument `verbose=2`.\n",
    "    \n",
    "    Assign the output to the object `grid`.\n",
    "    \n",
    "    \n",
    "2. Fit `grid` on the training data (`X_train` and `y_train`) and assign the result to variable `grid_search`.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running Grid Search...\n",
      "Fitting 3 folds for each of 6 candidates, totalling 18 fits\n",
      "[CV] model__C=0.1, vectorizer__ngram_range=(1, 1) ....................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ..... model__C=0.1, vectorizer__ngram_range=(1, 1), total=   0.3s\n",
      "[CV] model__C=0.1, vectorizer__ngram_range=(1, 1) ....................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.3s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ..... model__C=0.1, vectorizer__ngram_range=(1, 1), total=   0.3s\n",
      "[CV] model__C=0.1, vectorizer__ngram_range=(1, 1) ....................\n",
      "[CV] ..... model__C=0.1, vectorizer__ngram_range=(1, 1), total=   0.3s\n",
      "[CV] model__C=0.1, vectorizer__ngram_range=(1, 2) ....................\n",
      "[CV] ..... model__C=0.1, vectorizer__ngram_range=(1, 2), total=   1.1s\n",
      "[CV] model__C=0.1, vectorizer__ngram_range=(1, 2) ....................\n",
      "[CV] ..... model__C=0.1, vectorizer__ngram_range=(1, 2), total=   1.1s\n",
      "[CV] model__C=0.1, vectorizer__ngram_range=(1, 2) ....................\n",
      "[CV] ..... model__C=0.1, vectorizer__ngram_range=(1, 2), total=   1.1s\n",
      "[CV] model__C=1, vectorizer__ngram_range=(1, 1) ......................\n",
      "[CV] ....... model__C=1, vectorizer__ngram_range=(1, 1), total=   0.4s\n",
      "[CV] model__C=1, vectorizer__ngram_range=(1, 1) ......................\n",
      "[CV] ....... model__C=1, vectorizer__ngram_range=(1, 1), total=   0.4s\n",
      "[CV] model__C=1, vectorizer__ngram_range=(1, 1) ......................\n",
      "[CV] ....... model__C=1, vectorizer__ngram_range=(1, 1), total=   0.4s\n",
      "[CV] model__C=1, vectorizer__ngram_range=(1, 2) ......................\n",
      "[CV] ....... model__C=1, vectorizer__ngram_range=(1, 2), total=   1.2s\n",
      "[CV] model__C=1, vectorizer__ngram_range=(1, 2) ......................\n",
      "[CV] ....... model__C=1, vectorizer__ngram_range=(1, 2), total=   1.2s\n",
      "[CV] model__C=1, vectorizer__ngram_range=(1, 2) ......................\n",
      "[CV] ....... model__C=1, vectorizer__ngram_range=(1, 2), total=   1.2s\n",
      "[CV] model__C=10, vectorizer__ngram_range=(1, 1) .....................\n",
      "[CV] ...... model__C=10, vectorizer__ngram_range=(1, 1), total=   0.4s\n",
      "[CV] model__C=10, vectorizer__ngram_range=(1, 1) .....................\n",
      "[CV] ...... model__C=10, vectorizer__ngram_range=(1, 1), total=   0.4s\n",
      "[CV] model__C=10, vectorizer__ngram_range=(1, 1) .....................\n",
      "[CV] ...... model__C=10, vectorizer__ngram_range=(1, 1), total=   0.4s\n",
      "[CV] model__C=10, vectorizer__ngram_range=(1, 2) .....................\n",
      "[CV] ...... model__C=10, vectorizer__ngram_range=(1, 2), total=   1.3s\n",
      "[CV] model__C=10, vectorizer__ngram_range=(1, 2) .....................\n",
      "[CV] ...... model__C=10, vectorizer__ngram_range=(1, 2), total=   1.2s\n",
      "[CV] model__C=10, vectorizer__ngram_range=(1, 2) .....................\n",
      "[CV] ...... model__C=10, vectorizer__ngram_range=(1, 2), total=   1.2s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  18 out of  18 | elapsed:   13.9s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done\n"
     ]
    }
   ],
   "source": [
    "print('Running Grid Search...')\n",
    "\n",
    "# 1. Run a Grid Search with 3-fold cross-validation and assign the output to the \n",
    "# object 'grid_LR'.\n",
    "\n",
    "# YOUR CODE HERE\n",
    "grid_LR=GridSearchCV(model_pipeline, param_grid, cv=3, scoring=\"roc_auc\", verbose=2)\n",
    "\n",
    "\n",
    "# 2. Fit the model (grid_LR) on the training data and assign the fitted model to the \n",
    "# variable 'grid_search_LR'\n",
    "\n",
    "# YOUR CODE HERE\n",
    "grid_search_LR=grid_LR.fit(X_train, y_train)\n",
    "\n",
    "\n",
    "print('Done')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the code below to see the best pipeline configuration that was determined by the grid search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "         steps=[('vectorizer',\n",
       "                 TfidfVectorizer(analyzer='word', binary=False,\n",
       "                                 decode_error='strict',\n",
       "                                 dtype=<class 'numpy.float64'>,\n",
       "                                 encoding='utf-8', input='content',\n",
       "                                 lowercase=True, max_df=1.0, max_features=None,\n",
       "                                 min_df=10, ngram_range=(1, 2), norm='l2',\n",
       "                                 preprocessor=None, smooth_idf=True,\n",
       "                                 stop_words=None, strip_accents=None,\n",
       "                                 sublinear_tf=False,\n",
       "                                 token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "                                 tokenizer=None, use_idf=True,\n",
       "                                 vocabulary=None)),\n",
       "                ('model',\n",
       "                 LogisticRegression(C=10, class_weight=None, dual=False,\n",
       "                                    fit_intercept=True, intercept_scaling=1,\n",
       "                                    l1_ratio=None, max_iter=200,\n",
       "                                    multi_class='auto', n_jobs=None,\n",
       "                                    penalty='l2', random_state=None,\n",
       "                                    solver='lbfgs', tol=0.0001, verbose=0,\n",
       "                                    warm_start=False))],\n",
       "         verbose=False)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search_LR.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Task</b>: Print the best hyperparameters by accessing them by using the `best_params_` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'model__C': 10, 'vectorizer__ngram_range': (1, 2)}\n"
     ]
    }
   ],
   "source": [
    "print(grid_search_LR.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall that in the past, after we obtained the best hyperparameter values from a grid search, we re-trained a model with these values in order to evaluate the performance. This time we will do something different. Just as we can pass a pipeline object directly to `plot_roc_curve()` to evaluate the model, we can pass `grid_search.best_estimator_` to the function `plot_roc_curve()` to evaluate the model. We also pass in the test data (`X_test` and `y_test`). This allows the test data to be passed through the entire pipeline, using the best hyperparameter values.\n",
    "\n",
    "\n",
    "<b>Task</b>: In the code cell below plot the ROC curve and compute the AUC by calling the function `plot_roc_curve()` with the arguments `grid_search.best_estimator_` and the test data (`X_test` and  `y_test`). Note that you can simply just pass `grid_search` to the function as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAhW0lEQVR4nO3de5wU1Z338c9XRCEq8ij4KCALCrpcAzoKrJdIMErUlfCIeM8aTXQTSTarcR+NWSVs7qJ5JRuzSowZ3QBCvERivG0SWTZeQNQRZYyAijADXkAjEoJI/O0fVTNphrn0MFPd09Pf9+vVr+mqOl31K2aoX59zqs5RRGBmZuVrt2IHYGZmxeVEYGZW5pwIzMzKnBOBmVmZcyIwMytzuxc7gNbq1atXDBgwoNhhmJmVlKeffnpDRPRubFvJJYIBAwawdOnSYodhZlZSJL3W1DY3DZmZlTknAjOzMudEYGZW5pwIzMzKnBOBmVmZyywRSLpN0puSXmhiuyT9UNIqScskHZFVLGZm1rQsawSVwMRmtn8SGJy+LgH+I8NYzMysCZk9RxARiyQNaKbIJOCOSMbBflJST0kHRcT6rGIyMyuEOYvXcF9Vbbvvd2ifHlz398Pafb/FfKCsL7A2Z7kmXbdTIpB0CUmtgf79+xckODOzXb2gL371bQDGDNyvvUPKREk8WRwRs4BZABUVFZ5Jx6wTyOpbc3va1Qv6mIH7MWlUX84dUxpfXIuZCGqBg3OW+6XrzKwDyeqCXQrfmkvtgr6ripkIFgDTJN0JjAHedf+AlZNS+EYM2V2wy+UiWwoySwSS5gInAL0k1QDXAV0BIuJm4AHgFGAVsAX4TFaxmLWX9rx4l8I3YvAFuxxkedfQOS1sD+CyrI5vlo/WXtjb8+LtC6x1FCXRWWyWlfuqaqlev4mhB/XIq7wv3tYZORFYp7ErzTZ1SWDepeMyisqs43MisJKWe/HflWaboQf1YNKovpnEZlYqnAisaNqj4zX34u9mG7Nd40RgBdXWb/AN+eJv1nZOBNYmbbnrxhdxs47BicB2SV0CaO23el/8zToeJwLbJXW3XfrCblb6nAhsl/m2S7POwYnA6rWmvb81D2GZWcfmRFBmmrvYt6a93/ffm3UeTgSdWGMX/eYu9m7vNytPTgSdTEv36ftib2YNORF0MrmDqPmib2b5cCLohHw3j5m1xm7FDsDMzIrLicDMrMy5aaiTqOsk9v39ZtZaTgSdwJzFa/jqvc8Df70ryMwsX04EnUDd7aLfmjzCdwiZWas5EZSw3OagMQP3cxIws13izuISltsn4OYgM9tVrhGUOD8zYGZt5URQYnKHkPAdQmbWHpwIOqimRgnNHT/ITUJm1h6cCDqYlqaA9PhBZtbenAg6GE8BaWaF5kTQAbkD2MwKybePmpmVOSeCDmTO4jX1fQNmZoXiRNCB1N0l5DuBzKyQMk0EkiZKeknSKklXNbK9v6RHJT0raZmkU7KMpxR4qAgzK7TMEoGkLsBNwCeBocA5koY2KPY1YH5EjAbOBn6cVTxmZta4LGsERwOrIuKViNgG3AlMalAmgLpHY/cF1mUYT4c1Z/EazrrlCarXbyp2KGZWhrK8fbQvsDZnuQYY06DMdOARSV8E9gJObGxHki4BLgHo3790m03yeVrY/QNmVmjFfo7gHKAyIm6QNA74T0nDI+LD3EIRMQuYBVBRURFFiLNd1D0x7KeFzawjyTIR1AIH5yz3S9fluhiYCBART0jqBvQC3swwroJrOG+AHxYzs44kyz6Cp4DBkgZK2oOkM3hBgzJrgAkAkoYA3YC3MoypKDxvgJl1ZJnVCCJiu6RpwMNAF+C2iFguaQawNCIWAFcAP5H0zyQdxxdGRMk2/TTHw0aYWUeVaR9BRDwAPNBg3bU576uBY7KMwczMmucnizPmYSPMrKNzIsiYh40ws47OiaAAPGyEmXVkTgRmZmWu2A+UdSqNPTnsCebNrKNzjaCdzFm8hq/e+/xOHcN+dsDMOjrXCNpJXU3gW5NHuD/AzEqKawTtyJ3CZlaKnAjMzMqcE4GZWZnLu49A0kciYkuWwZSShncI+e4gMytVLdYIJP2dpGrgD+nyRyWV/ZSSdSOK1vHdQWZWqvKpEXwfOJl0COmIeE7S8ZlGVSI8oqiZdQZ59RFExNoGq/6SQSxmZlYE+dQI1kr6OyAkdQX+CXgx27DMzKxQ8qkR/CNwGclk9LXAKOALGcZkZmYFlE+N4PCIOC93haRjgMeyCcnMzAopnxrBv+e5zszMSlCTNQJJ44C/A3pLujxnUw+SOYjNzKwTaK5paA9g77TMPjnrNwFTsgzKzMwKp8lEEBH/Dfy3pMqIeK2AMZmZWQHl01m8RdL1wDCgW93KiPh4ZlGZmVnB5NNZPJtkeImBwNeB1cBTGcbU4c1ZvGanCWjMzEpVPolg/4j4KfBBRPx3RFwElHVtoG6wOY8tZGadQT5NQx+kP9dLOhVYB+yXXUgdV92Io9XrN3kSGjPrNPJJBN+QtC9wBcnzAz2AL2cZVEdVlwQ80qiZdSYtJoKIuD99+y4wHuqfLC4LufMO1CUBjzhqZp1Jcw+UdQGmkowx9FBEvCDpNOCrQHdgdGFCLJ45i9fw1XufB5L5iF0TMLPOqLkawU+Bg4ElwA8lrQMqgKsi4pcFiK3o6moC35o8wv0BZtZpNZcIKoCREfGhpG7A68ChEbGxMKF1DO4UNrPOrrnbR7dFxIcAEbEVeKW1SUDSREkvSVol6aomykyVVC1puaQ5rdm/mZm1XXM1gr+VtCx9L+DQdFlARMTI5nac9jHcBHwCqAGekrQgIqpzygwGrgaOiYh3JB3QhnMxM7Nd0FwiGNLGfR8NrIqIVwAk3QlMAqpzynwOuCki3gGIiDfbeEwzM2ul5gada+tAc32B3LmOa4AxDcocBiDpMZKhradHxEMNdyTpEuASgP793V5vZtae8pq8PkO7A4OBE4BzgJ9I6tmwUETMioiKiKjo3bt3YSM0M+vkskwEtSS3n9bpl67LVQMsiIgPIuJVYAVJYiiqOYvXcNYtT1C9flOxQzEzy1xeiUBSd0mHt3LfTwGDJQ2UtAdwNrCgQZlfktQGkNSLpKnolVYep915KAkzKyctJgJJfw9UAQ+ly6MkNbyg7yQitgPTgIeBF4H5EbFc0gxJp6fFHgY2SqoGHgWu7CjPKdQNJeFnCMyss8tn0LnpJHcALQSIiCpJA/PZeUQ8ADzQYN21Oe8DuDx9mZlZEeTTNPRBRLzbYF1kEYyZmRVePjWC5ZLOBbqkD4B9CXg827DMzKxQ8qkRfJFkvuL3gTkkw1F/OcOYzMysgPKpEfxtRFwDXJN1MGZmVnj51AhukPSipH+TNDzziMzMrKBaTAQRMZ5kZrK3gFskPS/pa5lHViRzFq9h8atvFzsMM7OCyeuBsoh4PSJ+CPwjyTMF1zb/idJVNxmNHyQzs3LRYh+BpCHAWcAZwEZgHslE9p1K3dzE1es3eTIaMysr+XQW30Zy8T85ItZlHE/ReFgJMytXLSaCiBhXiECKoa4WANQngXmXdtrTNTNrVJOJQNL8iJgq6Xl2fJI4rxnKSkFuLcA1ATMrV83VCP4p/XlaIQIpFtcCzKzcNXnXUESsT99+ISJey30BXyhMeGZmlrV8bh/9RCPrPtnegZiZWXE010fweZJv/odIWpazaR/gsawDMzOzwmiuj2AO8CDwbeCqnPXvRYQfvTUz6ySaSwQREaslXdZwg6T9nAzMzDqHlmoEpwFPk9w+qpxtARySYVxmZlYgTSaCiDgt/ZnXtJRmZlaa8pm8/hhJe6Xvz5d0oyQPxGNm1knkc/vofwBbJH2UZLC5l4H/zDQqMzMrmHwSwfaICGAS8KOIuInkFlIzM+sE8hl99D1JVwMXAMdJ2g3omm1YZmZWKPnUCM4imbj+ooh4HegHXJ9pVGZmVjD5TFX5OjAb2FfSacDWiLgj88gyNGfxGs665Qmq128qdihmZkWXz11DU4ElwJnAVGCxpClZB5YlT0JjZvZX+fQRXAMcFRFvAkjqDfwGuCvLwLLm4afNzBL59BHsVpcEUhvz/JyZmZWAfGoED0l6GJibLp8FPJBdSGZmVkj5zFl8paT/BxybrpoVEfdmG5aZmRVKc/MRDAZmAocCzwNfiYjaQgVmZmaF0Vxb/23A/cAZJCOQ/ntrdy5poqSXJK2SdFUz5c6QFJIqWnsMMzNrm+aahvaJiJ+k71+S9ExrdiypC3ATyVSXNcBTkhZERHWDcvsA/wQsbs3+zcysfTSXCLpJGs1f5yHonrscES0lhqOBVRHxCoCkO0nGK6puUO7fgO8CV7YydjMzawfNJYL1wI05y6/nLAfw8Rb23RdYm7NcA4zJLSDpCODgiPi1pCYTgaRLgEsA+vf3CNhmZu2puYlpxmd54HTwuhuBC1sqGxGzgFkAFRUVkWVcZmblJssHw2qBg3OW+6Xr6uwDDAcWSloNjAUWuMPYzKywskwETwGDJQ2UtAdwNrCgbmNEvBsRvSJiQEQMAJ4ETo+IpRnGZGZmDWSWCCJiOzANeBh4EZgfEcslzZB0elbHNTOz1mnxyWJJAs4DDomIGel8xQdGxJKWPhsRD9BgOIqIuLaJsifkFbGZmbWrfGoEPwbGAeeky++RPB9gZmadQD6Dzo2JiCMkPQsQEe+kbf5mZtYJ5FMj+CB9Sjigfj6CDzONyszMCiafRPBD4F7gAEnfBH4PfCvTqMzMrGDyGYZ6tqSngQkkw0t8KiJezDwyMzMriHzuGuoPbAF+lbsuItZkGZiZmRVGPp3FvybpHxDQDRgIvAQMyzAuMzMrkHyahkbkLqcDxX0hs4jMzKygWv1kcTr89JgWC5qZWUnIp4/g8pzF3YAjgHWZRWRmZgWVTx/BPjnvt5P0GdydTThmZlZozSaC9EGyfSLiKwWKx8zMCqzJPgJJu0fEX4BjChiPmZkVWHM1giUk/QFVkhYAvwD+VLcxIu7JODYzMyuAfPoIugEbSeYornueIICSTARzFq9h8atvM2bgfsUOxcysQ2guERyQ3jH0An9NAHVKdt7g+6qS2TInjepb5EjMzDqG5hJBF2BvdkwAdUo2EQCMGbgf547pX+wwzMw6hOYSwfqImFGwSMzMrCiae7K4sZqAmZl1Ms0lggkFi8LMzIqmyUQQEW8XMhAzMyuOVg86Z2ZmnYsTgZlZmXMiMDMrc04EZmZlzonAzKzMORGYmZU5JwIzszLnRGBmVuacCMzMylymiUDSREkvSVol6apGtl8uqVrSMkm/lfQ3WcZjZmY7yywRpPMd3wR8EhgKnCNpaINizwIVETESuAv4XlbxmJlZ47KsERwNrIqIVyJiG3AnMCm3QEQ8GhFb0sUngX4ZxmNmZo3IMhH0BdbmLNek65pyMfBgYxskXSJpqaSlb731VjuGaGZmHaKzWNL5QAVwfWPbI2JWRFREREXv3r0LG5yZWSeXz+T1u6oWODhnuV+6bgeSTgSuAT4WEe9nGI+ZmTUiyxrBU8BgSQMl7QGcDSzILSBpNHALcHpEvJlhLGZm1oTMEkFEbAemAQ8DLwLzI2K5pBmSTk+LXQ/sDfxCUpWkBU3szszMMpJl0xAR8QDwQIN11+a8PzHL45uZWcs6RGexmZkVjxOBmVmZcyIwMytzTgRmZmXOicDMrMw5EZiZlTknAjOzMudEYGZW5pwIzMzKnBOBmVmZcyIwMytzTgRmZmXOicDMrMw5EZiZlTknAjOzMudEYGZW5pwIzMzKnBOBmVmZy3Sqyo5kzuI13FdVS/X6TQw9qEexwzEz6zDKpkaQmwQmjepb7HDMzDqMsqkRAAw9qAfzLh1X7DDMzDqUskoEZqXigw8+oKamhq1btxY7FCsx3bp1o1+/fnTt2jXvzzgRmHVANTU17LPPPgwYMABJxQ7HSkREsHHjRmpqahg4cGDenyubPgKzUrJ161b2339/JwFrFUnsv//+ra5JOhGYdVBOArYrduXvxonAzKzMORGYWaO6dOnCqFGjGD58OGeeeSZbtmxh6dKlfOlLX9rlfe69994ArFu3jilTprRXqHz5y19m0aJF9csbNmyga9eu3HzzzY0ev05lZSXTpk2rX77jjjsYPnw4I0aMYPTo0cycObPNsT300EMcfvjhDBo0iO985zuNlnnttdeYMGECI0eO5IQTTqCmpgaAqqoqxo0bx7Bhwxg5ciTz5s2r/8zZZ5/NypUr2xwfkHQulNLryCOPjF0x9ebHY+rNj+/SZ80Krbq6utghxF577VX//txzz40bbrihXffZXjZs2BBjxozZYd2Pf/zjOPbYY+P4449v9vg/+9nP4rLLLouIiAceeCBGjx4dtbW1ERGxdevWmDVrVpti2759exxyyCHx8ssvx/vvvx8jR46M5cuX71RuypQpUVlZGRERv/3tb+P888+PiIiXXnopVqxYERERtbW1ceCBB8Y777wTERELFy6Mz372s40et7G/H2BpNHFd9V1DZh3c13+1nOp1m9p1n0P79OC6vx+Wd/njjjuOZcuWsXDhQmbOnMn999/P9OnTefnll1m1ahUbNmzgX/7lX/jc5z4HwPXXX8/8+fN5//33mTx5Ml//+td32N/q1as57bTTeOGFF6isrGTBggVs2bKFl19+mcmTJ/O9730PgEceeYTrrruO999/n0MPPZSf/exnO32rv/vuu5k4ceIO6+bOncsNN9zAueeeS01NDf369WvxHL/97W8zc+ZM+vTpA8Cee+5Zfz67asmSJQwaNIhDDjkESL7F33fffQwdOnSHctXV1dx4440AjB8/nk996lMAHHbYYfVl+vTpwwEHHMBbb71Fz549Oe6447jwwgvZvn07u+/etku5m4bMrFnbt2/nwQcfZMSIETttW7ZsGb/73e944oknmDFjBuvWreORRx5h5cqVLFmyhKqqKp5++ukdmm0aU1VVxbx583j++eeZN28ea9euZcOGDXzjG9/gN7/5Dc888wwVFRX1F8tcjz32GEceeWT98tq1a1m/fj1HH300U6dO3aE5pTkvvPDCDvtpyuzZsxk1atROr8aaumprazn44IPrl/v160dtbe1O5T760Y9yzz33AHDvvffy3nvvsXHjxh3KLFmyhG3btnHooYcCsNtuuzFo0CCee+65vM6vOa4RmHVwrfnm3p7+/Oc/M2rUKCCpEVx88cU8/vjjO5SZNGkS3bt3p3v37owfP54lS5bw+9//nkceeYTRo0cDsHnzZlauXMnxxx/f5LEmTJjAvvvuC8DQoUN57bXX+OMf/0h1dTXHHHMMANu2bWPcuJ1HBli/fj29e/euX543bx5Tp04Fkm/gF110EVdccUWTx27tXTbnnXce5513Xqs+05KZM2cybdo0KisrOf744+nbty9dunSp375+/XouuOACbr/9dnbb7a/f3w844ADWrVuXVwJrTqaJQNJE4AdAF+DWiPhOg+17AncARwIbgbMiYnWWMZlZfrp3705VVVWzZRpeRCUREVx99dVceumleR9rzz33rH/fpUsXtm/fTkTwiU98grlz57YYZ+5983PnzuX1119n9uzZQNIxvXLlSgYPHkz37t3Ztm0be+yxBwBvv/02vXr1AmDYsGE8/fTTfPzjH2/2eLNnz+b666/faf2gQYO46667dljXt29f1q5dW79cU1ND3747j3XWp0+f+hrB5s2bufvuu+nZsycAmzZt4tRTT+Wb3/wmY8eO3eFzW7dupXv37s3Gm4/MmoYkdQFuAj4JDAXOkTS0QbGLgXciYhDwfeC7WcVjZu3vvvvuY+vWrWzcuJGFCxdy1FFHcfLJJ3PbbbexefNmIGkeefPNN1u977Fjx/LYY4+xatUqAP70pz+xYsWKncoNGTKkvsyKFSvYvHkztbW1rF69mtWrV3P11VfXJ5OPfexj/PznPweSGs/8+fMZP348AFdffTVXXnklr7/+OpDUQG699dadjnfeeedRVVW106thEgA46qijWLlyJa+++irbtm3jzjvv5PTTT9+p3IYNG/jwww+BpK/ioosuqo9h8uTJfPrTn2606WnFihUMHz68hX/JlmXZR3A0sCoiXomIbcCdwKQGZSYBt6fv7wImyE/RmJWMkSNHMn78eMaOHcu//uu/0qdPH0466STOPfdcxo0bx4gRI5gyZQrvvfdeq/fdu3dvKisrOeeccxg5ciTjxo3jD3/4w07lTj31VBYuXAgktYHJkyfvsP2MM86oTwQ/+MEPuOeeexg1ahRjx47lzDPPrG+yOuWUU5g2bRonnngiw4YN44gjjmDTprZ10u++++786Ec/4uSTT2bIkCFMnTqVYcOSpr5rr72WBQsWALBw4UIOP/xwDjvsMN544w2uueYaAObPn8+iRYuorKys74uoq6W98cYbdO/enQMPPLBNMQIouauo/UmaAkyMiM+myxcAYyJiWk6ZF9IyNenyy2mZDQ32dQlwCUD//v2PfO2111odz9d/tRwoXnurWWu8+OKLDBkypNhhNGv69OnsvffefOUrXyl2KBx77LHcf//99c0p5eD73/8+PXr04OKLL95pW2N/P5KejoiKxvZVEp3FETELmAVQUVGxS5nLCcCs87rhhhtYs2ZNWSWCnj17csEFF7TLvrJMBLXAwTnL/dJ1jZWpkbQ7sC9Jp7GZdXDTp08vdgj1xowZU+wQCu4zn/lMu+0ryz6Cp4DBkgZK2gM4G1jQoMwC4B/S91OA30VWbVVmJcb/FWxX7MrfTWaJICK2A9OAh4EXgfkRsVzSDEl13eY/BfaXtAq4HLgqq3jMSkm3bt3YuHGjk4G1SqTzEXTr1q1Vn8usszgrFRUVsXTp0mKHYZYpz1Bmu6qpGcpKvrPYrNx07dq1VTNMmbWFxxoyMytzTgRmZmXOicDMrMyVXGexpLeA1j9anOgFbGixVOficy4PPufy0JZz/puI6N3YhpJLBG0haWlTveadlc+5PPicy0NW5+ymITOzMudEYGZW5sotEcwqdgBF4HMuDz7n8pDJOZdVH4GZme2s3GoEZmbWgBOBmVmZ65SJQNJESS9JWiVppxFNJe0paV66fbGkAUUIs13lcc6XS6qWtEzSbyX9TTHibE8tnXNOuTMkhaSSv9Uwn3OWNDX9XS+XNKfQMba3PP62+0t6VNKz6d/3KcWIs71Iuk3Sm+kMjo1tl6Qfpv8eyyQd0eaDRkSnegFdgJeBQ4A9gOeAoQ3KfAG4OX1/NjCv2HEX4JzHAx9J33++HM45LbcPsAh4EqgodtwF+D0PBp4F/k+6fECx4y7AOc8CPp++HwqsLnbcbTzn44EjgBea2H4K8CAgYCywuK3H7Iw1gqOBVRHxSkRsA+4EJjUoMwm4PX1/FzBBkgoYY3tr8Zwj4tGI2JIuPkkyY1wpy+f3DPBvwHeBzjCecz7n/Dngpoh4ByAi3ixwjO0tn3MOoEf6fl9gXQHja3cRsQh4u5kik4A7IvEk0FPSQW05ZmdMBH2BtTnLNem6RstEMoHOu8D+BYkuG/mcc66LSb5RlLIWzzmtMh8cEb8uZGAZyuf3fBhwmKTHJD0paWLBostGPuc8HThfUg3wAPDFwoRWNK39/94iz0dQZiSdD1QAHyt2LFmStBtwI3BhkUMptN1JmodOIKn1LZI0IiL+WMygMnYOUBkRN0gaB/ynpOER8WGxAysVnbFGUAscnLPcL13XaBlJu5NUJzcWJLps5HPOSDoRuAY4PSLeL1BsWWnpnPcBhgMLJa0maUtdUOIdxvn8nmuABRHxQUS8CqwgSQylKp9zvhiYDxARTwDdSAZn66zy+v/eGp0xETwFDJY0UNIeJJ3BCxqUWQD8Q/p+CvC7SHthSlSL5yxpNHALSRIo9XZjaOGcI+LdiOgVEQMiYgBJv8jpEVHK85zm87f9S5LaAJJ6kTQVvVLAGNtbPue8BpgAIGkISSJ4q6BRFtYC4NPp3UNjgXcjYn1bdtjpmoYiYrukacDDJHcc3BYRyyXNAJZGxALgpyTVx1UknTJnFy/itsvznK8H9gZ+kfaLr4mI04sWdBvlec6dSp7n/DBwkqRq4C/AlRFRsrXdPM/5CuAnkv6ZpOP4wlL+YidpLkky75X2e1wHdAWIiJtJ+kFOAVYBW4DPtPmYJfzvZWZm7aAzNg2ZmVkrOBGYmZU5JwIzszLnRGBmVuacCMzMypwTgXVIkv4iqSrnNaCZspvb4XiVkl5Nj/VM+oRqa/dxq6Sh6fuvNtj2eFtjTPdT9+/ygqRfSerZQvlRpT4ap2XPt49ahyRpc0Ts3d5lm9lHJXB/RNwl6SRgZkSMbMP+2hxTS/uVdDuwIiK+2Uz5C0lGXZ3W3rFY5+EagZUESXun8yg8I+l5STuNNCrpIEmLcr4xH5euP0nSE+lnfyGppQv0ImBQ+tnL0329IOnL6bq9JP1a0nPp+rPS9QslVUj6DtA9jWN2um1z+vNOSafmxFwpaYqkLpKul/RUOsb8pXn8szxBOtiYpKPTc3xW0uOSDk+fxJ0BnJXGclYa+22SlqRlGxux1cpNscfe9suvxl4kT8VWpa97SZ6C75Fu60XyVGVdjXZz+vMK4Jr0fReS8YZ6kVzY90rX/3/g2kaOVwlMSd+fCSwGjgSeB/YieSp7OTAaOAP4Sc5n901/LiSd86AuppwydTFOBm5P3+9BMopkd+AS4Gvp+j2BpcDARuLcnHN+vwAmpss9gN3T9ycCd6fvLwR+lPP5bwHnp+97koxFtFexf99+FffV6YaYsE7jzxExqm5BUlfgW5KOBz4k+Sb8f4HXcz7zFHBbWvaXEVEl6WMkk5U8lg6tsQfJN+nGXC/payTj1FxMMn7NvRHxpzSGe4DjgIeAGyR9l6Q56X9acV4PAj+QtCcwEVgUEX9Om6NGSpqSltuXZLC4Vxt8vrukqvT8XwT+K6f87ZIGkwyz0LWJ458EnC7pK+lyN6B/ui8rU04EVirOA3oDR0bEB0pGFO2WWyAiFqWJ4lSgUtKNwDvAf0XEOXkc48qIuKtuQdKExgpFxAolcx2cAnxD0m8jYkY+JxERWyUtBE4GziKZaAWS2aa+GBEPt7CLP0fEKEkfIRl/5zLghyQT8DwaEZPTjvWFTXxewBkR8VI+8Vp5cB+BlYp9gTfTJDAe2GnOZSXzML8RET8BbiWZ7u9J4BhJdW3+e0k6LM9j/g/wKUkfkbQXSbPO/0jqA2yJiJ+TDObX2JyxH6Q1k8bMIxkorK52AclF/fN1n5F0WHrMRkUy29yXgCv016HU64YivjCn6HskTWR1Hga+qLR6pGRUWitzTgRWKmYDFZKeBz4N/KGRMicAz0l6luTb9g8i4i2SC+NcSctImoX+Np8DRsQzJH0HS0j6DG6NiGeBEcCStInmOuAbjXx8FrCsrrO4gUdIJgb6TSTTL0KSuKqBZ5RMWn4LLdTY01iWkUzM8j3g2+m5537uUWBoXWcxSc2haxrb8nTZypxvHzUzK3OuEZiZlTknAjOzMudEYGZW5pwIzMzKnBOBmVmZcyIwMytzTgRmZmXufwHlBxiqXXY78AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# YOUR CODE HERE\n",
    "plot_roc_curve(grid_search_LR.best_estimator_, X_test, y_test)\n",
    "\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}