NativeBayes_word2vec.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run Import_Library.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run text_functions.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run evaluation_metrics.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Introduction**\n",
    "- There are two main training algorithms for word2vec: Continous Bag of Word (CBOW) and Skip-gram. The major difference between these two methods is that CBOW is using context to predict a target word while skip-gram is using a word to predict a target context. Generally, the skip-gram method can have a better performance compared with CBOW method, for it can capture two semantics for a single word\n",
    "- The Word2Vec Skip-gram model, for example, takes in pairs (word1, word2) generated by moving a window across text data, and trains a 1-hidden-layer neural network based on the synthetic task of given an input word, giving us a predicted probability distribution of nearby words to the input\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Data Preparation**\n",
    "\n",
    "- Convert values in predicted column to binary\n",
    "- Feature selection: len_text, digits, non_alpha_char, processed_text\n",
    "- Shuffle randomly the data with selected features in order to reduce the bias.\n",
    "- Split dataset into train and test subsets with the ratio 80:20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>text</th>\n",
       "      <th>len_text</th>\n",
       "      <th>digits</th>\n",
       "      <th>non_alpha_char</th>\n",
       "      <th>processed_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>ham</td>\n",
       "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
       "      <td>111</td>\n",
       "      <td>0</td>\n",
       "      <td>28</td>\n",
       "      <td>go jurong point crazy available bugis n great ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>ham</td>\n",
       "      <td>Ok lar... Joking wif u oni...</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>ok lar joking wif u oni</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>spam</td>\n",
       "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
       "      <td>155</td>\n",
       "      <td>25</td>\n",
       "      <td>33</td>\n",
       "      <td>free entry 2 wkly comp win fa cup final tkts 2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>ham</td>\n",
       "      <td>U dun say so early hor... U c already then say...</td>\n",
       "      <td>49</td>\n",
       "      <td>0</td>\n",
       "      <td>16</td>\n",
       "      <td>u dun say early hor u c already say</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>ham</td>\n",
       "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
       "      <td>61</td>\n",
       "      <td>0</td>\n",
       "      <td>14</td>\n",
       "      <td>nah dont think go usf life around though</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  label                                               text  len_text  digits  \\\n",
       "0   ham  Go until jurong point, crazy.. Available only ...       111       0   \n",
       "1   ham                      Ok lar... Joking wif u oni...        29       0   \n",
       "2  spam  Free entry in 2 a wkly comp to win FA Cup fina...       155      25   \n",
       "3   ham  U dun say so early hor... U c already then say...        49       0   \n",
       "4   ham  Nah I don't think he goes to usf, he lives aro...        61       0   \n",
       "\n",
       "   non_alpha_char                                     processed_text  \n",
       "0              28  go jurong point crazy available bugis n great ...  \n",
       "1              11                            ok lar joking wif u oni  \n",
       "2              33  free entry 2 wkly comp win fa cup final tkts 2...  \n",
       "3              16                u dun say early hor u c already say  \n",
       "4              14           nah dont think go usf life around though  "
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from joblib import load\n",
    "\n",
    "data = load('data.lib')\n",
    "\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "#print(type(X_text))\n",
    "\n",
    "def split_data(data, features, target, test_size=0.2):\n",
    "    '''\n",
    "    Shuffle and split data into trainset and testset\n",
    "    \n",
    "    '''\n",
    "    X = data[features]\n",
    "    Y = data[target]\n",
    "    \n",
    "    x_train, x_val, y_train, y_val = train_test_split(X , Y , shuffle = True, test_size = test_size)\n",
    "    \n",
    "    return x_train, x_val, y_train, y_val"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = ['len_text', 'digits', 'non_alpha_char', 'processed_text']\n",
    "\n",
    "target = 'label'\n",
    "\n",
    "data[target] = np.where(data[target]=='spam', 1, 0)\n",
    "\n",
    "x_train, x_val, y_train, y_val = split_data(data, features, target, test_size = 0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_doc_vec(words, w2vec_model):\n",
    "    \"\"\"\n",
    "    Function to take a document as a list of words and return the document vector\n",
    "    \n",
    "    Arg:\n",
    "        - words: a list of words, for example: [\"w1\", \"w2\", \"w3\"]\n",
    "        - w2vec_model: vector of vocabularies\n",
    "        \n",
    "    \"\"\"\n",
    "    good_words = []\n",
    "    for word in words:\n",
    "        # Words not in the original model will fair\n",
    "        try:\n",
    "            if word in w2vec_model:\n",
    "                good_words.append(word)\n",
    "        except:\n",
    "            print(word, 'does not exist in the model.')\n",
    "            continue\n",
    "    # If no words are in the original model\n",
    "    # print(\"good_words: {}\".format(good_words))\n",
    "    if len(good_words) == 0:\n",
    "        return None\n",
    "    # Return the mean of the vectors for all the good words\n",
    "    return w2vec_model[good_words].mean(axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Train Word2Vec Model with the corpus**\n",
    "- 1) Set the input as the entire corpus, the model will build the set of vocabulary\n",
    "- 2) Create multi-dimensional vector from each document in trainset and testset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus = data['processed_text'].str.split()\n",
    "\n",
    "size_vect = 100\n",
    "size_window = 30\n",
    "ch_sg = 1 # The default training algorithm is skip-gram\n",
    "min_word_cnt = 10\n",
    "# build the model with the entire corpus\n",
    "model = gensim.models.word2vec.Word2Vec(corpus\n",
    "                                        , min_count = min_word_cnt\n",
    "                                        , size = size_vect\n",
    "                                        , window = size_window\n",
    "                                        , iter = 1000\n",
    "                                        , sg = ch_sg\n",
    "                                       , workers = 5)\n",
    "\n",
    "x_train['w2vec'] = x_train['processed_text'].apply(lambda sent : get_doc_vec(sent, model))\n",
    "\n",
    "x_val['w2vec'] = x_val['processed_text'].apply(lambda sent : get_doc_vec(sent, model))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Let’s try to understand the parameters used in training the model**\n",
    "- size: The number of dimensions of the embeddings and the default is 100.\n",
    "- window: The maximum distance between a target word and words around it. The default window is 5.\n",
    "- min_count: The minimum count of words to consider when training the model; words with occurrence less than this count will be ignored. The default for min_count is 5.\n",
    "- workers: The number of partitions during training and the default workers is 3.\n",
    "- sg: The training algorithm, either CBOW(0) or skip gram(1). The default training algorithm is CBOW."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>len_text</th>\n",
       "      <th>digits</th>\n",
       "      <th>non_alpha_char</th>\n",
       "      <th>processed_text</th>\n",
       "      <th>w2vec</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>742</td>\n",
       "      <td>62</td>\n",
       "      <td>0</td>\n",
       "      <td>15</td>\n",
       "      <td>well little time thing good time ahead</td>\n",
       "      <td>[-0.47330487, -0.47312462, -0.5221913, -0.0153...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3602</td>\n",
       "      <td>28</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>jay tell already</td>\n",
       "      <td>[-0.08387119, -0.5157668, -0.02518104, 0.05996...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5531</td>\n",
       "      <td>60</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>compliment away system side</td>\n",
       "      <td>[-0.4088728, -0.70502704, 0.115013175, 0.24147...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3693</td>\n",
       "      <td>46</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>movie laptop</td>\n",
       "      <td>[-0.417161, -1.1797577, 0.039054543, 0.5233974...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4586</td>\n",
       "      <td>158</td>\n",
       "      <td>26</td>\n",
       "      <td>28</td>\n",
       "      <td>u secret admirer look 2 make contact ufind rre...</td>\n",
       "      <td>[-0.15474895, -0.63778716, 0.33145007, 0.17539...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      len_text  digits  non_alpha_char  \\\n",
       "742         62       0              15   \n",
       "3602        28       0               6   \n",
       "5531        60       0              13   \n",
       "3693        46       0              10   \n",
       "4586       158      26              28   \n",
       "\n",
       "                                         processed_text  \\\n",
       "742              well little time thing good time ahead   \n",
       "3602                                   jay tell already   \n",
       "5531                        compliment away system side   \n",
       "3693                                       movie laptop   \n",
       "4586  u secret admirer look 2 make contact ufind rre...   \n",
       "\n",
       "                                                  w2vec  \n",
       "742   [-0.47330487, -0.47312462, -0.5221913, -0.0153...  \n",
       "3602  [-0.08387119, -0.5157668, -0.02518104, 0.05996...  \n",
       "5531  [-0.4088728, -0.70502704, 0.115013175, 0.24147...  \n",
       "3693  [-0.417161, -1.1797577, 0.039054543, 0.5233974...  \n",
       "4586  [-0.15474895, -0.63778716, 0.33145007, 0.17539...  "
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Remarks**: we can't fit the data into machine learning algorithm because the data structure is not good.\n",
    "\n",
    "- Normalise the data\n",
    "- Addind new features\n",
    "- Remove the Nan Values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "def w2v_normalization(x_data, y_data, size_vector):\n",
    "    '''\n",
    "    x_data: either x_train or x_val\n",
    "    y_data: either y_train or y_val\n",
    "    '''\n",
    "    # Data Normalization\n",
    "    x_np_vecs = np.zeros((len(x_data), size_vector))\n",
    "    for i, vec in enumerate(x_data['w2vec']):\n",
    "        x_np_vecs[i, :] = vec\n",
    "\n",
    "    # Combine the full dataframe with the labels\n",
    "    x_data_w2v = pd.DataFrame(data = x_np_vecs\n",
    "                              , index = x_data.index)\n",
    "    \n",
    "    # Add new features\n",
    "    x_data_w2v = x_data_w2v.join(x_data[['len_text', 'digits', 'non_alpha_char']])\n",
    "    \n",
    "    # Join train data with label data in order to remove NaN values\n",
    "    x_data_w2v = x_data_w2v.join(y_data)\n",
    "    \n",
    "    x_data_w2v = x_data_w2v.dropna()\n",
    "    \n",
    "    y_data_train = x_data_w2v['label']\n",
    "    \n",
    "    x_data_train = x_data_w2v.drop(columns=['label'])\n",
    "      \n",
    "    return x_data_train, y_data_train\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train_w2v, y_train = w2v_normalization(x_train\n",
    "                                         , y_train\n",
    "                                         , size_vect)\n",
    "\n",
    "x_val_w2v, y_val = w2v_normalization(x_val\n",
    "                                     , y_val\n",
    "                                     , size_vect)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>93</th>\n",
       "      <th>94</th>\n",
       "      <th>95</th>\n",
       "      <th>96</th>\n",
       "      <th>97</th>\n",
       "      <th>98</th>\n",
       "      <th>99</th>\n",
       "      <th>len_text</th>\n",
       "      <th>digits</th>\n",
       "      <th>non_alpha_char</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>742</td>\n",
       "      <td>-0.473305</td>\n",
       "      <td>-0.473125</td>\n",
       "      <td>-0.522191</td>\n",
       "      <td>-0.015343</td>\n",
       "      <td>-0.311092</td>\n",
       "      <td>0.425733</td>\n",
       "      <td>0.560416</td>\n",
       "      <td>0.491478</td>\n",
       "      <td>0.096124</td>\n",
       "      <td>-0.249853</td>\n",
       "      <td>...</td>\n",
       "      <td>0.126400</td>\n",
       "      <td>0.263273</td>\n",
       "      <td>-0.740204</td>\n",
       "      <td>-0.215985</td>\n",
       "      <td>0.521319</td>\n",
       "      <td>-0.123448</td>\n",
       "      <td>-0.411699</td>\n",
       "      <td>62</td>\n",
       "      <td>0</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3602</td>\n",
       "      <td>-0.083871</td>\n",
       "      <td>-0.515767</td>\n",
       "      <td>-0.025181</td>\n",
       "      <td>0.059960</td>\n",
       "      <td>-0.029355</td>\n",
       "      <td>0.483920</td>\n",
       "      <td>0.456451</td>\n",
       "      <td>0.707481</td>\n",
       "      <td>-0.198413</td>\n",
       "      <td>-0.262070</td>\n",
       "      <td>...</td>\n",
       "      <td>0.130765</td>\n",
       "      <td>0.207721</td>\n",
       "      <td>-0.780777</td>\n",
       "      <td>-0.299856</td>\n",
       "      <td>0.419100</td>\n",
       "      <td>-0.007276</td>\n",
       "      <td>-0.455397</td>\n",
       "      <td>28</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 103 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             0         1         2         3         4         5         6  \\\n",
       "742  -0.473305 -0.473125 -0.522191 -0.015343 -0.311092  0.425733  0.560416   \n",
       "3602 -0.083871 -0.515767 -0.025181  0.059960 -0.029355  0.483920  0.456451   \n",
       "\n",
       "             7         8         9  ...        93        94        95  \\\n",
       "742   0.491478  0.096124 -0.249853  ...  0.126400  0.263273 -0.740204   \n",
       "3602  0.707481 -0.198413 -0.262070  ...  0.130765  0.207721 -0.780777   \n",
       "\n",
       "            96        97        98        99  len_text  digits  non_alpha_char  \n",
       "742  -0.215985  0.521319 -0.123448 -0.411699        62       0              15  \n",
       "3602 -0.299856  0.419100 -0.007276 -0.455397        28       0               6  \n",
       "\n",
       "[2 rows x 103 columns]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train_w2v.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>93</th>\n",
       "      <th>94</th>\n",
       "      <th>95</th>\n",
       "      <th>96</th>\n",
       "      <th>97</th>\n",
       "      <th>98</th>\n",
       "      <th>99</th>\n",
       "      <th>len_text</th>\n",
       "      <th>digits</th>\n",
       "      <th>non_alpha_char</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>3366</td>\n",
       "      <td>0.075215</td>\n",
       "      <td>-0.461036</td>\n",
       "      <td>0.207333</td>\n",
       "      <td>0.172093</td>\n",
       "      <td>0.015896</td>\n",
       "      <td>0.375577</td>\n",
       "      <td>0.299629</td>\n",
       "      <td>0.408385</td>\n",
       "      <td>0.015985</td>\n",
       "      <td>0.114019</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.151495</td>\n",
       "      <td>0.032117</td>\n",
       "      <td>-0.448398</td>\n",
       "      <td>-0.264202</td>\n",
       "      <td>0.262411</td>\n",
       "      <td>-0.151933</td>\n",
       "      <td>-0.054640</td>\n",
       "      <td>22</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3090</td>\n",
       "      <td>-0.541918</td>\n",
       "      <td>-0.281510</td>\n",
       "      <td>0.121438</td>\n",
       "      <td>0.022833</td>\n",
       "      <td>-0.474408</td>\n",
       "      <td>0.717466</td>\n",
       "      <td>0.197144</td>\n",
       "      <td>0.138646</td>\n",
       "      <td>0.155885</td>\n",
       "      <td>-0.054211</td>\n",
       "      <td>...</td>\n",
       "      <td>0.055396</td>\n",
       "      <td>-0.148287</td>\n",
       "      <td>-0.098618</td>\n",
       "      <td>-0.270382</td>\n",
       "      <td>0.403873</td>\n",
       "      <td>-0.357790</td>\n",
       "      <td>0.000982</td>\n",
       "      <td>50</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 103 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             0         1         2         3         4         5         6  \\\n",
       "3366  0.075215 -0.461036  0.207333  0.172093  0.015896  0.375577  0.299629   \n",
       "3090 -0.541918 -0.281510  0.121438  0.022833 -0.474408  0.717466  0.197144   \n",
       "\n",
       "             7         8         9  ...        93        94        95  \\\n",
       "3366  0.408385  0.015985  0.114019  ... -0.151495  0.032117 -0.448398   \n",
       "3090  0.138646  0.155885 -0.054211  ...  0.055396 -0.148287 -0.098618   \n",
       "\n",
       "            96        97        98        99  len_text  digits  non_alpha_char  \n",
       "3366 -0.264202  0.262411 -0.151933 -0.054640        22       0               5  \n",
       "3090 -0.270382  0.403873 -0.357790  0.000982        50       0              12  \n",
       "\n",
       "[2 rows x 103 columns]"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_val_w2v.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Select the best feature and fit the data to Bayes Classifier**\n",
    "\n",
    "- When we convert all of the texts in a dataset into word uni+bigram tokens, we may end up with tens of thousands of tokens. Not all of these tokens/features contribute to label prediction. So we can drop certain tokens, for instance those that occur extremely rarely across the dataset. We can also measure feature importance (how much each token contributes to label predictions), and only include the most informative tokens.\n",
    "\n",
    "- There are many statistical functions that take features and the corresponding labels and output the feature importance score. Two commonly used functions are f_classif and chi2. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "def select_best_features_and_fit(x_train, y_train, x_val):\n",
    "    '''\n",
    "    return predicted values\n",
    "    '''\n",
    "    \n",
    "    selector = SelectKBest(f_classif, k = min(TOP_K, x_train.shape[1]))\n",
    "    \n",
    "    selector.fit(x_train, y_train)\n",
    "\n",
    "    x_train_selector = selector.transform(x_train).astype('float32')\n",
    "\n",
    "    x_val_selector = selector.transform(x_val).astype('float32')\n",
    "\n",
    "    clf = GaussianNB()\n",
    "    # A sparse matrix was passed, but dense data is required.\n",
    "    clf.fit(x_train_selector, y_train)\n",
    "\n",
    "    y_preds = clf.predict(x_val_selector)\n",
    "\n",
    "    return y_preds\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_preds = select_best_features_and_fit(x_train_w2v, y_train, x_val_w2v)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.94\n"
     ]
    }
   ],
   "source": [
    "print(round(accuracy_score(y_val, y_preds), 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Evaluation Metrics**\n",
    "-  The accuracy will yield misleading results if the data set is unbalanced; that is, when the numbers of observations in different classes vary greatly. Confusion Matrix and Classification Report will help us to evaluate the performance for each class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**1) Confustion Matrix**\n",
    "\n",
    "- A confusion matrix, also known as an error matrix, is a specific table layout that allows visualization of the performance of an supervised machine learning algorithm.\n",
    "\n",
    "- Each row of the matrix represents the instances in a predicted class while each column represents the instances in an actual class. The diagonal elements show the number of correct classifications for each class meanwhile the off-diagonal elements provides the mis-classifications for each class.\n",
    "\n",
    "- There are four ways to check if the predictions are right or wrong:\n",
    "\n",
    "    - 1) True Positive (TP): The class was positive and predicted positive.\n",
    "    - 2) True Negative (TN): The class was negative and predicted negative.\n",
    "    - 3) False Negative (FN): The class was positive but predicted negative\n",
    "    - 4) False Positive (FP) : The case was negative but predicted positive\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Actual Ham</th>\n",
       "      <th>Actual Spam</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>Predicted Ham</td>\n",
       "      <td>896</td>\n",
       "      <td>56</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Predicted Spam</td>\n",
       "      <td>16</td>\n",
       "      <td>147</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                Actual Ham  Actual Spam\n",
       "Predicted Ham          896           56\n",
       "Predicted Spam          16          147"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_confusion_matrix =  make_confusion_matrix(confusion_matrix(y_val, y_preds)\n",
    "                                              , columns = ['Actual Ham', 'Actual Spam']\n",
    "                                              , index = ['Predicted Ham', 'Predicted Spam'])\n",
    "df_confusion_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The correction of prediction = TP + TN = 896 + 147 = 1043\n",
    "\n",
    "- The mis-classification = FN + FP = 84 + 13 = 72"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**2) Classification Report**\n",
    "- The report shows the main classification metrics precision, recall and f1-score on a per-class basis. The metrics are calculated by using true and false positives, true and false negatives. Positive and negative in this case are generic names for the predicted classes.\n",
    "\n",
    "- **Precision – What percent of your predictions were correct ?**\n",
    "\n",
    "    - Precision is the ability of a classifier not to label an instance positive that is actually negative. For each class it is defined as the ratio of true positives to the sum of true and false positives.\n",
    "\n",
    "    - Precision = Accuracy of positive predictions.\n",
    "\n",
    "    - Precision = TP/(TP + FP)\n",
    "\n",
    "- **Recall – What percent of the positive cases did you catch ?**\n",
    "\n",
    "    - Recall is the ability of a classifier to find all positive instances. For each class it is defined as the ratio of true positives to the sum of true positives (TP) and false negatives (FN).\n",
    "\n",
    "    - Recall: Fraction of positives that were correctly identified.\n",
    "\n",
    "    - Recall = TP/(TP+FN)\n",
    "\n",
    "- **F1 score – What percent of positive predictions were correct?**\n",
    "\n",
    "    - The F1 score is a weighted harmonic mean of precision and recall such that the best score is 1.0 and the worst is 0.0. Generally speaking, F1 scores are lower than accuracy measures as they embed precision and recall into their computation. As a rule of thumb, the weighted average of F1 should be used to compare classifier models, not global accuracy.\n",
    "\n",
    "    - F1 Score = 2*(Recall * Precision) / (Recall + Precision)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "         ham       0.98      0.94      0.96       952\n",
      "        spam       0.72      0.90      0.80       163\n",
      "\n",
      "    accuracy                           0.94      1115\n",
      "   macro avg       0.85      0.92      0.88      1115\n",
      "weighted avg       0.94      0.94      0.94      1115\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "\n",
    "print (classification_report(y_val , y_preds, target_names = ['ham', 'spam']))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}