Kaggle_Airbnb_Updated_201704181026_UploadToGithub.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Kaggle Competition Experience\n",
    "\n",
    "### Airbnb New User Bookings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### numerical data\n",
    "* merge train data and test data in order to do all the following things to them together. When we need to split them for train and test, then split them.\n",
    "* group data into numerical, categorical, text, datetime\n",
    "* extract datetime data\n",
    "* numerical data interaction, with time date, without time date\n",
    "* group data\n",
    "\n",
    "#### categorical data\n",
    "* label encoding\n",
    "* one hot encoding\n",
    "* leave one out encoding\n",
    "* group data\n",
    "\n",
    "#### train and test\n",
    "* create train and test data\n",
    "* cross validation\n",
    "* predict on test data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color = 'blue'>Links</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://www.kaggle.com/svpons/airbnb-recruiting-new-user-bookings/script-0-8655 <br />\n",
    "This is a very simple code and got 0.86\n",
    "<br /><br />\n",
    "http://brettromero.com/wordpress/data-science-kaggle-walkthrough-cleaning-data/ <br />\n",
    "This is someone telling how to clean data\n",
    "\n",
    "https://www.kaggle.com/c/airbnb-recruiting-new-user-bookings/data <br />\n",
    "The original link for this competition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# ------------------------------ import ------------------------------\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import datetime\n",
    "% matplotlib inline\n",
    "from sklearn import datasets\n",
    "from sklearn import svm\n",
    "from sklearn import cross_validation\n",
    "from sklearn import tree\n",
    "from sklearn import svm\n",
    "from sklearn import ensemble\n",
    "from sklearn import neighbors\n",
    "from sklearn import linear_model\n",
    "from sklearn import metrics\n",
    "from sklearn import preprocessing\n",
    "import xgboost as xgb\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 0 Read Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# -------------------------------- 0 read data --------------------------------\n",
    "sessionRawData = pd.read_csv('sessions.csv')\n",
    "countriesRawData = pd.read_csv('countries.csv')\n",
    "ageGenderRawData = pd.read_csv('age_gender_bkts.csv')\n",
    "trainUser2RawData = pd.read_csv('train_users_2.csv')\n",
    "testUserRawData = pd.read_csv('test_users.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# ----------------------------- merge train and test data -----------------------------\n",
    "# need to merge train data and test data here\n",
    "# then do the same thing to both of them\n",
    "train_size = trainUser2RawData.shape[0]\n",
    "test_size = testUserRawData.shape[0]\n",
    "\n",
    "fullData = pd.concat([trainUser2RawData, testUserRawData])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# print trainUser2RawData.shape\n",
    "# print testUserRawData.shape\n",
    "# print fullData.shape\n",
    "# print fullData['country_destination'].isnull().value_counts()\n",
    "# ----------------------------- check -----------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Step 1 Data Wrangling and exploration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                 age    signup_flow  timestamp_first_active\n",
      "count  158681.000000  275547.000000            2.755470e+05\n",
      "mean       47.145310       4.291965            2.013310e+13\n",
      "std       142.629468       8.794313            9.146438e+09\n",
      "min         1.000000       0.000000            2.009032e+13\n",
      "25%        28.000000       0.000000            2.013040e+13\n",
      "50%        33.000000       0.000000            2.014010e+13\n",
      "75%        42.000000       1.000000            2.014062e+13\n",
      "max      2014.000000      25.000000            2.014093e+13\n",
      "\n",
      " this is the null data in age column\n",
      "False    158681\n",
      "True     116866\n",
      "Name: age, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# -------------------------- have a look at the data --------------------------\n",
    "print fullData.describe()\n",
    "print '\\n this is the null data in age column\\n' , fullData.age.isnull().value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# ---------------------------- 1 data cleansing ----------------------------\n",
    "\n",
    "# convert all date columns to datetime data type \n",
    "fullData['date_account_created'] = pd.to_datetime(fullData.date_account_created)\n",
    "fullData.timestamp_first_active = pd.to_datetime(fullData.timestamp_first_active.astype('str'))\n",
    "fullData.date_first_booking = pd.to_datetime(fullData.date_first_booking)\n",
    "\n",
    "\n",
    "# replace the ones who greater than 100 years old with NaN\n",
    "for ii in fullData[fullData.age > 100].age.unique():\n",
    "    fullData.age.replace(to_replace = ii, value = np.nan, inplace = True)\n",
    "# replace the ones who less than 14 years old with NaN \n",
    "for ii in fullData[fullData.age < 14].age.unique():\n",
    "    fullData.age.replace(to_replace = ii, value = np.nan, inplace = True)\n",
    "    \n",
    "# print fullData[fullData.age > 100].age.value_counts() , 'the ones greater than 110 have been replaced'\n",
    "# print fullData[fullData.age < 14].age.value_counts() , 'the ones less than 10 have been replaced by 0'\n",
    "# print fullData[fullData.age < 0].age.value_counts() , 'so there is no one less than 0'\n",
    "# print fullData.age.value_counts()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "count    155932.000000\n",
      "mean         36.055043\n",
      "std          11.663664\n",
      "min          15.000000\n",
      "25%          28.000000\n",
      "50%          33.000000\n",
      "75%          42.000000\n",
      "max         100.000000\n",
      "Name: age, dtype: float64\n",
      "\n",
      "False    155932\n",
      "True     119615\n",
      "Name: age, dtype: int64 \n",
      "\n",
      "33.0  is median value of age\n"
     ]
    }
   ],
   "source": [
    "print fullData.age.describe()\n",
    "print '\\n' , fullData.age.isnull().value_counts() , '\\n'\n",
    "# print fullData.age.mean() , ' is mean value of age \\n'\n",
    "print fullData.age.median() , ' is median value of age'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# and then we find that there are these mean and median value of age\n",
    "# I have tried replace the NaN with very big number\n",
    "# now it's the time I replace them with the average value or median value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "count    275547.000000\n",
      "mean         19.969443\n",
      "std          20.354181\n",
      "min          -1.000000\n",
      "25%          -1.000000\n",
      "50%          25.000000\n",
      "75%          34.000000\n",
      "max         100.000000\n",
      "Name: age, dtype: float64\n",
      "\n",
      "False    275547\n",
      "Name: age, dtype: int64 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "# replace outliers with -1\n",
    "fullData.age.replace(to_replace = np.nan, value = -1, inplace = True)\n",
    "print fullData.age.describe()\n",
    "print '\\n' , fullData.age.isnull().value_counts() , '\\n'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>affiliate_channel</th>\n",
       "      <th>affiliate_provider</th>\n",
       "      <th>age</th>\n",
       "      <th>country_destination</th>\n",
       "      <th>date_account_created</th>\n",
       "      <th>date_first_booking</th>\n",
       "      <th>first_affiliate_tracked</th>\n",
       "      <th>first_browser</th>\n",
       "      <th>first_device_type</th>\n",
       "      <th>gender</th>\n",
       "      <th>id</th>\n",
       "      <th>language</th>\n",
       "      <th>signup_app</th>\n",
       "      <th>signup_flow</th>\n",
       "      <th>signup_method</th>\n",
       "      <th>timestamp_first_active</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>direct</td>\n",
       "      <td>direct</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>NDF</td>\n",
       "      <td>2010-06-28</td>\n",
       "      <td>NaT</td>\n",
       "      <td>untracked</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>Mac Desktop</td>\n",
       "      <td>-unknown-</td>\n",
       "      <td>gxn3p5htnn</td>\n",
       "      <td>en</td>\n",
       "      <td>Web</td>\n",
       "      <td>0</td>\n",
       "      <td>facebook</td>\n",
       "      <td>2009-03-19 04:32:55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>seo</td>\n",
       "      <td>google</td>\n",
       "      <td>38.0</td>\n",
       "      <td>NDF</td>\n",
       "      <td>2011-05-25</td>\n",
       "      <td>NaT</td>\n",
       "      <td>untracked</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>Mac Desktop</td>\n",
       "      <td>MALE</td>\n",
       "      <td>820tgsjxq7</td>\n",
       "      <td>en</td>\n",
       "      <td>Web</td>\n",
       "      <td>0</td>\n",
       "      <td>facebook</td>\n",
       "      <td>2009-05-23 17:48:09</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  affiliate_channel affiliate_provider   age country_destination  \\\n",
       "0            direct             direct  -1.0                 NDF   \n",
       "1               seo             google  38.0                 NDF   \n",
       "\n",
       "  date_account_created date_first_booking first_affiliate_tracked  \\\n",
       "0           2010-06-28                NaT               untracked   \n",
       "1           2011-05-25                NaT               untracked   \n",
       "\n",
       "  first_browser first_device_type     gender          id language signup_app  \\\n",
       "0        Chrome       Mac Desktop  -unknown-  gxn3p5htnn       en        Web   \n",
       "1        Chrome       Mac Desktop       MALE  820tgsjxq7       en        Web   \n",
       "\n",
       "   signup_flow signup_method timestamp_first_active  \n",
       "0            0      facebook    2009-03-19 04:32:55  \n",
       "1            0      facebook    2009-05-23 17:48:09  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fullData.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### check the null value in each columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "affiliate_channel \n",
      "False    275547\n",
      "Name: affiliate_channel, dtype: int64 \n",
      "\n",
      "affiliate_provider \n",
      "False    275547\n",
      "Name: affiliate_provider, dtype: int64 \n",
      "\n",
      "age \n",
      "False    275547\n",
      "Name: age, dtype: int64 \n",
      "\n",
      "country_destination \n",
      "False    213451\n",
      "True      62096\n",
      "Name: country_destination, dtype: int64 \n",
      "\n",
      "date_account_created \n",
      "False    275547\n",
      "Name: date_account_created, dtype: int64 \n",
      "\n",
      "date_first_booking \n",
      "True     186639\n",
      "False     88908\n",
      "Name: date_first_booking, dtype: int64 \n",
      "\n",
      "first_affiliate_tracked \n",
      "False    269462\n",
      "True       6085\n",
      "Name: first_affiliate_tracked, dtype: int64 \n",
      "\n",
      "first_browser \n",
      "False    275547\n",
      "Name: first_browser, dtype: int64 \n",
      "\n",
      "first_device_type \n",
      "False    275547\n",
      "Name: first_device_type, dtype: int64 \n",
      "\n",
      "gender \n",
      "False    275547\n",
      "Name: gender, dtype: int64 \n",
      "\n",
      "id \n",
      "False    275547\n",
      "Name: id, dtype: int64 \n",
      "\n",
      "language \n",
      "False    275547\n",
      "Name: language, dtype: int64 \n",
      "\n",
      "signup_app \n",
      "False    275547\n",
      "Name: signup_app, dtype: int64 \n",
      "\n",
      "signup_flow \n",
      "False    275547\n",
      "Name: signup_flow, dtype: int64 \n",
      "\n",
      "signup_method \n",
      "False    275547\n",
      "Name: signup_method, dtype: int64 \n",
      "\n",
      "timestamp_first_active \n",
      "False    275547\n",
      "Name: timestamp_first_active, dtype: int64 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "# print fullData.date_first_booking.head(2)\n",
    "# xxx = fullData.date_first_booking.dt.year\n",
    "# print xxx.head(2)\n",
    "for header in fullData:\n",
    "    print header , '\\n' , fullData[header].isnull().value_counts() , '\\n'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " From above we can see that there are so many NaN in some of the columns, like <font color = 'red'>age</font> and <font color = 'red'>date_first_booking</font>. Then how should we cope with these NaN data? If we simply replace them with 99999, I think it's not a appropriate way to do so.\n",
    "\n",
    " And after this, we know that there is no data about date_first_booking in test data, so what we have here in date_first_booking will not work in the test data, even worse, it will create a massive number of NaN data for us which can really mess with the model and prediction I think.<br />After all, there're only 60k rows in test data, and there're 186k NaN in full data. It could be a huge negtive effect for the model.\n",
    "\n",
    " Some null in first_affiliate_tracked, but just 6k of them, and 269k are valid value, so just ignore the null maybe."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Group Original variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\n",
    "target = ['country_destination']\n",
    "id_vars = ['id']\n",
    "num_vars_org = ['age'\n",
    "#                 , 'signup_flow'\n",
    "               ]\n",
    "datetime_vars_org = ['date_account_created', 'date_first_booking', 'timestamp_first_active']\n",
    "cat_vars_org = ['affiliate_channel', 'affiliate_provider', 'first_affiliate_tracked', 'first_browser', \\\n",
    "               'first_device_type', 'gender', 'language', 'signup_app', 'signup_method'\n",
    "               , 'signup_flow'\n",
    "               ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Step 2 Feature Engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Numerical variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### new numerical variables from date time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# ---------- created ----------\n",
    "fullData['created_year'] = fullData['date_account_created'].dt.year\n",
    "fullData['created_month'] = fullData['date_account_created'].dt.month\n",
    "fullData['created_day'] = fullData['date_account_created'].dt.day\n",
    "fullData['created_dayofweek'] = fullData['date_account_created'].dt.dayofweek\n",
    "fullData['created_dayofyear'] = fullData['date_account_created'].dt.dayofyear\n",
    "fullData['created_weekofyear'] = fullData['date_account_created'].dt.weekofyear\n",
    "\n",
    "# ---------- active ----------\n",
    "fullData['first_active_year'] = fullData['timestamp_first_active'].dt.year\n",
    "fullData['first_active_month'] = fullData['timestamp_first_active'].dt.month\n",
    "fullData['first_active_day'] = fullData['timestamp_first_active'].dt.day\n",
    "fullData['first_active_dayofweek'] = fullData['timestamp_first_active'].dt.dayofweek\n",
    "fullData['first_active_dayofyear'] = fullData['timestamp_first_active'].dt.dayofyear\n",
    "fullData['first_active_weekofyear'] = fullData['timestamp_first_active'].dt.weekofyear\n",
    "\n",
    "# ---------- booking ----------\n",
    "# fullData['first_booking_year'] = fullData['date_first_booking'].dt.year\n",
    "# fullData['first_booking_month'] = fullData['date_first_booking'].dt.month\n",
    "# fullData['first_booking_day'] = fullData['date_first_booking'].dt.day\n",
    "# fullData['first_booking_dayofweek'] = fullData['date_first_booking'].dt.dayofweek\n",
    "# fullData['first_booking_dayofyear'] = fullData['date_first_booking'].dt.dayofyear\n",
    "# fullData['first_booking_weekofyear'] = fullData['date_first_booking'].dt.weekofyear\n",
    "\n",
    "\n",
    "\n",
    "num_datetime_vars = ['created_year', 'created_month', 'created_day', 'created_dayofweek', 'created_dayofyear', \\\n",
    "                    'created_weekofyear', \\\n",
    "                    'first_active_year', 'first_active_month', 'first_active_day', 'first_active_dayofweek', \\\n",
    "                    'first_active_dayofyear', 'first_active_weekofyear', \\\n",
    "#                     'first_booking_year', 'first_booking_month', 'first_booking_day', 'first_booking_dayofweek', \\\n",
    "#                     'first_booking_dayofyear', 'first_booking_weekofyear'\n",
    "                    ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Numerical variables <font color = 'red'>Interaction</font> among date and time variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# ---------- active-created ----------\n",
    "fullData['diff_created_firstActive_year'] = fullData['first_active_year'] - fullData['created_year']\n",
    "fullData['diff_created_firstActive_month'] = fullData['first_active_month'] - fullData['created_month']\n",
    "fullData['diff_created_firstActive_day'] = fullData['first_active_day'] - fullData['created_day']\n",
    "\n",
    "# ---------- booking-active ----------\n",
    "# fullData['diff_firstActive_booking_year'] = fullData['first_booking_year'] - fullData['first_active_year']\n",
    "# fullData['diff_firstActive_booking_month'] = fullData['first_booking_month'] - fullData['first_active_month']\n",
    "# fullData['diff_firstActive_booking_day'] = fullData['first_booking_day'] - fullData['first_active_day']\n",
    "\n",
    "num_datetime_vars.append('diff_created_firstActive_year')\n",
    "num_datetime_vars.append('diff_created_firstActive_month')\n",
    "num_datetime_vars.append('diff_created_firstActive_day')\n",
    "# num_datetime_vars.append('diff_firstActive_booking_year')\n",
    "# num_datetime_vars.append('diff_firstActive_booking_month')\n",
    "# num_datetime_vars.append('diff_firstActive_booking_day')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Categorical variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### One-Hot encoding some of the categorical data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dummiedMergedCoutriesTrainUses = pd.get_dummies(fullData, columns = cat_vars_org)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "age\n",
      "country_destination\n",
      "date_account_created\n",
      "date_first_booking\n",
      "id\n",
      "timestamp_first_active\n",
      "created_year\n",
      "created_month\n",
      "created_day\n",
      "created_dayofweek\n",
      "created_dayofyear\n",
      "created_weekofyear\n",
      "first_active_year\n",
      "first_active_month\n",
      "first_active_day\n",
      "first_active_dayofweek\n",
      "first_active_dayofyear\n",
      "first_active_weekofyear\n",
      "diff_created_firstActive_year\n",
      "diff_created_firstActive_month\n",
      "diff_created_firstActive_day\n",
      "affiliate_channel_api\n",
      "affiliate_channel_content\n",
      "affiliate_channel_direct\n",
      "affiliate_channel_other\n",
      "affiliate_channel_remarketing\n",
      "affiliate_channel_sem-brand\n",
      "affiliate_channel_sem-non-brand\n",
      "affiliate_channel_seo\n",
      "affiliate_provider_baidu\n",
      "affiliate_provider_bing\n",
      "affiliate_provider_craigslist\n",
      "affiliate_provider_daum\n",
      "affiliate_provider_direct\n",
      "affiliate_provider_email-marketing\n",
      "affiliate_provider_facebook\n",
      "affiliate_provider_facebook-open-graph\n",
      "affiliate_provider_google\n",
      "affiliate_provider_gsp\n",
      "affiliate_provider_meetup\n",
      "affiliate_provider_naver\n",
      "affiliate_provider_other\n",
      "affiliate_provider_padmapper\n",
      "affiliate_provider_vast\n",
      "affiliate_provider_wayn\n",
      "affiliate_provider_yahoo\n",
      "affiliate_provider_yandex\n",
      "first_affiliate_tracked_linked\n",
      "first_affiliate_tracked_local ops\n",
      "first_affiliate_tracked_marketing\n",
      "first_affiliate_tracked_omg\n",
      "first_affiliate_tracked_product\n",
      "first_affiliate_tracked_tracked-other\n",
      "first_affiliate_tracked_untracked\n",
      "first_browser_-unknown-\n",
      "first_browser_AOL Explorer\n",
      "first_browser_Android Browser\n",
      "first_browser_Apple Mail\n",
      "first_browser_Arora\n",
      "first_browser_Avant Browser\n",
      "first_browser_BlackBerry Browser\n",
      "first_browser_Camino\n",
      "first_browser_Chrome\n",
      "first_browser_Chrome Mobile\n",
      "first_browser_Chromium\n",
      "first_browser_CometBird\n",
      "first_browser_Comodo Dragon\n",
      "first_browser_Conkeror\n",
      "first_browser_CoolNovo\n",
      "first_browser_Crazy Browser\n",
      "first_browser_Epic\n",
      "first_browser_Firefox\n",
      "first_browser_Flock\n",
      "first_browser_Google Earth\n",
      "first_browser_Googlebot\n",
      "first_browser_IBrowse\n",
      "first_browser_IE\n",
      "first_browser_IE Mobile\n",
      "first_browser_IceDragon\n",
      "first_browser_IceWeasel\n",
      "first_browser_Iron\n",
      "first_browser_Kindle Browser\n",
      "first_browser_Maxthon\n",
      "first_browser_Mobile Firefox\n",
      "first_browser_Mobile Safari\n",
      "first_browser_Mozilla\n",
      "first_browser_NetNewsWire\n",
      "first_browser_Nintendo Browser\n",
      "first_browser_OmniWeb\n",
      "first_browser_Opera\n",
      "first_browser_Opera Mini\n",
      "first_browser_Opera Mobile\n",
      "first_browser_Outlook 2007\n",
      "first_browser_PS Vita browser\n",
      "first_browser_Pale Moon\n",
      "first_browser_Palm Pre web browser\n",
      "first_browser_RockMelt\n",
      "first_browser_Safari\n",
      "first_browser_SeaMonkey\n",
      "first_browser_Silk\n",
      "first_browser_SiteKiosk\n",
      "first_browser_SlimBrowser\n",
      "first_browser_Sogou Explorer\n",
      "first_browser_Stainless\n",
      "first_browser_TenFourFox\n",
      "first_browser_TheWorld Browser\n",
      "first_browser_UC Browser\n",
      "first_browser_Yandex.Browser\n",
      "first_browser_wOSBrowser\n",
      "first_device_type_Android Phone\n",
      "first_device_type_Android Tablet\n",
      "first_device_type_Desktop (Other)\n",
      "first_device_type_Mac Desktop\n",
      "first_device_type_Other/Unknown\n",
      "first_device_type_SmartPhone (Other)\n",
      "first_device_type_Windows Desktop\n",
      "first_device_type_iPad\n",
      "first_device_type_iPhone\n",
      "gender_-unknown-\n",
      "gender_FEMALE\n",
      "gender_MALE\n",
      "gender_OTHER\n",
      "language_-unknown-\n",
      "language_ca\n",
      "language_cs\n",
      "language_da\n",
      "language_de\n",
      "language_el\n",
      "language_en\n",
      "language_es\n",
      "language_fi\n",
      "language_fr\n",
      "language_hr\n",
      "language_hu\n",
      "language_id\n",
      "language_is\n",
      "language_it\n",
      "language_ja\n",
      "language_ko\n",
      "language_nl\n",
      "language_no\n",
      "language_pl\n",
      "language_pt\n",
      "language_ru\n",
      "language_sv\n",
      "language_th\n",
      "language_tr\n",
      "language_zh\n",
      "signup_app_Android\n",
      "signup_app_Moweb\n",
      "signup_app_Web\n",
      "signup_app_iOS\n",
      "signup_method_basic\n",
      "signup_method_facebook\n",
      "signup_method_google\n",
      "signup_method_weibo\n",
      "signup_flow_0\n",
      "signup_flow_1\n",
      "signup_flow_2\n",
      "signup_flow_3\n",
      "signup_flow_4\n",
      "signup_flow_5\n",
      "signup_flow_6\n",
      "signup_flow_8\n",
      "signup_flow_10\n",
      "signup_flow_12\n",
      "signup_flow_14\n",
      "signup_flow_15\n",
      "signup_flow_16\n",
      "signup_flow_20\n",
      "signup_flow_21\n",
      "signup_flow_23\n",
      "signup_flow_24\n",
      "signup_flow_25\n"
     ]
    }
   ],
   "source": [
    "for header in dummiedMergedCoutriesTrainUses:\n",
    "    print header"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dummiedMergedCoutriesTrainUses.drop('date_first_booking', axis = 1, inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "age\n",
      "country_destination\n",
      "date_account_created\n",
      "id\n",
      "timestamp_first_active\n",
      "created_year\n",
      "created_month\n",
      "created_day\n",
      "created_dayofweek\n",
      "created_dayofyear\n",
      "created_weekofyear\n",
      "first_active_year\n",
      "first_active_month\n",
      "first_active_day\n",
      "first_active_dayofweek\n",
      "first_active_dayofyear\n",
      "first_active_weekofyear\n",
      "diff_created_firstActive_year\n",
      "diff_created_firstActive_month\n",
      "diff_created_firstActive_day\n",
      "affiliate_channel_api\n",
      "affiliate_channel_content\n",
      "affiliate_channel_direct\n",
      "affiliate_channel_other\n",
      "affiliate_channel_remarketing\n",
      "affiliate_channel_sem-brand\n",
      "affiliate_channel_sem-non-brand\n",
      "affiliate_channel_seo\n",
      "affiliate_provider_baidu\n",
      "affiliate_provider_bing\n",
      "affiliate_provider_craigslist\n",
      "affiliate_provider_daum\n",
      "affiliate_provider_direct\n",
      "affiliate_provider_email-marketing\n",
      "affiliate_provider_facebook\n",
      "affiliate_provider_facebook-open-graph\n",
      "affiliate_provider_google\n",
      "affiliate_provider_gsp\n",
      "affiliate_provider_meetup\n",
      "affiliate_provider_naver\n",
      "affiliate_provider_other\n",
      "affiliate_provider_padmapper\n",
      "affiliate_provider_vast\n",
      "affiliate_provider_wayn\n",
      "affiliate_provider_yahoo\n",
      "affiliate_provider_yandex\n",
      "first_affiliate_tracked_linked\n",
      "first_affiliate_tracked_local ops\n",
      "first_affiliate_tracked_marketing\n",
      "first_affiliate_tracked_omg\n",
      "first_affiliate_tracked_product\n",
      "first_affiliate_tracked_tracked-other\n",
      "first_affiliate_tracked_untracked\n",
      "first_browser_-unknown-\n",
      "first_browser_AOL Explorer\n",
      "first_browser_Android Browser\n",
      "first_browser_Apple Mail\n",
      "first_browser_Arora\n",
      "first_browser_Avant Browser\n",
      "first_browser_BlackBerry Browser\n",
      "first_browser_Camino\n",
      "first_browser_Chrome\n",
      "first_browser_Chrome Mobile\n",
      "first_browser_Chromium\n",
      "first_browser_CometBird\n",
      "first_browser_Comodo Dragon\n",
      "first_browser_Conkeror\n",
      "first_browser_CoolNovo\n",
      "first_browser_Crazy Browser\n",
      "first_browser_Epic\n",
      "first_browser_Firefox\n",
      "first_browser_Flock\n",
      "first_browser_Google Earth\n",
      "first_browser_Googlebot\n",
      "first_browser_IBrowse\n",
      "first_browser_IE\n",
      "first_browser_IE Mobile\n",
      "first_browser_IceDragon\n",
      "first_browser_IceWeasel\n",
      "first_browser_Iron\n",
      "first_browser_Kindle Browser\n",
      "first_browser_Maxthon\n",
      "first_browser_Mobile Firefox\n",
      "first_browser_Mobile Safari\n",
      "first_browser_Mozilla\n",
      "first_browser_NetNewsWire\n",
      "first_browser_Nintendo Browser\n",
      "first_browser_OmniWeb\n",
      "first_browser_Opera\n",
      "first_browser_Opera Mini\n",
      "first_browser_Opera Mobile\n",
      "first_browser_Outlook 2007\n",
      "first_browser_PS Vita browser\n",
      "first_browser_Pale Moon\n",
      "first_browser_Palm Pre web browser\n",
      "first_browser_RockMelt\n",
      "first_browser_Safari\n",
      "first_browser_SeaMonkey\n",
      "first_browser_Silk\n",
      "first_browser_SiteKiosk\n",
      "first_browser_SlimBrowser\n",
      "first_browser_Sogou Explorer\n",
      "first_browser_Stainless\n",
      "first_browser_TenFourFox\n",
      "first_browser_TheWorld Browser\n",
      "first_browser_UC Browser\n",
      "first_browser_Yandex.Browser\n",
      "first_browser_wOSBrowser\n",
      "first_device_type_Android Phone\n",
      "first_device_type_Android Tablet\n",
      "first_device_type_Desktop (Other)\n",
      "first_device_type_Mac Desktop\n",
      "first_device_type_Other/Unknown\n",
      "first_device_type_SmartPhone (Other)\n",
      "first_device_type_Windows Desktop\n",
      "first_device_type_iPad\n",
      "first_device_type_iPhone\n",
      "gender_-unknown-\n",
      "gender_FEMALE\n",
      "gender_MALE\n",
      "gender_OTHER\n",
      "language_-unknown-\n",
      "language_ca\n",
      "language_cs\n",
      "language_da\n",
      "language_de\n",
      "language_el\n",
      "language_en\n",
      "language_es\n",
      "language_fi\n",
      "language_fr\n",
      "language_hr\n",
      "language_hu\n",
      "language_id\n",
      "language_is\n",
      "language_it\n",
      "language_ja\n",
      "language_ko\n",
      "language_nl\n",
      "language_no\n",
      "language_pl\n",
      "language_pt\n",
      "language_ru\n",
      "language_sv\n",
      "language_th\n",
      "language_tr\n",
      "language_zh\n",
      "signup_app_Android\n",
      "signup_app_Moweb\n",
      "signup_app_Web\n",
      "signup_app_iOS\n",
      "signup_method_basic\n",
      "signup_method_facebook\n",
      "signup_method_google\n",
      "signup_method_weibo\n",
      "signup_flow_0\n",
      "signup_flow_1\n",
      "signup_flow_2\n",
      "signup_flow_3\n",
      "signup_flow_4\n",
      "signup_flow_5\n",
      "signup_flow_6\n",
      "signup_flow_8\n",
      "signup_flow_10\n",
      "signup_flow_12\n",
      "signup_flow_14\n",
      "signup_flow_15\n",
      "signup_flow_16\n",
      "signup_flow_20\n",
      "signup_flow_21\n",
      "signup_flow_23\n",
      "signup_flow_24\n",
      "signup_flow_25\n"
     ]
    }
   ],
   "source": [
    "for header in dummiedMergedCoutriesTrainUses:\n",
    "    print header"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Label encoding the target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "LE = preprocessing.LabelEncoder()\n",
    "labels_transed = LE.fit_transform(dummiedMergedCoutriesTrainUses['country_destination'][:train_size])\n",
    "    \n",
    "\n",
    "readyToLearnDataSet = pd.DataFrame(dummiedMergedCoutriesTrainUses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3 Train the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "1. realized that the model cannot learn datetime data type\n",
    "2. 'destination_language' is a mysterious variable that nobody can do things with it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "train_X = readyToLearnDataSet.drop(['country_destination', 'id',\\\n",
    "                                  'date_account_created',\\\n",
    "                                  'timestamp_first_active' ], axis = 1)[:train_size]\n",
    "\n",
    "train_y = labels_transed.reshape(-1,1)\n",
    "\n",
    "\n",
    "test_X = readyToLearnDataSet.drop(['country_destination', 'id',\\\n",
    "                                  'date_account_created',\\\n",
    "                                  'timestamp_first_active' ], axis = 1)[train_size:]\n",
    "\n",
    "test_y = readyToLearnDataSet[['country_destination']][train_size:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(213451, 169) train_X shape\n",
      "(213451, 1) train_y shape\n",
      "(62096, 169) test_X shape\n",
      "(62096, 1) test_y shape\n"
     ]
    }
   ],
   "source": [
    "print train_X.shape , 'train_X shape'\n",
    "print train_y.shape , 'train_y shape'\n",
    "print test_X.shape , 'test_X shape'\n",
    "print test_y.shape , 'test_y shape'\n",
    "\n",
    "# print test_y.country_destination.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False    275547\n",
      "Name: age, dtype: int64\n",
      "False    213451\n",
      "True      62096\n",
      "Name: country_destination, dtype: int64\n",
      "False    275547\n",
      "Name: date_account_created, dtype: int64\n",
      "False    275547\n",
      "Name: id, dtype: int64\n",
      "False    275547\n",
      "Name: timestamp_first_active, dtype: int64\n",
      "False    275547\n",
      "Name: created_year, dtype: int64\n",
      "False    275547\n",
      "Name: created_month, dtype: int64\n",
      "False    275547\n",
      "Name: created_day, dtype: int64\n",
      "False    275547\n",
      "Name: created_dayofweek, dtype: int64\n",
      "False    275547\n",
      "Name: created_dayofyear, dtype: int64\n",
      "False    275547\n",
      "Name: created_weekofyear, dtype: int64\n",
      "False    275547\n",
      "Name: first_active_year, dtype: int64\n",
      "False    275547\n",
      "Name: first_active_month, dtype: int64\n",
      "False    275547\n",
      "Name: first_active_day, dtype: int64\n",
      "False    275547\n",
      "Name: first_active_dayofweek, dtype: int64\n",
      "False    275547\n",
      "Name: first_active_dayofyear, dtype: int64\n",
      "False    275547\n",
      "Name: first_active_weekofyear, dtype: int64\n",
      "False    275547\n",
      "Name: diff_created_firstActive_year, dtype: int64\n",
      "False    275547\n",
      "Name: diff_created_firstActive_month, dtype: int64\n",
      "False    275547\n",
      "Name: diff_created_firstActive_day, dtype: int64\n",
      "False    275547\n",
      "Name: affiliate_channel_api, dtype: int64\n",
      "False    275547\n",
      "Name: affiliate_channel_content, dtype: int64\n",
      "False    275547\n",
      "Name: affiliate_channel_direct, dtype: int64\n",
      "False    275547\n",
      "Name: affiliate_channel_other, dtype: int64\n",
      "False    275547\n",
      "Name: affiliate_channel_remarketing, dtype: int64\n",
      "False    275547\n",
      "Name: affiliate_channel_sem-brand, dtype: int64\n",
      "False    275547\n",
      "Name: affiliate_channel_sem-non-brand, dtype: int64\n",
      "False    275547\n",
      "Name: affiliate_channel_seo, dtype: int64\n",
      "False    275547\n",
      "Name: affiliate_provider_baidu, dtype: int64\n",
      "False    275547\n",
      "Name: affiliate_provider_bing, dtype: int64\n",
      "False    275547\n",
      "Name: affiliate_provider_craigslist, dtype: int64\n",
      "False    275547\n",
      "Name: affiliate_provider_daum, dtype: int64\n",
      "False    275547\n",
      "Name: affiliate_provider_direct, dtype: int64\n",
      "False    275547\n",
      "Name: affiliate_provider_email-marketing, dtype: int64\n",
      "False    275547\n",
      "Name: affiliate_provider_facebook, dtype: int64\n",
      "False    275547\n",
      "Name: affiliate_provider_facebook-open-graph, dtype: int64\n",
      "False    275547\n",
      "Name: affiliate_provider_google, dtype: int64\n",
      "False    275547\n",
      "Name: affiliate_provider_gsp, dtype: int64\n",
      "False    275547\n",
      "Name: affiliate_provider_meetup, dtype: int64\n",
      "False    275547\n",
      "Name: affiliate_provider_naver, dtype: int64\n",
      "False    275547\n",
      "Name: affiliate_provider_other, dtype: int64\n",
      "False    275547\n",
      "Name: affiliate_provider_padmapper, dtype: int64\n",
      "False    275547\n",
      "Name: affiliate_provider_vast, dtype: int64\n",
      "False    275547\n",
      "Name: affiliate_provider_wayn, dtype: int64\n",
      "False    275547\n",
      "Name: affiliate_provider_yahoo, dtype: int64\n",
      "False    275547\n",
      "Name: affiliate_provider_yandex, dtype: int64\n",
      "False    275547\n",
      "Name: first_affiliate_tracked_linked, dtype: int64\n",
      "False    275547\n",
      "Name: first_affiliate_tracked_local ops, dtype: int64\n",
      "False    275547\n",
      "Name: first_affiliate_tracked_marketing, dtype: int64\n",
      "False    275547\n",
      "Name: first_affiliate_tracked_omg, dtype: int64\n",
      "False    275547\n",
      "Name: first_affiliate_tracked_product, dtype: int64\n",
      "False    275547\n",
      "Name: first_affiliate_tracked_tracked-other, dtype: int64\n",
      "False    275547\n",
      "Name: first_affiliate_tracked_untracked, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_-unknown-, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_AOL Explorer, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Android Browser, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Apple Mail, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Arora, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Avant Browser, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_BlackBerry Browser, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Camino, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Chrome, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Chrome Mobile, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Chromium, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_CometBird, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Comodo Dragon, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Conkeror, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_CoolNovo, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Crazy Browser, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Epic, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Firefox, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Flock, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Google Earth, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Googlebot, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_IBrowse, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_IE, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_IE Mobile, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_IceDragon, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_IceWeasel, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Iron, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Kindle Browser, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Maxthon, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Mobile Firefox, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Mobile Safari, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Mozilla, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_NetNewsWire, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Nintendo Browser, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_OmniWeb, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Opera, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Opera Mini, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Opera Mobile, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Outlook 2007, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_PS Vita browser, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Pale Moon, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Palm Pre web browser, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_RockMelt, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Safari, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_SeaMonkey, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Silk, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_SiteKiosk, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_SlimBrowser, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Sogou Explorer, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Stainless, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_TenFourFox, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_TheWorld Browser, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_UC Browser, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_Yandex.Browser, dtype: int64\n",
      "False    275547\n",
      "Name: first_browser_wOSBrowser, dtype: int64\n",
      "False    275547\n",
      "Name: first_device_type_Android Phone, dtype: int64\n",
      "False    275547\n",
      "Name: first_device_type_Android Tablet, dtype: int64\n",
      "False    275547\n",
      "Name: first_device_type_Desktop (Other), dtype: int64\n",
      "False    275547\n",
      "Name: first_device_type_Mac Desktop, dtype: int64\n",
      "False    275547\n",
      "Name: first_device_type_Other/Unknown, dtype: int64\n",
      "False    275547\n",
      "Name: first_device_type_SmartPhone (Other), dtype: int64\n",
      "False    275547\n",
      "Name: first_device_type_Windows Desktop, dtype: int64\n",
      "False    275547\n",
      "Name: first_device_type_iPad, dtype: int64\n",
      "False    275547\n",
      "Name: first_device_type_iPhone, dtype: int64\n",
      "False    275547\n",
      "Name: gender_-unknown-, dtype: int64\n",
      "False    275547\n",
      "Name: gender_FEMALE, dtype: int64\n",
      "False    275547\n",
      "Name: gender_MALE, dtype: int64\n",
      "False    275547\n",
      "Name: gender_OTHER, dtype: int64\n",
      "False    275547\n",
      "Name: language_-unknown-, dtype: int64\n",
      "False    275547\n",
      "Name: language_ca, dtype: int64\n",
      "False    275547\n",
      "Name: language_cs, dtype: int64\n",
      "False    275547\n",
      "Name: language_da, dtype: int64\n",
      "False    275547\n",
      "Name: language_de, dtype: int64\n",
      "False    275547\n",
      "Name: language_el, dtype: int64\n",
      "False    275547\n",
      "Name: language_en, dtype: int64\n",
      "False    275547\n",
      "Name: language_es, dtype: int64\n",
      "False    275547\n",
      "Name: language_fi, dtype: int64\n",
      "False    275547\n",
      "Name: language_fr, dtype: int64\n",
      "False    275547\n",
      "Name: language_hr, dtype: int64\n",
      "False    275547\n",
      "Name: language_hu, dtype: int64\n",
      "False    275547\n",
      "Name: language_id, dtype: int64\n",
      "False    275547\n",
      "Name: language_is, dtype: int64\n",
      "False    275547\n",
      "Name: language_it, dtype: int64\n",
      "False    275547\n",
      "Name: language_ja, dtype: int64\n",
      "False    275547\n",
      "Name: language_ko, dtype: int64\n",
      "False    275547\n",
      "Name: language_nl, dtype: int64\n",
      "False    275547\n",
      "Name: language_no, dtype: int64\n",
      "False    275547\n",
      "Name: language_pl, dtype: int64\n",
      "False    275547\n",
      "Name: language_pt, dtype: int64\n",
      "False    275547\n",
      "Name: language_ru, dtype: int64\n",
      "False    275547\n",
      "Name: language_sv, dtype: int64\n",
      "False    275547\n",
      "Name: language_th, dtype: int64\n",
      "False    275547\n",
      "Name: language_tr, dtype: int64\n",
      "False    275547\n",
      "Name: language_zh, dtype: int64\n",
      "False    275547\n",
      "Name: signup_app_Android, dtype: int64\n",
      "False    275547\n",
      "Name: signup_app_Moweb, dtype: int64\n",
      "False    275547\n",
      "Name: signup_app_Web, dtype: int64\n",
      "False    275547\n",
      "Name: signup_app_iOS, dtype: int64\n",
      "False    275547\n",
      "Name: signup_method_basic, dtype: int64\n",
      "False    275547\n",
      "Name: signup_method_facebook, dtype: int64\n",
      "False    275547\n",
      "Name: signup_method_google, dtype: int64\n",
      "False    275547\n",
      "Name: signup_method_weibo, dtype: int64\n",
      "False    275547\n",
      "Name: signup_flow_0, dtype: int64\n",
      "False    275547\n",
      "Name: signup_flow_1, dtype: int64\n",
      "False    275547\n",
      "Name: signup_flow_2, dtype: int64\n",
      "False    275547\n",
      "Name: signup_flow_3, dtype: int64\n",
      "False    275547\n",
      "Name: signup_flow_4, dtype: int64\n",
      "False    275547\n",
      "Name: signup_flow_5, dtype: int64\n",
      "False    275547\n",
      "Name: signup_flow_6, dtype: int64\n",
      "False    275547\n",
      "Name: signup_flow_8, dtype: int64\n",
      "False    275547\n",
      "Name: signup_flow_10, dtype: int64\n",
      "False    275547\n",
      "Name: signup_flow_12, dtype: int64\n",
      "False    275547\n",
      "Name: signup_flow_14, dtype: int64\n",
      "False    275547\n",
      "Name: signup_flow_15, dtype: int64\n",
      "False    275547\n",
      "Name: signup_flow_16, dtype: int64\n",
      "False    275547\n",
      "Name: signup_flow_20, dtype: int64\n",
      "False    275547\n",
      "Name: signup_flow_21, dtype: int64\n",
      "False    275547\n",
      "Name: signup_flow_23, dtype: int64\n",
      "False    275547\n",
      "Name: signup_flow_24, dtype: int64\n",
      "False    275547\n",
      "Name: signup_flow_25, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "for header11 in readyToLearnDataSet:\n",
    "    print readyToLearnDataSet[header11].isnull().value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### There is no <font color='red'>'Null'</font> value in this data set now"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### Implement Scorer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# ---------------------------- copy from someone else ----------------------------\n",
    "# https://github.com/davidgasquez/kaggle-airbnb/blob/master/kairbnb/metrics.py\n",
    "# https://www.kaggle.com/davidgasquez/airbnb-recruiting-new-user-bookings/ndcg-scorer/code\n",
    "\n",
    "\"\"\"Metrics to compute the model performance.\"\"\"\n",
    "\n",
    "import numpy as np\n",
    "from sklearn.preprocessing import LabelBinarizer\n",
    "from sklearn.metrics import make_scorer\n",
    "\n",
    "\n",
    "def dcg_score(y_true, y_score, k=5):\n",
    "    \"\"\"Discounted cumulative gain (DCG) at rank K.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    y_true : array, shape = [n_samples]\n",
    "        Ground truth (true relevance labels).\n",
    "    y_score : array, shape = [n_samples, n_classes]\n",
    "        Predicted scores.\n",
    "    k : int\n",
    "        Rank.\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    score : float\n",
    "    \"\"\"\n",
    "    order = np.argsort(y_score)[::-1]\n",
    "    y_true = np.take(y_true, order[:k])\n",
    "\n",
    "    gain = 2 ** y_true - 1\n",
    "\n",
    "    discounts = np.log2(np.arange(len(y_true)) + 2)\n",
    "    return np.sum(gain / discounts)\n",
    "\n",
    "\n",
    "def ndcg_score(ground_truth, predictions, k=5):\n",
    "    \"\"\"Normalized discounted cumulative gain (NDCG) at rank K.\n",
    "\n",
    "    Normalized Discounted Cumulative Gain (NDCG) measures the performance of a\n",
    "    recommendation system based on the graded relevance of the recommended\n",
    "    entities. It varies from 0.0 to 1.0, with 1.0 representing the ideal\n",
    "    ranking of the entities.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    ground_truth : array, shape = [n_samples]\n",
    "        Ground truth (true labels represended as integers).\n",
    "    predictions : array, shape = [n_samples, n_classes]\n",
    "        Predicted probabilities.\n",
    "    k : int\n",
    "        Rank.\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    score : float\n",
    "\n",
    "    Example\n",
    "    -------\n",
    "    >>> ground_truth = [1, 0, 2]\n",
    "    >>> predictions = [[0.15, 0.55, 0.2], [0.7, 0.2, 0.1], [0.06, 0.04, 0.9]]\n",
    "    >>> score = ndcg_score(ground_truth, predictions, k=2)\n",
    "    1.0\n",
    "    >>> predictions = [[0.9, 0.5, 0.8], [0.7, 0.2, 0.1], [0.06, 0.04, 0.9]]\n",
    "    >>> score = ndcg_score(ground_truth, predictions, k=2)\n",
    "    0.6666666666\n",
    "    \"\"\"\n",
    "    lb = LabelBinarizer()\n",
    "    lb.fit(range(len(predictions) + 1))\n",
    "    T = lb.transform(ground_truth)\n",
    "\n",
    "    scores = []\n",
    "\n",
    "    # Iterate over each y_true and compute the DCG score\n",
    "    for y_true, y_score in zip(T, predictions):\n",
    "        actual = dcg_score(y_true, y_score, k)\n",
    "        best = dcg_score(y_true, y_true, k)\n",
    "        score = float(actual) / float(best)\n",
    "        scores.append(score)\n",
    "\n",
    "    return np.mean(scores)\n",
    "\n",
    "\n",
    "# NDCG Scorer function\n",
    "ndcg_scorer = make_scorer(ndcg_score, needs_proba=True, k=5)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "XGBClassifier(base_score=0.5, colsample_bylevel=1, colsample_bytree=0.5,\n",
       "       gamma=1, learning_rate=0.3, max_delta_step=0, max_depth=6,\n",
       "       min_child_weight=1, missing=None, n_estimators=25, nthread=-1,\n",
       "       objective='multi:softprob', reg_alpha=0, reg_lambda=1,\n",
       "       scale_pos_weight=1, seed=1234, silent=True, subsample=0.5)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf_xgb = xgb.XGBClassifier(learning_rate = 0.3\n",
    "                  , n_estimators = 25\n",
    "                  , max_depth = 6\n",
    "                  , min_child_weight = 1\n",
    "                  , subsample = 0.5\n",
    "                  , colsample_bytree = 0.5\n",
    "                  , gamma = 1\n",
    "                  , seed = 1234\n",
    "                  , nthread = -1\n",
    "                  , objective='multi:softprob'\n",
    "                  )\n",
    "clf_xgb.fit(train_X, train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             0         1         2         3         4         5         6   \\\n",
      "0      0.002320  0.008866  0.003131  0.009130  0.024038  0.005922  0.006301   \n",
      "1      0.001227  0.003702  0.002116  0.003406  0.006146  0.003397  0.004107   \n",
      "2      0.001663  0.005109  0.007614  0.010596  0.009887  0.006460  0.009074   \n",
      "3      0.001603  0.004348  0.008125  0.009490  0.009530  0.006227  0.008747   \n",
      "4      0.001907  0.005501  0.003895  0.008014  0.014404  0.009445  0.010459   \n",
      "5      0.002963  0.011551  0.007303  0.013732  0.027112  0.016503  0.021612   \n",
      "6      0.003287  0.009839  0.006471  0.008893  0.018353  0.008912  0.016055   \n",
      "7      0.001913  0.006315  0.003932  0.008092  0.014184  0.008725  0.011135   \n",
      "8      0.001913  0.006315  0.003932  0.008092  0.014184  0.008725  0.011135   \n",
      "9      0.001680  0.004824  0.007254  0.006090  0.011431  0.005861  0.007441   \n",
      "10     0.001903  0.005820  0.006229  0.007831  0.011738  0.008236  0.010838   \n",
      "11     0.003114  0.007276  0.003989  0.012011  0.019810  0.007772  0.018891   \n",
      "12     0.002602  0.006104  0.006157  0.020641  0.026610  0.014314  0.010781   \n",
      "13     0.001739  0.005341  0.004754  0.010355  0.012425  0.007329  0.009219   \n",
      "14     0.001334  0.003361  0.002460  0.005152  0.005555  0.003600  0.004565   \n",
      "15     0.003194  0.012238  0.016321  0.014260  0.027163  0.015667  0.018471   \n",
      "16     0.001589  0.003411  0.003053  0.006046  0.012269  0.006217  0.009240   \n",
      "17     0.003657  0.013947  0.014951  0.016696  0.020877  0.013290  0.015702   \n",
      "18     0.003491  0.014301  0.007526  0.015592  0.030353  0.015534  0.021793   \n",
      "19     0.002868  0.007380  0.005931  0.010596  0.014959  0.010154  0.010075   \n",
      "20     0.002939  0.012547  0.011850  0.013570  0.037950  0.029232  0.016552   \n",
      "21     0.001967  0.006016  0.003845  0.008321  0.014585  0.008636  0.011213   \n",
      "22     0.002378  0.006937  0.004859  0.011076  0.014809  0.009176  0.009974   \n",
      "23     0.001967  0.006016  0.003845  0.008321  0.014585  0.008636  0.011213   \n",
      "24     0.001647  0.005022  0.007482  0.005972  0.011211  0.005572  0.007794   \n",
      "25     0.003330  0.022299  0.006675  0.010799  0.023938  0.007288  0.007642   \n",
      "26     0.002784  0.008332  0.009180  0.023837  0.036150  0.018795  0.017426   \n",
      "27     0.002226  0.004488  0.010069  0.006109  0.009933  0.006461  0.014811   \n",
      "28     0.003475  0.013625  0.012245  0.015742  0.029044  0.016475  0.019359   \n",
      "29     0.001565  0.004211  0.004892  0.010077  0.010039  0.006286  0.014288   \n",
      "...         ...       ...       ...       ...       ...       ...       ...   \n",
      "62066  0.001083  0.001647  0.001827  0.002625  0.003630  0.003951  0.002462   \n",
      "62067  0.010030  0.009875  0.007326  0.009939  0.036723  0.014836  0.019857   \n",
      "62068  0.004216  0.005317  0.005698  0.008505  0.014645  0.006716  0.011234   \n",
      "62069  0.004210  0.003254  0.002225  0.006770  0.017172  0.007240  0.006761   \n",
      "62070  0.008617  0.005896  0.004108  0.011625  0.031577  0.013418  0.032240   \n",
      "62071  0.001575  0.002782  0.002819  0.004119  0.005216  0.003562  0.004551   \n",
      "62072  0.001182  0.001876  0.002114  0.002727  0.003829  0.002344  0.002686   \n",
      "62073  0.001182  0.001876  0.002114  0.002727  0.003829  0.002344  0.002686   \n",
      "62074  0.001246  0.001916  0.002773  0.003225  0.004889  0.003108  0.003744   \n",
      "62075  0.004925  0.007165  0.005858  0.008951  0.020474  0.010280  0.010839   \n",
      "62076  0.005021  0.005677  0.006837  0.013392  0.029949  0.012993  0.014640   \n",
      "62077  0.004216  0.005317  0.005698  0.008505  0.014645  0.006716  0.011234   \n",
      "62078  0.003682  0.003650  0.003547  0.008876  0.010782  0.006537  0.017403   \n",
      "62079  0.005870  0.010690  0.016439  0.010792  0.030266  0.019539  0.017625   \n",
      "62080  0.006448  0.012681  0.009759  0.015897  0.026788  0.020051  0.015568   \n",
      "62081  0.005986  0.005096  0.003037  0.007016  0.011015  0.005408  0.005126   \n",
      "62082  0.003351  0.004638  0.004957  0.008174  0.014075  0.005459  0.012107   \n",
      "62083  0.003063  0.004902  0.005463  0.005657  0.009505  0.005135  0.004264   \n",
      "62084  0.003943  0.009653  0.007420  0.009552  0.027754  0.014668  0.011283   \n",
      "62085  0.005749  0.004031  0.003048  0.009061  0.025614  0.007800  0.009150   \n",
      "62086  0.001244  0.001957  0.002080  0.003121  0.004858  0.003394  0.004256   \n",
      "62087  0.006918  0.011476  0.014324  0.010740  0.024767  0.013994  0.024397   \n",
      "62088  0.005512  0.011271  0.009722  0.010730  0.030842  0.016748  0.019221   \n",
      "62089  0.005769  0.010289  0.008332  0.012784  0.021830  0.014756  0.011531   \n",
      "62090  0.003017  0.004113  0.004933  0.007848  0.009849  0.006230  0.008045   \n",
      "62091  0.006039  0.010939  0.006891  0.013282  0.026228  0.018085  0.017772   \n",
      "62092  0.001771  0.002568  0.004278  0.004189  0.006283  0.004929  0.004408   \n",
      "62093  0.002243  0.003556  0.017191  0.007134  0.009649  0.007280  0.006510   \n",
      "62094  0.001957  0.003911  0.005974  0.004780  0.012517  0.005353  0.010607   \n",
      "62095  0.005362  0.012753  0.008067  0.007174  0.013482  0.009768  0.010237   \n",
      "\n",
      "             7         8         9         10        11  \n",
      "0      0.585786  0.003548  0.002526  0.309618  0.038815  \n",
      "1      0.860099  0.001481  0.001456  0.096591  0.016272  \n",
      "2      0.767930  0.002227  0.001972  0.144297  0.033171  \n",
      "3      0.797136  0.001856  0.001901  0.122510  0.028527  \n",
      "4      0.761430  0.002719  0.002178  0.154269  0.025779  \n",
      "5      0.341753  0.005271  0.003302  0.490974  0.057923  \n",
      "6      0.418255  0.006937  0.003142  0.436760  0.063095  \n",
      "7      0.737482  0.003176  0.002199  0.169776  0.033071  \n",
      "8      0.737482  0.003176  0.002199  0.169776  0.033071  \n",
      "9      0.816164  0.001923  0.002216  0.107644  0.027472  \n",
      "10     0.748161  0.002203  0.002736  0.160244  0.034061  \n",
      "11     0.352550  0.004910  0.003091  0.506052  0.060533  \n",
      "12     0.585819  0.006296  0.002679  0.271144  0.046851  \n",
      "13     0.760068  0.002887  0.001999  0.153824  0.030062  \n",
      "14     0.851237  0.001706  0.001582  0.101141  0.018307  \n",
      "15     0.417126  0.005183  0.003251  0.410134  0.056992  \n",
      "16     0.804397  0.001962  0.001884  0.126723  0.023209  \n",
      "17     0.268818  0.005464  0.003271  0.538240  0.085085  \n",
      "18     0.365548  0.005663  0.003542  0.447981  0.068675  \n",
      "19     0.562726  0.004857  0.002390  0.314829  0.053235  \n",
      "20     0.395290  0.004797  0.002965  0.435821  0.036487  \n",
      "21     0.739285  0.002824  0.002742  0.170047  0.030520  \n",
      "22     0.570312  0.004808  0.002366  0.310602  0.052701  \n",
      "23     0.739285  0.002824  0.002742  0.170047  0.030520  \n",
      "24     0.820457  0.001787  0.001885  0.103078  0.028093  \n",
      "25     0.308518  0.005472  0.002661  0.516414  0.084966  \n",
      "26     0.313183  0.005596  0.003232  0.502201  0.059284  \n",
      "27     0.808555  0.001909  0.001902  0.109714  0.023823  \n",
      "28     0.319894  0.008217  0.012193  0.450223  0.099510  \n",
      "29     0.797445  0.002174  0.001910  0.123808  0.023307  \n",
      "...         ...       ...       ...       ...       ...  \n",
      "62066  0.926482  0.001864  0.001102  0.043785  0.009542  \n",
      "62067  0.424602  0.007075  0.002842  0.412314  0.044580  \n",
      "62068  0.765191  0.002525  0.001749  0.147173  0.027030  \n",
      "62069  0.754042  0.003967  0.001838  0.158296  0.034226  \n",
      "62070  0.637830  0.005310  0.002761  0.205272  0.041346  \n",
      "62071  0.873241  0.001497  0.001238  0.085094  0.014306  \n",
      "62072  0.921778  0.001183  0.000995  0.047882  0.011404  \n",
      "62073  0.921778  0.001183  0.000995  0.047882  0.011404  \n",
      "62074  0.906515  0.001185  0.000979  0.059012  0.011409  \n",
      "62075  0.624158  0.007699  0.002093  0.257406  0.040153  \n",
      "62076  0.642856  0.006130  0.002208  0.212064  0.048234  \n",
      "62077  0.765191  0.002525  0.001749  0.147173  0.027030  \n",
      "62078  0.821504  0.001978  0.001388  0.097956  0.022698  \n",
      "62079  0.401935  0.006116  0.002332  0.406288  0.072107  \n",
      "62080  0.336695  0.005420  0.002475  0.470052  0.078165  \n",
      "62081  0.678818  0.003786  0.002197  0.228045  0.044471  \n",
      "62082  0.800346  0.002427  0.001642  0.118530  0.024293  \n",
      "62083  0.753270  0.003505  0.001852  0.164593  0.038791  \n",
      "62084  0.502451  0.006763  0.002342  0.359802  0.044367  \n",
      "62085  0.675373  0.004249  0.002130  0.210953  0.042842  \n",
      "62086  0.926112  0.001246  0.000977  0.038045  0.012712  \n",
      "62087  0.413419  0.006751  0.002636  0.383821  0.086758  \n",
      "62088  0.297947  0.006962  0.003059  0.514445  0.073540  \n",
      "62089  0.336390  0.006635  0.002482  0.472755  0.096449  \n",
      "62090  0.809773  0.002012  0.001443  0.112356  0.030382  \n",
      "62091  0.318083  0.006074  0.002801  0.497125  0.076681  \n",
      "62092  0.808252  0.001942  0.001633  0.090518  0.069230  \n",
      "62093  0.799338  0.001887  0.001560  0.120016  0.023637  \n",
      "62094  0.823633  0.002193  0.001608  0.107417  0.020051  \n",
      "62095  0.473508  0.004963  0.002547  0.404347  0.047793  \n",
      "\n",
      "[62096 rows x 12 columns]\n"
     ]
    }
   ],
   "source": [
    "preds = clf_xgb.predict_proba(test_X)\n",
    "print pd.DataFrame(preds)\n",
    "# print pd.DataFrame(LE.inverse_transform(preds))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "id_test = readyToLearnDataSet['id'][train_size:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get the top 5 probabilities "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Taking the 5 classes with highest probabilities\n",
    "ids = []\n",
    "cts = []\n",
    "for i in range(len(id_test)):\n",
    "    idx = id_test[i]\n",
    "    ids += [idx] * 5\n",
    "    cts += LE.inverse_transform(np.argsort(preds[i])[::-1])[:5].tolist()\n",
    "    \n",
    "\n",
    "# Write csv files as submission\n",
    "sub = pd.DataFrame(np.column_stack((ids, cts)), columns=['id', 'country'])\n",
    "sub.to_csv('sub_201710041410.csv',index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature importance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                           feature_name  importance\n",
      "0                     created_dayofyear    0.106512\n",
      "1                                   age    0.098527\n",
      "2                first_active_dayofyear    0.083905\n",
      "3                           created_day    0.063660\n",
      "4                    created_weekofyear    0.040940\n",
      "5                     created_dayofweek    0.037341\n",
      "6                          created_year    0.034417\n",
      "7                      first_active_day    0.033180\n",
      "8                first_active_dayofweek    0.024857\n",
      "9               first_active_weekofyear    0.024857\n",
      "10                    first_active_year    0.024069\n",
      "11        first_device_type_Mac Desktop    0.018221\n",
      "12    first_affiliate_tracked_untracked    0.017658\n",
      "13                  signup_method_basic    0.015971\n",
      "14                        gender_FEMALE    0.014959\n",
      "15                     gender_-unknown-    0.014172\n",
      "16                          gender_MALE    0.013384\n",
      "17                        signup_flow_3    0.013159\n",
      "18                        created_month    0.011697\n",
      "19                       signup_app_Web    0.010348\n",
      "20       first_affiliate_tracked_linked    0.010010\n",
      "21    first_device_type_Windows Desktop    0.009898\n",
      "22              affiliate_channel_other    0.009785\n",
      "23              first_browser_-unknown-    0.009560\n",
      "24                first_browser_Firefox    0.009448\n",
      "25                 first_browser_Safari    0.009448\n",
      "26               signup_method_facebook    0.009448\n",
      "27                 first_browser_Chrome    0.008885\n",
      "28                        signup_flow_0    0.008885\n",
      "29          first_affiliate_tracked_omg    0.008773\n",
      "..                                  ...         ...\n",
      "76            affiliate_provider_meetup    0.001350\n",
      "77                          language_zh    0.001350\n",
      "78                       signup_flow_23    0.001125\n",
      "79        first_browser_Android Browser    0.000900\n",
      "80                          language_ja    0.000900\n",
      "81                          language_pt    0.000900\n",
      "82                       signup_flow_21    0.000787\n",
      "83       diff_created_firstActive_month    0.000562\n",
      "84         affiliate_provider_padmapper    0.000562\n",
      "85             affiliate_provider_yahoo    0.000562\n",
      "86               first_browser_Chromium    0.000562\n",
      "87                          language_nl    0.000562\n",
      "88   affiliate_provider_email-marketing    0.000450\n",
      "89                  first_browser_Opera    0.000450\n",
      "90                        signup_flow_5    0.000450\n",
      "91                          language_no    0.000337\n",
      "92                          language_ru    0.000337\n",
      "93                 signup_method_google    0.000337\n",
      "94    first_affiliate_tracked_marketing    0.000225\n",
      "95           first_browser_AOL Explorer    0.000225\n",
      "96                          language_sv    0.000225\n",
      "97        diff_created_firstActive_year    0.000112\n",
      "98             affiliate_provider_naver    0.000112\n",
      "99    first_affiliate_tracked_local ops    0.000112\n",
      "100    first_browser_BlackBerry Browser    0.000112\n",
      "101                first_browser_Camino    0.000112\n",
      "102             first_browser_IE Mobile    0.000112\n",
      "103              first_browser_RockMelt    0.000112\n",
      "104                  first_browser_Silk    0.000112\n",
      "105                         language_el    0.000112\n",
      "\n",
      "[106 rows x 2 columns]\n"
     ]
    }
   ],
   "source": [
    "full_vars = []\n",
    "for header in train_X:\n",
    "    full_vars.append(header)\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "feature_importance = pd.DataFrame(sorted(zip(full_vars,clf_xgb.feature_importances_)\n",
    "                          , key=lambda x: x[1], reverse = True),columns=['feature_name','importance']) \n",
    "\n",
    "print (feature_importance.query('importance>0'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check the prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 7  7 10 ...,  7  7  7]\n",
      "['NDF' 'NDF' 'US' ..., 'NDF' 'NDF' 'NDF']\n"
     ]
    }
   ],
   "source": [
    "print y\n",
    "print le.inverse_transform(y)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}