III__Model_XGBoost_non_recursive.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"accelerator":"GPU","colab":{"name":"III__Model_XGBoost_non_recursive.ipynb","provenance":[],"collapsed_sections":[]},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.7"}},"cells":[{"cell_type":"markdown","metadata":{"colab_type":"text","id":"TenfhbyBcz0o"},"source":["# XGBoost Modeling\n","\n","In this Notebook XGBoost can be used to make some magic!\n","Because each of the model needs to be safed properly. Please specify a name for the model, which describes briefly its specialities:"]},{"cell_type":"code","metadata":{"id":"5fMsSmIRk386","colab_type":"code","colab":{},"executionInfo":{"status":"ok","timestamp":1593370520301,"user_tz":-120,"elapsed":2509,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}}},"source":["model_name: str= \"Non_Recursive_XGBoost_Model\""],"execution_count":1,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"P94sbukY70Ka"},"source":["# 1. Setting up Colab\n","\n","Here we set up colab. U knooow it :D \n","\n","In case u don't use colab, please specify the variable ```use_colab``` to false. In case u don't use it its unavoidable to make sure that noone changed something online while u change it offline. The results of this would be worse than a division by zero.  "]},{"cell_type":"code","metadata":{"colab_type":"code","id":"9OwEnQ9q90xk","colab":{},"executionInfo":{"status":"ok","timestamp":1593370521021,"user_tz":-120,"elapsed":3143,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}}},"source":["use_colab: bool = True"],"execution_count":2,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"-aKdIiCP-DAh"},"source":["In case u accidentally run the following code twice u will get following weird and confusing error:\n","\n","```\n","OSError: [Errno 107] Transport endpoint is not connected\n","```\n","\n","In this case just restart the runtime above. \n"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"NW-V0LwRFBrz","colab":{"base_uri":"https://localhost:8080/","height":51},"executionInfo":{"status":"ok","timestamp":1593370522040,"user_tz":-120,"elapsed":4128,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}},"outputId":"34bdbe18-8d64-41b9-ddc6-f18483375c4c"},"source":["if use_colab:\n","  import os\n","  from google.colab import drive \n","  drive.mount(\"/content/gdrive\", force_remount=True)\n","  # Change directory for nicer imports\n","  %cd \"/content/gdrive/My Drive/Data_Mining_Cup/05 Code/\""],"execution_count":3,"outputs":[{"output_type":"stream","text":["Mounted at /content/gdrive\n","/content/gdrive/My Drive/Data_Mining_Cup/05 Code\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"o0G7khZfI-7B"},"source":["# 2. Notes and Ideas"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"FhkpJPxiJBhP"},"source":["## Ideas for new Features\n","Please specify assignment in bold and brackets behind. If nobody is assigned put the \"To Be Assigned (TBA)\" Flague.\n","*"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"xuI6qXPWJehT"},"source":["## Notes\n","\n","Currently no notes :( "]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"qg-T8Iok8z_X"},"source":["# 3. Setting up Notebook\n","\n","Here we setup some parameters for the notebook to work in the expected ways."]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"8LRU5vmI9eCt"},"source":["If we want to **use the final test phase** for the later submission, we need to specify the following parameter **```make_final_submission``` to true.** Otherwise the selfmade test set will be used."]},{"cell_type":"code","metadata":{"colab_type":"code","id":"LDWM8LmV_KkE","colab":{},"executionInfo":{"status":"ok","timestamp":1593370522057,"user_tz":-120,"elapsed":4093,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}}},"source":["make_final_submission: bool = False"],"execution_count":4,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"3JVVjiq__R7I"},"source":["In case we don't want to make a final submission, we now have to specify the test_set. In case you change it, please leave a comment on why and what the new one is. If the end is defined as None we use all the data from start_period (inclusive) till the end of dataset. \n","\n","In case make_final_submission is True we do not use those parameters."]},{"cell_type":"code","metadata":{"colab_type":"code","id":"KvKp58Ug_QwD","colab":{},"executionInfo":{"status":"ok","timestamp":1593370522058,"user_tz":-120,"elapsed":4045,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}}},"source":["# This will set the start of test period to 2 weeks before end of data\n","test_period_start: str=\"2018-06-02\"\n","# Use till the end\n","test_period_end: str=\"2018-06-15\""],"execution_count":5,"outputs":[]},{"cell_type":"code","metadata":{"colab_type":"code","id":"ThuLP_PksLxg","colab":{},"executionInfo":{"status":"ok","timestamp":1593370522059,"user_tz":-120,"elapsed":4003,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}}},"source":["# These datapoints are for the final submision\n","final_submission_start: str=\"2018-06-30\"\n","final_submission_end: str=\"2018-07-13\""],"execution_count":6,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"P01Jd3yw9GXT"},"source":["#### Imports\n","\n","Here you can add imports u need. Please use alphabetical order to not import new stuff twice."]},{"cell_type":"code","metadata":{"colab_type":"code","id":"vWcDzESD8HZ6","colab":{},"executionInfo":{"status":"ok","timestamp":1593370522842,"user_tz":-120,"elapsed":4725,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}}},"source":["from datetime import date, datetime, timedelta\n","import hyperopt\n","from hyperopt import fmin, tpe, hp, STATUS_OK, Trials\n","from itertools import product\n","import numpy as np\n","from math import sqrt \n","import matplotlib.pyplot as plt\n","import pandas as pd\n","import pickle\n","from sklearn import metrics\n","from sklearn.model_selection import train_test_split as tts\n","from sklearn.preprocessing import StandardScaler\n","from typing import List, Union, Tuple\n","\n","import xgboost as xgb \n","from xgboost import plot_importance \n","\n","%matplotlib inline"],"execution_count":7,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"cbSkLqn3eAmr"},"source":["In case of a \"Model not found\" Error concerning the Hyperopt package. Please uncomment and run the following line"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"4M6kXwVftN0o","colab":{},"executionInfo":{"status":"ok","timestamp":1593370522847,"user_tz":-120,"elapsed":4637,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}}},"source":["# !pip install hyperopt"],"execution_count":8,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"eaMSZ6KdG9v4"},"source":["## Loading data\n","\n","Here we now load the dataframes by the preceeding notebooks.\n","Please do not import single datasets and merge them here cause it will end in fatal inconsistencies.\n","\n","For comfort the following row will define a ```small_data``` attribute. is this True, only a part will be used for training, if False all the data points will be used in the following code.\n","\n","It will leave the testing and validation data in the same size, only will decrease the data for training.\n"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"IUqwkmndHxFl","colab":{},"executionInfo":{"status":"ok","timestamp":1593370522849,"user_tz":-120,"elapsed":4592,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}}},"source":["small_data: bool = True"],"execution_count":9,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"4AK5qsidKg3J"},"source":["In case small data should be used, the following line defines the start of this. "]},{"cell_type":"code","metadata":{"colab_type":"code","id":"tP16QRO8Kfud","colab":{},"executionInfo":{"status":"ok","timestamp":1593370522868,"user_tz":-120,"elapsed":4507,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}}},"source":["small_data_start_date: str = '20180301'"],"execution_count":10,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"cYJyv2lMIXEk"},"source":["For the training of XGBoost, we need to specify a validation set. Herefore we no can set the traning_end. Between this and the start of testing will be used as validation set. \n"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"SLEapT_nIbqz","colab":{},"executionInfo":{"status":"ok","timestamp":1593370522872,"user_tz":-120,"elapsed":4445,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}}},"source":["training_end: str = '20180519'"],"execution_count":11,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"1q8DC7gOOeob"},"source":["### Label\n","\n","The last thing which need to be specified is, which label should be used for the training, so what do u want to be predicted. Sadly because of xg boos we only can use exactly one label :( "]},{"cell_type":"code","metadata":{"colab_type":"code","id":"sLjArlOzOeKv","colab":{},"executionInfo":{"status":"ok","timestamp":1593370522873,"user_tz":-120,"elapsed":4418,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}}},"source":["labels: List[str] = [\"order\"]"],"execution_count":12,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"05O1lQn9OuLN"},"source":["Therefore this assert. Remove it if we change to another model"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"J9pzeNYlOyOw","colab":{},"executionInfo":{"status":"ok","timestamp":1593370522888,"user_tz":-120,"elapsed":4381,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}}},"source":["assert len(labels) == 1"],"execution_count":13,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"O4YTmnAvPTK3"},"source":["## Features to delete for training\n","\n","Turns out another parameter needs to be set. Which feature do we have in the data and should be deleted for training. The date will be deleted seperately"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"WT38nzS6Pglf","colab":{},"executionInfo":{"status":"ok","timestamp":1593370522889,"user_tz":-120,"elapsed":4355,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}}},"source":["# added later with all the lag features \n","# columns_to_delete: List[str] = []"],"execution_count":14,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"58iRvhQpH5N-"},"source":["# 4. Preparation\n","Here the data is finally fetched"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"sSC4rEXDSoqr","colab":{},"executionInfo":{"status":"ok","timestamp":1593370524568,"user_tz":-120,"elapsed":6010,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}}},"source":["# OPEN PANDAS DATAFRAMES FROM PREPROCESSING 2\n","try:\n","  if make_final_submission:\n","    dynamic_features = pd.read_pickle('./data/dynamic_features_final_nb2.pk')\n","    static_features = pd.read_pickle('./data/static_features_final_nb2.pk')\n","  else:\n","    dynamic_features = pd.read_pickle('./data/dynamic_features_nb2.pk')\n","    # dynamic_features = pd.read_pickle('./data/dynamic_features_nb2.pk_no_one_and_ws3')\n","    static_features = pd.read_pickle('./data/static_features_nb2.pk')\n","    # static_features = pd.read_pickle('./data/static_features_nb2.pk_no_one_and_ws3')\n","    labels_for_test = pd.read_pickle('./data/orders_for_test.pkl')\n","except FileNotFoundError as e:\n","  print(f\"One File could not be found. Please rerun the first Notebook, {e}\")"],"execution_count":15,"outputs":[]},{"cell_type":"code","metadata":{"id":"K7p9XKNsAmsM","colab_type":"code","colab":{},"executionInfo":{"status":"ok","timestamp":1593370524609,"user_tz":-120,"elapsed":6018,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}}},"source":["for i in range(dynamic_features.shape[1]):\n","  col_name = dynamic_features.columns[i]\n","  if dynamic_features[col_name].dtype == \"uint8\":\n","    dynamic_features[col_name] = dynamic_features[col_name].astype(int) "],"execution_count":16,"outputs":[]},{"cell_type":"code","metadata":{"id":"JP7Dww8tFBXg","colab_type":"code","colab":{},"executionInfo":{"status":"ok","timestamp":1593370524615,"user_tz":-120,"elapsed":6000,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}}},"source":["# static_features['recommendedRetailPrice_bucket'] = static_features['recommendedRetailPrice_bucket'].astype(int)"],"execution_count":17,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"u7sFhrLZgaim"},"source":["Combine the two datasets together"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"x02Z19f2gaEr","colab":{},"executionInfo":{"status":"ok","timestamp":1593370529047,"user_tz":-120,"elapsed":10406,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}}},"source":["data = dynamic_features.join(static_features.drop('avg_salesPrice_nonunique', axis=1), on=\"itemID\")\n","data = data.reset_index()"],"execution_count":18,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"iXfGKad5KtI5"},"source":["Here we make the dataframe a bit smaller:"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"u30_iB0yKT58","colab":{},"executionInfo":{"status":"ok","timestamp":1593370530968,"user_tz":-120,"elapsed":12295,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}}},"source":["if small_data: \n","  data = data[data['date'] > small_data_start_date]"],"execution_count":19,"outputs":[]},{"cell_type":"code","metadata":{"id":"4N8teBOZtzRS","colab_type":"code","colab":{},"executionInfo":{"status":"ok","timestamp":1593370530971,"user_tz":-120,"elapsed":12275,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}}},"source":["columns_to_delete = []\n","\n","for i in range(data.shape[1]):\n","  if \"order_t-\" in data.columns[i]:\n","    columns_to_delete.append(data.columns[i])"],"execution_count":20,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"jVrBGgpIkrMy"},"source":["#### Deletion of unused feature\n"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"Fc79L0Zskqxw","colab":{},"executionInfo":{"status":"ok","timestamp":1593370531554,"user_tz":-120,"elapsed":12836,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}}},"source":["try:\n","  # do the deletion\n","  data = data.drop(columns_to_delete, axis=1)\n","except KeyError as e:\n","  print(f\"I could not delete the columns {e} cause they were not there. \\n But hey, they are gone, like intented :D\")"],"execution_count":21,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"BkGtmL7RnaIc"},"source":["#### Adding the label\n","\n","After this step, data does not exist, only X, y left"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"kRPnmUpVnd7Z","colab":{},"executionInfo":{"status":"ok","timestamp":1593370532299,"user_tz":-120,"elapsed":13503,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}}},"source":["# We need to put date also into y to make the split\n","y = data[['date', *labels]].copy()\n","X = data.drop(labels, axis=1).copy()"],"execution_count":22,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"04RyklpNHf1P"},"source":["#### Split into train/val/test\n","\n","Here we create tuples for the respective datapoints.\n","(X,y) \n","In the end we will have train, val and test according to the variables defined above"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"jLTI2BfmFcHR","colab":{"base_uri":"https://localhost:8080/","height":68},"executionInfo":{"status":"ok","timestamp":1593370535275,"user_tz":-120,"elapsed":16256,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}},"outputId":"3a85a95b-a89a-4120-f5db-aa6a5e75c9fa"},"source":["def split(X, y, training_end, testing_start, testing_end, split_feature='date', labels_for_test=None, delete_split_feature=True):\n","  \n","  # Training\n","  training_begin = X['date'].min()\n","  print(f\"CREATING TRAINING from [{training_begin}, {training_end}]\")\n","  train_X = X[X[split_feature] <= training_end]\n","  train_y = y[y[split_feature] <= training_end]\n","\n","  # Validation\n","  print(f\"CREATING VALIDATION from ]{training_end}, {testing_start}[\")\n","  val_X = X[(X[split_feature] > training_end)]\n","  val_X = val_X[(val_X[split_feature] < testing_start)]\n","  val_y = y[(y[split_feature]> training_end)]\n","  val_y = val_y[(val_y[split_feature] < testing_start)]\n","\n","  assert len(val_X)>0, \"val_X must be greater 0\"\n","  assert len(val_y)>0, \"val_y must be greater 0\"\n","\n","   # Testing\n","  print(f\"CREATING TESTING from ({testing_start}, {testing_end}]\")\n","  test_X = X[(X[split_feature] >= testing_start) & (X[split_feature] <= testing_end)]\n","  test_y = y[(y[split_feature] >= testing_start) & (y[split_feature] <= testing_end)]\n","    \n","  assert len(test_X)>0, \"test_X must be greater 0\"\n","  assert len(test_y)>0, \"test_y must be greater 0\"\n","    \n","  if delete_split_feature:\n","    train_X = train_X.drop([split_feature,'itemID'], axis=1)\n","    train_y = train_y.drop(split_feature, axis=1)\n","    val_X = val_X.drop([split_feature,'itemID'], axis=1)\n","    val_y = val_y.drop(split_feature, axis=1)\n","    test_X = test_X.drop(split_feature, axis=1)\n","    test_y = test_y.drop(split_feature, axis=1)\n","\n","  if make_final_submission is False:\n","    labels_for_test = labels_for_test[labels_for_test.index.get_level_values(1) <= test_period_end]\n","    \n","  if labels_for_test is not None:\n","    test_y['order'] = labels_for_test.reset_index()['order'].values\n","  return (train_X, train_y), (val_X, val_y), (test_X, test_y)\n","\n","test_start = final_submission_start if make_final_submission else test_period_start\n","test_end =  final_submission_end if make_final_submission else test_period_end\n","\n","train, val, test = split(X,y, training_end, test_start, test_end, labels_for_test=labels_for_test)\n","del X\n","del y\n","#del data"],"execution_count":23,"outputs":[{"output_type":"stream","text":["CREATING TRAINING from [2018-03-02 00:00:00, 20180519]\n","CREATING VALIDATION from ]20180519, 2018-06-02[\n","CREATING TESTING from (2018-06-02, 2018-06-15]\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"edijbaQml9Yw","colab_type":"text"},"source":["### Save Test, train and val data\n","\n","Here we safe the preprocessing data.\n","\n","First the Test data:"]},{"cell_type":"code","metadata":{"id":"atVIbQIwl_5O","colab_type":"code","colab":{},"executionInfo":{"status":"ok","timestamp":1593370535277,"user_tz":-120,"elapsed":16185,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}}},"source":["# with open(f\"./data/{model_name}_test.pk\",\"wb\") as f:\n","#     pickle.dump(test, f)\n","#     f.close()"],"execution_count":24,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"Aw5H_f5Ll-v5","colab_type":"text"},"source":["Then the training data"]},{"cell_type":"code","metadata":{"id":"NqwqF1_PmEke","colab_type":"code","colab":{},"executionInfo":{"status":"ok","timestamp":1593370535278,"user_tz":-120,"elapsed":16146,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}}},"source":["# with open(f\"./data/{model_name}_train.pk\",\"wb\") as f:\n","#     pickle.dump(train, f)\n","#     f.close()"],"execution_count":25,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"-g-u9eM6mJtF","colab_type":"text"},"source":["At last the validation data"]},{"cell_type":"code","metadata":{"id":"hLRHgQXImHUp","colab_type":"code","colab":{},"executionInfo":{"status":"ok","timestamp":1593370535279,"user_tz":-120,"elapsed":16130,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}}},"source":["# with open(f\"./data/{model_name}_val.pk\",\"wb\") as f:\n","#     pickle.dump(val, f)\n","#     f.close()"],"execution_count":26,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"JHb2fdTZmjQ-","colab_type":"text"},"source":["# The Model\n","\n","### Load the data\n","Here we load the data from above step. __Why?__ This is to make sure that also one of the three main parts of the notebook can be run by themself :) In case all of the notebook is run, this part can be ignored. "]},{"cell_type":"code","metadata":{"id":"CIXjBUA-mi59","colab_type":"code","colab":{},"executionInfo":{"status":"ok","timestamp":1593370535280,"user_tz":-120,"elapsed":16105,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}}},"source":["# with open(f\"./data/{model_name}_train.pk\",\"rb\") as f:\n","#     train = pickle.load(f)\n","#     f.close()\n","\n","# with open(f\"./data/{model_name}_test.pk\",\"rb\") as f:\n","#     test = pickle.load(f)\n","#     f.close()\n","    \n","# with open(f\"./data/{model_name}_val.pk\",\"rb\") as f:\n","#     val = pickle.load(f)\n","#     f.close()   "],"execution_count":27,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"BNF7KlOMnQ07"},"source":["## The Custom Evaluation and Objective Function\n","\n","Here is space to create several custom evaluation and objective functions to later use in XGBoost."]},{"cell_type":"code","metadata":{"colab_type":"code","id":"JFIbtar6s8uR","colab":{},"executionInfo":{"status":"ok","timestamp":1593370535281,"user_tz":-120,"elapsed":16078,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}}},"source":["use_constant_price = False\n","\n","def monetary_value_objective(predt: np.ndarray, dtrain: Union[xgb.DMatrix, np.ndarray]) -> Tuple[np.ndarray, np.ndarray]:\n","  \"\"\"\n","  Loss is the difference to the optimal value (how much more would we get, if we \n","  predicted correctly). Minimal loss 0 is reached when the prediction matches \n","  the true value exactly.\n","\n","  Squared version (for nicer gradients).\n","\n","  predt = model prediction\n","  dtrain = labels and sample weights\n","  Currently, dtrain is a numpy array.\n","  \"\"\"\n","  item_price = 1\n","\n","  # Use the following if dtrain is in xgb.DMatrix format.\n","  # y = dtrain.get_label()\n","  y = dtrain\n","\n","  mask1 = predt <= y  # Predict too few\n","  mask2 = predt > y  # Predict too much\n","\n","  if use_constant_price or train[0][\"salesPrice\"].shape[0] != predt.shape[0]:\n","    if not use_constant_price:\n","      print(\"`amount` is not None. Using constant item price.\")\n","    item_price = 1\n","    item_price_1 = 1\n","    item_price_2 = 1\n","  else:\n","    # The following assertion fails if amount not None\n","    assert train[0][\"salesPrice\"].shape[0] == predt.shape[0]\n","    item_price = train[0][\"salesPrice\"]\n","    item_price_1 = train[0][\"salesPrice\"][mask1]\n","    item_price_2 = train[0][\"salesPrice\"][mask2]\n","\n","  grad = item_price**2 * (predt - y)  \n","  # Gradient is negative if prediction is too low, and positive if it is too high\n","  # Here scale it (0.72 = 0.6**2 * 2)\n","  grad[mask1] = 2 * grad[mask1]\n","  grad[mask2] = 0.72 * grad[mask2]\n","\n","  hess = np.empty_like(grad)\n","  hess[mask1] = 2 * item_price_1**2\n","  hess[mask2] = 0.72 * item_price_2**2\n","\n","  # For some reason, gradient must be negated, but hessian not\n","  grad = -grad\n","\n","  return grad, hess"],"execution_count":28,"outputs":[]},{"cell_type":"code","metadata":{"id":"PnJJjYRmo7-V","colab_type":"code","colab":{},"executionInfo":{"status":"ok","timestamp":1593370535282,"user_tz":-120,"elapsed":16056,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}}},"source":["def monetary_value_eval(predt: np.ndarray, dtrain: xgb.DMatrix) -> Tuple[str, float]:\n","  item_price = dtrain.get_weight()\n","  y = dtrain.get_label()\n","\n","  predt = np.maximum(predt, 0)  # Remove negative values\n","\n","  mask1 = predt <= y  # Predict too few\n","  mask2 = predt > y  # Predict too much\n"," \n","  revenue = np.empty_like(predt)\n","  revenue[mask1] = item_price[mask1] * predt[mask1]\n","  revenue[mask2] = item_price[mask2] * (1.6 * y[mask2] - 0.6 * predt[mask2])\n","\n","  total_revenue = revenue.sum()\n","\n","  return \"monetary-value\", total_revenue\n","\n","monetary_value = metrics.make_scorer(monetary_value_eval)"],"execution_count":29,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"yyTupCIAUJ2D"},"source":["## XGBoost Regressor Model\n","\n","In case we just need to fast train the model to see if it works we can set this following amount variable. This will set the training data to this amount. Set it to ```None``` if all the data should be used."]},{"cell_type":"code","metadata":{"colab_type":"code","id":"8Cenu-lTQBpn","colab":{},"executionInfo":{"status":"ok","timestamp":1593370535284,"user_tz":-120,"elapsed":16036,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}}},"source":["amount: Union[int, None] = None  # 1000"],"execution_count":30,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"Puz8I-hTV2ns"},"source":["### Hyper parameters tuning "]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"rfIdORwJQi-M"},"source":["more parameters can be added here in the parameter space: \n","range can also be changed for more fine tuning "]},{"cell_type":"code","metadata":{"colab_type":"code","id":"0HjG9wmstR0l","colab":{},"executionInfo":{"status":"ok","timestamp":1593370535285,"user_tz":-120,"elapsed":16014,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}}},"source":["para_space = {\n","            'n_estimators': hp.quniform('n_estimators', 10, 20, 1),\n","            'max_depth': hp.quniform(\"max_depth\", 3, 10, 1),\n","            'min_child_weight': hp.quniform('min_child_weight', 1, 5, 1),\n","            'learning_rate':hp.quniform('learning_rate', 0.05, 0.2, 0.05),  \n","            'gamma': hp.quniform('gamma', 0.8, 1, 0.1),\n","            'subsample':hp.quniform('subsample', 0.7, 1, 0.1),\n","            'eval_metric': monetary_value,            \n","            'tree_method':'auto', # Sadly I need to put this to auto\n","            'objective': monetary_value_objective,          \n","            'seed': 7\n","        }"],"execution_count":31,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"L3ohw7LCQr21"},"source":["The following parameter explains how many rounds for hyperparameter tuning should be used:"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"W7FKcr79QzvV","colab":{},"executionInfo":{"status":"ok","timestamp":1593370535286,"user_tz":-120,"elapsed":15998,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}}},"source":["evaluation_round: int = 5"],"execution_count":32,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"jHS9JHePRVHE"},"source":["### Actual Hyperparameter training\n","This method returns a evaluation function so it creates a function which creates a model"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"3K_mWKpstUV3","colab":{},"executionInfo":{"status":"ok","timestamp":1593370535565,"user_tz":-120,"elapsed":16224,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}}},"source":["def eval_scoring(amount=None):\n","    amount = amount or len(train[0])\n","        \n","    def eval_fc(para_space):\n","        xgb_model = xgb.XGBRegressor(\n","                              n_estimators = int(para_space['n_estimators']), \n","                              max_depth = int(para_space['max_depth']),\n","                              min_child_weight=int(para_space['min_child_weight']),\n","                              learning_rate = para_space['learning_rate'],\n","                              gamma = para_space['gamma'],\n","                              subsample = para_space['subsample'],\n","                              feval = para_space['eval_metric'],\n","                              tree_method = para_space['tree_method'],\n","                              objective = para_space['objective'],\n","                              seed = para_space['seed']\n","                            )\n","        sample_weights_train = train[0][:amount].salesPrice.values\n","        sample_weights_val = val[0][:amount].salesPrice.values\n","        \n","        eval_set = [(train[0][:amount],train[1][:amount]), (val[0][:amount], val[1][:amount])]\n","        eval_set_weights = [sample_weights_train, sample_weights_val]\n","        \n","        xgb_model.fit(train[0][:amount], train[1][:amount], eval_set = eval_set,\n","                      eval_metric = monetary_value_eval, \n","                      early_stopping_rounds = 10, verbose = False,\n","                      sample_weight=sample_weights_train, sample_weight_eval_set=eval_set_weights)\n","\n","        train_pred = xgb_model.predict(train[0][:amount])\n","        train_rmse = sqrt(metrics.mean_squared_error(train[1][:amount], train_pred))\n","\n","        val_pred = xgb_model.predict(val[0][:amount])\n","        val_rmse = sqrt(metrics.mean_squared_error(val[1][:amount], val_pred))\n","\n","        # CHANGED FROM HERE\n","        val_dmatrix = xgb.DMatrix(val[0][:amount], label=val[1][:amount], weight=sample_weights_val)\n","        _, val_mon = monetary_value_eval(val_pred, val_dmatrix)\n","        \n","        print(\"\\n-----------------------------------------------\")\n","        print(f\"Train RMSE: {train_rmse} | Val RMSE: {val_rmse} | Val monetary: {val_mon}\")\n","        print(f\"Trained with parameters: \\n  {para_space}\")\n","        print(\"--------------------------------------------------\\n\")\n","        \n","        # Return negative monetary value because hyperopt will select the model with the minimal loss\n","        return {'loss': -val_mon, 'status': STATUS_OK, 'model': xgb_model }\n","\n","        # CHANGED UNTIL HERE\n","    return eval_fc"],"execution_count":33,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"6v9_jcp4RpKi"},"source":["The actual hyperparameter tuning will be done now:"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"zythyo63tW2g","colab":{},"executionInfo":{"status":"ok","timestamp":1593370535568,"user_tz":-120,"elapsed":16196,"user":{"displayName":"Min Wu","photoUrl":"","userId":"11552317141468364459"}}},"source":["def hyperpara_tuning(para_space):\n","    \"\"\"\n","    Returns the best set of hyperparamters, \n","    given a parameter space and a evaluation function \n","    \"\"\"\n","    trials = Trials()\n","    best_model = fmin(fn=eval_scoring(amount=amount),\n","                      space=para_space,\n","                      algo=tpe.suggest,\n","                      max_evals=evaluation_round,\n","                      trials=trials\n","                     )\n","    \n","    return trials, best_model"],"execution_count":34,"outputs":[]},{"cell_type":"code","metadata":{"id":"F_pte4AvRnVD","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"outputId":"0aa40010-f1e4-400b-ed46-2a36ec99eb8e"},"source":["# set runtime environment to GPU at: Runtime -> Change runtime type \n","trials, best_hyperparams = hyperpara_tuning(para_space)\n","final_xgb_model = trials.best_trial['result']['model']\n","assert final_xgb_model is not None, \"Oooops there is no model created :O \"\n","\n","print(\"\\n\\n\\n############################################\\n\")\n","print(\"THE BEST PARAMETER WHERE:\")\n","print(best_hyperparams)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["\r  0%|          | 0/5 [00:00<?, ?it/s, best loss: ?]"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"pMhuAPxQWF2L"},"source":["### final xgb model based on Grid Search\n","\n","\n","Here we can train the model again based on the best hyperparameters. But this is only in some times useful and therefore commented out."]},{"cell_type":"code","metadata":{"colab_type":"code","id":"XxekGTg3FTgK","colab":{}},"source":["# def best_xgb_model(x_train, y_train, x_val, y_val, best_hyperparams):\n","#   xgb_model = xgb.XGBRegressor(n_estimators = int(best_hyperparams['n_estimators']), \n","#                                learning_rate = best_hyperparams['learning_rate'], \n","#                                gamma = best_hyperparams['gamma'],\n","#                                max_depth = int(best_hyperparams['max_depth']), \n","#                                min_child_weight = int(best_hyperparams['min_child_weight']),\n","#                                subsample = best_hyperparams['subsample'],\n","#                                tree_metric = 'gpu_hist', \n","#                                objective = monetary_value_objective,\n","#                                feval = monetary_value,\n","#                                seed = 7)\n","\n","#   xgb_model.fit(x_train, y_train, verbose = True, eval_metric = monetary_value_eval, #'rmse',\n","#                   eval_set = [(x_train, y_train), (x_val, y_val)])\n","    \n","#   return xgb_model"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab_type":"code","id":"-BX8s7Y1FTgP","colab":{}},"source":["# final_xgb_model = best_xgb_model(train[0], train[1], val[0], val[1], best_hyperparams)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"fPpgzPheWOIJ"},"source":["### Model Results \n","Here we can have a analytic look on the trained model. Was it good ? Did it make sense? Hungry for answers? Then take a look at following crazy lines of code"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"J1JNwriXFTgV","colab":{}},"source":["train_predictions = final_xgb_model.predict(train[0])\n","train_predictions = [round(i) for i in train_predictions]\n","print(\"Train RMSE : %.8g\" % sqrt(metrics.mean_squared_error(train[1], train_predictions)))\n","\n","val_predictions = final_xgb_model.predict(val[0])\n","val_predictions = [round(i) for i in val_predictions]\n","print(\"Val RMSE : %.8g\" % sqrt(metrics.mean_squared_error(val[1], val_predictions)))"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab_type":"code","id":"wEtx2KLyTrEm","colab":{}},"source":["results = final_xgb_model.evals_result()\n","epochs = len(results['validation_0']['rmse'])\n","x_axis = range(0, epochs)\n","fig, ax = plt.subplots()\n","ax.plot(x_axis, results['validation_0']['rmse'], label='Train')\n","ax.plot(x_axis, results['validation_1']['rmse'], label='Val')\n","ax.legend()\n","plt.ylabel('rmse')\n","plt.title('XGBoost Regression Error')\n","plt.show()"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"9cpTdj7wppQ4","colab_type":"code","colab":{}},"source":["fig, ax = plt.subplots()\n","ax.plot(x_axis, results['validation_0']['monetary-value'], label='Train')\n","ax.plot(x_axis, results['validation_1']['monetary-value'], label='Val')\n","ax.legend()\n","plt.ylabel('monetary value')\n","plt.title('Monetary value')\n","plt.show()"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab_type":"code","id":"fP-v-V2yFTge","colab":{}},"source":["ax = plot_importance(final_xgb_model)\n","fig = ax.figure\n","fig.set_size_inches(54, 42)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"YiZwmgXfm5Oz"},"source":["### Save the model\n","\n","Here we now save the model. To distinguish between used models, we can use suffixes. A suffix will then be used to find the right model. If the following variable equals None, simply the current date will be taken as suffix. The final file then is called {modelname}_{suffix}."]},{"cell_type":"code","metadata":{"colab_type":"code","id":"XxyEdzfgnv4p","colab":{}},"source":["suffix: str= '28062020'"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"-bDcPWednnxC","colab_type":"text"},"source":["---"]},{"cell_type":"code","metadata":{"id":"Mevn2BE1nkJf","colab_type":"code","colab":{}},"source":["suffix = suffix if suffix else datetime.now().strftime('%m%d%Y')\n","with open(f\"./models/{model_name}_{suffix}.pk\",\"wb\") as f:\n","    pickle.dump(final_xgb_model, f)\n","    f.close()"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"yucX_Bn5nwzy","colab_type":"text"},"source":["# Prediction\n","\n","This part is used to save the model, and create the predictions already. In case we run the whole notebook, no changes need to be made. \n","In case **only this part** should be executed, we need to input the name of the model.  Because we know already the name of the model (see upthere somewhere) we only need the suffix (Most probably the date of training).\n"]},{"cell_type":"code","metadata":{"id":"ZNsrVDsBnq43","colab_type":"code","colab":{}},"source":["# suffix_for_import = None"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"K5urqdJknzza","colab_type":"text"},"source":["### Load Model"]},{"cell_type":"code","metadata":{"id":"qJlN_EhUn1Mn","colab_type":"code","colab":{}},"source":["# if final_xgb_model is None:\n","#     suffix_for_import = suffix_for_import if suffix_for_import else datetime.now().strftime('%m%d%Y')\n","#     with open(f\"./models/{model_name}_{suffix_for_import}.pk\",\"rb\") as f:\n","#         final_xgb_model = pickle.load(f)\n","#         f.close()\n","#     print(\"Loaded model\")\n","# else:\n","#     print(\"Use the already created model\")"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"0DQTm72Un2lM","colab_type":"code","colab":{}},"source":["# with open(f\"./data/{model_name}_test.pk\",\"rb\") as f:\n","#     test = pickle.load(f)\n","#     f.close()"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"SlavyOmen6lE","colab_type":"text"},"source":["Here we also load the data again"]},{"cell_type":"code","metadata":{"id":"C10lzCMHn4Hh","colab_type":"code","colab":{}},"source":["# with open(f\"./data/{model_name}_train.pk\",\"rb\") as f:\n","#     train = pickle.load(f)\n","#     f.close()\n","\n","# with open(f\"./data/{model_name}_test.pk\",\"rb\") as f:\n","#     test = pickle.load(f)\n","#     f.close()\n","    \n","# with open(f\"./data/{model_name}_val.pk\",\"rb\") as f:\n","#     val = pickle.load(f)\n","#     f.close()   \n","\n","# labels_for_test = pd.read_pickle('./data/orders_for_test.pkl')"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"Ww6f9kY6n944","colab_type":"text"},"source":["## Non-recursive Prediction\n"]},{"cell_type":"code","metadata":{"id":"4jXRXg4jFv65","colab_type":"code","colab":{}},"source":["info = pd.read_csv('/content/gdrive/My Drive/Data_Mining_Cup/02 Data/DMC20_Data/infos.csv', delimiter='|', index_col=\"itemID\")\n","\n","# creating the sample_submission dataframe (with predition, true_demand, and simulationPrice -> to generate total revenue directly in this notebook)\n","\n","# add prediction \n","test_predictions = final_xgb_model.predict(test[0].drop('itemID', axis = 1))\n","test[0]['demandPrediction'] = pd.Series(test_predictions, index = test[0].index)\n","test[0]['demandPrediction'] = test[0]['demandPrediction'].round().astype(int) # round first, then sum \n","sample_submission = test[0][['itemID','demandPrediction']].groupby('itemID').sum()\n","\n","# add target \n","test[0]['true_demand'] = test[1]\n","sample_submission['true_demand'] = test[0][['itemID','true_demand']].groupby('itemID').sum()\n","\n","# add simulation price \n","sample_submission['simulationPrice'] = info['simulationPrice']"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"kIsBqmGWbQYC","colab_type":"code","colab":{}},"source":["# calculate total revenue:\n","\n","mask1 = sample_submission['demandPrediction'] <= sample_submission['true_demand']  # Predict too few\n","mask2 = sample_submission['demandPrediction'] > sample_submission['true_demand']  # Predict too much\n","\n","revenue = np.empty_like(sample_submission['demandPrediction'])\n","revenue[mask1] = sample_submission['simulationPrice'][mask1] * sample_submission['demandPrediction'][mask1]\n","revenue[mask2] = sample_submission['simulationPrice'][mask2] * (1.6 * sample_submission['true_demand'][mask2] - 0.6 * sample_submission['demandPrediction'][mask2])\n","\n","total_revenue = revenue.sum()\n","\n","total_revenue"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"VHjtak3stmKi","colab_type":"code","colab":{}},"source":["sample_submission"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"pBHeSdDV9t5D","colab_type":"text"},"source":["### Safe the sample submission"]},{"cell_type":"code","metadata":{"id":"qhQl8ZPt9trF","colab_type":"code","colab":{}},"source":["sample_submission['demandPrediction'].to_csv(f'./model_submissions/submission_non_recursive_28062020.csv', sep='|')"],"execution_count":null,"outputs":[]}]}