Fastai PetFinder.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Merge image, structured and text data in the same neural net with fast.ai\n",
    "\n",
    "In this notebook we will predict the adoption speed of pets in the [PetFinder Kaggle competition](https://www.kaggle.com/c/petfinder-adoption-prediction/).  This competition give access to tree kind of data, **image** of the pets, **structured** data like their age, breed, color etc and finally **text** data in the form of a description of the pet.\n",
    "\n",
    "It would be very interesting to be able to merge all this data inside the same neural network so that the network can use whatever information from all data to actually predictic how fast a pet is going to get adopted.\n",
    "\n",
    "Keep in mind that **this is my first Kaggle competition**, so I might not be using the best strategies or validation schemes, but I just wanted to explore this idea of merging different type of data inside the same neural network.\n",
    "\n",
    "## Fast.ai\n",
    "We are going to use fast.ai to do that because it offers a lot of stuff we need to do this.  Mainly a very intuitive [data block](https://docs.fast.ai/data_block.html) that we will use to get our various data from disk, line them up and pass them as input to our neural network.  It also provide with easily accessible pre-trained models we will be able to use for our tasks.\n",
    "\n",
    "## Leveraging pre-trained models\n",
    "![caption](Diagram.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import json\n",
    "import os\n",
    "import feather\n",
    "from fastai.text import *\n",
    "\n",
    "from petfinder.data import *\n",
    "\n",
    "%matplotlib inline\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Get the structured data\n",
    "The method get_data contains all the data wrangling rather boring stuff.  We open the structured data train.csv where we have information for each pet (identified by a PetID).  We have information like the age of the pet, the breed, the color, was it vaccinated, a textual description of the pet etc.  The PetFinder competition also ran the description inside the google sentiment analysis service and provided us with that.  I use some of this information and create some new columns for that too.\n",
    "\n",
    "We also find images in the train_images folder.  We create a dataframe where we have a row containing the PetID of the image and the path on disk of the image.  We then merge this dataframe to the main structured data by PetID.  This yield a dataframe with one row per image where all the structured information about the pet is there for each row.\n",
    "\n",
    "Kaggle also provided some metadata for each pet, but I didn't spend the time parsing those files...\n",
    "\n",
    "We have to predict between 5 AdoptionSpeed.  This is a classification problem, but a lot of people in the competition used a regression and then found the best rounding using the class OptimizedRounder at the of this notebook.  I tried using multi-class classification with this model but didn't have good results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = 'C:\\\\work\\\\ML\\\\PetFinder\\\\'\n",
    "bs=64\n",
    "\n",
    "pets = get_data(isTest=False)\n",
    "petsTest = get_data(isTest=True)\n",
    "\n",
    "pets.AdoptionSpeed = pets.AdoptionSpeed.astype(float)\n",
    "\n",
    "petsTest['AdoptionSpeed'] = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Language Model\n",
    "\n",
    "See the notebook *PetFinder Language Model* on how we train and fine tune a text language model on the pet description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Structured data\n",
    "\n",
    "Here we have some decisions to make for our structured variables.  We need to decide which one is going to be a categorical variable and which one is going to be contiuous.\n",
    "\n",
    "Even if a variable is a number doesnt mean it should be continuous variable.  If the variable only contains a small amount of unique values, it might be better to model it as a categorical variable.  We can use [embeddings](https://www.fast.ai/2018/04/29/categorical-embeddings/) for categorical data which will allow us to learn a far richer representation for them and is sometimes more powerful than using a continuous variable.\n",
    "\n",
    "Fastai takes care of defining those embeddings size, it also fill missing values and normalize the structured data for the neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.tabular import *\n",
    "from fastai.vision import *\n",
    "from fastai.metrics import *\n",
    "from fastai.text import *\n",
    "\n",
    "dep_var = 'AdoptionSpeed'\n",
    "cont_names, cat_names = cont_cat_split(pets, dep_var=dep_var, max_card=10)\n",
    "procs = [FillMissing, Categorify, Normalize]\n",
    "cat_names.remove('Filename')\n",
    "cat_names.remove('PicturePath')\n",
    "cat_names.remove('PetID')\n",
    "cat_names.remove('Description')\n",
    "\n",
    "# for name in cont_names:\n",
    "#     pets[name] = np.log(pets[name] - pets[name].min() + 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(['Age',\n",
       "  'Quantity',\n",
       "  'Fee',\n",
       "  'VideoAmt',\n",
       "  'PhotoAmt',\n",
       "  'RescuerDogCount',\n",
       "  'AvgSentenceSentimentMagnitude',\n",
       "  'AvgSentenceSentimentScore',\n",
       "  'SentimentMagnitude',\n",
       "  'SentimentScore',\n",
       "  'state_gdp',\n",
       "  'state_population',\n",
       "  'gdp_vs_population'],\n",
       " ['Type',\n",
       "  'Name',\n",
       "  'Breed1',\n",
       "  'Breed2',\n",
       "  'Gender',\n",
       "  'Color1',\n",
       "  'Color2',\n",
       "  'Color3',\n",
       "  'MaturitySize',\n",
       "  'FurLength',\n",
       "  'Vaccinated',\n",
       "  'Dewormed',\n",
       "  'Sterilized',\n",
       "  'Health',\n",
       "  'State',\n",
       "  'RescuerID',\n",
       "  'NoImage',\n",
       "  'NoDescription'])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cont_names, cat_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from petfinder.model import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loading and lining up the data\n",
    "\n",
    "We want to load our data.  Ideally we would like to re-use existing functionnality and not have to write custom data loader.  fast.ai got us covered, thanks to the amazing [data block api](https://docs.fast.ai/data_block.html)!\n",
    "\n",
    "First we need to am ItemList per type of data.  One for image, structured and text.  Each of them do pre-processing to the input, keep track of processing they do on data like normalization etc.\n",
    "\n",
    "But then we merge them using a MixedItemList.  MixedItemList simply get an item from each ItemList it contains and merge them together into one Item.  Then when fast.ai pass data to our model in the forward method, we can expect as many input as we have ItemList in our MixedItemList.\n",
    "\n",
    "I pickle the MixedItemList to avoid having to recompute it when I reload the notebook because some of the ItemList pre-processing can be long (like TextItemList)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "byPetID = pets.groupby('PetID').size().reset_index()\n",
    "byPetID = byPetID.sample(frac=.1, random_state=42).drop([0], axis=1)\n",
    "byPetID['IsValidation'] = True\n",
    "pets = pd.merge(pets, byPetID, how='left', on='PetID')\n",
    "pets.IsValidation = pets.IsValidation.fillna(False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.callbacks import *\n",
    "\n",
    "bs = 32\n",
    "size = 224\n",
    "np.random.seed(42)\n",
    "\n",
    "data_lm = load_data(path, 'data_lm_descriptions.pkl', bs=bs)\n",
    "vocab = data_lm.vocab\n",
    "\n",
    "imgList = ImageList.from_df(pets, path=path, cols='PicturePath')\n",
    "tabList = TabularList.from_df(pets, cat_names=cat_names, cont_names=cont_names, procs=procs, path=path)\n",
    "textList = TextList.from_df(pets, cols='Description', path=path, vocab=vocab)\n",
    "\n",
    "if os.path.isfile(path + 'mixed_img_tab_text.pkl') != True :\n",
    "    mixed = (MixedItemList([imgList, tabList, textList], path, inner_df=tabList.inner_df)\n",
    "            .split_from_df(col='IsValidation')\n",
    "            .label_from_df(cols='AdoptionSpeed', label_cls=FloatList)\n",
    "            .transform([[get_transforms()[0], [], []], [get_transforms()[1], [], []]], size=size))\n",
    "\n",
    "    outfile = open(path + 'mixed_img_tab_text.pkl', 'wb')\n",
    "    pickle.dump(mixed, outfile)\n",
    "    outfile.close()\n",
    "else:\n",
    "    infile = open(path + 'mixed_img_tab_text.pkl','rb')\n",
    "    mixed = pickle.load(infile)\n",
    "    infile.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This makes a text databunch used later on to create our learner (for the text portion of our learner).  We need this to construct a pre-trained RNN for classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "if os.path.isfile(path + 'text-classification-databunch.pkl'):\n",
    "    data_text = load_data(path, 'text-classification-databunch.pkl')\n",
    "else:\n",
    "    petsAll = pd.concat([pets, petsTest])\n",
    "    petsAll = petsAll.dropna(subset=['Description'])\n",
    "    \n",
    "    data_text = (TextList.from_df(petsAll, cols='Description', path=path, vocab=vocab)).split_none().label_from_df(cols='AdoptionSpeed').databunch(bs=bs)\n",
    "    data_text.save('text-classification-databunch.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Special functions\n",
    "Neural network frameworks like to process data in batches.  Batches have to have a pre-defined size.  In our case we are using image and structured data which should always have the same size, but our text data can vary in size.  The description for each pet will be different.\n",
    "\n",
    "We have to modify some function in fastai to make it work with our inputs.  First since we are using a pre-trained resnet34 network for our images, we need to normalize our images using statistics from ImageNet.  But the normalize method for images from fastai expects a certain tensor shape.  We need to create a custom normalize function to take into account our custom tensor shape.\n",
    "\n",
    "Each row in our batch will contain an array of stuff, first the image data, then the structured data and last the text data.\n",
    "\n",
    "``` python\n",
    "\n",
    "def _normalize_images_batch(b:Tuple[Tensor,Tensor], mean:FloatTensor, std:FloatTensor)->Tuple[Tensor,Tensor]:\n",
    "    \"`b` = `x`,`y` - normalize `x` array of imgs and `do_y` optionally `y`.\"\n",
    "    x,y = b\n",
    "    mean,std = mean.to(x[0].device),std.to(x[0].device)\n",
    "    x[0] = normalize(x[0],mean,std)\n",
    "    return x,y\n",
    "\n",
    "def normalize_custom_funcs(mean:FloatTensor, std:FloatTensor, do_x:bool=True, do_y:bool=False)->Tuple[Callable,Callable]:\n",
    "    \"Create normalize/denormalize func using `mean` and `std`, can specify `do_y` and `device`.\"\n",
    "    mean,std = tensor(mean),tensor(std)\n",
    "    return (partial(_normalize_images_batch, mean=mean, std=std),\n",
    "            partial(denormalize, mean=mean, std=std))\n",
    "```\n",
    "\n",
    "**collate_mixed** is the method responsible to take a batch with variable size rows (because of the variable Description text size) and make them all of equal length so that we can have uniform batch sizes.  We basically find the row in the batch which have to longest text, take its length and make all other rows the same length by padding them with zeroes at the end.\n",
    "\n",
    "``` python\n",
    "def collate_mixed(samples, pad_idx:int=0):\n",
    "    # Find max length of the text from the MixedItemList\n",
    "    max_len = max([len(s[0].data[2]) for s in samples])\n",
    "\n",
    "    for s in samples:\n",
    "        res = np.zeros(max_len + pad_idx, dtype=np.int64)\n",
    "        res[:len(s[0].data[2])] = s[0].data[2]\n",
    "        s[0].data[2] = res\n",
    "\n",
    "    return data_collate(samples)\n",
    "```\n",
    "\n",
    "Then we transform our MixedItemList into a databunch with our collate function for equal size batches and we also normalize the images using our custom normalize function from earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = mixed.databunch(bs=bs, collate_fn=collate_mixed)\n",
    "\n",
    "norm, denorm = normalize_custom_funcs(*imagenet_stats)\n",
    "data.add_tfm(norm) # normalize images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When fastai process your structured data, it creates new columns for any columns that had NaN values.  This new column is True when the other column was NaN, otherwise false.  If you want to use those columns, simply uncomment the next cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# cat_names = mixed.train.x.item_lists[1].cat_names\n",
    "# cont_names = mixed.train.x.item_lists[1].cont_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Custom model\n",
    "Here is the custom PyTorch model I created.  It expects a list of embeddings size for each categorical variable (emb_szs), the number of continuous variable (n_cont), the size of the text vocabulary for the language model and finally we have our pre-trained language model encoder that gets passed (encoder).\n",
    "\n",
    "**self.cnn** is responsible for the image data.  Notice the we use AdaptiveConcatPool2d to be able to have any image size as input.\n",
    "\n",
    "**self.lm_encoder** is responsible for the text data.  It uses our fine-tuned language model encoder we trained in the notebook PetFinder Language Model.\n",
    "\n",
    "**self.tab** is responsible for the structured data.  It will create embeddings for categorical variables.\n",
    "\n",
    "**self.reduce** is simply to reduce the size of the output of the cnn to a more manageable size.\n",
    "\n",
    "Once the data is passed through each specialist network (cnn, encoder and tabular), we concatenate their output into a single vector.\n",
    "\n",
    "**self.merge and self.final** are then responsible to reduce this concatenated vector to the final size of 5 which is the number of possible AdoptionSpeed we want to predict.  AdoptionSpeed is a categorical variable with 5 unique values.\n",
    "\n",
    "**use_trainer** is set to true if we are using RNNTrainer\n",
    "\n",
    "The **reset** method is used to reset the internal state of the RNN in self.lm_encoder.\n",
    "\n",
    "We are outputing one output for regression and forcing it in the range 0-4.\n",
    "\n",
    "``` python\n",
    "class ImageTabularTextModel(nn.Module):\n",
    "    def __init__(self, emb_szs:ListSizes, n_cont:int, vocab_sz:int, encoder, use_trainer):\n",
    "        super().__init__()\n",
    "        self.use_trainer = use_trainer\n",
    "        self.cnn = create_body(models.resnet34)\n",
    "        nf = num_features_model(self.cnn) * 2\n",
    "        drop = .5\n",
    "\n",
    "        self.lm_encoder = SequentialRNN(encoder[0], PoolingLinearClassifier([400 * 3] + [32], [.4]))\n",
    "\n",
    "        self.tab = TabularModel(emb_szs, n_cont, 128, [512, 256])\n",
    "\n",
    "        self.reduce = nn.Sequential(*([AdaptiveConcatPool2d(), Flatten()] + bn_drop_lin(nf, 512, bn=True, p=drop, actn=nn.ReLU(inplace=True))))\n",
    "        self.merge = nn.Sequential(*bn_drop_lin(512 + 128 + 32, 128, bn=True, p=drop, actn=nn.ReLU(inplace=True)))\n",
    "        self.final = nn.Sequential(*bn_drop_lin(128, 1, bn=False, p=0., actn=None))\n",
    "\n",
    "    def forward(self, img:Tensor, x:Tensor, text:Tensor) -> Tensor:\n",
    "        imgCnn = self.cnn(img)\n",
    "        imgLatent = self.reduce(imgCnn)\n",
    "        tabLatent = self.tab(x[0], x[1])\n",
    "        textLatent = self.lm_encoder(text)\n",
    "\n",
    "        cat = torch.cat([imgLatent, F.relu(tabLatent), F.relu(textLatent[0])], dim=1)\n",
    "\n",
    "        pred = self.final(self.merge(cat))\n",
    "        pred = torch.sigmoid(pred) * 4 # making sure this is in the range 0-4\n",
    "\n",
    "        if(not self.use_trainer):\n",
    "            return pred\n",
    "        else:\n",
    "            return pred, textLatent\n",
    "    \n",
    "    def reset(self):\n",
    "        for c in self.children():\n",
    "            if hasattr(c, 'reset'): c.reset()\n",
    "```\n",
    "\n",
    "# Custom learner functions\n",
    "\n",
    "We need a split_layer function to tell fastai how to split the layers when doing [discriminative learning rates](https://towardsdatascience.com/understanding-learning-rates-and-how-it-improves-performance-in-deep-learning-d0d4059c1c10).  This is also what determines which layer to freeze when when we call the Learner.freeze method.  This one could certainly be better...  Looking at other split layers for the pre-trained RNN and reset, we should probably structure this differently.\n",
    "\n",
    "``` python\n",
    "def split_layers(model:nn.Module) -> List[nn.Module]:\n",
    "    groups = [[model.cnn, model.lm_encoder]]\n",
    "    groups += [[model.tab, model.reduce, model.merge, model.final]]\n",
    "    return groups\n",
    "```\n",
    "\n",
    "We create our custom Learner class to be able to set some custom parameters.  I added an option to use RNNTrainer which is supposed to help if the language model is overfitting.  It is based on the [AWD_LSTM paper](https://arxiv.org/abs/1708.02182).  I had to modify the default version because of how I was passing data to it.\n",
    "\n",
    "``` python\n",
    "class RNNTrainerCustom(RNNTrainer):\n",
    "    def on_loss_begin(self, last_output:Tuple[Tensor,Tensor,Tensor], **kwargs):\n",
    "        \"Save the extra outputs for later and only returns the true output.\"\n",
    "        self.raw_out,self.out = last_output[1][1],last_output[1][2]\n",
    "        return {'last_output': last_output[0]}\n",
    "\n",
    "class ImageTabularTextLearner(Learner):\n",
    "    def __init__(self, data:DataBunch, model:nn.Module, use_trainer:bool=False, alpha:float=2., beta:float=1., **learn_kwargs):\n",
    "        super().__init__(data, model, **learn_kwargs)\n",
    "        if(use_trainer):\n",
    "            self.callbacks.append(RNNTrainerCustom(self, alpha=alpha, beta=beta))\n",
    "        self.split(split_layers)\n",
    "```\n",
    "\n",
    "Finally an helper method constructing our model and learner.  We use the text_classifier_learner method from fastai to construct a pre-trained language model where we load our fine-tuned encoder.  This method returns a learner though, but we only care about the model it returns which we use in our own model.\n",
    "\n",
    "The metric this Kaggle competition [evaluate on the quadratic weighted kappa](https://www.kaggle.com/c/petfinder-adoption-prediction/overview/evaluation).  So we will track it to see how we are doing.\n",
    "\n",
    "``` python\n",
    "def image_tabular_text_learner(data, len_cont_names, vocab_sz, data_lm, use_trainer:bool=False):\n",
    "    l = text_classifier_learner(data_lm, AWD_LSTM, drop_mult=0.5)\n",
    "    l.load_encoder('fine_tuned_enc')\n",
    "\n",
    "    emb = data.train_ds.x.item_lists[1].get_emb_szs()\n",
    "    model = ImageTabularTextModel(emb, len_cont_names, vocab_sz, l.model, use_trainer)\n",
    "\n",
    "    learn = ImageTabularTextLearner(data, model, use_trainer, metrics=[mae])\n",
    "    return learn\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = image_tabular_text_learner(data, len(cont_names), len(vocab.itos), data_text, use_trainer=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# learn.callback_fns +=[partial(EarlyStoppingCallback, monitor='accuracy', min_delta=0.005, patience=3)]\n",
    "# learn.callback_fns += [(partial(LearnerTensorboardWriter, base_dir=Path(path + 'logs\\\\'), name='mixed-metadata'))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# learn.lr_find()\n",
    "# learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = 1e-3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 20:36 <p><table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>mean_absolute_error</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.426630</td>\n",
       "      <td>1.140555</td>\n",
       "      <td>0.857836</td>\n",
       "      <td>10:22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.181755</td>\n",
       "      <td>1.148888</td>\n",
       "      <td>0.837642</td>\n",
       "      <td>10:13</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Better model found at epoch 0 with mean_absolute_error value: 0.8578364849090576.\n",
      "Better model found at epoch 1 with mean_absolute_error value: 0.8376424312591553.\n"
     ]
    }
   ],
   "source": [
    "# learn.to_fp16 doesn't work with this model for some reason\n",
    "# learn = learn.to_fp16()\n",
    "learn.freeze()\n",
    "learn.fit_one_cycle(2, lr, callbacks=SaveModelCallback(learn, every='improvement', mode='min', monitor='mean_absolute_error', name='mixed'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.purge()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# learn.lr_find()\n",
    "# learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs=8\n",
    "data = mixed.databunch(bs=bs, collate_fn=collate_mixed)\n",
    "\n",
    "norm, denorm = normalize_custom_funcs(*imagenet_stats)\n",
    "data.add_tfm(norm) # normalize images\n",
    "\n",
    "learn = image_tabular_text_learner(data, len(cont_names), len(vocab.itos), data_text, use_trainer=True)\n",
    "# learn.callback_fns +=[partial(EarlyStoppingCallback, monitor='kappa_score', min_delta=0.005, patience=3)]\n",
    "learn.load('mixed-unfrozen')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='1' class='' max='4', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      25.00% [1/4 45:29<2:16:27]\n",
       "    </div>\n",
       "    \n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>mean_absolute_error</th>\n",
       "      <th>root_mean_squared_error</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.252965</td>\n",
       "      <td>1.149033</td>\n",
       "      <td>0.857493</td>\n",
       "      <td>0.961548</td>\n",
       "      <td>45:28</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>\n",
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='3794' class='' max='6580', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      57.66% [3794/6580 25:16<18:33 0.2089]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Better model found at epoch 0 with mean_absolute_error value: 0.8574932813644409.\n"
     ]
    }
   ],
   "source": [
    "learn.unfreeze()\n",
    "learn.fit_one_cycle(4, max_lr=slice(1e-6,1e-4), callbacks=SaveModelCallback(learn, every='improvement', mode='min', monitor='mean_absolute_error', name='mixed-unfrozen'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.load('mixed')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "p,y = learn.get_preds(ds_type=DatasetType.Valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "from petfinder.test import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "optR = OptimizedRounder()\n",
    "optR.fit(p.numpy()[:, 0], y.numpy())\n",
    "coeff = optR.coefficients()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds = optR.predict(p.numpy()[:, 0], coeff).astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = pets[pets.IsValidation == True][['PetID', 'AdoptionSpeed']]\n",
    "predictions['Prediction'] = preds\n",
    "predictions = predictions.groupby('PetID').mean()[['Prediction', 'AdoptionSpeed']]\n",
    "# preds, y = predictions['Prediction'], predictions['AdoptionSpeed']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4232357217350937"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "quadratic_weighted_kappa(predictions['Prediction'], predictions['AdoptionSpeed'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generating a submission for the competition\n",
    "\n",
    "Unfortunately fastai export does not support MixedItemList yet.  So to test my code on the test set I had to trick fastai in thinking that the test set is actually the validation set.  I just set all labels of the test set to be 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "pets['IsTest'] = False\n",
    "petsTest['IsTest'] = True\n",
    "petsTest['AdoptionSpeed'] = 0\n",
    "\n",
    "petsAll = pd.concat([pets, petsTest])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is pretty much the same code as training, but here we use .split_from_df(col='IsTest') to tell fastai that the validation are only the rows in the dataframe where the column IsTest is True."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "imgListTest = ImageList.from_df(petsAll, path=path, cols='PicturePath')\n",
    "tabListTest = TabularList.from_df(petsAll, cat_names=cat_names, cont_names=cont_names, procs=procs, path=path)\n",
    "textListTest = TextList.from_df(petsAll, cols='Description', path=path, vocab=vocab)\n",
    "\n",
    "mixedTest = (MixedItemList([imgListTest, tabListTest, textListTest], path, inner_df=tabListTest.inner_df)\n",
    "            .split_from_df(col='IsTest')\n",
    "            .label_from_df(cols='AdoptionSpeed', label_cls=FloatList)\n",
    "            .transform([[get_transforms()[0], [], []], [get_transforms()[1], [], []]], size=size))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataTest = mixedTest.databunch(bs=bs, collate_fn=collate_mixed)\n",
    "dataTest.add_tfm(norm) # normalize images\n",
    "\n",
    "learn = image_tabular_text_learner(dataTest, len(cont_names), len(vocab.itos), data_text, use_trainer=True)\n",
    "learn.load('mixed')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds,y = learn.get_preds(ds_type=DatasetType.Valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [],
   "source": [
    "p,y = preds.numpy()[:, 0], y.numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [],
   "source": [
    "optR = OptimizedRounder()\n",
    "preds = optR.predict(p, coeff).astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = petsTest\n",
    "predictions['AdoptionSpeed'] = preds\n",
    "predictions = predictions.groupby('PetID').mean()['AdoptionSpeed'].reset_index()\n",
    "predictions['AdoptionSpeed'] = predictions['AdoptionSpeed'].astype(int)\n",
    "predictions.to_csv('submission.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}