BERT.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "BERT.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "mount_file_id": "1uedKqXll7IbJTxxtdX82ESNvP1Zy9vWa",
      "authorship_tag": "ABX9TyPIw+nvDYM9f4eKdMVxftrl",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/atherfawaz/BERT-Supervised/blob/master/BERT.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ca7jIEhZNU_Q",
        "colab_type": "text"
      },
      "source": [
        "# Supervised Learning with BERT\n",
        "\n",
        "***Question*: Supervised Learning with BERT\n",
        "The Astrological department believes that a person's astrological sign can be guessed from their behavior. An organization is collecting blog-posts of different people from various sources. You have been tasked to build a Deep Learning model that can use these posts data of individuals to predict which star group out of 12 does an individual belong to. You also need to tell the gender of that person.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4PIs-jEzMCT4",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "b40a55fb-d7f9-4803-a413-837a9a48e0ae"
      },
      "source": [
        "#from google.colab import drive\n",
        "#drive.mount('/content/gdrive')\n",
        "%cd /content/drive/My Drive/Ebryx/blogs_train"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/drive/My Drive/Ebryx/blogs_train\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4GW1LBluMU3t",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!tar -xvf  'blogs_train.tar.xz'\n",
        "%cd /content/drive/My Drive/Ebryx/blogs_train"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2ArTrha5U1RA",
        "colab_type": "text"
      },
      "source": [
        "# Dataset\n",
        "The file name would contain the gender, age, occupation, and astrological sign of the blooger. For example, 4115891.male.24.Student.Leo.xml is one file. A single file will contain a set of blogs separated by date. To illustrate, this is what a sample file looks like:\n",
        "\n",
        "```\n",
        "<Blog>\n",
        "  <date>31,May,2004</date>\n",
        "    <post>\n",
        "      Well, everyone got up and going this morning.  It's still raining, but that's okay with me.  Sort of suits my mood.  I could easily have stayed home in bed with my book and the cats.  This has been a lot of rain though!..\n",
        "    </post>\n",
        "</Blog\n",
        "```\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NN1hRWkRd6vI",
        "colab_type": "text"
      },
      "source": [
        "# Parsing the dataset\n",
        "Parsing the dataset from separate files into a Pandas Dataframe for displaying and easy access. Some XML files contain encoding issues and the problematic contents of those files have been replaced by random number. While this could impact the accuracy of the model later, the effect would not be that big."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jYh3rv4SnJ1R",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import os\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import codecs\n",
        "from bs4 import BeautifulSoup\n",
        "from progressbar import ProgressBar\n",
        "pbar = ProgressBar()\n",
        "\n",
        "print('PARSING FILES....')\n",
        "\n",
        "FILES = os.listdir()\n",
        "#print('File count: ', len(FILES))\n",
        "#print(FILES)\n",
        "#FILES = ['4115891.male.24.Student.Leo.xml', '4115958.male.16.Communications-Media.Libra.xml', '4116071.male.26.Arts.Sagittarius.xml', '4116243.female.24.Manufacturing.Sagittarius.xml']\n",
        "\n",
        "posts_arr = []\n",
        "sign_arr = []\n",
        "gender_arr = []\n",
        "age_arr = []\n",
        "occupation_arr = []\n",
        "\n",
        "for to_fetch in pbar(FILES):\n",
        "    #print('Parsing file:', to_fetch)\n",
        "    gender = to_fetch.split('.')[-5]\n",
        "    age = to_fetch.split('.')[-4]\n",
        "    occupation = to_fetch.split('.')[-3]\n",
        "    sign = to_fetch.split('.')[-2]\n",
        "    with codecs.open(to_fetch, 'r', encoding='utf-8', errors='ignore') as fp:\n",
        "      soup = BeautifulSoup(fp, 'lxml',\n",
        "                           from_encoding='utf8')\n",
        "      posts = soup.find_all('post')\n",
        "      #print(soup.prettify())\n",
        "      for post in posts:\n",
        "        clean_str = post.text\n",
        "        clean_str = clean_str.replace('\\r', '')\n",
        "        clean_str = clean_str.replace('\\n', '')\n",
        "        posts_arr.append(clean_str)\n",
        "        sign_arr.append(sign)\n",
        "        age_arr.append(age)\n",
        "        occupation_arr.append(occupation)\n",
        "        gender_arr.append(gender)\n",
        "\n",
        "df = pd.DataFrame({'Gender': gender_arr, 'Age': age_arr, 'Occupation': occupation_arr, 'Post': posts_arr, 'Sign': sign_arr})\n",
        "\n",
        "#df.head(50)\n",
        "df.to_csv('/content/drive/My Drive/Ebryx/dataset.csv', encoding='utf-8', index=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jjGpg5ZfnmcH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df.head(100)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "knjTjaAtcx-H",
        "colab_type": "text"
      },
      "source": [
        "# Loading BERT for finetuning\n",
        "\n",
        "Taking help from the model implementation from huggingface and [this repository](https://colab.research.google.com/github/abhimishra91/transformers-tutorials/blob/master/transformers_multiclass_classification.ipynb#scrollTo=JrBr2YesGdO_).\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BKyIOfh5jmCz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!pip install transformers;"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "32pVsuaBj12B",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 357
        },
        "outputId": "d88c724e-364c-4c4c-c658-4b6960d4011c"
      },
      "source": [
        "!nvidia-smi"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Tue Aug 18 07:49:07 2020       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 450.57       Driver Version: 418.67       CUDA Version: 10.1     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   58C    P8    10W /  70W |      0MiB / 15079MiB |      0%      Default |\n",
            "|                               |                      |                 ERR! |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bkjgmBG-jHPs",
        "colab_type": "text"
      },
      "source": [
        "# Run from here."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C7amTtuAjhZM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Importing the libraries needed\n",
        "# Importing stock ml libraries\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "from sklearn import metrics\n",
        "import transformers\n",
        "import torch\n",
        "from torch.utils.data import Dataset, DataLoader, RandomSampler, SequentialSampler\n",
        "from transformers import BertTokenizer, BertModel, BertConfig\n",
        "from progressbar import ProgressBar\n",
        "pbar = ProgressBar()\n",
        "\n",
        "# Defining some key variables that will be used later on in the training\n",
        "MAX_LEN = 512\n",
        "TRAIN_BATCH_SIZE = 8\n",
        "VALID_BATCH_SIZE = 4\n",
        "EPOCHS = 1\n",
        "LEARNING_RATE = 1e-05\n",
        "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')\n",
        "\n",
        "# Setting up the device for GPU usage\n",
        "\n",
        "from torch import cuda\n",
        "device = 'cuda' if cuda.is_available() else 'cpu'"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SBPNbR2tl8sZ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        },
        "outputId": "5cb2b458-b50c-44b8-d09e-63c18b4c0000"
      },
      "source": [
        "# Creating the dataset and dataloader for the neural network\n",
        "df = pd.read_csv('/content/drive/My Drive/Ebryx/dataset.csv')\n",
        "df = df[['Post', 'Sign', 'Gender']]\n",
        "\n",
        "encode_dict = {}\n",
        "gender_dict = {}\n",
        "\n",
        "iden = np.identity(13, dtype=int)\n",
        "\n",
        "def encode_cat(x):\n",
        "    if x not in encode_dict.keys():\n",
        "      encode_dict[x]=len(encode_dict)\n",
        "    return encode_dict[x]\n",
        "\n",
        "def encode_gender(x):\n",
        "  if x == 'male':\n",
        "    return 0\n",
        "  else:\n",
        "    return 1\n",
        "\n",
        "def get_onehot(x):\n",
        "  return iden[x]\n",
        "\n",
        "df['Sign'] = df['Sign'].apply(lambda x: encode_cat(x))\n",
        "df['Sign'] = df['Sign'].apply(lambda x: get_onehot(x))\n",
        "df['Gender'] = df['Gender'].apply(lambda x: encode_gender(x))\n",
        "\n",
        "i = 0\n",
        "for item in df['Gender']:\n",
        "  if (item == 0):\n",
        "    df['Sign'][i][12] = 1\n",
        "  else:\n",
        "    df['Sign'][i][12] = 0\n",
        "  i += 1\n",
        "\n",
        "\n",
        "df = df [['Post', 'Sign']]\n",
        "\n",
        "print('Here sign means sign + gender, last bit is for gender:')\n",
        "df.head()"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Here sign means sign + gender, last bit is for gender:\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Post</th>\n",
              "      <th>Sign</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>I just watched Beauty and the beast...</td>\n",
              "      <td>[1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>This picture shows a Vietnamese ...</td>\n",
              "      <td>[1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>So I just used the term “Bad Ass...</td>\n",
              "      <td>[1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>This is a dumb little story I whipp...</td>\n",
              "      <td>[1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Listen, you fuckers, you screwhe...</td>\n",
              "      <td>[1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                Post                                     Sign\n",
              "0             I just watched Beauty and the beast...  [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
              "1                This picture shows a Vietnamese ...  [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
              "2                So I just used the term “Bad Ass...  [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
              "3             This is a dumb little story I whipp...  [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
              "4                Listen, you fuckers, you screwhe...  [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NT4MFsu3kW8h",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "outputId": "bc82fe53-9376-43ce-faff-dc6cfbb892be"
      },
      "source": [
        "class CustomDataset(Dataset):\n",
        "\n",
        "    def __init__(self, dataframe, tokenizer, max_len):\n",
        "        self.tokenizer = tokenizer\n",
        "        self.data = dataframe\n",
        "        self.post = dataframe.Post\n",
        "        self.targets = self.data.Sign\n",
        "        self.max_len = max_len\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.post)\n",
        "\n",
        "    def __getitem__(self, index):\n",
        "        post = str(self.post[index])\n",
        "        post = \" \".join(post.split())\n",
        "\n",
        "        inputs = self.tokenizer.encode_plus(\n",
        "            post,\n",
        "            None,\n",
        "            add_special_tokens=True,\n",
        "            max_length=self.max_len,\n",
        "            pad_to_max_length=True,\n",
        "            return_token_type_ids=True,\n",
        "            truncation=True,\n",
        "        )\n",
        "        ids = inputs['input_ids']\n",
        "        mask = inputs['attention_mask']\n",
        "        token_type_ids = inputs[\"token_type_ids\"]\n",
        "\n",
        "        return {\n",
        "            'ids': torch.tensor(ids, dtype=torch.long),\n",
        "            'mask': torch.tensor(mask, dtype=torch.long),\n",
        "            'token_type_ids': torch.tensor(token_type_ids, dtype=torch.long),\n",
        "            'targets': torch.tensor(self.targets[index], dtype=torch.float)\n",
        "        }\n",
        "\n",
        "# Creating the dataset and dataloader for the neural network\n",
        "\n",
        "train_size = 0.8\n",
        "train_dataset=df.sample(frac=train_size,random_state=200)\n",
        "test_dataset=df.drop(train_dataset.index).reset_index(drop=True)\n",
        "train_dataset = train_dataset.reset_index(drop=True)\n",
        "\n",
        "\n",
        "print(\"FULL Dataset: {}\".format(df.shape))\n",
        "print(\"TRAIN Dataset: {}\".format(train_dataset.shape))\n",
        "print(\"TEST Dataset: {}\".format(test_dataset.shape))\n",
        "\n",
        "training_set = CustomDataset(train_dataset, tokenizer, MAX_LEN)\n",
        "testing_set = CustomDataset(test_dataset, tokenizer, MAX_LEN)"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "FULL Dataset: (380720, 2)\n",
            "TRAIN Dataset: (304576, 2)\n",
            "TEST Dataset: (76144, 2)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nNFZh2WQkhl5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "train_params = {'batch_size': TRAIN_BATCH_SIZE,\n",
        "                'shuffle': True,\n",
        "                'num_workers': 0\n",
        "                }\n",
        "\n",
        "test_params = {'batch_size': VALID_BATCH_SIZE,\n",
        "                'shuffle': True,\n",
        "                'num_workers': 0\n",
        "                }\n",
        "\n",
        "training_loader = DataLoader(training_set, **train_params)\n",
        "testing_loader = DataLoader(testing_set, **test_params)"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YB0uZ-dgnJm9",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "25fe4afe-3c7a-4bbf-bffb-89bda4727135"
      },
      "source": [
        "# Creating the customized model, by adding a drop out and a dense layer on top of distil bert to get the final output for the model. \n",
        "\n",
        "class BERTClass(torch.nn.Module):\n",
        "    def __init__(self):\n",
        "        super(BERTClass, self).__init__()\n",
        "        self.l1 = transformers.BertModel.from_pretrained('bert-base-uncased')\n",
        "        self.l2 = torch.nn.Dropout(0.3)\n",
        "        self.l3 = torch.nn.Linear(768, 13)\n",
        "    \n",
        "    def forward(self, ids, mask, token_type_ids):\n",
        "        _, output_1= self.l1(ids, attention_mask = mask, token_type_ids = token_type_ids)\n",
        "        output_2 = self.l2(output_1)\n",
        "        output = self.l3(output_2)\n",
        "        return output\n",
        "\n",
        "model = BERTClass()\n",
        "model.to(device)\n"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "BERTClass(\n",
              "  (l1): BertModel(\n",
              "    (embeddings): BertEmbeddings(\n",
              "      (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
              "      (position_embeddings): Embedding(512, 768)\n",
              "      (token_type_embeddings): Embedding(2, 768)\n",
              "      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "      (dropout): Dropout(p=0.1, inplace=False)\n",
              "    )\n",
              "    (encoder): BertEncoder(\n",
              "      (layer): ModuleList(\n",
              "        (0): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (1): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (2): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (3): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (4): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (5): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (6): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (7): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (8): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (9): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (10): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (11): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "      )\n",
              "    )\n",
              "    (pooler): BertPooler(\n",
              "      (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "      (activation): Tanh()\n",
              "    )\n",
              "  )\n",
              "  (l2): Dropout(p=0.3, inplace=False)\n",
              "  (l3): Linear(in_features=768, out_features=13, bias=True)\n",
              ")"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rwhi-s_AnR4K",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 272
        },
        "outputId": "cca7d9ea-7b60-41da-f8af-3140d91fdfec"
      },
      "source": [
        "def loss_fn(outputs, targets):\n",
        "    return torch.nn.BCEWithLogitsLoss()(outputs, targets)\n",
        "\n",
        "optimizer = torch.optim.Adam(params =  model.parameters(), lr=LEARNING_RATE)\n",
        "\n",
        "def train(epoch):\n",
        "    model.train()\n",
        "    for i,data in enumerate(training_loader, 0):\n",
        "        ids = data['ids'].to(device, dtype = torch.long)\n",
        "        mask = data['mask'].to(device, dtype = torch.long)\n",
        "        token_type_ids = data['token_type_ids'].to(device, dtype = torch.long)\n",
        "        targets = data['targets'].to(device, dtype = torch.float)\n",
        "\n",
        "        outputs = model(ids, mask, token_type_ids)\n",
        "\n",
        "        optimizer.zero_grad()\n",
        "        loss = loss_fn(outputs, targets)\n",
        "        \n",
        "        if i%5==0:\n",
        "            print(f' [{i}] Loss:  {loss.item():.3f}')\n",
        "        \n",
        "        optimizer.zero_grad()\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "\n",
        "for epoch in range(EPOCHS):\n",
        "    train(epoch)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            " [0] Loss:  0.736\n",
            " [5] Loss:  0.700\n",
            " [10] Loss:  0.671\n",
            " [15] Loss:  0.626\n",
            " [20] Loss:  0.605\n",
            " [25] Loss:  0.580\n",
            " [30] Loss:  0.572\n",
            " [35] Loss:  0.571\n",
            " [40] Loss:  0.524\n",
            " [45] Loss:  0.492\n",
            " [50] Loss:  0.465\n",
            " [55] Loss:  0.455\n",
            " [60] Loss:  0.447\n",
            " [65] Loss:  0.415\n",
            " [70] Loss:  0.394\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cHQAidgKktN3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}