lab1_NLP.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://www.nvidia.com/dli\"> <img src=\"images/DLI Header.png\" alt=\"Header\" style=\"width: 400px;\"/> </a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Natural Language Processing for Signal Generation on News Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src = \"images/RNFC-logo.png\" width=\"800\" height=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### RN Financial Corporation\n",
    "#### Andrew Tan\n",
    "#### Rafael Nicolas Fermin Cota"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Natural Language Processing (NLP) is a field of artificial intelligence that models the interaction between human (natural) language and computers. The various tasks NLP models can be split into several categories:\n",
    "1. Syntax: Parsing, part-of-speech tagging, morphological segmentation, stemming...\n",
    "2. Semantics: Machine translation, sentiment analysis, natural language understanding...\n",
    "3. Discourse: Automatic summarization (TL:DR)...\n",
    "4. Speech: Speech recognition, text-to-speech...\n",
    "\n",
    "In the early days of NLP (pre-1980s) most of these tasks were accomplished with many hand-crafted rules. With the introduction of machine learning and the steady increase of computational power, more NLP models were being built with statistical learning on natural language corpus.\n",
    "\n",
    "Fast forward to today, where Deep learning models are achieving state-of-the-art performance in a variety of vision and NLP tasks. The combination of larger datasets, advancement in GPU technology, increased research into deep learning architectures/applications and the vast number of problems a modern company deals with means that today's Data Scientist must be capable of deploying Deep Learning solutions. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lab we will focus on the following:\n",
    "\n",
    "1. Building Deep Neural Networks to process and interpret news data.\n",
    "2. Understand the various building blocks of making a NLP system.\n",
    "3. Backtest and apply the models to news data for signal generation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Word Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "A Word Embedding is a mathematical mapping from a vast dimensional space where each word occupies a dimension to a reduced-dimension, continuous vector space. Typically a large corpus of text is used to train and develop these embeddings. \n",
    "There are various methods to generating word embeddings. These include neural networks, probabilistic models, dimensionality reduction etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### GloVe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "GloVe (Global Vectors for Word Representation) is an unsupervised learning  algorithm. It trains on the word-to-word co-occurrence statistics from a corpus and attempts to learn word vectors such that their dot product equals the logarithms of the words' probability of co-occurrence. GloVe has several neat features:\n",
    "* Nearest Neighbors - the cosine similarity between two word vectors is can be an effective measure of the linguistic or semantic similarity of the corresponding words.\n",
    "* Linear substructures - in contrast to the cosine similarity, an great deal of information is captured in the vector differences between word vectors. GloVe tries to captures the information pertaining to the relationships between words and this can be showcased through the vector differences. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/Word-Vectors.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### FastText\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "FastText is a library for text classification and representation learning developed by Facebook AI Research. Its focus is on speed and scalability while maintaining comparable levels of performance compared to other methods. \n",
    "\n",
    "FastText provides two methods for computing word representations from a corpus. Both define a supervised learning task in which by learning this task well will generate useful word vectors. \n",
    "\n",
    "* Skipgram\n",
    "   \n",
    "The Skipgram model attempts to utilize given word to predict the word(s) surrounding it. Skipgram thus learns the likelihood of a word being present based on the occurence of the word(s) that appear near it in the corpus. You can think of the task as predicting the context given a word.\n",
    "    \n",
    "* Continuous Bag of Words (Cbow)\n",
    "\n",
    "The CBow method is the inverse, instead it takes a bag of words surrounding the target word and attempts to make a prediction. You can think of this as predicting the word given the context. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Applications of Natural Language Processing to News Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Sentiment Analysis for News"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "It is undeniable that following the news release of a story with strong impact on an industry or company the market prices intraday will react accordingly. For the average person with an investment account, the news is a substantial signal in the decision making process to buy/sell a certain stock. However to make a systematic approach to trading on news signals is simply impossible for a human to do manually. \n",
    "\n",
    "- There are 92000+ news article released per day\n",
    "- An average human can read at a speed of 200-250 words per minute\n",
    "- Reading at this rate, for 8 hours continuously, a human may process up to 40-50 articles per day. This calculation disregards the time it takes to find the articles and make any analysis.\n",
    "\n",
    "In Finance, the efficiency and speed at which you process information can be vital for making well-informed, smart decisions. We can leverage deep learning in order to train models that provide sentiment scores for headlines, articles, tweets, and posts. These sentiments can produce valuable signals to support a buy/sell/hold decision as well as valuation models. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multi-channel LSTM network for Sentiment Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following code, we will use the Keras + Tensorflow libraries to construct and train a multi-channel LSTM network for classifying sentiments. A multi-channel network simply means we can use more than one type of embedding. This means that the network has access to more features from seperate word embeddings. The idea is that a single type of embedding may not contain enough information, By utilizing another embedding, that is either trained with a different corpus or a different algorithm entirely we can have stronger more robust features. In the model below we will utilize pretrained embeddings from both GloVe and FastText Skipgram models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
      "[nltk_data]   Unzipping corpora/stopwords.zip.\n",
      "Keras version: 2.2.4\n",
      "Tensorflow version: 1.11.0\n",
      "Sklearn version: 0.20.0\n"
     ]
    }
   ],
   "source": [
    "## Import the necessary libraries\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from datetime import date\n",
    "from tqdm import tqdm\n",
    "\n",
    "import sklearn\n",
    "from sklearn.model_selection import train_test_split, StratifiedKFold\n",
    "\n",
    "\n",
    "import tensorflow as tf\n",
    "from tensorflow.python import pywrap_tensorflow\n",
    "import keras\n",
    "\n",
    "from IPython.display import Image\n",
    "\n",
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from keras.utils.np_utils import to_categorical\n",
    "\n",
    "import nltk\n",
    "nltk.download('stopwords')\n",
    "from nltk import RegexpTokenizer\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "from numpy.random import seed\n",
    "from tensorflow import set_random_seed\n",
    "seed(42)\n",
    "set_random_seed(42)\n",
    "MAX_SEQUENCE_LENGTH = 32\n",
    "EMBEDDING_DIM = 300\n",
    "\n",
    "print(\"Keras version:\",keras.__version__)\n",
    "print(\"Tensorflow version:\",tf.__version__)\n",
    "print(\"Sklearn version:\",sklearn.__version__)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be utilizing open source news data from Bloomberg and Reuters between 2006 and 2012.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>headline</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>url</th>\n",
       "      <th>tldr</th>\n",
       "      <th>Class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Exxon Mobil offers plan to end Alaska dispute</td>\n",
       "      <td>2006-10-20 06:15:00</td>\n",
       "      <td>http://www.reuters.com/article/2006/10/20/busi...</td>\n",
       "      <td>In a proposal sent earlier this week to the Al...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Hey buddy, can you spare $600 for a Google share?</td>\n",
       "      <td>2006-10-20 04:25:00</td>\n",
       "      <td>http://www.reuters.com/article/2006/10/20/busi...</td>\n",
       "      <td>SAN FRANCISCO/NEW YORK  (Reuters) - Wall Stree...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Ford posts biggest loss in 14 years</td>\n",
       "      <td>2006-10-23 06:42:00</td>\n",
       "      <td>http://www.reuters.com/article/2006/10/23/us-a...</td>\n",
       "      <td>Ford also said it was considering raising new ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Shell looks to buy out Canada unit for C$7.7 b...</td>\n",
       "      <td>2006-10-23 04:34:00</td>\n",
       "      <td>http://www.reuters.com/article/2006/10/23/us-e...</td>\n",
       "      <td>In July, Shell Canada rattled the industry and...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>U.S. venture investors betting on energy, Web 2.0</td>\n",
       "      <td>2006-10-23 08:36:00</td>\n",
       "      <td>http://www.reuters.com/article/2006/10/23/us-f...</td>\n",
       "      <td>SAN FRANCISCO  (Reuters) - U.S. venture capita...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            headline            timestamp  \\\n",
       "0      Exxon Mobil offers plan to end Alaska dispute  2006-10-20 06:15:00   \n",
       "1  Hey buddy, can you spare $600 for a Google share?  2006-10-20 04:25:00   \n",
       "2                Ford posts biggest loss in 14 years  2006-10-23 06:42:00   \n",
       "3  Shell looks to buy out Canada unit for C$7.7 b...  2006-10-23 04:34:00   \n",
       "4  U.S. venture investors betting on energy, Web 2.0  2006-10-23 08:36:00   \n",
       "\n",
       "                                                 url  \\\n",
       "0  http://www.reuters.com/article/2006/10/20/busi...   \n",
       "1  http://www.reuters.com/article/2006/10/20/busi...   \n",
       "2  http://www.reuters.com/article/2006/10/23/us-a...   \n",
       "3  http://www.reuters.com/article/2006/10/23/us-e...   \n",
       "4  http://www.reuters.com/article/2006/10/23/us-f...   \n",
       "\n",
       "                                                tldr  Class  \n",
       "0  In a proposal sent earlier this week to the Al...      2  \n",
       "1  SAN FRANCISCO/NEW YORK  (Reuters) - Wall Stree...      1  \n",
       "2  Ford also said it was considering raising new ...      1  \n",
       "3  In July, Shell Canada rattled the industry and...      1  \n",
       "4  SAN FRANCISCO  (Reuters) - U.S. venture capita...      1  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Read and sample the data.\n",
    "df = pd.read_csv(\"/dli/data/news_data/news_data_labelled.csv\")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting timestamp: 2006-10-20 04:25:00 \n",
      "Ending timestamp: 2012-12-31 23:06:28\n"
     ]
    }
   ],
   "source": [
    "print(\"Starting timestamp: {} \\nEnding timestamp: {}\".format(df.timestamp.min(),df.timestamp.max()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Split the dataset into testing and training datasets\n",
    "X = df.tldr\n",
    "y = df.Class\n",
    "X_train, X_test, y_train, y_test = train_test_split(X,y, stratify = y, test_size=0.10, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Tokenize training set and apply to train + test sets. Pad the sequences with zeros.\n",
    "tokenizer = Tokenizer(num_words=None,\n",
    "                       filters='!\"#$%&()*+,-./:;<=>?@[\\\\]^_`{|}~\\t\\n',\n",
    "                       lower=True,\n",
    "                       split=\" \",\n",
    "                       char_level=False)\n",
    "\n",
    "tokenizer.fit_on_texts(X_train)\n",
    "\n",
    "X_train = pad_sequences(tokenizer.texts_to_sequences(X_train),maxlen=MAX_SEQUENCE_LENGTH,value=0.)\n",
    "X_test = pad_sequences(tokenizer.texts_to_sequences(X_test),maxlen=MAX_SEQUENCE_LENGTH,value=0.)\n",
    "y_train = to_categorical(y_train)\n",
    "y_test = to_categorical(y_test)\n",
    "\n",
    "word_index = tokenizer.word_index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "## Define the embedding matrices for Glove and FastText\n",
    "def embedding_matrix(path_to_embedding : str, embedding_dim: int, word_index : dict) -> np.array:\n",
    "    \"\"\"\n",
    "    This function creates an embedding matrix.\n",
    "    \n",
    "    Inputs:\n",
    "    path_to_embedding - path to text file of word embeddings\n",
    "    embedding_dim - dimension of word embeddings\n",
    "    word_index - dictionary mapping words to indices\n",
    "    \n",
    "    Outputs:\n",
    "    embedding_matrix - numpy matrix containing the embeddings\n",
    "    \n",
    "    \"\"\"\n",
    "    embeddings_index = {}\n",
    "    f = open(path_to_embedding, encoding='utf-8')\n",
    "    for line in f:\n",
    "        try:\n",
    "            values = line.split()\n",
    "            word = values[0]\n",
    "            coefs = np.asarray(values[1:], dtype='float32')\n",
    "            embeddings_index[word] = coefs\n",
    "        except:\n",
    "            pass\n",
    "        \n",
    "    f.close()\n",
    "\n",
    "    embedding_matrix = np.zeros((len(word_index) + 1,embedding_dim))\n",
    "    found = 0\n",
    "    for word, i in word_index.items():\n",
    "        embedding_vector = embeddings_index.get(word)\n",
    "        if embedding_vector is not None:\n",
    "            # words not found in embedding index will be all-zeros.\n",
    "            found +=1\n",
    "            embedding_matrix[i] = embedding_vector\n",
    "\n",
    "    return embedding_matrix\n",
    "                                \n",
    "                                \n",
    "glove_embedding_matrix = embedding_matrix(\"/dli/data/news_data/glove/glove.840B.300d.txt\",EMBEDDING_DIM,tokenizer.word_index)\n",
    "fasttext_embedding_matrix = embedding_matrix(\"/dli/data/news_data/fasttext/wiki-news-300d-1M.vec\",EMBEDDING_DIM,tokenizer.word_index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Import libraries + tools for building the multi-channel LSTM\n",
    "from keras.models import Model\n",
    "from keras import layers\n",
    "from keras.optimizers import Adam\n",
    "from keras.layers import Input, Dense, Dropout, Activation, Embedding, BatchNormalization\n",
    "from keras.layers import LSTM, concatenate, Bidirectional, PReLU, ELU, LeakyReLU, GRU, SimpleRNN\n",
    "from keras.callbacks import ModelCheckpoint, ReduceLROnPlateau, TensorBoard\n",
    "from keras.utils.vis_utils import plot_model\n",
    "import datetime\n",
    "import pydot, graphviz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Define the neural network architecture\n",
    "\n",
    "## Subchannel network for encoding sequential information\n",
    "def subnetwork_channel(input_layer : layers, RNN_architecture : str, units : int, dropout_rate : float) -> layers:\n",
    "    \"\"\"\n",
    "    This function creates a sub network for encoding sequences.\n",
    "    \n",
    "    Inputs:\n",
    "    input_layer - The input keras layer into the subnetwork\n",
    "    RNN_architecture - Name of the RNN type to use\n",
    "    units - Number of units in the RNN\n",
    "    dropout_rate - dropout rate\n",
    "    \n",
    "    Outputs:\n",
    "    batch - Batch Normalized output layer\n",
    "    \n",
    "    \"\"\"\n",
    "    assert RNN_architecture in [\"LSTM\", \"GRU\", \"RNN\"]\n",
    "    \n",
    "    dropout1 = Dropout(rate = dropout_rate)(input_layer)\n",
    "    \n",
    "    if RNN_architecture == \"LSTM\":\n",
    "        rnn_layer = Bidirectional(LSTM(units = units, return_sequences = False))(dropout1)\n",
    "    elif RNN_architecture == \"GRU\":\n",
    "        rnn_layer = Bidirectional(GRU(units = units, return_sequences = False))(dropout1)\n",
    "    elif RNN_architecture == \"RNN\":\n",
    "        rnn_layer = Bidirectional(SimpleRNN(units = units, return_sequences = False))(dropout1)\n",
    "    \n",
    "    dropout2 = Dropout(rate = dropout_rate)(rnn_layer)\n",
    "    batch = BatchNormalization()(dropout2)\n",
    "    return batch\n",
    "\n",
    "## Output layer network\n",
    "def output_channel(input_layer : layers, activation : str, units : int, dropout_rate : float) -> layers:\n",
    "    \"\"\"\n",
    "    This function creates a sub network for outputing classification probabilities.\n",
    "    \n",
    "    Inputs:\n",
    "    input_layer - The input keras layer into the subnetwork\n",
    "    activation  - Name of the activation type to use\n",
    "    units - Number of units in the Dense network\n",
    "    dropout_rate - dropout rate\n",
    "    \n",
    "    Outputs:\n",
    "    output - Softmax output layer\n",
    "    \n",
    "    \"\"\"\n",
    "    assert activation in [\"ReLU\",\"PReLU\", \"ELU\", \"LeakyReLU\"]\n",
    "    \n",
    "    dense = Dense(units)(input_layer)\n",
    "    \n",
    "    if activation == \"PReLU\":\n",
    "        act = PReLU()(dense)\n",
    "    elif activation == \"ELU\":\n",
    "        act = ELU()(dense)\n",
    "    elif activation == \"LeakyReLU\":\n",
    "        act = LeakyReLU()(dense)\n",
    "    elif activation == \"ReLU\":\n",
    "        act = Dense(units, activation='relu')(input_layer)\n",
    "    \n",
    "        \n",
    "    dropout = Dropout(rate = dropout_rate)(act)\n",
    "    batch = BatchNormalization()(dropout)\n",
    "    output = Dense(3,activation='softmax', name = \"Output\")(batch)\n",
    "    \n",
    "    return output\n",
    "    \n",
    "## Define full model.\n",
    "def define_model(RNN_architecture : str = \"LSTM\", rnn_units : int = 256, dense_units : int = 128,dense_activation : str = \"PReLU\" ,dropout_rate : float = 0.4) -> Model:\n",
    "    \"\"\"\n",
    "    This function defines and compiles a Multichannel RNN for Sentiment Classification.\n",
    "    \n",
    "    Inputs:\n",
    "    RNN_architecture - Name of the RNN type to use\n",
    "    rnn_units - Number of units in the RNN\n",
    "    dense_units - Number of units in the Dense network\n",
    "    dense_activation  - Name of the activation type to use\n",
    "    dropout_rate - dropout rate\n",
    "    \n",
    "    Outputs:\n",
    "    model - A Keras model\n",
    "    \n",
    "    \"\"\"\n",
    "    # Input Layer\n",
    "    shape = (MAX_SEQUENCE_LENGTH,)\n",
    "    input1 = Input(shape = shape, name = \"Main_input\")\n",
    "    \n",
    "    # Channel 1 - GLoVe\n",
    "    embedding1 = Embedding(len(word_index) + 1,\n",
    "              EMBEDDING_DIM,\n",
    "              weights=[glove_embedding_matrix],\n",
    "              input_length=MAX_SEQUENCE_LENGTH,\n",
    "              trainable=False,\n",
    "              input_shape=X_train.shape[1:], name = \"GLoVe_Embedding\")(input1)\n",
    "\n",
    "    net1 = subnetwork_channel(embedding1, RNN_architecture = RNN_architecture, units = rnn_units, dropout_rate = dropout_rate)\n",
    "    \n",
    "    # Channel 2 - Fast Text\n",
    "    embedding2 = Embedding(len(word_index) + 1,\n",
    "              EMBEDDING_DIM,\n",
    "              weights=[fasttext_embedding_matrix],\n",
    "              input_length=MAX_SEQUENCE_LENGTH,\n",
    "              trainable=False,\n",
    "              input_shape=shape, name = \"FastText_Embedding\")(input1)\n",
    "\n",
    "    net2 = subnetwork_channel(embedding2, RNN_architecture = RNN_architecture, units = rnn_units, dropout_rate = dropout_rate)\n",
    "    \n",
    "    # Merge\n",
    "    merged = concatenate([net1,net2], name =\"Merge\")\n",
    "    # Output channel\n",
    "    output = output_channel(merged, activation = dense_activation, units = dense_units, dropout_rate = dropout_rate)\n",
    "    \n",
    "    # Compile \n",
    "    model = Model(inputs = input1, outputs = output)\n",
    "    model.compile(loss = 'categorical_crossentropy', optimizer = Adam(0.002), metrics = ['categorical_accuracy'])\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "Main_input (InputLayer)         (None, 32)           0                                            \n",
      "__________________________________________________________________________________________________\n",
      "GLoVe_Embedding (Embedding)     (None, 32, 300)      116584800   Main_input[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "FastText_Embedding (Embedding)  (None, 32, 300)      116584800   Main_input[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "dropout_1 (Dropout)             (None, 32, 300)      0           GLoVe_Embedding[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "dropout_3 (Dropout)             (None, 32, 300)      0           FastText_Embedding[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "bidirectional_1 (Bidirectional) (None, 512)          1140736     dropout_1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "bidirectional_2 (Bidirectional) (None, 512)          1140736     dropout_3[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dropout_2 (Dropout)             (None, 512)          0           bidirectional_1[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "dropout_4 (Dropout)             (None, 512)          0           bidirectional_2[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_1 (BatchNor (None, 512)          2048        dropout_2[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_2 (BatchNor (None, 512)          2048        dropout_4[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "Merge (Concatenate)             (None, 1024)         0           batch_normalization_1[0][0]      \n",
      "                                                                 batch_normalization_2[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "dense_2 (Dense)                 (None, 128)          131200      Merge[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "dropout_5 (Dropout)             (None, 128)          0           dense_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_3 (BatchNor (None, 128)          512         dropout_5[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "Output (Dense)                  (None, 3)            387         batch_normalization_3[0][0]      \n",
      "==================================================================================================\n",
      "Total params: 235,587,267\n",
      "Trainable params: 2,415,363\n",
      "Non-trainable params: 233,171,904\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "## Compile and display the model.\n",
    "model = define_model(RNN_architecture = \"LSTM\", rnn_units= 256, dense_units = 128, dense_activation = \"ReLU\", dropout_rate = 0.4)\n",
    "model.summary()\n",
    "pic_name = 'images/multichannel-bidirectionalLSTM.png'\n",
    "plot_model(model,show_shapes=True,to_file=pic_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/multichannel-bidirectionalLSTM.png\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "123061/123061 [==============================] - 18s 143us/step - loss: 0.5400 - categorical_accuracy: 0.7623\n",
      "Epoch 2/10\n",
      "123061/123061 [==============================] - 17s 142us/step - loss: 0.5351 - categorical_accuracy: 0.7646\n",
      "Epoch 3/10\n",
      "123061/123061 [==============================] - 17s 142us/step - loss: 0.5334 - categorical_accuracy: 0.7660\n",
      "Epoch 4/10\n",
      "123061/123061 [==============================] - 17s 142us/step - loss: 0.5308 - categorical_accuracy: 0.7667\n",
      "Epoch 5/10\n",
      "123061/123061 [==============================] - 17s 142us/step - loss: 0.5292 - categorical_accuracy: 0.7690\n",
      "Epoch 6/10\n",
      "123061/123061 [==============================] - 17s 142us/step - loss: 0.5247 - categorical_accuracy: 0.7707\n",
      "Epoch 7/10\n",
      "123061/123061 [==============================] - 17s 142us/step - loss: 0.5224 - categorical_accuracy: 0.7717\n",
      "Epoch 8/10\n",
      "123061/123061 [==============================] - 17s 142us/step - loss: 0.5235 - categorical_accuracy: 0.7706\n",
      "Epoch 9/10\n",
      "123061/123061 [==============================] - 18s 142us/step - loss: 0.5288 - categorical_accuracy: 0.7700\n",
      "Epoch 10/10\n",
      "123061/123061 [==============================] - 18s 142us/step - loss: 0.5265 - categorical_accuracy: 0.7710\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f3863208e80>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Train\n",
    "tensorboard = TensorBoard(log_dir='/dli/tasks/tensorboard/logs/')\n",
    "model.fit(X_train,y_train,epochs = 10, batch_size = 1024,callbacks = [tensorboard])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Click [here](/tensorboard/) to start TensorBoard."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "13674/13674 [==============================] - 13s 966us/step\n",
      "Validation Accuracy: 77.1\n"
     ]
    }
   ],
   "source": [
    "## Find the testing accuracy\n",
    "val_loss, val_catergorical_accuracy = model.evaluate(X_test,y_test)\n",
    "print(\"Validation Accuracy: {:.1f}\".format(val_catergorical_accuracy * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Our model was able to achieve ~79% accuracy. According to research on sentiment analysis and classification, human raters may only agree with each other about 80% of the time. Due to the nature of sentiment analysis, the outcome a reader arrives at can be very subjective depending on how the reader interprets the words, tone or phrasing of the text. Thus, a model that predicts with 100% accuracy may still disagree with a human 20% of the time. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise: Re-tune Neural Network Parameters\n",
    "Try experimenting with different parameters in the neural network.\n",
    "In the function 'define_model'\n",
    "    - 'RNN_architecture' can be one of: \"RNN\", \"GRU\", \"LSTM\".\n",
    "    - 'rnn_units' are the number of units in the RNN\n",
    "    - 'dense_units' are the number of units in the dense network\n",
    "    - 'dense_activation' can be one of: \"PReLU\", \"LeakyReLU\", \"ELU\", \"ReLU\"\n",
    "    - 'dropout_rate' rate of dropout throughout the network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "Main_input (InputLayer)         (None, 32)           0                                            \n",
      "__________________________________________________________________________________________________\n",
      "GLoVe_Embedding (Embedding)     (None, 32, 300)      116584800   Main_input[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "FastText_Embedding (Embedding)  (None, 32, 300)      116584800   Main_input[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "dropout_11 (Dropout)            (None, 32, 300)      0           GLoVe_Embedding[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "dropout_13 (Dropout)            (None, 32, 300)      0           FastText_Embedding[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "bidirectional_5 (Bidirectional) (None, 512)          285184      dropout_11[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "bidirectional_6 (Bidirectional) (None, 512)          285184      dropout_13[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "dropout_12 (Dropout)            (None, 512)          0           bidirectional_5[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "dropout_14 (Dropout)            (None, 512)          0           bidirectional_6[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_7 (BatchNor (None, 512)          2048        dropout_12[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_8 (BatchNor (None, 512)          2048        dropout_14[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "Merge (Concatenate)             (None, 1024)         0           batch_normalization_7[0][0]      \n",
      "                                                                 batch_normalization_8[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "dense_6 (Dense)                 (None, 128)          131200      Merge[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "dropout_15 (Dropout)            (None, 128)          0           dense_6[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_9 (BatchNor (None, 128)          512         dropout_15[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "Output (Dense)                  (None, 3)            387         batch_normalization_9[0][0]      \n",
      "==================================================================================================\n",
      "Total params: 233,876,163\n",
      "Trainable params: 704,259\n",
      "Non-trainable params: 233,171,904\n",
      "__________________________________________________________________________________________________\n",
      "Epoch 1/10\n",
      "123061/123061 [==============================] - 19s 153us/step - loss: 0.8139 - categorical_accuracy: 0.6458\n",
      "Epoch 2/10\n",
      "123061/123061 [==============================] - 17s 142us/step - loss: 0.5886 - categorical_accuracy: 0.7421\n",
      "Epoch 3/10\n",
      "123061/123061 [==============================] - 17s 142us/step - loss: 0.5725 - categorical_accuracy: 0.7464\n",
      "Epoch 4/10\n",
      "123061/123061 [==============================] - 17s 142us/step - loss: 0.5691 - categorical_accuracy: 0.7464\n",
      "Epoch 5/10\n",
      "123061/123061 [==============================] - 18s 142us/step - loss: 0.5619 - categorical_accuracy: 0.7499\n",
      "Epoch 6/10\n",
      "123061/123061 [==============================] - 18s 142us/step - loss: 0.5523 - categorical_accuracy: 0.7540\n",
      "Epoch 7/10\n",
      "123061/123061 [==============================] - 17s 142us/step - loss: 0.5652 - categorical_accuracy: 0.7433\n",
      "Epoch 8/10\n",
      "123061/123061 [==============================] - 17s 142us/step - loss: 0.5644 - categorical_accuracy: 0.7488\n",
      "Epoch 9/10\n",
      "123061/123061 [==============================] - 17s 142us/step - loss: 0.5500 - categorical_accuracy: 0.7568\n",
      "Epoch 10/10\n",
      "123061/123061 [==============================] - 18s 142us/step - loss: 0.5431 - categorical_accuracy: 0.7608\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f38cc853358>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Compile and display the model.\n",
    "model = define_model(RNN_architecture = \"RNN\", rnn_units= 256, dense_units = 128, dense_activation = \"ReLU\", dropout_rate = 0.4)\n",
    "model.summary()\n",
    "pic_name = 'images/multichannel-bidirectionalLSTM.png'\n",
    "plot_model(model,show_shapes=True,to_file=pic_name)\n",
    "\n",
    "## Train\n",
    "tensorboard = TensorBoard(log_dir='/dli/tasks/tensorboard/logs/')\n",
    "model.fit(X_train,y_train,epochs = 10, batch_size = 1024,callbacks = [tensorboard])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "13674/13674 [==============================] - 14s 997us/step\n",
      "Validation Accuracy: 75.3\n"
     ]
    }
   ],
   "source": [
    "## Find the testing accuracy\n",
    "val_loss, val_catergorical_accuracy = model.evaluate(X_test,y_test)\n",
    "print(\"Validation Accuracy: {:.1f}\".format(val_catergorical_accuracy * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Conclusion:** Given parameters of 2 classificators (LSTM and RNN), best performance on the validation subset is achieved using LSTM model. Further parameters' optimization is needed. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Intraday Sentiment Strategy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To illustrate the power of Sentiment Analysis we'll construct and backtest a simple strategy.\n",
    "- Trade intraday over the year of 2013\n",
    "- Companies: \n",
    "    - Apple, Microsoft, Boeing, JPMorgan, Google, GM, Citigroup, Ford, Toyota, HSBC, ICAP\n",
    "- Assume perfect market entry and exit, no transaction fees\n",
    "- Sentiment score is the confidence of a text being positive or negative.\n",
    "- Basic strategy: \n",
    "    - BUY when 'sentiment_score' >= 'sentiment_cutoff' and SELL 'time_to_close_position' minutes later. \n",
    "    - SHORT SELL when 'sentiment_score' <= -'sentiment_cutoff' and BUY 'time_to_close_position' minutes later. \n",
    "    - If news is released when market is closed then BUY as soon as it is open."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f992c7428a174d97806c228b7918e26a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(IntSlider(value=30, description='time_till_close_position', max=60, min=5, step=5), Floa…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from utils import interactive_backtest\n",
    "interactive_backtest()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## News Processing for Risk Management (Optional)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another use case of News in trading is the ability to monitor portfolio holdings and mitigate risk. Being able to identify the possibility of a drop in a stock's price or observing that the market is reacting to the release of particular news can be a useful component in managing risk."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Case Study: Apple cuts iPhone X production due to weak demand:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "News reported on the Nikkei on Monday, January 29th revealed that Apple would cut its production target for the iPhone X from 40 to 20 million units. Apple's stock did not react well, in the wake of the reports stock fell even further even after it was already on the downtrend due to earnings reports. In this case we can see that Apple's stock price is correlated to the sentiment on the news articles related to the iPhone.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/apple_price.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/apple_sentiment.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Case Study:  Aimia Inc Receives Notice of Contract Non-renewal from Air Canada"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Aimia Inc is a data-driven marketing and loyalty analytics company. On May 11th, 2017 the company announced that Air Canada, its largest client had given its notice of non-renewal. The market responded accordingly with a sharp drop in price. The relative volume of articles on Aimia on the few days leading up to the announcement skyrocketed. A drastic change in the volume can be a signal for redirecting attention to certain companies. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/aimia_price.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/aimia_vol.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, don't forget to save your work from this lab before time runs out and the instance shuts down!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Applying Natural Language Processing on News Data using Deep Learning.ipynb\r\n",
      "utils.py\r\n",
      "images/\r\n",
      "images/aim_price.png\r\n",
      "images/apple_price.png\r\n",
      "images/aimia_price.png\r\n",
      "images/aimia_vol.png\r\n",
      "images/multichannel-bidirectionalLSTM.png\r\n",
      "images/Word-Vectors.png\r\n",
      "images/DLI Header.png\r\n",
      "images/apple_sentiment.png\r\n",
      "images/RNFC-logo.png\r\n"
     ]
    }
   ],
   "source": [
    "!tar -cvf output.zip  \"Applying Natural Language Processing on News Data using Deep Learning.ipynb\" utils.py images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Download output.zip](output.zip)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Glove: https://nlp.stanford.edu/projects/glove/\n",
    "- Fasttext: https://fasttext.cc/\n",
    "- News articles per day: https://www.slideshare.net/chartbeat/mockup-infographicv4-27900399\n",
    "- News data source: https://github.com/philipperemy/financial-news-dataset\n",
    "- Word embeddings: https://www.analyticsvidhya.com/blog/2017/06/word-embeddings-count-word2veec/, \n",
    "- Natural Language Processing: https://en.wikipedia.org/wiki/Natural-language_processing\n",
    "- Sentiment Analysis: https://en.wikipedia.org/wiki/Sentiment_analysis\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://www.nvidia.com/dli\"> <img src=\"images/DLI Header.png\" alt=\"Header\" style=\"width: 400px;\"/> </a>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}