Decision_Tree_and_Random_Forest.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/google/applied-machine-learning-intensive/blob/master/content/06_other_models/00_decision_trees_and_random_forests/colab.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "copyright"
      },
      "source": [
        "#### Copyright 2020 Google LLC."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oXbDqPstu1RM"
      },
      "outputs": [],
      "source": [
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yLFFlvePlEsJ"
      },
      "source": [
        "# Decision Trees and Random Forests"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1Qm3o8i5lnAH"
      },
      "source": [
        "In this lab we will apply decision trees and random forests to perform machine learning tasks. These two model types are relatively easy to understand, but they are very powerful tools.\n",
        "\n",
        "Random forests build upon decision tree models, so we'll start by creating a decision tree and then move to random forests."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5Dj52_5Wm1Oa"
      },
      "source": [
        "## Load Data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qlxCUdTql0aW"
      },
      "source": [
        "Let's start by loading some data. We'll use the familiar iris dataset from scikit-learn."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oyP6DVIEjQZL"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "\n",
        "from sklearn.datasets import load_iris\n",
        "\n",
        "iris_bunch = load_iris()\n",
        "\n",
        "feature_names = iris_bunch.feature_names\n",
        "target_name = 'species'\n",
        "\n",
        "iris_df = pd.DataFrame(\n",
        "    iris_bunch.data,\n",
        "    columns=feature_names\n",
        ")\n",
        "\n",
        "iris_df[target_name] = iris_bunch.target\n",
        "\n",
        "iris_df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jwJoZY7hm3Rh"
      },
      "source": [
        "## Decision Trees"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lW7jPu6cm5Vs"
      },
      "source": [
        "Decision trees are models that create a tree structure that has a condition at each non-terminal leaf in the tree. The condition is used to choose which branch to traverse down the tree.\n",
        "\n",
        "Let's see what this would look like with a simple example.\n",
        "\n",
        "Let's say we want to determine if a piece of fruit is a lemon, lime, orange, or grapefruit. We might have a tree that looks like:\n",
        "\n",
        "```txt\n",
        "                      ----------\n",
        "           -----------| color? |-----------\n",
        "          |           ----------           |\n",
        "          |               |                |\n",
        "       <green>         <orange>        <yellow>\n",
        "          |               |                |\n",
        "          |               |                |\n",
        "       ========           |            =========\n",
        "       | lime |           |            | lemon |\n",
        "       ========       ---------        =========\n",
        "                 -----| size? |-----\n",
        "                 |    ---------    |\n",
        "                 |                 |\n",
        "              <small>           <large>\n",
        "                 |                 |\n",
        "                 |                 |\n",
        "            ==========       ==============\n",
        "            | orange |       | grapefruit |\n",
        "            ==========       ==============\n",
        "```\n",
        "\n",
        "This would roughly translate to the following code:\n",
        "\n",
        "```python\n",
        "\n",
        "def fruit_type(fruit):\n",
        "  if fruit.color == \"green\":\n",
        "    return \"lime\"\n",
        "  if fruit.color == \"yellow\":\n",
        "    return \"lemon\"\n",
        "  if fruit.color == \"orange\":\n",
        "    if fruit.size == \"small\":\n",
        "      return \"orange\"\n",
        "    if fruit.size == \"large\":\n",
        "      return \"grapefruit\"\n",
        "```\n",
        "\n",
        "As you can see, the decision tree is very easy to interpret. If you use a decision tree to make predictions and then need to determine why the tree made the decision that it did, it is very easy to inspect.\n",
        "\n",
        "Also, decision trees don't benefit from scaling or normalizing your data, which is different from many types of models."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E_ZO5q4xmOMU"
      },
      "source": [
        "### Create a Decision Tree"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LWYYYG0prrPb"
      },
      "source": [
        "Now that we have the data loaded, we can create a decision tree. We'll use the [`DecisionTreeClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html) from scikit-learn to perform this task.\n",
        "\n",
        "Note that there is also a [`DecisionTreeRegressor`](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeRegressor.html) that can be used for regression models. In practice, you'll typically see decision trees applied to classification problems more than regression.\n",
        "\n",
        "To build and train the model, we create an instance of the classifier and then call the `fit()` method that is used for all scikit-learn models."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PjkabO7nkCjt"
      },
      "outputs": [],
      "source": [
        "from sklearn import tree\n",
        "\n",
        "dt = tree.DecisionTreeClassifier()\n",
        "\n",
        "dt.fit(\n",
        "    iris_df[feature_names],\n",
        "    iris_df[target_name]\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zt0GmCTOsUx8"
      },
      "source": [
        "If this were a real application, we'd keep some data to the side for testing."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K24b3sUdmj17"
      },
      "source": [
        "### Visualize the Tree\n",
        "\n",
        "We now have a decision tree and can use it to make predictions. But before we do that, let's take a look at the tree itself.\n",
        "\n",
        "To do this we create a [`StringIO`](https://docs.python.org/3/library/io.html) object that we can export dot data to. [DOT](https://www.graphviz.org/doc/info/lang.html) is a graph description language with Python-graphing utilities that we can plot with.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xmPNDQvKkeOd"
      },
      "outputs": [],
      "source": [
        "import io\n",
        "import pydotplus\n",
        "\n",
        "from IPython.display import Image  \n",
        "\n",
        "dot_data = io.StringIO()  \n",
        "\n",
        "tree.export_graphviz(\n",
        "    dt,\n",
        "    out_file=dot_data,  \n",
        "    feature_names=feature_names\n",
        ")  \n",
        "\n",
        "graph = pydotplus.graph_from_dot_data(dot_data.getvalue())  \n",
        "\n",
        "Image(graph.create_png())  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f2ipXFP7n8Xg"
      },
      "source": [
        "That tree looks pretty complex. Many branches in the tree is a sign that we may have overfit the model. Let's create the tree again; this time we'll limit the depth."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WA0lUiePoIuZ"
      },
      "outputs": [],
      "source": [
        "from sklearn import tree\n",
        "\n",
        "dt = tree.DecisionTreeClassifier(max_depth=2)\n",
        "\n",
        "dt.fit(\n",
        "    iris_df[feature_names],\n",
        "    iris_df[target_name]\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LOU_tzOyoO6A"
      },
      "source": [
        "And plot to see the branching."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "plW0crKvoSY6"
      },
      "outputs": [],
      "source": [
        "import io\n",
        "import pydotplus\n",
        "\n",
        "from IPython.display import Image  \n",
        "\n",
        "dot_data = io.StringIO()  \n",
        "\n",
        "tree.export_graphviz(\n",
        "    dt,\n",
        "    out_file=dot_data,  \n",
        "    feature_names=feature_names\n",
        ")  \n",
        "\n",
        "graph = pydotplus.graph_from_dot_data(dot_data.getvalue())  \n",
        "\n",
        "Image(graph.create_png())  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "67yJDrz9oUo6"
      },
      "source": [
        "This tree is less likely to be overfitting since we forced it to have a depth of 2. Holding out a test sample and performing validation would be a good way to check."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yTw-HAn5syz4"
      },
      "source": [
        "What are the `gini`, `samples`, and `value` items shown in the tree?\n",
        "\n",
        "`gini` is is the *Gini impurity*. This is a measure of the chance that you'll misclassify a random element in the dataset at this decision point. Smaller `gini` is better.\n",
        "\n",
        "`samples` is a count of the number of samples that have met the criteria to reach this leaf.\n",
        "\n",
        "Within `value` is the count of each class of data that has made it to this leaf. Summing `value` should equal `sample`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yvbBaidx39o-"
      },
      "source": [
        "### Hyperparameters"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cgJIfXZ44BW0"
      },
      "source": [
        "There are many hyperparameters you can tweak in your decision tree models. One of those is `criterion`. `criterion` determines the quality measure that the model will use to determine the shape of the tree.\n",
        "\n",
        "The possible `criterion` values are `gini` and `entropy`. `gini` is the [Gini Impuirty](https://en.wikipedia.org/wiki/Decision_tree_learning#Gini_impurity) while `entropy` is a measure of [Information Gain](https://en.wikipedia.org/wiki/Decision_tree_learning#Information_gain).\n",
        "\n",
        "In the example below, we switch the classifier to use \"entropy\" for `criterion`. You'll see in the resultant tree that we now see \"entropy\" instead of \"gini\", but the resultant trees are the same. For more complex models, though, it may be worthwhile to test the different criterion."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CktGFTok5cFE"
      },
      "outputs": [],
      "source": [
        "import io\n",
        "import pydotplus\n",
        "\n",
        "from IPython.display import Image  \n",
        "from sklearn import tree\n",
        "\n",
        "dt = tree.DecisionTreeClassifier(\n",
        "    max_depth=2, \n",
        "    criterion=\"entropy\"\n",
        ")\n",
        "\n",
        "dt.fit(\n",
        "    iris_df[feature_names],\n",
        "    iris_df[target_name]\n",
        ")\n",
        "\n",
        "dot_data = io.StringIO()  \n",
        "\n",
        "tree.export_graphviz(\n",
        "    dt,\n",
        "    out_file=dot_data,  \n",
        "    feature_names=feature_names\n",
        ")  \n",
        "\n",
        "graph = pydotplus.graph_from_dot_data(dot_data.getvalue())  \n",
        "\n",
        "Image(graph.create_png())  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vVzXWuwg8x__"
      },
      "source": [
        "We've limited the depth of the tree using `max_depth`. We can also limit the number of samples required to be present in a node for it to be considered for splitting using `min_samples_split`. We can also limit the minimum size of a leaf node using `min_samples_leaf`. All of these hyperparameters help you to prevent your model from overfitting.\n",
        "\n",
        "There are many other hyperparameters that can be found in the [`DecisionTreeClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html) documentation."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uY_juAqE9emQ"
      },
      "source": [
        "### Exercise 1: Tuning Decision Tree Hyperparameters"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hgRtUMCA91UI"
      },
      "source": [
        "In this exercise we will use a decision tree to classify wine quality in the [Red Wine Quality dataset](https://www.kaggle.com/uciml/red-wine-quality-cortez-et-al-2009).\n",
        "\n",
        "The target column in the dataset is `quality`. Quality is an integer value between 1 and 10 (inclusive). You'll use the other columns in the dataset to build a decision tree to predict wine quality.\n",
        "\n",
        "For this exercise:\n",
        "\n",
        "* Hold out some data for final testing of model generalization.\n",
        "* Use [`GridSearchCV`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html) to compare some hyperparameters for your model. You can choose which parameters to test.\n",
        "* Print the hyperparameters of the best performing model.\n",
        "* Print the accuracy of the best performing model and the holdout dataset.\n",
        "* Visualize the best performing tree.\n",
        "\n",
        "Use as many text and code cells as you need to perform this exercise. We'll get you started with the code to authenticate and download the dataset."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bf2Da0VnKJqh"
      },
      "source": [
        "First upload your `kaggle.json` file, and then run the code block below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QnVpaP7XKODL"
      },
      "outputs": [],
      "source": [
        "! chmod 600 kaggle.json && (ls ~/.kaggle 2>/dev/null || mkdir ~/.kaggle) && mv kaggle.json ~/.kaggle/ && echo 'Done'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8BPOFhblKPwM"
      },
      "source": [
        "Next, download the wine quality dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mFkRa-83KTIK"
      },
      "outputs": [],
      "source": [
        "! kaggle datasets download uciml/red-wine-quality-cortez-et-al-2009\n",
        "! ls"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xlxj0X-q-THj"
      },
      "source": [
        "##### **Student Solution**"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "\n",
        "wine_df = pd.read_csv('red-wine-quality-cortez-et-al-2009.zip')\n",
        "\n",
        "\n",
        "wine_df['quality']"
      ],
      "metadata": {
        "id": "hQ50d_KMzSwg"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "feature_names = wine_df.columns[:1]\n",
        "feature_names"
      ],
      "metadata": {
        "id": "BS_SOLtnE-BM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "target_name = ['quality']"
      ],
      "metadata": {
        "id": "FiI0SCgFEMpo"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "X_train,X_test,y_train,y_test = train_test_split(wine_df[feature_names],wine_df[target_name])"
      ],
      "metadata": {
        "id": "JPiLS9fjET2w"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn import tree\n",
        "\n",
        "dt = tree.DecisionTreeClassifier()\n"
      ],
      "metadata": {
        "id": "3c2Mc9UGu8oM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn import tree\n",
        "from sklearn.model_selection import GridSearchCV\n",
        "\n",
        "params = {'criterion':['gini','entropy'], 'splitter':['best','random']}\n",
        "\n",
        "dt = tree.DecisionTreeClassifier(max_depth=4)\n",
        "\n",
        "search = GridSearchCV(dt,params)\n",
        "\n",
        "dt = dt.fit(\n",
        "    wine_df[feature_names],\n",
        "    wine_df[target_name]\n",
        ")\n",
        "\n",
        "\n",
        "search = search.fit(X_train,y_train)\n",
        "\n",
        "print(search.best_estimator_)"
      ],
      "metadata": {
        "id": "gl40NgCkFkjq"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import io\n",
        "import pydotplus\n",
        "\n",
        "from IPython.display import Image  \n",
        "\n",
        "dot_data = io.StringIO()  \n",
        "\n",
        "tree.export_graphviz(\n",
        "    dt,\n",
        "    out_file=dot_data,  \n",
        "    feature_names=feature_names\n",
        ")  \n",
        "\n",
        "graph = pydotplus.graph_from_dot_data(dot_data.getvalue())  \n",
        "\n",
        "Image(graph.create_png())  "
      ],
      "metadata": {
        "id": "OgdUekSCHBJF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pCT2HzaZ-W_d"
      },
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cege4SOkmo4A"
      },
      "source": [
        "## Random Forests"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7XivwzvBmqlS"
      },
      "source": [
        "Random forests are a simple yet powerful machine learning tool based on decision trees. Random forests are easy to understand, yet they touch upon many advanced machine learning concepts, such as ensemble learning and bagging. These models can be used for both classification and regression. Also, since they are built from decision trees, they are not sensitive to unscaled data.\n",
        "\n",
        "You can think of a random forest as a group decision made by a number of decision trees. For classification problems, the random forest creates multiple decision trees with different subsets of the data. When it is asked to classify a data point, it will ask all of the trees what they think and then take the majority decision.\n",
        "\n",
        "For regression problems, the random forest will again use the opinions of multiple decision trees, but it will take the mean (or some other summation) of the responses and use that as the regression value.\n",
        "\n",
        "This type of modeling, where one model consists of other models, is called *ensemble learning*. Ensemble learning can often lead to better models because taking the combined, differing opinions of a group of models can reduce overfitting."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_2J1CZ4uohf2"
      },
      "source": [
        "### Create a Random Forest\n",
        "\n",
        "Creating a random forest is as easy as creating a decision tree.\n",
        "\n",
        "scikit-learn provides a [`RandomForestClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) and a [`RandomForestRegressor`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html), which can be used to combine the predictive power of multiple decision trees."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZH9xJB4ikyfv"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "\n",
        "from sklearn.datasets import load_iris\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "\n",
        "iris_bunch = load_iris()\n",
        "\n",
        "feature_names = iris_bunch.feature_names\n",
        "target_name = 'species'\n",
        "\n",
        "iris_df = pd.DataFrame(\n",
        "    iris_bunch.data,\n",
        "    columns=feature_names\n",
        ")\n",
        "\n",
        "iris_df[target_name] = iris_bunch.target\n",
        "\n",
        "rf = RandomForestClassifier()\n",
        "rf.fit(\n",
        "    iris_df[feature_names],\n",
        "    iris_df[target_name]\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PIoGwH2Lp2F4"
      },
      "source": [
        "You can look at different trees in the random forest to see how their decision branching differs. By default there are `100` decision trees created for the model.\n",
        "\n",
        "Let's view a few.\n",
        "\n",
        "Run the code below a few times, and see if you notice a difference in the trees that are shown."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XI_DkBXSpKDl"
      },
      "outputs": [],
      "source": [
        "import pydotplus\n",
        "import random\n",
        "\n",
        "from IPython.display import Image  \n",
        "from six import StringIO  \n",
        "\n",
        "dot_data = StringIO()  \n",
        "\n",
        "tree.export_graphviz(\n",
        "    random.choice(rf.estimators_),\n",
        "    out_file=dot_data,  \n",
        "    feature_names=feature_names\n",
        ")  \n",
        "\n",
        "graph = pydotplus.graph_from_dot_data(dot_data.getvalue())  \n",
        "\n",
        "Image(graph.create_png())  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i_xl-Et8qjTA"
      },
      "source": [
        "### Make Predictions\n",
        "\n",
        "Just like any other scikit-learn model, you can use the `predict()` method to make predictions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ll4flhAWqA4W"
      },
      "outputs": [],
      "source": [
        "print(rf.predict([iris_df.iloc[121][feature_names]]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fMYt-B05PQ2B"
      },
      "source": [
        "### Hyperparameters"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bD0KX-pcPTHZ"
      },
      "source": [
        "Many of the hyperparameters available in decision trees are also available in random forest models. There are, however, some hyperparameters that are only available in random forests.\n",
        "\n",
        "The two most important are `bootstrap` and `oob_score`. These two hyperparameters are relevant to ensemble learning.\n",
        "\n",
        "`bootstrap` determines if the model will use [bootstrap sampling](https://en.wikipedia.org/wiki/Bootstrapping_(statistics)). When you bootstrap, only a sample of the dataset will be used for training each tree in the forest. The full dataset will be used as the source of the sampling for each tree, but each sample will have a different set of data points, perhaps with some repetition. In bootstrapping, there is also \"replacement\" of the data, which means a data point can occur in more that one tree.\n",
        "\n",
        "`oob_score` stands for \"Out of bag score.\" When you create a bootstrap sample, this is referred to as a *bag* in machine learning parlance. When the tree is being scored, only data points in the bag sampled for the tree will be used unless `oob_score` is set to true."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FLf1S7Hirvyy"
      },
      "source": [
        "### Exercise 2: Feature Importance"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2lToTdAbJA-M"
      },
      "source": [
        "In this exercise we will use the [UCI Abalone  dataset](https://www.kaggle.com/hurshd0/abalone-uci) to determine the age of sea snails.\n",
        "\n",
        "The target feature in the dataset is `rings`, which is a proxy for age in the snails. This is a numeric value, but it is stored as an integer and has a biological limit. So we can think of this as a classification problem and use a [`RandomForestClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html).\n",
        "\n",
        "You will download the dataset and train a random forest classifier. After you have fit the classifier, the `feature_importances_` attribute of the model will be populated. Use the importance scores to print the least important feature.\n",
        "\n",
        "*Note that some of the features are categorical string values. You'll need to convert these to numeric values to use them in the model.*\n",
        "\n",
        "Use as many text and code blocks as you need to perform this exercise."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NlOyoIK8r6kc"
      },
      "source": [
        "#### **Student Solution**"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "! chmod 600 kaggle.json && (ls ~/.kaggle 2>/dev/null || mkdir ~/.kaggle) && mv kaggle.json ~/.kaggle/ && echo 'Done'"
      ],
      "metadata": {
        "id": "v_j39wTuC2-t"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QNdrDofHr_XQ"
      },
      "outputs": [],
      "source": [
        "# Your Code Goes Here\n",
        "\n",
        "! kaggle datasets download hurshd0/abalone-uci\n",
        "! 1s"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "\n",
        "snail_df = pd.read_csv('abalone-uci.zip')\n",
        "\n",
        "snail_df['sex'] = snail_df['sex'].replace(['M','F','I'],[0,1,2])\n",
        "\n",
        "snail_df"
      ],
      "metadata": {
        "id": "EVHuw_Q4HMvP"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "features = snail_df.columns[:-1]\n",
        "target_name = 'rings'\n",
        "snail_df[target_name]"
      ],
      "metadata": {
        "id": "smpqbUgNH2mO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "X_train,X_test,y_train,y_test = train_test_split(snail_df[features],snail_df[target_name])"
      ],
      "metadata": {
        "id": "CN__C4MsIOPV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.ensemble import RandomForestClassifier\n",
        "from sklearn.datasets import make_classification\n",
        "\n",
        "\n",
        "\n",
        "feature_names = features\n",
        "target_name = 'rings'\n",
        "\n",
        "\n",
        "\n",
        "rf = RandomForestClassifier()\n",
        "\n",
        "rf.fit(\n",
        "    snail_df[feature_names],\n",
        "    snail_df['rings']\n",
        "\n",
        ")"
      ],
      "metadata": {
        "id": "6s127E-jIe7s"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.metrics import accuracy_score\n",
        "\n",
        "predictions = rf.predict(X_train)\n",
        "\n",
        "accuracy_score(y_train,predictions)\n"
      ],
      "metadata": {
        "id": "q9DdjAKt_3mY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "predictions = rf.predict(X_test)\n",
        "\n",
        "accuracy_score(y_test,predictions)"
      ],
      "metadata": {
        "id": "VMqb1givBsA2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from numpy.ma.core import array\n",
        "rf.feature_importances_"
      ],
      "metadata": {
        "id": "7kr6OjbhByUq"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XSyu1rcWA9qo"
      },
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "snail_df.columns[rf.feature_importances_.argmin()]"
      ],
      "metadata": {
        "id": "PpazOeZ3CCBr"
      },
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [
        "copyright",
        "YdlbjnrP-Xuw",
        "exercise-1-key-1"
      ],
      "name": "Decision Tree and Random Forest.ipynb",
      "private_outputs": true,
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}