examples/01.1_Optimizing_RandomForest_using_DEHB.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optimizing RandomForest using DEHB\n",
    "This notebook aims to build on the template from `00_interfacing_DEHB` and use it on an actual problem, to optimize the hyperparameters of a Random Forest model, for a dataset.\n",
    "\n",
    "Additional requirements:\n",
    "* scikit-learn>=0.24\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import numpy as np\n",
    "import warnings\n",
    "\n",
    "\n",
    "seed = 123\n",
    "np.random.seed(seed)\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The problem defined here is to optimize a [Random Forest model](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html), on any given dataset, using DEHB. The hyperparameters chosen to be optimized are:\n",
    "* `max_depth`\n",
    "* `min_samples_split`\n",
    "* `max_features`\n",
    "* `min_samples_leaf`\n",
    "while the `n_estimators` hyperparameter to the Random Forest is chosen to be a fidelity parameter instead. Lesser number of trees ($<10$) in the Random Forest may not allow adequate ensembling for the grouped prediction to be significantly better than the individual tree predictions. Whereas a large number of trees (~$100$) often give accurate predictions but is naturally slower to train and predict on account of more trees to train. Therefore, a smaller `n_estimators` can be used as a cheaper approximation of the actual fidelity of `n_estimators=100`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining fidelity range"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "min_fidelity, max_fidelity = 2, 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the remaining $4$ hyperparameters, the search space can be created as a `ConfigSpace` object, with the domain of individual parameters defined."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating search space"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ConfigSpace as CS\n",
    "\n",
    "\n",
    "def create_search_space(seed=123):\n",
    "    \"\"\"Parameter space to be optimized --- contains the hyperparameters\n",
    "    \"\"\"\n",
    "    cs = CS.ConfigurationSpace(seed=seed)\n",
    "\n",
    "    cs.add_hyperparameters([\n",
    "        CS.UniformIntegerHyperparameter(\n",
    "            'max_depth', lower=1, upper=15, default_value=2, log=False\n",
    "        ),\n",
    "        CS.UniformIntegerHyperparameter(\n",
    "            'min_samples_split', lower=2, upper=128, default_value=2, log=True\n",
    "        ),\n",
    "        CS.UniformFloatHyperparameter(\n",
    "            'max_features', lower=0.1, upper=0.9, default_value=0.5, log=False\n",
    "        ),\n",
    "        CS.UniformIntegerHyperparameter(\n",
    "            'min_samples_leaf', lower=1, upper=64, default_value=1, log=True\n",
    "        ),\n",
    "    ])\n",
    "    return cs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Configuration space object:\n",
      "  Hyperparameters:\n",
      "    max_depth, Type: UniformInteger, Range: [1, 15], Default: 2\n",
      "    max_features, Type: UniformFloat, Range: [0.1, 0.9], Default: 0.5\n",
      "    min_samples_leaf, Type: UniformInteger, Range: [1, 64], Default: 1, on log-scale\n",
      "    min_samples_split, Type: UniformInteger, Range: [2, 128], Default: 2, on log-scale\n",
      "\n"
     ]
    }
   ],
   "source": [
    "cs = create_search_space(seed)\n",
    "print(cs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dimensionality of search space: 4\n"
     ]
    }
   ],
   "source": [
    "dimensions = len(cs.get_hyperparameters())\n",
    "print(\"Dimensionality of search space: {}\".format(dimensions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the primary black/gray-box interface to the Random Forest model needs to be built for DEHB to query. As given in the `00_interfacing_DEHB` notebook, this function will have a signature akin to: `target_function(config, fidelity)`, and return a two-element tuple of the `score` and `cost`. It must be noted that DEHB **minimizes** and therefore the `score` being returned by this `target_function` should account for it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, the target function trains a Random Forest model on a dataset. We load a dataset here and maintain a fixed, train-validation-test split for one complete run. Multiple DEHB runs can therefore optimize on the same validation split, and evaluate final performance on the same test set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating target function to optimize (2 parts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1 ) Preparing dataset and splits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_iris, load_digits, load_wine\n",
    "\n",
    "\n",
    "classification = {\"iris\": load_iris, \"digits\": load_digits, \"wine\": load_wine}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "\n",
    "def prepare_dataset(model_type=\"classification\", dataset=None):\n",
    "\n",
    "    if model_type == \"classification\":\n",
    "        if dataset is None:\n",
    "            dataset = np.random.choice(list(classification.keys())) \n",
    "        _data = classification[dataset]()\n",
    "    else:\n",
    "        if dataset is None:\n",
    "            dataset = np.random.choice(list(regression.keys()))\n",
    "        _data = regression[dataset]()\n",
    "\n",
    "    train_X, rest_X, train_y, rest_y = train_test_split(\n",
    "      _data.get(\"data\"), \n",
    "      _data.get(\"target\"), \n",
    "      train_size=0.7, \n",
    "      shuffle=True, \n",
    "      random_state=seed\n",
    "    )\n",
    "    \n",
    "    # 10% test and 20% validation data\n",
    "    valid_X, test_X, valid_y, test_y = train_test_split(\n",
    "      rest_X, rest_y,\n",
    "      test_size=0.3333, \n",
    "      shuffle=True, \n",
    "      random_state=seed\n",
    "    )\n",
    "    return train_X, train_y, valid_X, valid_y, test_X, test_y, dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "wine\n",
      "Train size: (124, 13)\n",
      "Valid size: (36, 13)\n",
      "Test size: (18, 13)\n"
     ]
    }
   ],
   "source": [
    "train_X, train_y, valid_X, valid_y, test_X, test_y, dataset = \\\n",
    "    prepare_dataset(model_type=\"classification\")\n",
    "\n",
    "print(dataset)\n",
    "print(\"Train size: {}\\nValid size: {}\\nTest size: {}\".format(\n",
    "    train_X.shape, valid_X.shape, test_X.shape\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2 ) Function interface with DEHB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.metrics import accuracy_score, make_scorer\n",
    "\n",
    "\n",
    "accuracy_scorer = make_scorer(accuracy_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def target_function(config, fidelity, **kwargs):\n",
    "    # Extracting support information\n",
    "    seed = kwargs[\"seed\"]\n",
    "    train_X = kwargs[\"train_X\"]\n",
    "    train_y = kwargs[\"train_y\"]\n",
    "    valid_X = kwargs[\"valid_X\"]\n",
    "    valid_y = kwargs[\"valid_y\"]\n",
    "    max_fidelity = kwargs[\"max_fidelity\"]\n",
    "    \n",
    "    if fidelity is None:\n",
    "        fidelity = max_fidelity\n",
    "    \n",
    "    start = time.time()\n",
    "    # Building model \n",
    "    model = RandomForestClassifier(\n",
    "        **config.get_dictionary(),\n",
    "        n_estimators=int(fidelity),\n",
    "        bootstrap=True,\n",
    "        random_state=seed,\n",
    "    )\n",
    "    # Training the model on the complete training set\n",
    "    model.fit(train_X, train_y)\n",
    "    \n",
    "    # Evaluating the model on the validation set\n",
    "    valid_accuracy = accuracy_scorer(model, valid_X, valid_y)\n",
    "    cost = time.time() - start\n",
    "    \n",
    "    # Evaluating the model on the test set as additional info\n",
    "    test_accuracy = accuracy_scorer(model, test_X, test_y)\n",
    "    \n",
    "    result = {\n",
    "        \"fitness\": -valid_accuracy,  # DE/DEHB minimizes\n",
    "        \"cost\": cost,\n",
    "        \"info\": {\n",
    "            \"test_score\": test_accuracy,\n",
    "            \"fidelity\": fidelity\n",
    "        }\n",
    "    }\n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have all components to define the problem to be optimized. DEHB can be initialized using all these information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running DEHB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m2024-06-18 22:23:54.633\u001b[0m | \u001b[33m\u001b[1mWARNING \u001b[0m | \u001b[36mdehb.optimizers.dehb\u001b[0m:\u001b[36m__init__\u001b[0m:\u001b[36m264\u001b[0m - \u001b[33m\u001b[1mA checkpoint already exists, results could potentially be overwritten.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "from dehb import DEHB\n",
    "\n",
    "dehb = DEHB(\n",
    "    f=target_function, \n",
    "    cs=cs, \n",
    "    dimensions=dimensions, \n",
    "    min_fidelity=min_fidelity, \n",
    "    max_fidelity=max_fidelity,\n",
    "    n_workers=1,\n",
    "    output_path=\"./temp\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m2024-06-18 22:24:04.776\u001b[0m | \u001b[33m\u001b[1mWARNING \u001b[0m | \u001b[36mdehb.optimizers.dehb\u001b[0m:\u001b[36m_timeout_handler\u001b[0m:\u001b[36m351\u001b[0m - \u001b[33m\u001b[1mRuntime budget exhausted. Saving optimization checkpoint now.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "trajectory, runtime, history = dehb.run(\n",
    "    total_cost=10,\n",
    "    # parameters expected as **kwargs in target_function is passed here\n",
    "    seed=123,\n",
    "    train_X=train_X,\n",
    "    train_y=train_y,\n",
    "    valid_X=valid_X,\n",
    "    valid_y=valid_y,\n",
    "    max_fidelity=dehb.max_fidelity\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "480 480 480\n",
      "\n"
     ]
    },
    {
     "ename": "ValueError",
     "evalue": "too many values to unpack (expected 5)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[13], line 5\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[38;5;66;03m# Last recorded function evaluation\u001b[39;00m\n\u001b[1;32m      4\u001b[0m last_eval \u001b[38;5;241m=\u001b[39m history[\u001b[38;5;241m-\u001b[39m\u001b[38;5;241m1\u001b[39m]\n\u001b[0;32m----> 5\u001b[0m config, score, cost, fidelity, _info \u001b[38;5;241m=\u001b[39m last_eval\n\u001b[1;32m      7\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mLast evaluated configuration, \u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m      8\u001b[0m \u001b[38;5;28mprint\u001b[39m(dehb\u001b[38;5;241m.\u001b[39mvector_to_configspace(config), end\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n",
      "\u001b[0;31mValueError\u001b[0m: too many values to unpack (expected 5)"
     ]
    }
   ],
   "source": [
    "print(len(trajectory), len(runtime), len(history), end=\"\\n\\n\")\n",
    "\n",
    "# Last recorded function evaluation\n",
    "last_eval = history[-1]\n",
    "config_id, config, score, cost, fidelity, _info = last_eval\n",
    "\n",
    "print(\"Last evaluated configuration, \")\n",
    "print(dehb.vector_to_configspace(config), end=\"\")\n",
    "print(\"got a score of {}, was evaluated at a fidelity of {:.2f} and \"\n",
    "      \"took {:.3f} seconds to run.\".format(score, fidelity, cost))\n",
    "print(\"The additional info attached: {}\".format(_info))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below, we let DEHB optimize for $5$ different runs. The `reset()` allows DEHB to begin optimization from the beginning by cleaning all history and starting with random samples. Each run of DEHB optimization is for just $10$ seconds as set by `total_cost=10`. We then report the mean and the standard deviation of the best score seen across these $5$ runs. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "runs = 5\n",
    "\n",
    "best_config_list = []\n",
    "\n",
    "for i in range(runs):\n",
    "    # Resetting to begin optimization again\n",
    "    dehb.reset()\n",
    "    # Executing a run of DEHB optimization lasting for 10s\n",
    "    trajectory, runtime, history = dehb.run(\n",
    "        total_cost=10,\n",
    "        seed=123,\n",
    "        train_X=train_X,\n",
    "        train_y=train_y,\n",
    "        valid_X=valid_X,\n",
    "        valid_y=valid_y,\n",
    "        max_fidelity=dehb.max_fidelity\n",
    "    )\n",
    "    best_config = dehb.vector_to_configspace(dehb.inc_config)\n",
    "    \n",
    "    # Creating a model using the best configuration found\n",
    "    model = RandomForestClassifier(\n",
    "        **best_config.get_dictionary(),\n",
    "        n_estimators=int(max_fidelity),\n",
    "        bootstrap=True,\n",
    "        random_state=seed,\n",
    "    )\n",
    "    # Training the model on the complete training set\n",
    "    model.fit(\n",
    "        np.concatenate((train_X, valid_X)), \n",
    "        np.concatenate((train_y, valid_y))\n",
    "    )\n",
    "    # Evaluating the model on the held-out test set\n",
    "    test_accuracy = accuracy_scorer(model, test_X, test_y)\n",
    "    best_config_list.append((best_config, test_accuracy))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Mean score across trials: \", np.mean([score for _, score in best_config_list]))\n",
    "print(\"Std. dev. of score across trials: \", np.std([score for _, score in best_config_list]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for config, score in best_config_list:\n",
    "    print(\"{} got an accuracy of {} on the test set.\".format(config, score))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}