NYCTaxi.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7eNHyhQtRwUR"
   },
   "source": [
    "# Analyzing NYC Taxi Fares with RAPIDS\n",
    "\n",
    "[RAPIDS](https://rapids.ai/) is a suite of GPU accelerated data science libraries with APIs that should be familiar to users of Pandas, scikit-learn, and Dask.\n",
    "\n",
    "This notebook builds a simple data pipeline:\n",
    "\n",
    "- we'll use cuDF to load, clean and prep some data, as well as for some feature engineering\n",
    "- we'll use cuML to explore patterns the data using unsupervised learning, and build a simple predictive model on top of it\n",
    "\n",
    "We'll learn that cuDF and cuML are GPU-accelerated libraries that are very similar to Pandas and scikit-learn (their respective CPU counterparts). The benefit of using RAPIDS libraries like cuDF and cuML is that they are much faster!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import glob\n",
    "import os\n",
    "\n",
    "import cudf\n",
    "import cuml\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import requests\n",
    "import seaborn as sns\n",
    "import sklearn\n",
    "from tqdm.auto import tqdm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The very first operation performed using cuDF initializes the library, which has some overhead. To ensure that this initialization time is not included when we measure code execution time, we \"warm up\" cuDF before using it for any real work:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = cudf.Series([1])  # warmup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main dataset we'll be using in this notebook comes from the [New York City Taxi and Limousine Commision (TLC)](https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page). The TLC publishes data about taxi rides, including pick-up and drop-off dates/times, pick-up and drop-off locations, trip distances, itemized fares, rate types, payment types, and driver-reported passenger counts. The data is made available as PARQUET files, and is published monthly.\n",
    "\n",
    "The code below downloads the PARQUET files containing trip data for \"Yellow\" Taxis from the year 2021. It also downloads a second, smaller dataset: a CSV file containing geographical information that will be useful in later parts of our analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def download(url, fname):\n",
    "    \"\"\"\n",
    "    Download file from `url`, writing the result to `fname`.\n",
    "    If `fname` already exists, do nothing.\n",
    "    \"\"\"\n",
    "    # this code adapted from the tqdm examples\n",
    "    # https://github.com/tqdm/tqdm/blob/master/examples/tqdm_requests.py\n",
    "    if os.path.exists(fname):\n",
    "        return\n",
    "    response = requests.get(url, stream=True)\n",
    "    with tqdm.wrapattr(\n",
    "        open(fname, \"wb\"),\n",
    "        \"write\",\n",
    "        unit=\"B\",\n",
    "        unit_scale=True,\n",
    "        unit_divisor=1024,\n",
    "        miniters=1,\n",
    "        desc=fname,\n",
    "        total=int(response.headers.get(\"content-length\", 0)),\n",
    "    ) as fout:\n",
    "        for chunk in response.iter_content(chunk_size=4096):\n",
    "            fout.write(chunk)\n",
    "\n",
    "\n",
    "def download_taxi_data(n):\n",
    "    \"\"\"\n",
    "    Download `n` months of taxi data.\n",
    "    \"\"\"\n",
    "    base = \"https://d37ci6vzurychx.cloudfront.net/trip-data/\"\n",
    "    fname = \"yellow_tripdata_2021-{i:02d}.parquet\"\n",
    "    url = base + fname\n",
    "    for i in range(1, n + 1):\n",
    "        download(url.format(i=i), fname.format(i=i))\n",
    "\n",
    "\n",
    "def download_taxi_zones():\n",
    "    download(\n",
    "        \"https://gist.githubusercontent.com/shwina/72d79165ce9605d8f6e3378ae717b16b/raw/84a47bc587c99c6736f38a97f9dcc32ba8f89b05/taxi_zones.csv\",\n",
    "        \"taxi_zones.csv\",\n",
    "    )\n",
    "\n",
    "\n",
    "# adjust n between 1-12 depending on the size of analysis\n",
    "download_taxi_data(n=6)\n",
    "download_taxi_zones()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1: Loading and Preparing Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our dataset is a collection of Parquet files containing NYC taxi trip data. Another smaller dataset `taxi_zones.csv` contains some geographical information that we'll use later:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Reading the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we'll read the Parquet data into a Pandas dataframe using the `pandas.read_parquet()` function. Then, we'll see how to do the same thing with cuDF."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: Use Pandas to read all the parquet files into a variable\n",
    "# named `df`. Use cUDF to do the same thing, and save the result to a\n",
    "# variable named `gdf`\n",
    "\n",
    "df = pd.read_parquet(sorted(list(glob.glob(\"*.parquet\"))))\n",
    "gdf = cudf.read_parquet(sorted(list(glob.glob(\"*.parquet\"))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "aYcFIiSnRwUY",
    "outputId": "6872a33a-da7c-4609-8cd8-b1ec029311c0",
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(type(df))\n",
    "print(type(gdf))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "display(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "display(gdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A few things to note:\n",
    "\n",
    "- cuDF offers a Pandas-like API. It doesn't require you to learn a new library to take advantage of the GPU.\n",
    "- cuDF has a `cudf.DataFrame` type that is analogous to `pd.DataFrame`. The primary difference between the two is that `cudf.DataFrame` lives on the GPU and any operations on it utilize the GPU rather than the CPU (and are hence much faster)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "svYLGK5JRwUZ"
   },
   "source": [
    "### 1.2 Data Cleanup\n",
    "\n",
    "Next, we're going to do some preliminary cleanup of our data.\n",
    "\n",
    "We've defined a helper function called `clean_columns` below. This helper function accepts a Pandas DataFrame as its first parameter, and does four things:\n",
    "\n",
    "1. Converts the column names to all lowercase, and removes any extraneous spaces in the column names.\n",
    "2. Discards columns that we won't use in our analysis.\n",
    "3. Casts columns to the appropriate data type.\n",
    "3. Replace missing values (\"nulls\") in columns with a sentinel value -1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In particular, note the operations invoked on the input DataFrame `df`:\n",
    "\n",
    "- `df.columns`\n",
    "- `df.rename`\n",
    "- `df.drop`\n",
    "- `df[]` (indexing operation returning a `Series`)\n",
    "- `Series.fillna()`\n",
    "- `Series.asype()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def clean_columns(df, columns_to_keep, column_renames):\n",
    "    \"\"\"\n",
    "    Perform column cleanup on the input DataFrame `df`.\n",
    "    Drop any columns not present in `columns_to_keep`.\n",
    "    Then, rename columns according to the mapping `column_renames`.\n",
    "    Finally, cast any numeric columns from 64-bit to 32-bit data types,\n",
    "    while filling any nulls that may be present with the sentinel\n",
    "    value -1.\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    df: DataFrame\n",
    "        Input dataframe\n",
    "    columns_to_keep: list\n",
    "        List of columns to keep in the result\n",
    "    column_renames: dict\n",
    "        Mapping that specifies the column renaming to apply\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    DataFrame\n",
    "    \"\"\"\n",
    "    # rename columns\n",
    "    colname_cleanup = {col: col.strip().lower() for col in df.columns}\n",
    "    df = df.rename(columns=colname_cleanup)\n",
    "    df = df.rename(column_renames, axis=1)\n",
    "\n",
    "    # Simplify the payment_type column\n",
    "    df[\"is_credit_card\"] = df[\"payment_type\"] == 1\n",
    "\n",
    "    # Drop unwanted columns, and cast data down from\n",
    "    # 64-bit type to 32-bit type when possible\n",
    "    for col in df.columns:\n",
    "        if col not in columns_to_keep:\n",
    "            print(f\"Dropping ({col})\")\n",
    "            df = df.drop(columns=col)\n",
    "            continue\n",
    "\n",
    "        # cast int64->int32, float64->float32\n",
    "        # and fill nulls with -1\n",
    "        dtype = df[col].dtype\n",
    "        if dtype.kind in {\"i\", \"f\"}:\n",
    "            if dtype.itemsize == 8:\n",
    "                df[col] = df[col].astype(dtype.kind + str(dtype.itemsize))\n",
    "            df[col] = df[col].fillna(-1)\n",
    "\n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's call the `clean_columns` on our Pandas dataframe `df`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "columns_to_keep = [\n",
    "    \"pickup_datetime\",\n",
    "    \"dropoff_datetime\",\n",
    "    \"passenger_count\",\n",
    "    \"pickup_longitude\",\n",
    "    \"pickup_latitude\",\n",
    "    \"rate_code\",\n",
    "    \"fare_amount\",\n",
    "    \"pickup_location\",\n",
    "    \"dropoff_location\",\n",
    "    \"is_credit_card\",\n",
    "    \"airport_fee\",\n",
    "]\n",
    "\n",
    "column_renames = {\n",
    "    \"tpep_pickup_datetime\": \"pickup_datetime\",\n",
    "    \"tpep_dropoff_datetime\": \"dropoff_datetime\",\n",
    "    \"ratecodeid\": \"rate_code\",\n",
    "    \"pulocationid\": \"pickup_location\",\n",
    "    \"dolocationid\": \"dropoff_location\",\n",
    "}\n",
    "\n",
    "df = clean_columns(df, columns_to_keep, column_renames)\n",
    "display(df.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1c2NSk3oRwUa",
    "outputId": "65a2cd9b-a60d-4fba-8115-1ca99cfc4b14",
    "tags": []
   },
   "source": [
    "What about `gdf`? How can we do the same cleanup on `gdf` as we did with `df`? Do we need to define a new helper function that can work with cuDF DataFrames?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "1Gwi2R3kRwUa",
    "outputId": "d4bf2198-5598-47f6-cadb-b31c2456b946",
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "# TODO: do the same cleanup on `gdf` as we did with `df`\n",
    "\n",
    "gdf = clean_columns(gdf, columns_to_keep, column_renames)\n",
    "display(gdf.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that `clean_columns` can be used with a cuDF DataFrame without any changes! This is because cuDF DataFrames support many of the same operations (like `drop`, `rename`, `astype`, `fillna`) as Pandas DataFrames."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mc22qD3DRwUb",
    "tags": []
   },
   "source": [
    "### 1.3 Visualizing and filtering the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "The next step in our analysis is to identify any statistical anomalies in the data, e.g., data that was entered incorrectly due to human error?\n",
    "\n",
    "Data Visualization is a great tool to find these kinds of issues.\n",
    "\n",
    "As an example, we'll use the [Seaborn](https://seaborn.pydata.org/) library to make a boxplot from our data, which will show us the outliers in the dataset. \n",
    "\n",
    "Seaborn does not currently support cuDF DataFrames as input, so we need to use the `.to_pandas()` method to convert to a Pandas DataFrame before passing it to Seaborn.\n",
    "\n",
    "For a more general guide on how to use RAPIDS with most of the popular visualization libraries, see the viz gallery in our docs: https://docs.rapids.ai/visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "pVs3cuccRwUb",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# TODO: using the cuDF DataFrame gdf, make a boxplot\n",
    "# of the fare_amount to passenger_count using seaborn\n",
    "\n",
    "sns.boxplot(x=\"passenger_count\", y=\"fare_amount\", data=gdf.to_pandas())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset clearly has some significant outliers. Let's filter the data to throw out some of the outliers and also to remove missing values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "UyjuYoWRRwUc",
    "outputId": "c5d7d545-6aa2-4244-a712-1a9f63eb3f3f",
    "tags": []
   },
   "outputs": [],
   "source": [
    "query_frags = [\n",
    "    \"fare_amount > 0\",\n",
    "    \"fare_amount < 500\",\n",
    "    \"passenger_count > 0\",\n",
    "    \"passenger_count < 6\",\n",
    "]\n",
    "query = \" and \".join(query_frags)\n",
    "display(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "UyjuYoWRRwUc",
    "outputId": "c5d7d545-6aa2-4244-a712-1a9f63eb3f3f",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# TODO: apply the query above to the DataFrame `gdf`\n",
    "# remember to reset the index afterwards.\n",
    "# (any time you filter the dataframe, it's a good idea to reset its index)\n",
    "\n",
    "\n",
    "gdf = gdf.query(query)\n",
    "gdf = gdf.reset_index(drop=True)\n",
    "\n",
    "# inspect the results of cleaning\n",
    "display(gdf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sns.boxplot(x=\"passenger_count\", y=\"fare_amount\", data=gdf.to_pandas())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've gotten rid of the more obvious outliers, we can continue with the rest of the analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4 Takeaways\n",
    "\n",
    "- cuDF offers similar classes, functions, and methods to Pandas\n",
    "  - This includes functions for reading and writing data!\n",
    "- cuDF DataFrames and other objects utilize GPU memory (rather than CPU memory or \"RAM\")\n",
    "- Code that is written to use Pandas can easily be changed to use cuDF instead\n",
    "- You can use cuDF with third-party libraries that work with Pandas, using the `.to_pandas()` and `.from_pandas()` functions\n",
    "- By virtue of using the GPU, cuDF can be much faster than Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IieCGSP2RwUc"
   },
   "source": [
    "## Part 2: Feature engineering and user-defined functions\n",
    "\n",
    "We're going to add a few more features to our dataset, and show off some of the more complex functionality offered by cuDF.\n",
    "\n",
    "1. Our dataset has the \"location IDs\" to represent the pickup and dropoff locations. We would like the actual coordinates (latitude and longitude) of the pickup and dropoff locations. Of course, that information will come from another dataset (`taxi_zones.csv`). We'll see how to use cuDF to combine the two datasets.\n",
    "\n",
    "2. Next, we'll compute the trip distance from the pickup and dropoff coordinates using the [Haversine Distance Formula](https://en.wikipedia.org/wiki/Haversine_formula).\n",
    "\n",
    "3. Finally, we'll extract additional useful variables from the `pickup_datetime` field using cuDF's datetime functionality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "gdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Combining datasets with `merge()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data in `taxi_zones.csv` contains the coordinates for each location. The `'x'` and `'y'` columns refer to the longitute and latitude respectively of each location:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "zones = cudf.read_csv(\"taxi_zones.csv\")\n",
    "zones"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we can combine `zones` and `gdf`, we have to do some pre-processing of both. The helper function below discards the rows in `zones` that have duplicate location information. It also discards rows in `gdf` for which the `pickup_location` and `dropoff_location` cannot be found in `zones`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def drop_rows_with_repeated_or_missing_locations(zones, df):\n",
    "    dup_rows = zones[(zones[\"LocationID\"] == 103) | (zones[\"LocationID\"] == 56)]\n",
    "    zones = zones.drop(dup_rows.index, axis=0)\n",
    "    zones = zones.reset_index(drop=True)\n",
    "\n",
    "    df = df[df[\"pickup_location\"].isin(zones[\"LocationID\"])]\n",
    "    df = df[df[\"dropoff_location\"].isin(zones[\"LocationID\"])]\n",
    "    df = df.reset_index(drop=True)\n",
    "    return zones, df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "zones, gdf = drop_rows_with_repeated_or_missing_locations(zones, gdf)\n",
    "zones = zones.set_index(\"LocationID\")\n",
    "display(zones)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have the coordinate data in `zones` for every pickup and dropoff location in our main dataset `gdf`. Let's use a merge (\"join\") to combine the two datasets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "# TODO: merge `gdf` with `zones`, using the `pickup_location` column\n",
    "# and the index of `zones` respectively as the merge keys\n",
    "# the result should contain two columns 'x' and `y`, representing\n",
    "# the coordinates (longitude and lattitude respectively) of each\n",
    "# pickup location\n",
    "\n",
    "gdf.merge(zones, left_on=\"pickup_location\", right_index=True, how=\"left\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's compare how long Pandas takes to do the same merge:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df = gdf.to_pandas()\n",
    "zones_cpu = zones.to_pandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "df.merge(zones_cpu, left_on=\"pickup_location\", right_index=True, how=\"left\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A couple of things to note:\n",
    "\n",
    "1. The merge operation on the GPU is _fast_ (about 20x faster than CPU).\n",
    "\n",
    "2. Unlike Pandas, `merge()` in cuDF returns rows in a non-deterministic order. In particular, the original ordering of the join keys is not preserved. We can recover the original ordering by sorting the resulting DataFrame by its index."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below, we define a helper function that adds both pickup and dropoff coordinates `gdf`, as well as the borough name corresponding to the pickup and dropoff locations (a total of 6 new columns):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_pickup_and_dropoff_info(df, zones):\n",
    "    pickup_info = (\n",
    "        df[[\"pickup_location\"]]\n",
    "        .merge(\n",
    "            zones[[\"x\", \"y\", \"borough\"]],\n",
    "            left_on=\"pickup_location\",\n",
    "            right_index=True,\n",
    "            how=\"left\",\n",
    "        )\n",
    "        .sort_index()\n",
    "    )\n",
    "    dropoff_info = (\n",
    "        df[[\"dropoff_location\"]]\n",
    "        .merge(\n",
    "            zones[[\"x\", \"y\", \"borough\"]],\n",
    "            left_on=\"dropoff_location\",\n",
    "            right_index=True,\n",
    "            how=\"left\",\n",
    "        )\n",
    "        .sort_index()\n",
    "    )\n",
    "    df[\"pickup_latitude\"] = pickup_info[\"y\"]\n",
    "    df[\"pickup_longitude\"] = pickup_info[\"x\"]\n",
    "    df[\"pickup_borough\"] = pickup_info[\"borough\"]\n",
    "    df[\"dropoff_latitude\"] = dropoff_info[\"y\"]\n",
    "    df[\"dropoff_longitude\"] = dropoff_info[\"x\"]\n",
    "    df[\"dropoff_borough\"] = dropoff_info[\"borough\"]\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "gdf = add_pickup_and_dropoff_info(gdf, zones)\n",
    "display(gdf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df = gdf.to_pandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OxpuZnFWRwUd"
   },
   "source": [
    "### 2.2 Computing the trip distance using a user-defined function: `apply()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "The Haversine Distance formula gives the distance between two points on a sphere:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Haversine distance](https://upload.wikimedia.org/wikipedia/commons/c/cb/Illustration_of_great-circle_distance.svg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'd like to use the haversine distance between the pickup and dropoff coordinates as a measure of trip distance. Of course, this is a (bad) approximation, as most taxi trips are not a straight line. But it is enough to yield some useful insights later, and helps illustrate how to use a user-defined function with cuDF.\n",
    "\n",
    "**Note**: The original dataset does contain the actual trip distance; we filtered it out early in this notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's define a function that accepts a single record (row) from our dataframe, and computes the Haversine distance for that trip."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from math import asin, cos, pi, sin, sqrt\n",
    "\n",
    "\n",
    "def haversine_distance(row):\n",
    "    x_1, y_1, x_2, y_2 = (\n",
    "        row[\"pickup_latitude\"],\n",
    "        row[\"pickup_longitude\"],\n",
    "        row[\"dropoff_latitude\"],\n",
    "        row[\"dropoff_longitude\"],\n",
    "    )\n",
    "    x_1 = pi / 180 * x_1\n",
    "    y_1 = pi / 180 * y_1\n",
    "    x_2 = pi / 180 * x_2\n",
    "    y_2 = pi / 180 * y_2\n",
    "\n",
    "    dlon = y_2 - y_1\n",
    "    dlat = x_2 - x_1\n",
    "    a = sin(dlat / 2) ** 2 + cos(x_1) * cos(x_2) * sin(dlon / 2) ** 2\n",
    "\n",
    "    c = 2 * asin(sqrt(a))\n",
    "    r = 6371  # Radius of earth in kilometers\n",
    "\n",
    "    return c * r"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "# TODO: Apply the Haversine distance function to each row\n",
    "# of the Pandas DataFrame `df`, saving the result to a new\n",
    "# column \"h_distance\". Hint: use the `apply()` function\n",
    "# with `axis=1`.\n",
    "#\n",
    "# This can take a while to run (a couple of minutes)\n",
    "# df[\"h_distance\"] = df.apply(haversine_distance, axis=1)\n",
    "\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "# need to remove string columns before using `apply()`\n",
    "# in cuDF\n",
    "gdf_numeric = gdf.select_dtypes(exclude=\"object\")\n",
    "gdf[\"h_distance\"] = gdf_numeric.apply(haversine_distance, axis=1)\n",
    "gdf.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The call to `apply()` is much faster on the GPU.\n",
    "\n",
    "In fact, much of the time spent in the call to `apply()` is in JIT compilation. If you run the UDF again, we'll see that it takes a very small fraction of time compared to the first run."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more about user-defined functions in cuDF, features and limitations, see the [cuDF documentation](https://docs.rapids.ai/api/cudf/stable/user_guide/guide-to-udfs.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eNGgW9q7RwUe"
   },
   "source": [
    "For advanced GPU-accelerated spatial calculations, check out [cuSpatial](https://medium.com/rapids-ai/releasing-cuspatial-to-accelerate-geospatial-and-spatiotemporal-processing-b686d8b32a9).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 Extracting datetime features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we'll use the datetime functionality provided by cuDF to extract some features from the `pickup_datetime` column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "qwtLbAyWRwUc",
    "tags": []
   },
   "outputs": [],
   "source": [
    "def add_datetime_features(df):\n",
    "    df[\"hour\"] = df[\"pickup_datetime\"].dt.hour\n",
    "    df[\"year\"] = df[\"pickup_datetime\"].dt.year\n",
    "    df[\"month\"] = df[\"pickup_datetime\"].dt.month\n",
    "    df[\"day\"] = df[\"pickup_datetime\"].dt.day\n",
    "\n",
    "    df[\"day_of_week\"] = df[\"pickup_datetime\"].dt.dayofweek\n",
    "\n",
    "    df = df.drop(columns=[\"pickup_datetime\", \"dropoff_datetime\"])\n",
    "\n",
    "    df[\"is_weekend\"] = (df[\"day_of_week\"] >= 5).astype(np.int32)\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "gdf = add_datetime_features(gdf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "display(gdf.tail())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df = gdf.to_pandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 Takeaways\n",
    "\n",
    "- In addition to supporting simple operations like selecting, sorting and filtering data, cuDF has GPU-accelerated support for complex operations like `join` (or `merge`), `groupby`, rolling window calculations, etc.,\n",
    "\n",
    "- Custom operations using user-defined functions (UDFs) and the `apply()` function are supported. This can be extremely convenient and fast, but cuDF has many more limitations compared to Pandas when it comes to UDFs.\n",
    "\n",
    "- In addition to numeric data types, cuDF has rich support for datetimes, categoricals, strings, and even \"nested\" types like lists."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IMABXzlxRwUf"
   },
   "source": [
    "# Section 3: Exploratory Analysis and Machine Learning\n",
    "\n",
    "Let's say we're studying consumer behavior on behalf of the taxi commission. Are there a couple of clear \"types\" of rides that come up again and again? What are some of the key patterns we see in the data? Can we reliably predict some elements of user behavior?\n",
    "\n",
    "Let's just do this on GPU, because it's faster and the code only differs by using `cuml.` instead of `sklearn.`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "G7zrB6uXRwUf",
    "tags": []
   },
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Unsupervised learning - UMAP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll use the UMAP algorithm to explore the data in an unsupervised manner. UMAP takes high-dimensional data, like our taxi dataset with 20 features, and maps it down to a lower dimensional space so we can view all of the points together in a 2d plot while preserving much of the structure of the dataset. This means that similar points will be clearly grouped together in the plot and we can see how patterns change across "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "# cuML can handle the full dataset nicely, but it's hard to visualize a million points sometimes\n",
    "# so let's start with a subset. Note that doing UMAP over 100k points on CPU can easily take minutes\n",
    "gdf_sample = gdf.sample(100_000).reset_index()\n",
    "\n",
    "# Fit\n",
    "umap = cuml.manifold.UMAP()\n",
    "umap_out = umap.fit_transform(\n",
    "    cuml.preprocessing.RobustScaler().fit_transform(\n",
    "        gdf_sample.drop([\"pickup_borough\", \"dropoff_borough\"], axis=1)\n",
    "    )\n",
    ")\n",
    "\n",
    "gdf_sample[\"umap_x\"] = umap_out.iloc[:, 0]\n",
    "gdf_sample[\"umap_y\"] = umap_out.iloc[:, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import holoviews as hv\n",
    "import hvplot.cudf\n",
    "\n",
    "# Let's use hvplot to plot our UMAP results along with some useful ancillary data\n",
    "# Because hvplot has native support for cudf (via hvplot.cudf), we don't have to do any conversion\n",
    "gdf_sample.hvplot.scatter(\n",
    "    x=\"umap_x\",\n",
    "    y=\"umap_y\",\n",
    "    color=[\"is_credit_card\"],\n",
    "    groupby=[\"day_of_week\"],\n",
    "    hover_cols=[\n",
    "        \"pickup_latitude\",\n",
    "        \"pickup_longitude\",\n",
    "        \"pickup_borough\",\n",
    "        \"airport_fee\",\n",
    "        \"h_distance\",\n",
    "        \"fare_amount\",\n",
    "        \"pickup_location\",\n",
    "        \"is_credit_card\",\n",
    "        \"dropoff_borough\",\n",
    "    ],\n",
    "    s=0.5,\n",
    ").opts(title=\"Taxi Data\", width=700, height=500, cmap=\"Category10\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9avYs6sVRwUe",
    "tags": []
   },
   "source": [
    "### 3.2 Build a supervised model to predict payment form\n",
    "\n",
    "Let's imagine you're a taxi operator trying to understand consumer payment behavior. What types of rides will be paid by credit card and what types via cash? Let's build a predictive model to help us understand.\n",
    "\n",
    "#### Splitting train and test data\n",
    "\n",
    "We'll want to know how accurate our model is, so let's start by splitting the dataset into \"train\" and \"test\" subsets randomly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# We'll use a subset of the data because the kernel crashes using scikit-learn on the full dataset\n",
    "df = df.sample(1_000_000).reset_index(drop=True)\n",
    "gdf = gdf.sample(1_000_000).reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%time\n",
    "\n",
    "# First let's split using Pandas and scikit-learn\n",
    "\n",
    "y_df = df[\"is_credit_card\"]\n",
    "X_df = df.drop(columns=[\"is_credit_card\", \"pickup_borough\", \"dropoff_borough\"])\n",
    "\n",
    "# Split our dataframes\n",
    "X_train_df, X_test_df, y_train_df, y_test_df = sklearn.model_selection.train_test_split(\n",
    "    X_df, y_df\n",
    ")\n",
    "\n",
    "# Create array versions of these dataframes\n",
    "X_train_np, X_test_np = (\n",
    "    X_train_df.to_numpy(np.float32),\n",
    "    X_test_df.to_numpy(np.float32),\n",
    ")\n",
    "y_train_np, y_test_np = (\n",
    "    y_train_df.to_numpy(np.float32),\n",
    "    y_test_df.to_numpy(np.float32),\n",
    ")\n",
    "\n",
    "len(X_train_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we think of ML, we often think of running large predictive models. But scikit-learn also provides a huge toolbox of utilities like preprocessing, data splitting, and more that are essential for ML practitioners. cuML contains GPU-based analogues of these utilities so that your data does not need to take a round trip back to CPU just to partition it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "# Now do the same on GPU\n",
    "\n",
    "y_gdf = gdf[\"is_credit_card\"]\n",
    "X_gdf = gdf.drop(columns=[\"is_credit_card\", \"pickup_borough\", \"dropoff_borough\"])\n",
    "\n",
    "# Split our dataframes\n",
    "(\n",
    "    X_train_gdf,\n",
    "    X_test_gdf,\n",
    "    y_train_gdf,\n",
    "    y_test_gdf,\n",
    ") = cuml.model_selection.train_test_split(X_gdf, y_gdf)\n",
    "\n",
    "# Create array versions of these dataframes\n",
    "X_train_gpu, X_test_gpu = (\n",
    "    X_train_gdf.to_cupy(np.float32),\n",
    "    X_test_gdf.to_cupy(np.float32),\n",
    ")\n",
    "y_train_gpu, y_test_gpu = (\n",
    "    y_train_gdf.to_cupy(np.float32),\n",
    "    y_test_gdf.to_cupy(np.float32),\n",
    ")\n",
    "\n",
    "len(X_train_gdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2sXlkFcHRwUf"
   },
   "source": [
    "#### Fit a simple supervised model with cuML\n",
    "\n",
    "cuML supports a large range of supervised models, all emulating the scikit-learn interfaces. See the README (https://github.com/rapidsai/cuml) for a recent list. Here, we'll try a very common but powerful model - a random forest ensemble for classification. Specifically, we want to classify whether the trip was paid for by credit card (the positive class) or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Hkv9TO8-RwUf"
   },
   "outputs": [],
   "source": [
    "from cuml.ensemble import RandomForestClassifier as cuRandomForestClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier as skRandomForestClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2ycwFRHbRwUg",
    "outputId": "8a8e8fdd-0fec-4441-db56-584baa3b0657"
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "# Scikit-learn will parallelize over all CPU cores with n_jobs=-1\n",
    "\n",
    "sk_model = skRandomForestClassifier(n_estimators=250, n_jobs=-1)\n",
    "sk_model.fit(X_train_np, y_train_np)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Fpeth5n6RwUg",
    "outputId": "efb47e42-9408-455b-8421-aa82c6b1c59d"
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "# TODO: Build a similar model on GPU with cuML\n",
    "# This is RandomForest estimator builds 250 separate trees, each of which tries to predict whether or\n",
    "# not a transaction uses a credit card, then averages the results together. Note again that its' the same\n",
    "# API as sklearn but significantly faster. These speedups typically get greater as the dataset grows.\n",
    "\n",
    "cuml_model = cuRandomForestClassifier(n_estimators=250)\n",
    "cuml_model.fit(X_train_gpu, y_train_gpu)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "RomZDHWdRwUg",
    "outputId": "60e6d468-1b01-4088-e05f-0a574163b07e",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Let's use the model to predict from the test set and evaluate the predictions' accuracy\n",
    "rf_predictions = cuml_model.predict(X_test_gpu)\n",
    "\n",
    "# Compute the probability that the transaction used a credit card, according to our model\n",
    "rf_probabilities = cuml_model.predict_proba(X_test_gpu)[:, 1]\n",
    "\n",
    "## Just as cuML provides utilities for data preprocessing and splitting, it also provides a\n",
    "## wide set of GPU-accelerated metrics, like accuracy and AUC score. These are fast and\n",
    "## ensure you don't need to fall back to CPU for any of your pipeline.\n",
    "print(\"Accuracy: \", cuml.metrics.accuracy.accuracy_score(y_test_gpu, rf_predictions))\n",
    "print(\"AUC: \", cuml.metrics.roc_auc_score(y_test_gpu, rf_probabilities))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, so we're right about 75% of the time about whether the transaction will use a credit card. The AUC (area under the curve) statistic shows us in a bit more detail that our model has value\n",
    "but isn't particularly amazing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Takeaways"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- cuML provides functionality and API similar to the popular `scikit-learn` library for machine learning\n",
    "- In conjunction with tools like hvPlot, cuML allows you to explore and find patterns in large amounts of data using unsupervised learning techniques\n",
    "- cuML not only provides powerful, GPU-acclerated predictive models, but also useful utilities for data preprocessing and metrics, which keeps data on the GPU and accelerates every part of your machine learning pipeline"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}