notebooks/Metrics with Multiple Features.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Metrics with Multiple Features\n",
    "\n",
    "This notebook demonstrates the new API for metrics, which supports multiple sensitive and conditional features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data\n",
    "\n",
    "To demonstrate the API, we use the well-known 'Adult' dataset, and we train a simple model on it. We start with some uncontroversial `import` statements:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from sklearn.datasets import fetch_openml\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import LabelEncoder, StandardScaler"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we import the data, dropping some of the values to help maintain clarity:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = fetch_openml(data_id=1590, as_frame=True)\n",
    "X_raw = data.data\n",
    "X_raw = X_raw[ X_raw.race!='Other' ]\n",
    "Y = data.target[ data.data.race!='Other' ]\n",
    "Y = (Y == '>50K') * 1\n",
    "display(X_raw)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can select some columns which we might want to use for our sensitive and conditional features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def marriage_transform(m_s_string):\n",
    "    result = 'A'\n",
    "    if m_s_string.startswith(\"Married\"):\n",
    "        result = 'B'\n",
    "    elif m_s_string.startswith(\"Widowed\"):\n",
    "        result = 'C'\n",
    "    return result\n",
    "\n",
    "def occupation_transform(occ_string):\n",
    "    result = 'pp'\n",
    "    if occ_string.startswith(\"Machine\"):\n",
    "        result = 'qq'\n",
    "    return result\n",
    "    \n",
    "colA = X_raw['marital-status'].map(marriage_transform).fillna('C')\n",
    "colA.name=\"Feature A\"\n",
    "colB = X_raw['occupation'].map(occupation_transform).fillna('qq')\n",
    "colB.name=\"Feature B\"\n",
    "\n",
    "A = X_raw[['race', 'sex']]\n",
    "A['Feature A'] = colA\n",
    "A['Feature B'] = colB\n",
    "display(A)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the data imported, we perform some standard processing, and a test/train split:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "le = LabelEncoder()\n",
    "Y = le.fit_transform(Y)\n",
    "\n",
    "le = LabelEncoder()\n",
    "\n",
    "sc = StandardScaler()\n",
    "X_dummies = pd.get_dummies(X_raw)\n",
    "X_scaled = sc.fit_transform(X_dummies)\n",
    "X_scaled = pd.DataFrame(X_scaled, columns=X_dummies.columns)\n",
    "\n",
    "X_train, X_test, Y_train, Y_test, A_train, A_test = \\\n",
    "        train_test_split(X_scaled, Y, A,\n",
    "                         test_size=0.3,\n",
    "                         random_state=12345,\n",
    "                         stratify=Y)\n",
    "\n",
    "# Ensure indices are aligned\n",
    "X_train = X_train.reset_index(drop=True)\n",
    "X_test = X_test.reset_index(drop=True)\n",
    "A_train = A_train.reset_index(drop=True)\n",
    "A_test = A_test.reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we train a simple model on the data, and generate some predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "unmitigated_predictor = LogisticRegression(solver='liblinear', fit_intercept=True)\n",
    "unmitigated_predictor.fit(X_train, Y_train)\n",
    "\n",
    "Y_pred = unmitigated_predictor.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple Metrics\n",
    "\n",
    "We now start computing metrics. The new API is based around the `MetricFrame` object. The constructor specifies the data, the metric and the sensitive feature:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sklearn.metrics as skm\n",
    "\n",
    "from fairlearn.metrics.experimental import MetricFrame\n",
    "\n",
    "basic_metric = MetricFrame(skm.recall_score, Y_test, Y_pred, sensitive_features=A_test['sex'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `MetricFrame` object has properties of `overall` and `by_group`, which show the overall value of the metric (evaluated on the entire dataset), as well as the metric evaluated on each of the unique values of the specified sensitive feature:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Overall:\")\n",
    "display(basic_metric.overall)\n",
    "print(\"For comparison, calculate from recall_score:\", skm.recall_score(Y_test, Y_pred))\n",
    "\n",
    "print(\"\\nBy Group\")\n",
    "display(basic_metric.by_group)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose that we have sample weights which we want to use in the metric calculation. For this we use the `sample_params=` argument in the constructor. This contains a dictionary of arrays which need to be sliced up with the `y_true` and `y_pred` arrays, before being passed into the metric function. The dictionary keys are the names of the arguments in the metric function signature:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wgts = np.random.random(size=len(Y_test))\n",
    "\n",
    "basic_metric_wgts = MetricFrame(skm.recall_score, \n",
    "                                  Y_test, Y_pred, \n",
    "                                  sensitive_features=A_test['sex'], \n",
    "                                  sample_params={ 'sample_weight':wgts })\n",
    "\n",
    "print(\"Overall:\")\n",
    "display(basic_metric_wgts.overall)\n",
    "print(\"For comparison, calculate from recall_score:\", skm.recall_score(Y_test, Y_pred, sample_weight=wgts))\n",
    "\n",
    "print(\"\\nBy Group\")\n",
    "display(basic_metric_wgts.by_group)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the metric function has other arguments, then it will need to be wrapped. An example is `fbeta_score()` which requires a value for `beta`. The `functools.partial` routine makes this easy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import functools\n",
    "\n",
    "fbeta_05 = functools.partial(skm.fbeta_score, beta=0.5)\n",
    "\n",
    "basic_metric_wrapped = MetricFrame(fbeta_05, Y_test, Y_pred, sensitive_features=A_test['sex'])\n",
    "\n",
    "print(\"Overall\")\n",
    "display(basic_metric_wrapped.overall)\n",
    "\n",
    "print(\"\\nBy Group\")\n",
    "display(basic_metric_wrapped.by_group)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can evaluate multiple metrics at once by passing in a dictionary of metric functions. If we have sample parameters as well, then that argument becomes a dictionary of dictionaries, with the top set of keys matching those in the metrics dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "metric_dict = { 'recall':skm.recall_score, 'fbeta_0.5':fbeta_05 }\n",
    "sample_params = { 'recall':{ 'sample_weight':wgts }, 'fbeta_0.5':{ 'sample_weight':wgts } }\n",
    "\n",
    "basic_metric_two = MetricFrame(metric_dict,\n",
    "                                 Y_test, Y_pred,\n",
    "                                 sensitive_features=A_test['sex'],\n",
    "                                 sample_params=sample_params)\n",
    "\n",
    "print(\"Overall\")\n",
    "display(basic_metric_two.overall)\n",
    "\n",
    "print(\"\\nBy Group\")\n",
    "display(basic_metric_two.by_group)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Aggregates\n",
    "\n",
    "We provide some aggregating functions, which provide means of obtaining scalar measures. First are the `group_min()` and `group_max()` methods which compute the minimum and maximum values of each metric across the sensitive feature:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Group Min\")\n",
    "display(basic_metric_two.group_min())\n",
    "print(\"\\nGroup Max\")\n",
    "display(basic_metric_two.group_max())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is also a `difference()` method, which calculates the difference between the minimum and maximum. Alternatively, its `method=` argument can compute the difference relative to the overall value of the metric (returning the largest absolute value):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Basic difference\")\n",
    "display(basic_metric_two.difference())\n",
    "print(\"\\nDifference to overall\")\n",
    "display(basic_metric_two.difference(method='to_overall'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multiple Sensitive Features\n",
    "\n",
    "The new metrics are not constrained to considering a single sensitive feature at a time. Multiple columns can be passed into as senstive features, and the intersections of all subgroups will be computed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "two_sf = MetricFrame(metric_dict,\n",
    "                       Y_test, Y_pred,\n",
    "                       sensitive_features=A_test[['sex', 'race']])\n",
    "\n",
    "print(\"Overall\")\n",
    "display(two_sf.overall)\n",
    "print(\"\\nBy Group\")\n",
    "display(two_sf.by_group)\n",
    "print(\"Difference to overall\")\n",
    "display(two_sf.difference(method='to_overall'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conditional Features\n",
    "\n",
    "Conditional features denote groups for which outcomes are allowed to differ. For example, in a loan scenario, it is acceptable for people in a high income group to be offered loans more often than those in a low income group. While this could be monitored by splitting the input array, `MetricFrame` supports this directly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cond_metric = MetricFrame(skm.recall_score,\n",
    "                            Y_test, Y_pred,\n",
    "                            sensitive_features=A_test[['sex', 'race']],\n",
    "                            control_levels=A_test['Feature A'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This changes the `overall` property to be a DataFrame. The rows correspond to the unique values of the conditional feature:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display(cond_metric.overall)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `by_group` property still looks similar - indeed, we can compare it to a metric which moves the conditional feature into the sensitive feature list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display(cond_metric.by_group)\n",
    "\n",
    "cond_metric_alt = MetricFrame(skm.recall_score,\n",
    "                                Y_test, Y_pred,\n",
    "                                sensitive_features=A_test[['Feature A', 'sex', 'race']])\n",
    "print(\"\\nFor comparision\")\n",
    "display(cond_metric_alt.by_group)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The aggregates are also evaluated for each unique value of the conditional feature:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Group Max\")\n",
    "display(cond_metric.group_max())\n",
    "print(\"\\nDifference\")\n",
    "display(cond_metric.difference())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also support multiple conditional features, and evaluate multiple metrics at once:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cond_metric_two = MetricFrame(metric_dict,\n",
    "                                Y_test, Y_pred,\n",
    "                                sensitive_features=A_test[['sex', 'race']],\n",
    "                                control_levels=A_test[['Feature A', 'Feature B']])\n",
    "\n",
    "print(\"Overall\")\n",
    "display(cond_metric_two.overall)\n",
    "print(\"\\nBy Group\")\n",
    "display(cond_metric_two.by_group)\n",
    "print(\"\\nDifference to overall\")\n",
    "display(cond_metric_two.difference(method='to_overall'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scoring Functions\n",
    "\n",
    "We also provide a metafunction which can be used to create functions suitable for SciKit-Learn's `make_scorer()` routine:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fairlearn.metrics.experimental import make_derived_metric\n",
    "\n",
    "my_fn = make_derived_metric('difference', skm.recall_score)\n",
    "print(my_fn(Y_test, Y_pred, sensitive_features=A_test['sex'], sample_weight=wgts))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For comparison:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "score_compare = MetricFrame(skm.recall_score,\n",
    "                              Y_test, Y_pred,\n",
    "                              sensitive_features=A_test['sex'],\n",
    "                              sample_params={ 'sample_weight': wgts})\n",
    "print(score_compare.difference())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}