diff --git a/README.md b/README.md index 235032d..d2c32e9 100644 --- a/README.md +++ b/README.md @@ -49,7 +49,7 @@ Corrections or suggestions? Please file a [GitHub issue](https://github.com/jpha * [*Accurate Intelligible Models with Pairwise Interactions*](http://www.cs.cornell.edu/~yinlou/papers/lou-kdd13.pdf) * [*This Looks Like That: Deep Learning for Interpretable Image Recognition*](https://arxiv.org/pdf/1806.10574.pdf) -* **Links from Lecture**: +* **Links from Lecture 1**: * [Tay (bot)](https://en.wikipedia.org/wiki/Tay_(bot)) * [New York Regulator Probes UnitedHealth Algorithm for Racial Bias](https://www.wsj.com/articles/new-york-regulator-probes-unitedhealth-algorithm-for-racial-bias-11572087601) * [When a Computer Program Keeps You in Jail](https://www.nytimes.com/2017/06/13/opinion/how-computers-are-harming-criminal-justice.html) @@ -89,7 +89,7 @@ Corrections or suggestions? Please file a [GitHub issue](https://github.com/jpha * **Interpretable Machine Learning** - [Chapter 5](https://christophm.github.io/interpretable-ml-book/agnostic.html) and [Chapter 6](https://christophm.github.io/interpretable-ml-book/example-based.html) * [*Towards Better Understanding of Gradient-based Attribution Methods for Deep Neural Networks*](https://arxiv.org/pdf/1711.06104.pdf) -* **Links from Lecture**: +* **Links from Lecture 2**: * [On the Art and Science of Explainable Machine Learning](https://arxiv.org/pdf/1810.02909.pdf) * [Access Denied: Faulty Automated Background Checks Freeze Out Renters](https://themarkup.org/locked-out/2020/05/28/access-denied-faulty-automated-background-checks-freeze-out-renters) * [ML Attack Cheatsheet](https://github.com/jphall663/secure_ML_ideas/blob/master/img/cheatsheet.png) @@ -102,6 +102,10 @@ Corrections or suggestions? Please file a [GitHub issue](https://github.com/jpha ## Lecture 3: Discrimination Testing and Remediation +![Two hundred neural networks from a random grid search trained on the UCI Credit Card Default dataset](/img/lecture_3.png) +**Source:** [Lecture 3 Notes](tex/lecture_3.pdf) + + ### Lecture 3 Class Materials * [Lecture Notes]() @@ -119,9 +123,9 @@ Corrections or suggestions? Please file a [GitHub issue](https://github.com/jpha * **Introduction and Background**: -* **Post-hoc Explanation Techniques**: +* **Discrimination Testing and Remediation Techniques**: -* **Links from Lecture**: +* **Links from Lecture 3**: diff --git a/img/blueprint.png b/img/blueprint.png deleted file mode 100644 index e00313f..0000000 Binary files a/img/blueprint.png and /dev/null differ diff --git a/img/lecture_3.png b/img/lecture_3.png new file mode 100644 index 0000000..5e26717 Binary files /dev/null and b/img/lecture_3.png differ diff --git a/img/trust_understanding.png b/img/trust_understanding.png deleted file mode 100644 index 94de810..0000000 Binary files a/img/trust_understanding.png and /dev/null differ diff --git a/tests/200_ANNs_acc_air.ipynb b/tests/200_ANNs_acc_air.ipynb new file mode 100644 index 0000000..340de7f --- /dev/null +++ b/tests/200_ANNs_acc_air.ipynb @@ -0,0 +1,159 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": 20, + "metadata": {}, + "outputs": [], + "source": [ + "import pandas as pd\n", + "import os\n", + "import matplotlib.pyplot as plt\n", + "\n", + "%matplotlib inline" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
accair
00.762800.860397
10.760450.848089
20.758400.863557
30.752000.813363
40.737600.942831
\n", + "
" + ], + "text/plain": [ + " acc air\n", + "0 0.76280 0.860397\n", + "1 0.76045 0.848089\n", + "2 0.75840 0.863557\n", + "3 0.75200 0.813363\n", + "4 0.73760 0.942831" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "df = pd.DataFrame(columns=['acc', 'air'])\n", + "\n", + "for time_stamp in ['2020-05-15-10:45:26', '2020-05-15-21:07:03', '2020-05-16-09:19:32', '2020-05-16-20:18:11', \n", + " '2020-05-17-09:21:18']:\n", + " \n", + " path = 'out-friedman-mlp-' + time_stamp + os.sep + 'friedman-mlp-' + time_stamp + '.csv'\n", + " df = df.append(pd.read_csv(path)[['acc', 'air']], ignore_index=True)\n", + " \n", + "df.head()" + ] + }, + { + "cell_type": "code", + "execution_count": 39, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgMAAAHwCAYAAAA7NWx9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XucXHV9//H3Z2Y3m0BAQoJVcoEgCQgUom5BwGLkJmIItlAI3qBVgxdQFARUUARFvGCpP5GSWmqLF4ikakAsBSWgCDRLSSKJXEKoZAMqhCSQEHazu5/fH+fM5uzszO6Z3Tl7Jvt9PR+Pw8ycOefMZ06G/X7O93bM3QUAAMJVyDsAAACQL5IBAAACRzIAAEDgSAYAAAgcyQAAAIEjGQAAIHAkA8AOyszGmpmb2ZRGPF7WzOzDZnZX3nEM14523jE6kQxgVDKzzYmlx8y2Jl6/J6PPvMrMvlu2btT8oTezB8xsi5m9JrFujpk9mmdclZjZ/vF5/8+y9beY2cUpj/FHM3tLNhECjYVkAKOSu48vLZKelnRSYt0P8o5vJJhZMYPDviLpsxkctw8za6rDYXokzTazN9XhWJmo0/cEho1kAMExs/Fm9oqZ7Rq/vsLMOsxsXPz662Z2Vfx8dzP7oZk9Z2ZPmdmFZmbD+OybzOySxOsTzGx14vUfzeyTZvaImW0ysx+Y2ZjE+58zsz+ZWbuk91Y49rfM7L/NbIukw81snJldY2Zr42P/PzNrSXO8Kq6R9PdmNq3K95tqZj8zs+fNbI2ZfbjG736Bma2U9GK87vPxeX8pPifvTBFjiUu6WtKXqm1gZn9jZivMbKOZ/drMDojX/1jSqyX9d1yb9HEzu9nMPha//7q45uED8esDzeyPieN+zMyeNLP1ZvafZvYX8fpSTdFHzOxJSY9UiOlt8b/XkWZWNLNr49/fJjNbbmb71XAOgFRIBhAcd98saYWkv45XvVVSu6Q3J17fEz//Z0nNkqZLOk7SRyS9O+MQT5V0jKR9JR1W+jwze5ekj8bx7S/pHRX2fa+kSyXtImmppG9KmiLpLyXtJ2mmpItrOF65pyTdKOnz5W/ENRG3S/qtpD0lnSDps2b21hTHLTld0XmeGL9+TNIRkl4l6auSbjKzSTUc758kvalSdb+ZvVnSdyT9ffx5N0r6qZk1ufvfSfqzpOPj2qRvKfpNzI53f6ukNZKOSry+Jz7uiYr+Df5G0mRJz8fHTpoj6U2S3lAW01xJ/y5prrvfF2/3RkmvkzRB0W9hQw3fH0iFZAChukfSW+Or5BmSrotf7yLpYEn3xe+dIukid9/s7qsVXRm/b4Djvi++ytxoZhsl/XGAbav5R3f/k7s/p6hwnRWvP03Sv7j7o3FC88UK+97i7g+6e4+kbkkfkPQJd9/o7pskXSVpXg3Hq+RLkk43sxll698iaay7f9XdO939cUn/lvi8NP7R3Z9x962S5O43u/uz7t7j7jdKWqeoEE0l/l5XqXLtwNmSvu3uD7l7t7svkNQywPHv0fbC/6j4uMmEspRAvkfSAndf4e6vSLpQ0rHJvhaSvhz/m2xNrHuPouTleHd/OF63TdKuipI1uftKd/9zmu8O1IJkAKEqXeUdJqlN0q8U/UE/UtLv3P1FSa9R9P/I04n9/qDoaq+aG919t9ISH6NWyQTiZUnj4+d7SlpbFku55Pt7KqrVWJlITn6qqPo77fH6cfdnJC2QdFnZW3tJ2rssGfqUajsHyXhkZh9IVONvVFRbUkvNgBRd/e9rZsdViPezZfHuoer/vqskFeKmhLdI+omkzWa2t6LkoJQM7KnEuXT3jYqaPZLH7fM9Y5+S9H13T3bI/IWkf5V0vaQ/mtl3zGx8hX2BYSEZQKh+I+kQSe9U9Ed8maKrr+O1/Y/6HxV1Qku2j09TdHU6VFsk7ZR4XUtB+aykqWWxlEvehvRZSV2SXpdIUF7l7hMT7w92vGquUlSFfVBi3VpJjyaTIXffxd3/Jn4/zXfvjd/MZkr6f5LmS9o9Tq5WS6qpz0Z8df4l9a8dWCvp82Xx7uTupREIXnYcl3SvoqaYV9z9eUW/lfmSmhQlC5L0jKJEo/Q9dlN0dZ/83VS6XezfSHpvsp+FR77p7m9QVGN1iKRPpP/2QDokAwhSfLW2UlEfgHviavU2SR9UnAy4e4eiq78rzWxnM3udoj/E3x/GRy+TNMfMdjOzyZLOrWHfhZI+aGYz46vDfu32Se6+TdINkv7JzCZZZGriCrmm45Ud+zlJ35L06cTq30iSmZ0Xd5RrMrODzeyN8fu1fvfxipKx5xRdkX9YUc3AUPyroqv+tyXWLZB0rpm1xudmvJnNNbNSwvInSfuUHeceSedoe8K4JH59r2+/H/yPJH3IzA4ys7GKEqdfuftgTUZPK+or8lkz+wcp6tcQx9ekKJnqVHROgLoiGUDI7lF0lfm/idc7Ky7UYmfHj39Q1JTwXUnDGZp4g6Kr26cl3aao4EjF3X+iqAD7taRHJd2RYrfzFF2ptknaJOm/FBeoQzxe0jeUuEqPk48TFXX4+4OiQvw6bW/mqOm7u/v/KurA2aaoFmN6/LxmcWyXSdo9se4+SR9XVAW/UdLjijrolQr1L0v6ctyEcE687h5FnTPvjV/fG3+/exPHvU3SVyQtVnTuX6OB+5kk41yjKCG43MzeJ2k3Sd+L41uj6Lz+U+ovDqRk25NZAAAQImoGAAAIXKbJQDypyGNmttoqTAFqZtPM7G4zezjuMXxi4r3PxPs9ZmZvzzJOAABCllkzQTwByeOKJhBpVzQByhnuviqxzQJJD7v7dfFwndvdfe/4+Y8kHapomM5dkma6e3cmwQIAELAsawYOlbTa3de4e6ekmySdXLaNKxpyI0UzjD0TPz9Z0k3u3uHuTynqdHRohrECABCsLJOByeo7sUa7+k/mcZmicbXtimZaKw01SrMvAACog7zvmHWGpO+5+9VmdrikG83soMF2KjGz+Yom/NDOO+/8pv333z+jMEeRp56KHqdPzzcOAMCwPfTQQ8+7+x7DPU6WycA69Z3dbIr6z9z2AUU3M5G73x9P0DEp5b6K5xJfIEmtra3e1jakIchhmT07elyyJM8oAAB1YGapphEfTJbNBEslzTCz6RbdgnWeokk4kkozbsnMXi9prKKJShZLmmdmLWY2XdGNZP4nw1gBAAhWZjUD7t4Vz9p1h6SipBvcfaWZXS6pzd0XSzpf0r+Y2ScVdSY8K57Sc6WZLVQ013eXpI8xkgAAgGxk2mfA3W9X1DEwue7zieerFN0lrtK+X1Y0HSgAAMhQ3h0IMdIOPzzvCAAADYZkIDRf+UreEQAAGgz3JgAAIHAkA6E55ZRoAQAgRjNBaNavzzsCAECDoWYAAIDAkQwAABA4kgEAAAJHn4HQHHNM3hEAABoMyUBoLr007wgAAA2GZgIAAAJHMhCad7wjWgAAiNFMEJqtW/OOAADQYKgZAAAgcCQDAAAEjmQAAIDA0WcgNHPm5B0BAKDBkAyE5oIL8o4AANBgaCYAACBwJAOhmT07WgAAiJEMAAAQOJIBAAACRzIAAEDgSAYAAAgcQwtDc9ppeUcAAGgwJAOh+ehH844AANBgaCYIzcsvRwsAADFqBkJz4onR45IluYYBAGgc1AwAABA4kgE0hPWbO7R87Uat39zREMerdzwA0MhoJkDufrZsnS5atELNhYK29fToa6ccrLmzJg+63/rNHWrfsFVTJozTxPEtQz5e+XGS+3d29+ict+2rdx82rc9nDCWutIa7PwDUimQAuVq/uUMXLVqhV7b16BX1SJIuXLRCR+47qU9BOFCBnSzw0x6vdMwfPPi0rr17tcYUo+Nc+s4DdMXPV/XZ/+o7H9e3735CXz/1kEGTlJ8tW6cLb1mhYsHU3eP6+qnbE5E0hXy0/3IVZOryHl120kF6z5v3GtrJBYCUSAZCc9ZZeUfQR/uGrWouFHoLXklqLhTUvmFrb4FZXvBfOucAXXHbqooFfvuGrVU/Z+L4lt4C+ZF1m3T5bSvV0eWSpI6u6DhfvHWlxjT1bz3r6HJduGiFDnjtrtrS2V2xQF+/uUMX/Hi5tnV777pPLVymI/edpN+sfl4XLVqhopm2dffoCycdqPe8eS+t39yhlc9skmTaqbmgT928TNHu0TE+99NHJJPec1hjJwTUZgA7NpKB0DRYMjBlwjht6+nps25bT4+mTBgnqXLNwRdvXaUm63uc5kJBK5/ZpHUbtuqVbX2P98q2Hu08ptibVDQVTJs7uivG01wsqDNRmCe5Syf8071qaSqq211fO+Xg3gRk5zFF/Wb1c30SAUnq6pE+edPDevD/XuhNPKSokF/evlE/eXhdv33KXbZ4paZO2EkH7rlrQxS0aWtpAOw4zH3gP0Q7itbWVm9ra8s7jMb3/PPR46RJ+caRsHjZOl1YpTBZvnaj3vvdB/VSR1fv9mOK1q/Abi6a3F1dffMASVJL0fQvZ/6V5t/Y1i9RKDe2uaC3H/AX+tnyZweNu2hSoWCSu7b1VI6rXnZqLqpHXrWgLS+gs7pSH6iWpmRsc0H3XXR0ZokLtRDAdmb2kLu3Dvc41AyE5tRTo8cRmGcg7R/tubMm68h9J2nlMy9Kch2456t636tUc1CpwB3o6toKJsn7NUeUay6a/mqvCakSAUnqdqk78blZJQKS9PK2qCajUv+H8gL6tDdN0cKH2ut+pV6tlqa50LeapryZp56ohQCyQTKATNT6R7vUpp7cvlQFX+rU11woqKOrW2bW28Y/GJN06TsPkGR6ubN/00CzSdviMnxbt+vXq9cP4duOnFJBK6m3eaK8gP6PB56WpEE7UNaqYv+OomlbV/VmnnqqpXMogNqQDKDuav2jXWn783+8XAWTxhSLvdXRB+35Kj24Zr2u/MWjqWNxSV+89RE1FwvqrtAkti3HVrLJu7Vo3cba5jHY1tOjR9Zt0ukL7o+So+4e2SBNfUUz3f3on/W2/V89rKGOO48p9qul6e5xfeGkA3uTtVIil0XhnKazKYChIRlA3dX6R7vS9qVq/46uqK/A5beu0tV/d4iuvvPxmuPp7JY6u9PVJIykWhMBSTr7r/fpN/RxMFs6u3XZrSt1yc8e6U2q0ra3l8+5cOz+r9Zdj/65dyhmqcbnhINek6pJaDjt/YN1Nh0q+iAAJAPIQK1/tCttX66jq0fnL1yujiqFekuT6f2H761/+fVTQwt6B+FSv8Sppcn6jFQo2WlMQS93RtuVRk987iePaOcx20dDDDYZU3mNzc8f+aPGFKX5R/WdiGni+JZBC9LhtvdPHN+ir51ycL/OpsMpwOmDAESYjjg0H/lItGSo9Ed7bHNBu7Q0aWxzYcA/2uXbtzQVVGGof9VEYFxzUf/y/lbNOXjPivvVk0k6rXWyWrL+oCqOmjGpX+LkLu3U3DeenccU9cG37KPxLcV+x9jS2a1XtvXowkUrBpxuuVRjU66zW7p2yeqa4k4mFi91dKX6/Ermzpqs+y46Wt//4GG676Kjh1Vw1ysmYDSgZiA0p58+Ih9TGiGQtvq1fPv7Vj+vCxetUMEqd/xL8sQIhKZiQV09fTu4JfsefOq4mXr02Rf1nw8/M6TvNaapoItOeH20LFqhu37/5yEdZyjef/g0tU6f2O/qeO4he2phW3ufbbvi9Qt+vabq8QZrbx+oxia5b5pq9nq296ephUiDPgjAdiQDoVm7NnqcOjXzj6r1j3Zy++3DDTfpQ//R1qcavGhSsWBqbiqouyeq7paiP+6lce+VRiUkC6s37b27vnjrKpmiJojShXWxWBhwLoJiwbTymU06auar9d0z/0oL7nlSX/nFo6qlH+KR+0xU29MbZKY+n9VcNH3wLdN1w31P9fm+TQXppg+9Wa3TJ/Y5N6VOfXO+/Zt+n9Hj0qpnX+xNHIoF05ayiZYGa28v1dh8+pYV/UZvlPatVM1e6Xxn1d4/HI0YE5AXJh0KzezZ0eMIzDNQL5UmJUoWOOXDEi995wE6aPLgneSSveS3dHb3FqyDTUzU0mR97lOw+k8v6R3f+vWgMwlG+xb024uPlrR9aOAzm15RaX6FieNbBpyEqVylSZlKSpP/lD7rkXWb+vX6L93PodqVfTRd8ou6f83zuuE3T/XWsJT+DY786q/6nK+mglQsFPp1MFy/uUM/fPBpffvuJ/ocI+/2+VrONdCImHQIDWEkemJXa3IoVVGXd3K74uerUs2AV6nmotqVcFLpPgWloZJbOrs1tqmobd19C+QxxUKfUQxNBenrpx7cJ35J2vcvdkn1fStJU5V/yNTdNHF8iw6Zulu/Xv8DdaArf+/zcw7sk2QtX7uxXzV7V0/URFE6fxcuWqGXXunqTUIk0/yj9unT+TDP3vxzZ03WAa/dVcvWbtSsqbtpws5jtHztxtxHFlQ7J4x8QFZIBjBkI9kTu1qTQ73bfT3+b3NB2tYTXV1393jvlMOVPqNSgTymGPVlSCoWCjpy33TTQKdtYtlelb+834iCSlXeyeMONB+EpEGTrDSjQIoF0xdvXanObu89zrVLVuvdh02TlH9v/uTnb93WJTOLErscawmqnZO8zxVGt0y7RJvZCWb2mJmtNrOLK7z/j2a2LF4eN7ONife6E+8tzjJO1C7vntjrN3do+dqNFSfCGWq7b+k7dXRtL/h7elw/+uBhKhb7/q+S/IxKoyfOPXqmxjb17ck/prh99sB6mjtrsn578TE6/7iZamlKN4JDqjxaoJTkDPReSf9RIKbmYt+pibd1u5qLlY8znN9Q6d9/OL+38s/v6onizfL3PFjc1c7J6j+9xMgHZCqzmgEzK0q6VtJxktolLTWzxe6+qrSNu38ysf25kt6QOMRWd5+VVXwYnjx7Yvebi791iha29Z2LfygxVPpOLU1FNTcVBx3fXl61L/UfftfZ3a1NWzu1fnPHsM5RpariieNbdO4xM/Tuw6alrkYerANdmiSr2iiQ8hsZVTrOUH9D9bpCrvT5SfX+PaeJu9o5WVahSYaRD6inLJsJDpW02t3XSJKZ3STpZEmrqmx/hqQvZBgPJOn88+tymLx6Yleq2l7Y1q7bznmLtnR2D6stdaDvdMjU3QZtxy+v2k8mEFu3danHpY/94OFhFWCDFSi1jOAYbBKfoUzwU6m/wy4tTVWPU+tvqJ73JxismaOev+e0cVf7Dc6auhsjH5CpLJOByZLWJl63Szqs0oZmtpek6ZJ+lVg91szaJHVJusrdf1phv/mS5kvStGnT6hT2KHfSSXU5TBazwaVR7cppS2e3Dpm627COPdh3qnWo5JH7TtKC97Xqxa2dOv/HUafEUifDCxet0AGv3bWmBGb95g5dGPcNGG5BWKpdOOC1u2rB+1qVHM1QkqYjY7XkJM1xhvIbqvd8BeUJW3mfgXr9ntPGXe2c7PsXu+Ty/xvC0SgdCOdJusXdkwOh93L3dWa2j6Rfmdnv3P3J5E7uvkDSAikaWjhy4e7AHnssetxvv2EfqtaJheoh6xqJ8t7l5T3900oWkh1d3SqU3eZXkk781q/VUkNntR88+HS/ToJDKQhLsXmPq6PbNTaeZKFSDAMlQLVcpVc7Tq2/oXr/+5d//oYtncP+t6+klrirnZM8/n9DOLJMBtZJSs5sMyVeV8k8SR9LrnD3dfHjGjNboqg/wZP9d0VNzj47eqzTPAP1mg2uls/L8gqpHu3RlQpJlc1BUBqb35moKRjoCn/95g5de3f/KYA7u7trKghX/+klffqWFepMDJ0sxVJrLUO9rtLr2bQxFKXPz7K3fq1xVzsnI/3/G8KRZTKwVNIMM5uuKAmYJ+nd5RuZ2f6SJki6P7FugqSX3b3DzCZJOlLS1zKMFTuQrK6Q6tUeXbEjYtHkZmopbr/tcEciQRisEG3fsFVjioV+8x+c87YZqWP72bJ1+vSPl6uzyuRItRbk9bxKL58AaqB/13r++yc/t159Earhyh6NLLNkwN27zOwcSXdIKkq6wd1XmtnlktrcvTRccJ6km7zvVIivl3S9mfUoGv54VXIUApDFFVK9rnQrFZJWMP087uTYO4VwolAerBCtdMyWpkLveP3BlBKdaolAmhjK1esqvZZmi+RnD/fff7CmnCx669frd8vkQ6i3TPsMuPvtkm4vW/f5steXVdjvt5L+MsvYgHL1utIdqBNYSa2F6HAL3oGG0SUL31oLluFe7SZrY0qG2mwx1M+t1pTTqL3169WcQUKBpEbpQAjkrp7t0YMVkkMpRIdT8FZKdJoL0uUnH6TJE8b1G0lQi+Fc7Q6UpGQ5jn6wppxG7a1fr6YsZjNEOZKB0FxySd4RNLR6tusOVkgOpRBNu0/5VV+lYXQy05W3P5prYTDQWP80cw7U+u+U7CMwUFNOo14t16Mpq55zNWD0IBkIzbHH5h1Bw9vRe2xXu+rbflvoF+PbQved9yCPwiCZpFTqM1AtnqFc2aaZubKewwmzUI+mrDxnD0XjIhkIzbJl0eMsZnoejQa76ps4vkWvGtfcb2RCnoVBsjYmzWiCoVzZDjZzZelzhztVdDX1ap+vR1NWXrOHorGRDITmvPOixzrNM4DGkuaqrxELg1pqY4ZyZTvQzJX/t35Lpu3naWoxakkWhtuUldfsoWhsJAPAKJKmoN/RC4OhJDPV9qk0v8Cnb1mh3XYaowP33HXQczJYIZ6mFmMoTR7DbcpizgOUIxkARpG0Bf2OXBgMJZmpts+Wzu5+NQYdXT368I0PqUc+YME8nLsQlmox8uzMt6P3jUF9kQwAo0zagn5HLgzqNTRz/eaOiqMZXt4W3SalWsE83LsQlmox6MyHRlHIOwAA9TdxfIsOmbrbqC5QhvIdy/cp1RiMbS5opzHFftuXCuZypUJ8sG2Tx9+lpUljmwt9ajEasf8GwkTNQGiuvDLvCICGsn3I5aZ4yOXg00TX4y6EUt/mi6KZtnX36NJ3HjCqkzg0JmoGQnPEEdECoNfE8S06auar9fVTD6l6FV++/UBX/JW2r1aLMXfWZH3y2Jnq7O5Rc7GgK36+SouXVbvBK5AN63t/oB1Xa2urt7W15R1G4/vtb6NHEgKgolqG+VXattY5BX7wwB/0uZ8+0mfd2OaC7rvoaGoIMCgze8jdW4d7HJoJQvPZz0aPzDOAnOVxo5w0n1lLx8rybWsdJrj6Ty/pC4sf6be+WDA6EWJEkQwAGHF53Cgn68+sdZjgz5at06dvWaGuCrdm2NbtdCLEiKLPAIARlSw0X+ro0ivbenThohVav7ljh/7MtCMMkvF0VsoEJH3hJDoRYmSRDAAYUbUUmjvSZ9YywqBSPJLUVJC+/K6D9J7D9hpyHOs3d2j52o2ZJlcYfWgmADCi8hhbP5zPTNu3oZaZEadMGKfO7r7xjGkq6PZz3zKsOyfm0fyC0YFkIDTXXJN3BAhcHvdGGOpn1lq4VpvlcOUzL0pyHbjnqzRxfIt+s/p5dSeSk+ai6RunDu8Wymn6LOTRaRM7BpKB0HDrYjSAPO6NUOtnDvW+AckRBj9btk7nL1zW20mwuWi6bO6BuuK2VX06DhZMOnLfSQPGMljcg01tnGWtAUnGjo9kIDR33RU9HntsvnEgeHncGyHrWyUnrd/coQtvWd6n0N/W7fri4lVqLlqfbccUi1WPO1ghXiqIdx5TrNoUkuUNkWiaGB1IBkLzpS9FjyQDwICG27ehfcNWFa0gqbvP+mJB2tZd/bjJq2xJAxbi5QXxaa1TtLCtvV9TyPK1G/slNgWZVj7zoo6auUe/z02bIOR510XUF8kAAFQw3L4NUyaMU7f3HzrY49IXTjpQV/x8Vb/jlhfuH5u9b79CvGimux/9s2ZN3a1fQbywrV23nfMWbens7lOoV+qw+PK2bn3oP9r09VMPlktDurrnroujB8kAAFQxnL4NE8e36OunHqJPlfUZ+PqpUUF72PTdtWztRu09cSc1NxW1+k8v9Svcv333E5L6Nils6ezWZbeuVGe3q7usgPce15bObh0ydbc+68s7LJZ0dPXogh8vl+Tq7Fbv515wywrt+aqxam4q9vveyRqEtLUn9e5TQB+F+iMZAIABDKdvw5H7TtINZx2qF7d2atdxzb2jCUo1AN7j6uh2jW0uqMelnrKC1cz0sdn76tolq1U005bOqMlhc0d3pY9TR7dr57JbMZeq8qvMb6TO7v73p+ns6tGp1z+gsc3RXAiXzjlAB+35Kj2yblO/Go2Bak/Wb+7QDx58Wtfe/YTGFIt16VNAH4VskAwAQAYqFVqloYalGoCS5POkV7b16PB9dtchU1u15rmX9I3/frxqIiBFNzgqJQylq+dNWzv7VeWnVYrrcz95RDuPKWhLZ/Q62T/gvouO1n0XHd3vSv0HD/xBl926UtviZKOjq6t3n6H2KaCPQnZIBkJz/fV5RwCMegMVWpXa2aspmvTuf/0ftRQL6uzuVk+Km8xOmTCuTyKSdr/BlBKBpFL/gPLbM1e6E2P5PkMpvOmjkB2SgdDst1/eEQCj3kCFVqV29mq6Xeru6um9h0FTQWppKmhMsfroAan/CITSfl3dParQKiAper+7p6dqc0Il5f0DogmWNumyW1em3qcWecxeGQqSgdDcemv0eNJJ+cYBjGIDFVrJUQrJPgOS+hTuHV3dKhSsTxPCuOYmXfueN+pV45p7j/WJY2b2qaKvNIwwud+2rm793/qX9cKWTn3zrsf7JBJ/WP+yrr7z8VTfsaWp0Kd/QKk2omDW2zRQyWmtU1JN+FSpg2Aes1eGwtzrUH/UAFpbW72trS3vMBrf7NnR45IleUYBjHqLl63rV2hVmywoORQwuX7Ot3/TJxkY21zQfRcdPWDht35zh4786q9S7Vde6K7f3KEjrvqlOrq2lwsFqV+Dxk7NRf3z+97UZ46C8s+sZrDvkKaDIKMJtjOzh9y9dbjHoWYAADIw2LDEaqMUkuuHchVcy9VzeQyl4ZDJfS+dc4Auv3VlnwShR64D99y193W1fhBjitZvtMJAbfxpOwjmMXvlaEcyAAAZGW6hNdR5DoYzP0KlfXdpaRowuajULNLSVNDVf3eIzv/xcnUkOiIM1MZPB8H8kAwAQAMbakIxnESkfN80tRySs95DAAAdfElEQVSVaiPmHLKnetxT127QQTA/JAMAgEENllxUSxhqqaWgg2B+6EAYmrVro8epU/ONAwCqoINgenQgxNCQBABocHQQHHmFvAPACLv55mgBACBGzUBorrsuejz99HzjAAA0DGoGAAAIHMkAAACBIxkAACBwJAMAAASODoShueWWvCMAADQYkoHQTJqUdwQAgAZDM0Fovve9aAEAIEYyEBqSAQBAmUyTATM7wcweM7PVZnZxhff/0cyWxcvjZrYx8d6ZZvZEvJyZZZwAAIQssz4DZlaUdK2k4yS1S1pqZovdfVVpG3f/ZGL7cyW9IX6+u6QvSGqV5JIeivfdkFW8AACEKsuagUMlrXb3Ne7eKekmSScPsP0Zkn4UP3+7pDvd/YU4AbhT0gkZxgoAQLCyTAYmS1qbeN0er+vHzPaSNF3Sr2rdFwAADE+jDC2cJ+kWd++uZSczmy9pviRNmzYti7hGn9tvzzsCAECDybJmYJ2kqYnXU+J1lczT9iaC1Pu6+wJ3b3X31j322GOY4QZip52iBQCAWJbJwFJJM8xsupmNUVTgLy7fyMz2lzRB0v2J1XdIOt7MJpjZBEnHx+swXN/5TrQAABDLLBlw9y5J5ygqxH8vaaG7rzSzy81sbmLTeZJucndP7PuCpCsUJRRLJV0er8NwLVwYLQAAxDLtM+Dut0u6vWzd58teX1Zl3xsk3ZBZcAAAQBIzEAIAEDySAQAAAkcyAABA4BplngGMlCVL8o4AANBgqBkAACBwJAOh+cY3ogUAgBjJQGhuuy1aAACIkQwAABA4kgEAAAJHMgAAQOAYWhiacePyjgAA0GBIBkLzi1/kHQEAoMHQTAAAQOBIBkJzxRXRAgBAjGQgNL/8ZbQAABAjGQAAIHAkAwAABI5kAACAwDG0MDQTJ+YdAQCgwZAMhGbRorwjAAA0GJoJAAAIHMlAaD7zmWgBACBGM0Fo7r8/7wgAAA2GmgEAAAJHMgAAQOBIBgAACBx9BkIzZUreEQAAGgzJQGi+//28IwAANBiaCQAACBzJQGjOOy9aAACI0UwQmmXL8o4AANBgqBkAACBwJAMAAASOZAAAgMDRZyA0M2fmHQEAoMGQDIRmwYK8IwAANBiaCQAACBzJQGjmz48WAABiNBOE5vHH844AANBgqBkAACBwJAMAAASOZAAAgMDRZyA0s2blHQEAoMGQDITmmmvyjgAA0GBoJgAAIHCZJgNmdoKZPWZmq83s4irbnGZmq8xspZn9MLG+28yWxcviLOMMynvfGy0AAMQyayYws6KkayUdJ6ld0lIzW+zuqxLbzJD0GUlHuvsGM3t14hBb3Z0G7nprb887AgBAg8myZuBQSavdfY27d0q6SdLJZdt8SNK17r5Bktz9zxnGAwAAKsgyGZgsaW3idXu8LmmmpJlmdp+ZPWBmJyTeG2tmbfH6d2UYJwAAQct7NEGTpBmSZkuaIuleM/tLd98oaS93X2dm+0j6lZn9zt2fTO5sZvMlzZekadOmjWzkAACMElnWDKyTNDXxekq8Lqld0mJ33+buT0l6XFFyIHdfFz+ukbRE0hvKP8DdF7h7q7u37rHHHvX/BqPR4YdHCwAAsSxrBpZKmmFm0xUlAfMkvbtsm59KOkPSv5nZJEXNBmvMbIKkl929I15/pKSvZRhrOL7ylbwjAAA0mMySAXfvMrNzJN0hqSjpBndfaWaXS2pz98Xxe8eb2SpJ3ZI+7e7rzewISdebWY+i2ourkqMQAABA/Zi75x1DXbS2tnpbW1veYTS+U06JHhctyjcOAMCwmdlD7t463OPk3YEQI239+rwjAAA0GKYjBgAgcCQDAAAEjmQAAIDA0WcgNMcck3cEAIAGQzIQmksvzTsCAECDoZkAAIDAkQyE5h3viBYAAGI0E4Rm69a8IwAANBhqBgAACBzJAAAAgSMZAAAgcPQZCM2cOXlHAABoMCQDobnggrwjAAA0GJoJAAAIHMlAaGbPjhYAAGIkAwAABI5kAACAwJEMAAAQOJIBAAACx9DC0Jx2Wt4RAAAaDMlAaD760bwjAAA0GJoJQvPyy9ECAECMmoHQnHhi9LhkSa5hAAAaBzUDAAAEjmQAAIDAkQwAABA4kgEAAAJHB8LQnHVW3hEAABrMoMmAmZ0r6fvuvmEE4kHWSAYAAGXSNBP8haSlZrbQzE4wM8s6KGTo+eejBQCA2KDJgLtfImmGpH+VdJakJ8zsSjN7XcaxIQunnhotAADEUnUgdHeX9Md46ZI0QdItZva1DGMDAAAjIE2fgU9Ier+k5yV9V9Kn3X2bmRUkPSHpwmxDBAAAWUozmmB3SX/r7n9IrnT3HjObk01YAABgpKRpJviFpBdKL8xsVzM7TJLc/fdZBQYAAEZGmpqB6yS9MfF6c4V12FF85CN5RwAAaDBpkgGLOxBK6m0eYLKiHdXpp+cdAQCgwaRpJlhjZh83s+Z4+YSkNVkHhoysXRstAADE0iQDH5Z0hKR1ktolHSZpfpZBIUPve1+0AAAQG7S6393/LGneCMQCAABykGaegbGSPiDpQEljS+vd/R8yjAsAAIyQNM0EN0p6jaS3S7pH0hRJL2UZFAAAGDlpkoF93f1SSVvc/d8lvVNRvwEAADAKpBkiuC1+3GhmBym6P8GrswsJmTr//LwjAAA0mDTJwAIzmyDpEkmLJY2XdGmmUSE7J52UdwQAgAYzYDNBfDOiF919g7vf6+77uPur3f36NAc3sxPM7DEzW21mF1fZ5jQzW2VmK83sh4n1Z5rZE/FyZk3fCtU99li0AAAQG7BmIJ5t8EJJC2s9sJkVJV0r6ThF8xMsNbPF7r4qsc0MSZ+RdKS7bzCzV8frd5f0BUmtklzSQ/G+G2qNA2XOPjt6XLIk1zAAAI0jTQfCu8zsAjObama7l5YU+x0qabW7r3H3Tkk3STq5bJsPSbq2VMjHcxpI0ciFO939hfi9OyWdkOobAQCAmqTpM1CazP5jiXUuaZ9B9pssKTnvbWn2wqSZkmRm90kqSrrM3f+ryr6Tyz/AzOYrng1x2rRpg4QDAAAqSTMD4fSMP3+GpNmK5i+418z+Mu3O7r5A0gJJam1t9UE2BwAAFaSZgfD9lda7+38Msus6SVMTr6fE65LaJT3o7tskPWVmjytKDtYpShCS+y4ZLFYAAFC7NM0Ef5V4PlbSMZL+V9JgycBSSTPMbLqiwn2epHeXbfNTSWdI+jczm6So2WCNpCclXRkPaZSk4xV1NMRwXXJJ3hEAABpMmmaCc5OvzWw3RZ0BB9uvy8zOkXSHov4AN7j7SjO7XFKbuy+O3zvezFZJ6pb0aXdfH3/OFYoSCkm63N1fqOF7oZpjj807AgBAgzH32prazaxZ0iPuvl82IQ1Na2urt7W15R1G41u2LHqcNSvfOAAAw2ZmD7l763CPk6bPwK2KRg9I0VDEAzSEeQfQIM47L3pkngEAQCxNn4FvJJ53SfqDu7dnFA8AABhhaZKBpyU96+6vSJKZjTOzvd39/zKNDAAAjIg0MxD+WFJP4nV3vA4AAIwCaZKBpng6YUlS/HxMdiEBAICRlKaZ4DkzmxsPBZSZnSzp+WzDQmauvDLvCAAADSZNMvBhST8ws2/Hr9slVZyVEDuAI47IOwIAQINJM+nQk5LebGbj49ebM48K2fntb6NHkgIAQGzQPgNmdqWZ7ebum919s5lNMLMvjURwyMBnPxstAADE0nQgfIe7byy9cPcNkk7MLiQAADCS0iQDRTNrKb0ws3GSWgbYHgAA7EDSdCD8gaRfmtm/STJJZ0n69yyDAgAAIydNB8KvmtlySccqukfBHZL2yjowAAAwMtLUDEjSnxQlAn8n6SlJizKLCNm65pq8IwAANJiqyYCZzZR0Rrw8L+lmRbc8ftsIxYYscOtiAECZgWoGHpX0a0lz3H21JJnZJ0ckKmTnrruix2OPzTcOAEDDGCgZ+FtJ8yTdbWb/JekmRR0IsSP7UjxFBMkAACBWdWihu//U3edJ2l/S3ZLOk/RqM7vOzI4fqQABAEC2Bp1nwN23uPsP3f0kSVMkPSzposwjAwAAIyLNpEO93H2Duy9w92OyCggAAIysmpIBAAAw+qSdZwCjxfXX5x0BAKDBkAyEZr/98o4AANBgaCYIza23RgsAADFqBkJz9dXR40kn5RsHAKBhUDMAAEDgSAYAAAgcyQAAAIEjGQAAIHB0IAzNjTfmHQEAoMGQDIRm6tS8IwAANBiaCUJz883RAgBAjJqB0Fx3XfR4+un5xgEAaBjUDAAAEDiSAQAAAkcyAABA4EgGAAAIHB0IQ3PLLXlHAABoMCQDoZk0Ke8IAAANhmaC0Hzve9ECAECMZCA0JAMAgDIkAwAABI5kAACAwJEMAAAQOJIBAAACl2kyYGYnmNljZrbazC6u8P5ZZvacmS2Llw8m3utOrF+cZZxBuf32aAEAIJbZPANmVpR0raTjJLVLWmpmi919VdmmN7v7ORUOsdXdZ2UVX7B22invCAAADSbLmoFDJa129zXu3inpJkknZ/h5SOM734kWAABiWSYDkyWtTbxuj9eVO8XMVpjZLWY2NbF+rJm1mdkDZvauDOMMy8KF0QIAQCzvDoS3Strb3Q+WdKekf0+8t5e7t0p6t6RrzOx15Tub2fw4YWh77rnnRiZiAABGmSyTgXWSklf6U+J1vdx9vbt3xC+/K+lNiffWxY9rJC2R9IbyD3D3Be7e6u6te+yxR32jBwAgEFkmA0slzTCz6WY2RtI8SX1GBZjZaxMv50r6fbx+gpm1xM8nSTpSUnnHQwAAUAeZjSZw9y4zO0fSHZKKkm5w95VmdrmkNndfLOnjZjZXUpekFySdFe/+eknXm1mPooTlqgqjEAAAQB2Yu+cdQ120trZ6W1tb3mEAADBizOyhuH/dsOTdgRAAAOSMZCA03/hGtAAAECMZCM1tt0ULAAAxkgEAAAJHMgAAQOBIBgAACFxm8wygQY0bl3cEAIAGQzIQml/8Iu8IAAANhmYCAAACRzIQmiuuiBYAAGIkA6H55S+jBQCAGMkAAACBIxkAACBwJAMAAASOoYWhmTgx7wgAAA2GZCA0ixblHQEAoMHQTAAAQOBIBkLzmc9ECwAAMZoJQnP//XlHAABoMNQMAAAQOJIBAAACRzIAAEDg6DMQmilT8o4AANBgSAZC8/3v5x0BAKDB0EwAAEDgSAZCc9550QIAQIxmgtAsW5Z3BACABkPNAAAAgSMZAAAgcCQDAAAEjj4DoZk5M+8IAAANhmQgNAsW5B0BAKDB0EwAAEDgSAZCM39+tAAAEKOZIDSPP553BACABkPNAAAAgSMZAAAgcCQDAAAEjj4DoZk1K+8IAAANhmQgNNdck3cEAIAGQzMBAACBIxkIzXvfGy0AAMRoJghNe3veEQAAGgw1AwAABI5kAACAwGWaDJjZCWb2mJmtNrOLK7x/lpk9Z2bL4uWDiffONLMn4uXMLOMEACBkmfUZMLOipGslHSepXdJSM1vs7qvKNr3Z3c8p23d3SV+Q1CrJJT0U77shq3iDcfjheUcAAGgwWXYgPFTSandfI0lmdpOkkyWVJwOVvF3Sne7+QrzvnZJOkPSjjGINx1e+kncEAFDV9Xc/oZ+ueFbvOvi1OvttM/IOJxhZJgOTJa1NvG6XdFiF7U4xs6MkPS7pk+6+tsq+k7MKFACQv9dfcru2drkk6ffPvqRrfvmEfv+lE3OOKgx5dyC8VdLe7n6wpDsl/XstO5vZfDNrM7O25557LpMAR51TTokWAGgg19/9RG8iULK1y3X93U/kFFFYskwG1kmamng9JV7Xy93Xu3tH/PK7kt6Udt94/wXu3ururXvssUfdAh/V1q+PFgBoID9d8WxN61FfWSYDSyXNMLPpZjZG0jxJi5MbmNlrEy/nSvp9/PwOSceb2QQzmyDp+HgdAGAUetfBr61pPeors2TA3bsknaOoEP+9pIXuvtLMLjezufFmHzezlWa2XNLHJZ0V7/uCpCsUJRRLJV1e6kwIABh9zn7bDI1rsj7rxjUZnQhHiLn74FvtAFpbW72trS3vMBrf7NnR45IleUYBABUxmqA2ZvaQu7cO9zjcmyA0xxyTdwQAUNXZb5tBEpADkoHQXHpp3hEAABpM3kMLAQBAzkgGQvOOd0QLAAAxmglCs3Vr3hEAABoMNQMAAASOZAAAgMCRDAAAEDj6DIRmzpy8IwAANBiSgdBccEHeEQAAGgzNBAAABI5kIDSzZ2+/PwEAACIZAAAgeCQDAAAEjmQAAIDAkQwAABA4hhaG5rTT8o4AANBgSAZC89GP5h0BAKDB0EwQmpdfjhYAAGLUDITmxBOjxyVLcg0DANA4qBkAACBwJAMAAASOZAAAgMCRDAAAEDg6EIbmrLPyjgAA0GBIBkJDMgAAKEMzQWiefz5aAACIUTMQmlNPjR6ZZwAAEKNmAACAwJEMAAAQOJIBAAACRzIAAEDg6EAYmo98JO8IAAANhmQgNKefnncEAIAGQzNBaNaujRYAAGLUDITmfe+LHplnAAAQo2YAAIDAkQwAABA4kgEAAAJHMgAAQODoQBia88/POwIAQIMhGQjNSSflHQEAoMHQTBCaxx6LFgAAYtQMhObss6NH5hkAAMSoGQAAIHAkAwAABC7TZMDMTjCzx8xstZldPMB2p5iZm1lr/HpvM9tqZsvi5Z+zjBMAgJBl1mfAzIqSrpV0nKR2SUvNbLG7ryrbbhdJn5D0YNkhnnT3WVnFBwAAIll2IDxU0mp3XyNJZnaTpJMlrSrb7gpJX5X06QxjQckll+QdAQCgwWTZTDBZUvJeue3xul5m9kZJU9395xX2n25mD5vZPWb215U+wMzmm1mbmbU999xzdQt8VDv22GgBACCWWwdCMytI+qakSlPiPStpmru/QdKnJP3QzHYt38jdF7h7q7u37rHHHtkGPFosWxYtAADEsmwmWCdpauL1lHhdyS6SDpK0xMwk6TWSFpvZXHdvk9QhSe7+kJk9KWmmpLYM4w3DeedFj8wzAACIZVkzsFTSDDObbmZjJM2TtLj0prtvcvdJ7r63u+8t6QFJc929zcz2iDsgysz2kTRD0poMYwUAIFiZ1Qy4e5eZnSPpDklFSTe4+0ozu1xSm7svHmD3oyRdbmbbJPVI+rC7v5BVrAAAhCzT6Yjd/XZJt5et+3yVbWcnni+StCjL2AAAQIQZCAEACBw3KgrNlVfmHQEAoMGQDITmiCPyjgAA0GBoJgjNb38bLQAAxKgZCM1nPxs9Ms8AACBGzQAAAIEjGQAAIHAkAwAABI5kAACAwNGBMDTXXJN3BACABkMyEJpZs/KOAADQYGgmCM1dd0ULAAAxagZC86UvRY/HHptvHACAhkHNAAAAgSMZAAAgcCQDAAAEjmQAAIDA0YEwNNdfn3cEAIAGQzIQmv32yzsCAECDoZkgNLfeGi0AAMSoGQjN1VdHjyedlG8cAICGQc0AAACBIxkAACBwJAMAAASOZAAAgMDRgTA0N96YdwQAgAZDMhCaqVPzjgAA0GBoJgjNzTdHCwAAMWoGQnPdddHj6afnGwcAoGFQMwAAQOBIBgAACBzJAAAAgSMZAAAgcHQgDM0tt+QdAQCgwZAMhGbSpLwjAAA0GJoJQvO970ULAAAxkoHQkAwAAMqQDAAAEDiSAQAAAkcyAABA4EgGAAAIHEMLQ3P77XlHAABoMCQDodlpp7wjAAA0GJoJQvOd70QLAAAxkoHQLFwYLQAAxDJNBszsBDN7zMxWm9nFA2x3ipm5mbUm1n0m3u8xM3t7lnECABCyzPoMmFlR0rWSjpPULmmpmS1291Vl2+0i6ROSHkysO0DSPEkHStpT0l1mNtPdu7OKFwCAUGVZM3CopNXuvsbdOyXdJOnkCttdIemrkl5JrDtZ0k3u3uHuT0laHR8PAADUWZbJwGRJaxOv2+N1vczsjZKmuvvPa90XAADUR25DC82sIOmbks4axjHmS5ofv+wws0fqEFoIJsns+byD2AFMksR5SodzlQ7nKR3OU3r71eMgWSYD6yRNTbyeEq8r2UXSQZKWmJkkvUbSYjObm2JfSZK7L5C0QJLMrM3dW8u3QX+cq3Q4T+lxrtLhPKXDeUrPzNrqcZwsmwmWSpphZtPNbIyiDoGLS2+6+yZ3n+Tue7v73pIekDTX3dvi7eaZWYuZTZc0Q9L/ZBgrAADByqxmwN27zOwcSXdIKkq6wd1XmtnlktrcffEA+640s4WSVknqkvQxRhIAAJCNTPsMuPvtkm4vW/f5KtvOLnv9ZUlfruHjFtQaX8A4V+lwntLjXKXDeUqH85ReXc6VuXs9jgMAAHZQTEcMAEDgdohkYLBpjc3sH81sWbw8bmYbE+91J96r2k9hNEhxnqaZ2d1m9rCZrTCzExPvBTX981DPlZntbWZbE7+pfx756EdOivO0l5n9Mj5HS8xsSuK9M83siXg5c2QjH3nDPFch/Z26wcz+XG0ouEW+FZ/HFfF8NKX3gvlNDfM81f57cveGXhR1PnxS0j6SxkhaLumAAbY/V1FnxdLrzXl/h0Y5T4ralj4SPz9A0v8lni+X1CJpenycYt7fqUHP1d6SHsn7OzTQefqxpDPj50dLujF+vrukNfHjhPj5hLy/UyOeq/h1EH+n4u96lKQ3Vvv/SNKJkn4hySS9WdKDgf6mhnSehvp72hFqBtJOa1xyhqQfjUhkjSXNeXJJu8bPXyXpmfh5aNM/D+dchSTNeTpA0q/i53cn3n+7pDvd/QV33yDpTkknjEDMeRnOuQqKu98r6YUBNjlZ0n945AFJu5nZaxXYb2oY52lIdoRkIPXUxGa2l6Ir218lVo81szYze8DM3pVdmLlLc54uk/ReM2tXNMrj3Br2HU2Gc64kaXrcfHCPmf11ppHmK815Wi7pb+PnfyNpFzObmHLf0WQ450oK5+9UGtXOZWi/qcEMdD5q/j3tCMlALeZJusX7zkmwl0czWb1b0jVm9rp8QmsIZ0j6nrtPUVTFdKNF00Kjv2rn6llJ09z9DZI+JemHZrbrAMcZ7S6Q9FYze1jSWxXNFMqcIJUNdK74O4V6qvn3tCMUBKmmJo7NU1kTgbuvix/XSFoi6Q31D7EhpDlPH5C0UJLc/X5JYxXNAV7LOR4Nhnyu4qaU9fH6hxS1E8/MPOJ8DHqe3P0Zd//bODn6XLxuY5p9R5nhnKuQ/k6lUe1chvabGkzV8zGU39OOkAwMOK1xiZntr6hTyf2JdRPMrCV+PknSkYpmNRyN0pynpyUdI0lm9npFBdxzCm/65yGfKzPbw8yK8fp9FJ2rNSMW+cga9DyZ2aRE7dJnJN0QP79D0vHx/4MTJB0frxuthnyuAvs7lcZiSe+Pe8u/WdImd39W4f2mBlPxPA3595R3j8mUvSpPlPS4oquwz8XrLld0L4PSNpdJuqpsvyMk/U5RW93vJH0g7++S53lS1IHpvvh8LJN0fGLfz8X7PSbpHXl/l0Y9V5JOkbQyXve/kk7K+7vkfJ5OlfREvM13JbUk9v0HRZ1RV0v6+7y/S6OeqwD/Tv1IUXPbNkXt3B+Q9GFJH47fN0nXxufxd5JaQ/xNDfU8DfX3xAyEAAAEbkdoJgAAABkiGQAAIHAkAwAABI5kAACAwJEMAAAQOJIBAKmY2bvMzOM5PUp3cHwkfj7bzDbFd0l71My+kW+0AGpBMgAgrTMk/SZ+rOTX7j5L0Wxnc8zsyBGLDMCwkAwAGJSZjZf0FkUTn8wbaFt336poUqaQbyID7FBIBgCkcbKk/3L3xyWtN7M3Vdswnip2hqR7Ryo4AMNDMgAgjTMk3RQ/v0mVmwr+2syWK7pZyh3u/seRCg7A8DTlHQCAxmZmu0s6WtJfmplLKkpyRfOiJ/3a3efEN7t6wMwWuvuyEQ4XwBBQMwBgMKdKutHd93L3vd19qqSn1Pf2qb3c/SlJV0m6aARjBDAMJAMABnOGpJ+UrVuk6Da81fyzpKPMbO+MYgJQR9y1EACAwFEzAABA4EgGAAAIHMkAAACBIxkAACBwJAMAAASOZAAAgMCRDAAAEDiSAQAAAvf/AZ23rG1ZYo4jAAAAAElFTkSuQmCC\n", + "text/plain": [ + "" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "fig, ax = plt.subplots(figsize=(8,8))\n", + "_ = df.plot(kind='scatter', x='air', y='acc', title='Two Hundred Neural Networks', ax=ax)\n", + "_ = ax.axvline(x=0.8, color='r', linestyle='--')\n", + "_ = ax.set_ylim([0.4, 0.8])\n", + "_ = ax.set_xlim([0.75, 1.05])\n", + "_ = ax.set_xlabel('AIR')\n", + "_ = ax.set_ylabel('Accuracy')\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.9" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/tex/lecture_3.bib b/tex/lecture_3.bib index 6154adb..faf7f2c 100644 --- a/tex/lecture_3.bib +++ b/tex/lecture_3.bib @@ -1,3 +1,10 @@ + @article{fair_washing, + title = {Fairwashing: the {R}isk of {R}ationalization}, + author = {A{\"\i}vodji, Ulrich and Arai, Hiromi and Fortineau, Olivier and Gambs, S{\'e}bastien and Hara, Satoshi and Tapp, Alain}, + journal = {arXiv preprint arXiv:1901.09749}, + year = {2019}, + note = {URL: \url{https://arxiv.org/pdf/1901.09749.pdf}}} + @inproceedings{gender_shades, title={Gender {S}hades: {I}ntersectional {A}ccuracy {D}isparities in {C}ommercial {G}ender {C}lassification}, author={Buolamwini, Joy and Gebru, Timnit}, @@ -39,6 +46,24 @@ @misc{kangdebugging year={2019}, note={URL: \url{https://www-cs.stanford.edu/~matei/papers/2018/mlsys_model_assertions.pdf}}} +@article{obermeyer2019dissecting, + title={Dissecting {R}acial {B}ias in an {A}lgorithm {U}sed to {M}anage the {H}ealth of {P}opulations}, + author={Obermeyer, Ziad and Powers, Brian and Vogeli, Christine and Mullainathan, Sendhil}, + journal={Science}, + volume={366}, + number={6464}, + pages={447--453}, + year={2019}, + publisher={American Association for the Advancement of Science}, + note={URL: \url{https://bit.ly/36XK6yk}}} + +@article{scaffolding, + title={How {C}an {W}e {F}ool {L}{I}{M}{E} and {S}{H}{A}{P}? {A}dversarial {A}ttacks on {P}ost-hoc {E}xplanation {M}ethods}, + author={Slack, Dylan and Hilgard, Sophie and Jia, Emily and Singh, Sameer and Lakkaraju, Himabindu}, + journal={arXiv preprint arXiv:1911.02508}, + year={2019}, + note={URL: \url{https://arxiv.org/pdf/1911.02508.pdf}}} + @inproceedings{zemel2013learning, title={Learning {F}air {R}epresentations}, author={Zemel, Rich and Wu, Yu and Swersky, Kevin and Pitassi, Toni and Dwork, Cynthia}, diff --git a/tex/lecture_3.pdf b/tex/lecture_3.pdf index 9cdf437..9dae02f 100644 Binary files a/tex/lecture_3.pdf and b/tex/lecture_3.pdf differ diff --git a/tex/lecture_3.tex b/tex/lecture_3.tex index 9cec838..206c3f4 100644 --- a/tex/lecture_3.tex +++ b/tex/lecture_3.tex @@ -13,8 +13,6 @@ linkcolor=[rgb]{0,0,0.61}} \usepackage[natbib=true,style=numeric,backend=bibtex,useprefix=true]{biblatex} -%\usepackage{blindtext} - %---------------------------------------------------------------------------------- \definecolor{OwlGreen}{RGB}{51,0,102} %---------------------------------------------------------------------------------- @@ -26,18 +24,20 @@ \renewcommand*{\bibfont}{\scriptsize} \addbibresource{lecture_3.bib} +%------------------------------------------------------------------------------------------ + \usenavigationsymbolstemplate{} \setbeamertemplate{footline}{% \raisebox{5pt}{\makebox{\hfill\makebox[20pt]{\color{gray} \scriptsize\insertframenumber}}}\hspace*{5pt}} +%------------------------------------------------------------------------------------------ \renewcommand*{\thefootnote}{\fnsymbol{footnote}} - %------------------------------------------------------------------------------------------ \author{Patrick Hall} \title{Responsible Machine Learning\footnote{\tiny{This material is shared under a \href{https://creativecommons.org/licenses/by/4.0/deed.ast}{CC By 4.0 license} which allows for editing and redistribution, even for commercial purposes. However, any derivative work should attribute the author.}}} -\subtitle{Lecture 3: Fairness} +\subtitle{Lecture 3: Discrimination Testing and Remediation} \institute{The George Washington University} \date{\today} @@ -74,39 +74,21 @@ \end{frame} - \begin{frame} - \frametitle{Why Care About Discrimination in ML?} + \frametitle{Why Care About Discrimination in Machine Learning?} \begin{itemize} \Large - \item \textbf{Reputational risk}: - \begin{itemize} - \Large - \item{\normalsize{Upon encountering a perceived unethical ML system, 34\% of consumers are likely to, ``stop interacting with the company.''\footnote{\scriptsize{See: \href{https://www.capgemini.com/research/why-addressing-ethical-questions-in-ai-will-benefit-organizations/}{Why addressing ethical questions in AI will benefit organizations}.}}}} - \end{itemize} - \item Non-compliance fines and litigation costs. - \item Responsible practice of ML. + \item \textbf{Responsible practice of machine learning (ML)}: ML can affect millions of people! \cite{obermeyer2019dissecting} + \item \textbf{Discrimination is often illegal (in the U.S.)}: Non-compliance fines and litigation costs. + \item \textbf{Reputational risk}: Upon encountering a perceived unethical ML system, 34\% of consumers are likely to, ``stop interacting with the company.''\footnote{\scriptsize{See: \href{https://www.capgemini.com/research/why-addressing-ethical-questions-in-ai-will-benefit-organizations/}{Why addressing ethical questions in AI will benefit organizations}.}} \end{itemize} \end{frame} - - \begin{frame} - - \frametitle{Elements of Responsible Practice of ML} - - \begin{figure}[htb] - \begin{center} - \includegraphics[height=180pt]{../img/trust_understanding.png} - \end{center} - \end{figure} - - \end{frame} - %------------------------------------------------------------------------------- - \section{Discrimination \& Bias} + \section{Bias and Discrimination} %------------------------------------------------------------------------------- \subsection*{} @@ -116,82 +98,101 @@ \frametitle{What Is Bias?} \begin{itemize} - \Large - \item Almost all data, statistical models, and machine learning (ML) models encode different types of \textit{bias}, i.e., systematic misrepresentations of reality.\\ - \item Sometimes, bias is helpful. + \item Almost \textit{all} data, statistical models, and ML models encode different types of \textit{bias}, i.e., \textbf{systematic misrepresentations of reality}.\\ + \item Sometimes, bias is \textit{helpful}. \begin{itemize} - \Large - \item{shrunken and robust $\beta_j$ coefficients in penalized linear models} \end{itemize} - \item Other types of bias might be unwanted, unhelpful, or illegal discrimination. + \item{Shrunken and robust $\beta_j$ coefficients in penalized linear models.} \end{itemize} + \item Other types of bias can be unwanted, unhelpful, discriminatory, or illegal. + \item Many instances of discrimination in ML arise from sociologically biased data collection, labeling, or storage processes. \end{itemize} \end{frame} - \subsection*{} - - \begin{frame}[allowframebreaks]{Title} - + \begin{frame} + \frametitle{What is Discrimination in ML?} - - \noindent In some applications\footnote{\small{e.g., Under the Equal Credit Opportunity Act (ECOA), as implemented by Regulation B, and the Fair Credit Reporting Act (FCRA})}, model predictions should \textbf{\textit{ideally}} be independent of demographic group membership.\\ + + \noindent In many applications\footnote{\small{e.g., Under the Equal Credit Opportunity Act (ECOA), as implemented by Regulation B, and the Fair Credit Reporting Act (FCRA})}, model predictions should \textbf{\textit{ideally}} be independent of demographic group membership.\\ \vspace{5pt} \noindent In these applications, a model exhibits discrimination if: \begin{enumerate} \item Demographic group membership is not independent of the likelihood of receiving a favorable or accurate model prediction. - \item Membership in a \textit{subset} of a demographic group is not independent of the likelihood of receiving a favorable or accurate model prediction (i.e., \textit{local bias}).\cite{hall2019guidelines} + \item Membership in a \textit{subset} of a demographic group is not independent of the likelihood of receiving a favorable or accurate model prediction (i.e., \textit{local or individual discrimination}).\cite{hall2019guidelines} \end{enumerate} + \end{frame} \begin{frame} - \frametitle{What is Discrimination in ML?} + \frametitle{What Kinds of Discrimination Occur in ML?} \noindent \Large Several forms of discrimination may manifest in ML, including: \begin{itemize} - \item Overt discrimination, i.e. \textit{disparate treatment}. - \item Unintentional discrimination, i.e. \textit{disparate impact} (DI). - \item Discrimination in ML may or may not be illegal, depending on how it arises and applicable discrimination laws.\cite{hall2019guidelines} + + \item Group disparities: + + \begin{itemize} + + \item Overt discrimination against groups, i.e., \textit{disparate treatment}. + + % More careful definition: Disparate treatment occurs when a lender treats a (potential) customer differently based on a prohibited basis, such as the person’s age, gender, or race. + + \item Unintentional discrimination against groups, i.e., \textit{disparate impact} (DI). + + % More careful definition: Disparate impact occurs when a protected class experiences a larger share of less favorable outcomes as a result of an otherwise non-discriminatory and legitimate decision-making process. Disparate impact is not necessarily a violation of law and may be justified by a “business necessity,” such as cost or profitability. However, there may still be a violation if the lenders could have used an alternative policy or practice that had a less discriminatory effect. + + % All DI is discrimination, I'm not sure that all discrimination is DI. + + \end{itemize} + + \item Local or individual discrimination. + \end{itemize} - \end{frame} + + \end{frame} \begin{frame} - \frametitle{What is Discrimination in ML?} + \frametitle{How Does Discrimination Arise in ML?} \noindent Discrimination originates from training data:\\ \begin{itemize} - \item Incomplete or inaccurate data, particularly under-representation of minorities, e.g. \href{http://gendershades.org/}{Gender Shades}\cite{gender_shades}. - \item Accurate but differing patterns of causation, correlation, or dependency between demographic groups and past outcomes, e.g. traditional FICO credit scores.\footnote{\scriptsize{See: \href{https://www.youtube.com/watch?v=rToFuhI6Nlw}{Responsible Data Science: Identifying and Fixing Biased AI}.}} - \item Explicit encoding of historical social biases into training data, e.g. criminal records.\textsuperscript{$\mathparagraph$} + \item Incomplete or inaccurate data, e.g., under-representation of minorities. See \href{http://gendershades.org/}{Gender Shades} \cite{gender_shades}. + \item Accurate but differing patterns of causation, correlation, or dependency between demographic groups and past outcomes, e.g., traditional FICO credit scores.\footnote{\scriptsize{See: \href{https://www.youtube.com/watch?v=rToFuhI6Nlw}{Responsible Data Science: Identifying and Fixing Biased AI}.}} + \item Explicit encoding of historical social biases into training data, e.g., criminal records.\textsuperscript{$\mathparagraph$} \end{itemize} \end{frame} \begin{frame} - \frametitle{What is Discrimination in ML?} + \frametitle{How Does Discrimination Arise in ML?} ML models can perpetuate or exacerbate discrimination.\\ \vspace{10pt} - \noindent \textbf{Group disparities}, i.e. different or inaccurate treatment of entire demographic groups:\\ + \noindent \textbf{Group disparities}, i.e., different or inaccurate treatment of entire demographic groups:\\ \begin{itemize} - \item Learning different correlations between demographic groups and favorable model outcomes, i.e. \textit{adverse impact}. - \item Exhibiting different accuracies across demographic groups, i.e. \textit{differential validity}.\textsuperscript{$\mathparagraph$} + \item Learning different correlations between demographic groups and favorable model outcomes, i.e., \textit{DI}. + \item Exhibiting different accuracies across demographic groups, i.e., \textit{differential validity}.\textsuperscript{$\mathparagraph$} \end{itemize} \vspace{5pt} - \noindent \textbf{Locally}, i.e. different or inaccurate treatment of similar individuals:\\ + \noindent \textbf{Locally}, i.e., different or inaccurate treatment of similar individuals:\\ \begin{itemize} \item Local response function or decision boundary form. - \item Capacity to form local complex or latent demographic proxies. + \item Capacity to form local complex demographic proxies on a row-by-row basis. \end{itemize} \end{frame} - + +%------------------------------------------------------------------------------- + \section{Testing for Discrimination in ML} +%------------------------------------------------------------------------------- + \begin{frame} \frametitle{Common Metrics of Discrimination in ML} - Common metrics of \textbf{\textit{group}} disparities:\\ + Common metrics for DI and \textbf{\textit{group}} disparities:\\ \begin{itemize} \item Accuracy disparity: $\frac{\text{accuracy}_p}{\text{accuracy}_r}$ \item Adverse impact ratio: $\frac{\text{\% accepted}_p }{ \text{\% accepted}_r}$ @@ -199,13 +200,43 @@ \item Standardized mean difference: $\frac{\bar{\hat{y}}_p - \bar{\hat{y}}_r}{\sigma_{\hat{y}}}$ \end{itemize} \noindent - \scriptsize where, $p \equiv \text{protected group}$ and $r \equiv \text{reference group}$ (often white males),\\ + \scriptsize{where, $p \equiv \text{protected group}$ and $r \equiv \text{reference group}$ (often white males).}\\ \vspace{5pt} - $\text{\% accepted}_\text{group} = 100 \cdot \frac{\text{tn}_\text{group}~+~\text{fn}_\text{group}}{N_\text{group}}$, and $\text{accuracy}_\text{group} = \frac{\text{tp}_\text{group}~+~\text{tn}_\text{group}}{N_\text{group}}$.\textsuperscript{$\mathparagraph$}\\ - \vspace{10pt} - \normalsize - \textbf{\textit{Local bias}} is much trickier to measure and often an unmitigated risk for consumer-facing ML systems. - \end{frame} + \normalsize{There are many other, sometimes conflicting, mathematical definitions of discrimination. + See \href{https://www.youtube.com/watch?v=wqamrPkF5kk}{21 Definitions of Fairness and Their Politics}.} + + \end{frame} + + \begin{frame} + + \frametitle{Additional Considerations for Discrimination Testing} + + \begin{itemize} + + \item Local discrimination, i.e., the model treats a small number of similar people differently. + + \begin{itemize} + \item Search around probability thresholds. + \item Adversarial models. + \end{itemize} + + \item Post-hoc explanation to understand drivers of discrimination: + \begin{itemize} + + \item To be conducted after discrimination is confirmed by standard tests. + + \item Be aware of: + \begin{itemize} + \item No demographic features in model. + \item Fairwashing \cite{fair_washing} and scaffolding \cite{scaffolding}. + \end{itemize} + + \end{itemize} + + \end{itemize} + + \end{frame} + %------------------------------------------------------------------------------- \section{Remediation} @@ -217,12 +248,13 @@ \frametitle{How to Fix Discrimination in ML?} \Large \noindent - \textbf{Fix the process}: ensure diversity of experience in design, training, and review of ML systems.\\ + \textbf{Fix organizational processes}: Lecture 6\\ \noindent \textbf{Fix the data}: \begin{itemize} \item Collect demographically representative training data. - \item Select features judiciously, e.g. using \texttt{time\_on\_file} as an input variable as opposed to \texttt{bankruptcy\_flag}.\textsuperscript{$\mathparagraph$} + \item Label and annotate data carefully. + \item Select features judiciously. \item Sample and reweigh training data to minimize discrimination.\cite{kamiran2012data} \end{itemize} @@ -242,17 +274,25 @@ \item Learning fair representations (LFR) and adversarial de-biasing.\cite{zemel2013learning}, \cite{zhang2018mitigating} \item Use dual objective functions that consider both accuracy and fairness metrics. \end{itemize} - \item Edit model mechanisms to ensure less biased predictions, e.g. with \href{https://github.com/interpretml/interpret}{GA2M} models. + \item Edit model mechanisms to ensure less biased predictions, e.g., with \href{https://github.com/interpretml/interpret}{GA2M/EBM} models. \end{itemize} \noindent \textbf{Fix the predictions}: \begin{itemize} - \item Balance model predictions, e.g. reject-option classification.\cite{kamiran2012decision} + \item Balance model predictions, e.g., reject-option classification.\cite{kamiran2012decision} \item Correct or override predictions with model assertions or appeal mechansims.\cite{hall2019guidelines}, \cite{kangdebugging} \end{itemize} \end{frame} - - \subsection*{} + + \begin{frame} + + \frametitle{How to Fix Discrimination in ML?} + \centering + Consider discrimination measures during model selection.\\ + \vspace{10pt} + {\includegraphics[scale=0.09]{../img/lecture_3.png}} + + \end{frame} \begin{frame} @@ -260,7 +300,7 @@ \centering As part of a responsible ML workflow.\\ \vspace{10pt} - {\includegraphics[scale=0.08]{../img/blueprint.png}} + {\includegraphics[scale=0.08]{../img/rml_diagram_no_hilite.png}} \end{frame} @@ -273,8 +313,11 @@ \begin{frame} \frametitle{Acknowledgements} - This presentation borrows heavily from the expertise of Nicholas Schmidt and Bryce Stephens of \href{https://www.bldsllc.com/}{BLDS, LLC}, a leading fair lending compliance firm. + This presentation borrows heavily from the expertise of Nicholas Schmidt and Bryce Stephens of \href{https://www.bldsllc.com/}{BLDS, LLC}, a leading fair lending compliance firm.\\ + \vspace{10pt} + Thanks to Lisa Song for her continued assistance in developing these course materials. + \end{frame} %-------------------------------------------------------------------------------