Materials for a technical, nuts-and-bolts course about increasing transparency, fairness, security and privacy in machine learning.
- Lecture 1: Explainable Machine Learning Models
- Lecture 2: Post-hoc Explanation
- Lecture 3: Discrimination Testing and Remediation
- Lecture 4: Machine Learning Security
- Lecture 5: Machine Learning Model Debugging
- Lecture 6: Responsible Machine Learning Best Practices
Corrections or suggestions? Please file a GitHub issue.
Source: Simple Explainable Boosting Machine Example
- Introduction:
- Lecture Notes
- Assignment 1:
- Reading: Machine Learning for High-Risk Applications, Chapter 2 (pp. 33 - 50) and Chapter 6 (pp. 189 - 217)
- Python PiML-Toolbox
- Python explainable boosting machine (EBM)/GA2M
- R
gam
h2o
penalized GLM (R and Python)- Monotonic gradient boosting machine (GBM):
h2o
andxgboost
(R and Python) - R
rpart
- Python
skope-rules
- Building from Penalized GLM to Monotonic GBM (simple)
- Building from Penalized GLM to Monotonic GBM
- Simple Explainable Boosting Machine Example
- PiML Assignment 1 Example and simple requirements.txt
-
Introduction and Background:
- An Introduction to Machine Learning Interpretability
- Designing Inherently Interpretable Machine Learning Models
- Psychological Foundations of Explainability and Interpretability in Artificial Intelligence
- Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead
-
Explainable Machine Learning Techniques:
- Accurate Intelligible Models with Pairwise Interactions
- Elements of Statistical Learning - Chapters 3,4, and 9
- Fast Interpretable Greedy-Tree Sums (FIGS)
- Interpretable Machine Learning - Chapter 5
- GAMI-Net: An Explainable Neural Network Based on Generalized Additive Models with Structured Interactions
- Neural Additive Models: Interpretable Machine Learning with Neural Nets
- A Responsible Machine Learning Workflow with Focus on Interpretable Models, Post-hoc Explanation, and Discrimination Testing
- This Looks Like That: Deep Learning for Interpretable Image Recognition
- Unwrapping The Black Box of Deep ReLU Networks: Interpretability, Diagnostics, and Simplification
Source: Global and Local Explanations of a Constrained Model
- Lecture Notes
- Software Examples:
- Assignment 2
- Python:
- R:
-
Introduction and Background:
-
Post-hoc Explanation Techniques:
- Interpretable Machine Learning - Chapter 6 and Chapter 7
- Towards Better Understanding of Gradient-based Attribution Methods for Deep Neural Networks
Source: Lecture 3 Notes
- Lecture Notes
- Software Example: Testing a Constrained Model for Discrimination and Remediating Discovered Discrimination
- Assignment 3
Python:
-
Introduction and Background:
- Fairness and Machine Learning - Introduction
- NIST SP1270: Towards a Standard for Identifying and Managing Bias in Artificial Intelligence
- Fairness Through Awareness
-
Discrimination Testing and Remediation Techniques:
- Certifying and Removing Disparate Impact
- Data Preprocessing Techniques for Classification Without Discrimination
- Decision Theory for Discrimination-aware Classification
- Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification Without Disparate Mistreatment
- Learning Fair Representations
- Mitigating Unwanted Biases with Adversarial Learning
Source: Responsible Machine Learning
- Lecture Notes
- Software Example: Attacking a Machine Learning Model
- Assignment 4
Python:
-
Introduction and Background:
-
Machine Learning Attacks:
- Membership Inference Attacks Against Machine Learning Models
- Stealing Machine Learning Models via Prediction APIs
- Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures
- Hacking Smart Machines with Smarter Ones: How to Extract Meaningful Data from Machine Learning Classifiers
-
Links from Lecture 4:
Source: Real-World Strategies for Model Debugging
- Lecture Notes
- Software Example: Debugging a Machine Learning Model
- Assignment 5
DALEX
(R)SALib
(Python)themis-ml
(Python)What-if Tool
(Browser)
-
Introduction and Background:
-
Links from Lecture 5:
A Responsible Machine Learning Workflow Diagram. Source: Information, 11(3) (March 2020).
-
Introduction and Background:
-
Links from Lecture 6: