开源软件名称（OpenSource Name）：

lopusz/awesome-interpretable-machine-learning

开源软件地址(OpenSource Url)：

https://github.com/lopusz/awesome-interpretable-machine-learning

开源编程语言(OpenSource Language)：

Python 98.3%

开源软件介绍(OpenSource Introduction)：

Awesome Interpretable Machine Learning

Opinionated list of resources facilitating model interpretability (introspection, simplification, visualization, explanation).

Interpretable Models

Interpretable models
- Simple decision trees
- Rules
- (Regularized) linear regression
- k-NN
(2008) Predictive learning via rule ensembles by Jerome H. Friedman, Bogdan E. Popescu
- https://dx.doi.org/10.1214/07-AOAS148
(2014) Comprehensible classification models by Alex A. Freitas
- https://dx.doi.org/10.1145/2594473.2594475
- http://www.kdd.org/exploration_files/V15-01-01-Freitas.pdf
- Interesting discussion of interpretability for a few classification models (decision trees, classification rules, decision tables, nearest neighbors and Bayesian network classifier)
(2015) Interpretable classifiers using rules and Bayesian analysis: Building a better stroke prediction model by Benjamin Letham, Cynthia Rudin, Tyler H. McCormick, David Madigan
- https://arxiv.org/pdf/1511.01644
- https://dx.doi.org/10.1214/15-AOAS848
(2017) Learning Explanatory Rules from Noisy Data by Richard Evans, Edward Grefenstette
- https://arxiv.org/pdf/1711.04574
(2019) Transparent Classification with Multilayer Logical Perceptrons and Random Binarization by Zhuo Wang, Wei Zhang, Ning Liu, Jianyong Wang
- https://arxiv.org/pdf/1912.04695
- Code: https://github.com/12wang3/mllp

Feature Importance

Models offering feature importance measures
- Random forest
- Boosted trees
- Extremely randomized trees
  - (2006) Extremely randomized trees by Pierre Geurts, Damien Ernst, Louis Wehenkel
    - https://dx.doi.org/10.1007/s10994-006-6226-1
- Random ferns
  - (2015) rFerns: An Implementation of the Random Ferns Method for General-Purpose Machine Learning by Miron B. Kursa
- Linear regression (with a grain of salt)
(2007) Bias in random forest variable importance measures: Illustrations, sources and a solution by Carolin Strobl, Anne-Laure Boulesteix, Achim Zeileis, Torsten Hothorn
- https://dx.doi.org/10.1186/1471-2105-8-25
(2008) Conditional Variable Importance for Random Forests by Carolin Strobl, Anne-Laure Boulesteix, Thomas Kneib, Thomas Augustin, Achim Zeileis
- https://dx.doi.org/10.1186/1471-2105-9-307
(2018) Model Class Reliance: Variable Importance Measures for any Machine Learning Model Class, from the “Rashomon” Perspective by Aaron Fisher, Cynthia Rudin, Francesca Dominici
- https://arxiv.org/pdf/1801.01489
- https://github.com/aaronjfisher/mcr
- Universal (model agnostic) variable importance measure
(2019) Please Stop Permuting Features: An Explanation and Alternatives by Giles Hooker, Lucas Mentch
- https://arxiv.org/pdf/1905.03151
- Paper advocating against feature permutation for importance
(2018) Visualizing the Feature Importance for Black Box Models by Giuseppe Casalicchio, Christoph Molnar, Bernd Bischl
- https://arxiv.org/pdf/1804.06620
- https://github.com/giuseppec/featureImportance
- Global and local (model agnostic) variable importance measure (based on Model Reliance)
Very good blog post describing deficiencies of random forest feature importance and the permutation importance
- http://explained.ai/rf-importance/index.html
Permutation importance - simple model agnostic approach is described in Eli5 documentation
- https://eli5.readthedocs.io/en/latest/blackbox/permutation_importance.html

Feature Selection

Classification of feature selection methods
- Filters
- Wrappers
- Embedded methods
(2003) An Introduction to Variable and Feature Selection by Isabelle Guyon, André Elisseeff
- http://www.jmlr.org/papers/volume3/guyon03a/guyon03a.pdf
- Be sure to read this very illustrative introduction to feature selection
Filter Methods
- (2006) On the Use of Variable Complementarity for Feature Selection in Cancer Classification by Patrick Meyer, Gianluca Bontempi
  - https://dx.doi.org/10.1007/11732242_9
  - https://pdfs.semanticscholar.org/d72f/f5063520ce4542d6d9b9e6a4f12aafab6091.pdf
  - Introduces information theoretic methods - double input symmetrical relevance (DISR)
- (2012) Conditional Likelihood Maximisation: A Unifying Framework for Information Theoretic Feature Selection by Gavin Brown, Adam Pocock, Ming-Jie Zhao, Mikel Luján
  - http://www.jmlr.org/papers/volume13/brown12a/brown12a.pdf
  - Code: https://github.com/Craigacp/FEAST
  - Discusses various approaches based on mutual information (MIM, mRMR, MIFS, CMIM, JMI, DISR, ICAP, CIFE, CMI)
- (2012) Feature selection via joint likelihood by Adam Pocock
  - http://www.cs.man.ac.uk/~gbrown/publications/pocockPhDthesis.pdf
- (2017) Relief-Based Feature Selection: Introduction and Review by Ryan J. Urbanowicz, Melissa Meeker, William LaCava, Randal S. Olson, Jason H. Moore
  - https://arxiv.org/pdf/1711.08421
- (2017) Benchmarking Relief-Based Feature Selection Methods for Bioinformatics Data Mining by Ryan J. Urbanowicz, Randal S. Olson, Peter Schmitt, Melissa Meeker, Jason H. Moore
  - https://arxiv.org/pdf/1711.08477
Wrapper methods
- (2015) Feature Selection with theBorutaPackage by Miron B. Kursa, Witold R. Rudnicki
  - https://dx.doi.org/10.18637/jss.v036.i11
  - https://cran.r-project.org/web/packages/Boruta/
  - Code (official, R): https://notabug.org/mbq/Boruta/
  - Code (Python): https://github.com/scikit-learn-contrib/boruta_py
- Boruta for those in a hurry
  - https://cran.r-project.org/web/packages/Boruta/vignettes/inahurry.pdf
General
- (1994) Irrelevant Features and the Subset Selection Problem by George John, Ron Kohavi, Karl Pfleger
  - https://pdfs.semanticscholar.org/a83b/ddb34618cc68f1014ca12eef7f537825d104.pdf
  - Classic paper discussing weakly relevant features, irrelevant features, strongly relevant features
- (2003) Special issue of JMLR of feature selection - oldish (2003)
  - http://www.jmlr.org/papers/special/feature03.html
- (2004) Result Analysis of the NIPS 2003 Feature Selection Challenge by Isabelle Guyon, Steve Gunn, Asa Ben-Hur, Gideon Dror
  - Paper: https://papers.nips.cc/paper/2728-result-analysis-of-the-nips-2003-feature-selection-challenge.pdf
  - Website http://clopinet.com/isabelle/Projects/NIPS2003/
- (2007) Consistent Feature Selection for Pattern Recognition in Polynomial Time by Roland Nilsson, José Peña, Johan Björkegren, Jesper Tegnér
  - http://www.jmlr.org/papers/volume8/nilsson07a/nilsson07a.pdf
  - Discusses minimal optimal vs all-relevant approaches to feature selection
Feature Engineering and Selection by Kuhn & Johnson
- Sligtly off-topic, but very interesting book
- http://www.feat.engineering/index.html
- https://bookdown.org/max/FES/
- https://github.com/topepo/FES
Feature Engineering presentation by H. J. van Veen
- Slightly off-topicm but very interesting deck of slides
- Slides: https://www.slideshare.net/HJvanVeen/feature-engineering-72376750

Model Explanations

Philosophy

Magnets by R. P. Feynman https://www.youtube.com/watch?v=wMFPe-DwULM
(2002) Looking Inside the Black Box, presentation of Leo Breiman
- https://www.stat.berkeley.edu/users/breiman/wald2002-2.pdf
(2011) To Explain or to Predict? by Galit Shmueli
- https://arxiv.org/pdf/1101.0891
- https://dx.doi.org/10.1214/10-STS330
(2016) The Mythos of Model Interpretability by Zachary C. Lipton
- https://arxiv.org/pdf/1606.03490
- https://www.youtube.com/watch?v=mvzBQci04qA
(2017) Towards A Rigorous Science of Interpretable Machine Learning by Finale Doshi-Velez, Been Kim
- https://arxiv.org/pdf/1702.08608
(2017) The Promise and Peril of Human Evaluation for Model Interpretability by Bernease Herman
- https://arxiv.org/pdf/1711.07414
(2018) The Book of Why: The New Science of Cause and Effect by Judea Pearl
(2018) Please Stop Doing the “Explainable” ML by Cynthia Rudin
- Video (starts 17:30, lasts 10 min): https://zoom.us/recording/play/0y-iI9HamgyDzzP2k_jiTu6jB7JgVVXnjWZKDMbnyRTn3FsxTDZy6Wkrj3_ekx4J
- Linked at: https://users.cs.duke.edu/~cynthia/mediatalks.html
(2018) Explaining Explanations: An Approach to Evaluating Interpretability of Machine Learning by Leilani H. Gilpin, David Bau, Ben Z. Yuan, Ayesha Bajwa, Michael Specter, Lalana Kagal
- https://arxiv.org/pdf/1806.00069
(2019) Interpretable machine learning: definitions, methods, and applications by W. James Murdoch, Chandan Singh, Karl Kumbier, Reza Abbasi-Asl, Bin Yu
- https://arxiv.org/pdf/1901.04592
(2019) On Explainable Machine Learning Misconceptions A More Human-Centered Machine Learning by Patrick Hall
- https://github.com/jphall663/xai_misconceptions/blob/master/xai_misconceptions.pdf
- https://github.com/jphall663/xai_misconceptions
(2019) An Introduction to Machine Learning Interpretability. An Applied Perspective on Fairness, Accountability, Transparency, and Explainable AI by Patrick Hall and Navdeep Gill
- https://www.h2o.ai/wp-content/uploads/2019/08/An-Introduction-to-Machine-Learning-Interpretability-Second-Edition.pdf

Model Agnostic Explanations

(2009) How to Explain Individual Classification Decisions by David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, Klaus-Robert Mueller
- https://arxiv.org/pdf/0912.1128
(2013) Peeking Inside the Black Box: Visualizing Statistical Learning with Plots of Individual Conditional Expectation by Alex Goldstein, Adam Kapelner, Justin Bleich, Emil Pitkin
- https://arxiv.org/pdf/1309.6392
(2016) “Why Should I Trust You?”: Explaining the Predictions of Any Classifier by Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin
- https://arxiv.org/pdf/1602.04938
- Code: https://github.com/marcotcr/lime
- https://github.com/marcotcr/lime-experiments
- https://www.youtube.com/watch?v=bCgEP2zuYxI
- Introduces the LIME method (Local Interpretable Model-agnostic Explanations)
(2016) A Model Explanation System: Latest Updates and Extensions by Ryan Turner
- https://arxiv.org/pdf/1606.09517
- http://www.blackboxworkshop.org/pdf/Turner2015_MES.pdf
(2017) Understanding Black-box Predictions via Influence Functions by Pang Wei Koh, Percy Liang
- https://arxiv.org/pdf/1703.04730
(2017) A Unified Approach to Interpreting Model Predictions by Scott Lundberg, Su-In Lee
- https://arxiv.org/pdf/1705.07874
- Code: https://github.com/slundberg/shap
- Introduces the SHAP method (SHapley Additive exPlanations), generalizing LIME
(2018) Anchors: High-Precision Model-Agnostic Explanations by Marco Ribeiro, Sameer Singh, Carlos Guestrin
- https://homes.cs.washington.edu/~marcotcr/aaai18.pdf
- Code: https://github.com/marcotcr/anchor-experiments
(2018) Learning to Explain: An Information-Theoretic Perspective on Model Interpretation by Jianbo Chen, Le Song, Martin J. Wainwright, Michael I. Jordan
- https://arxiv.org/pdf/1802.07814
(2018) Explanations of model predictions with live and breakDown packages by Mateusz Staniak, Przemyslaw Biecek
- https://arxiv.org/pdf/1804.01955
- Docs: https://mi2datalab.github.io/live/
- Code: https://github.com/MI2DataLab/live
- Docs: https://pbiecek.github.io/breakDown
- Code: https://github.com/pbiecek/breakDown
(2018) A review book - Interpretable Machine Learning. A Guide for Making Black Box Models Explainable by Christoph Molnar
- https://christophm.github.io/interpretable-ml-book/
(2018) Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead by Cynthia Rudin
- https://arxiv.org/pdf/1811.10154
(2019) Quantifying Interpretability of Arbitrary Machine Learning Models Through Functional Decomposition by Christoph Molnar, Giuseppe Casalicchio, Bernd Bischl
- https://arxiv.org/pdf/1904.03867

Model Specific Explanations - Neural Networks

(2013) Visualizing and Understanding Convolutional Networks by Matthew D Zeiler, Rob Fergus
- https://arxiv.org/pdf/1311.2901
(2013) Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps by Karen Simonyan, Andrea Vedaldi, Andrew Zisserman
- https://arxiv.org/pdf/1312.6034
(2015) Understanding Neural Networks Through Deep Visualization by Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, Hod Lipson
- https://arxiv.org/pdf/1506.06579
- https://github.com/yosinski/deep-visualization-toolbox
(2016) Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization by Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, Dhruv Batra
- https://arxiv.org/pdf/1610.02391
(2016) Generating Visual Explanations by Lisa Anne Hendricks, Zeynep Akata, Marcus Rohrbach, Jeff Donahue, Bernt Schiele, Trevor Darrell
- https://arxiv.org/pdf/1603.08507
(2016) Rationalizing Neural Predictions by Tao Lei, Regina Barzilay, Tommi Jaakkola
- https://arxiv.org/pdf/1606.04155
- https://people.csail.mit.edu/taolei/papers/emnlp16_rationale_slides.pdf
- Code: https://github.com/taolei87/rcnn/tree/master/code/rationale
(2016) Gradients of Counterfactuals by Mukund Sundararajan, Ankur Taly, Qiqi Yan
- https://arxiv.org/pdf/1611.02639
Pixel entropy can be used to detect relevant picture regions (for CovNets)
- See Visualization section and Fig. 5 of the paper
  - (2017) High-Resolution Breast Cancer Screening with Multi-View Deep Convolutional Neural Networks by Krzysztof J. Geras, Stacey Wolfson, Yiqiu Shen, Nan Wu, S. Gene Kim, Eric Kim, Laura Heacock, Ujas Parikh, Linda Moy, Kyunghyun Cho
    - https://arxiv.org/pdf/1703.07047
(2017) SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability by Maithra Raghu, Justin Gilmer, Jason Yosinski, Jascha Sohl-Dickstein
- https://arxiv.org/pdf/1706.05806
- https://research.googleblog.com/2017/11/interpreting-deep-neural-networks-with.html
(2017) Visual Explanation by Interpretation: Improving Visual Feedback Capabilities of Deep Neural Networks by Jose Oramas, Kaili Wang, Tinne Tuytelaars
- https://arxiv.org/pdf/1712.06302
(2017) Axiomatic Attribution for Deep Networks by Mukund Sundararajan, Ankur Taly, Qiqi Yan
- https://arxiv.org/pdf/1703.01365
- Code: https://github.com/ankurtaly/Integrated-Gradients
- Proposes Integrated Gradients Method
- See also: Gradients of Counterfactuals https://arxiv.org/pdf/1611.02639.pdf
(2017) Learning Important Features Through Propagating Activation Differences by Avanti Shrikumar, Peyton Greenside, Anshul Kundaje
- https://arxiv.org/pdf/1704.02685
- Proposes Deep Lift method
- Code: https://github.com/kundajelab/deeplift
- Videos: https://www.youtube.com/playlist?list=PLJLjQOkqSRTP3cLB2cOOi_bQFw6KPGKML
(2017) The (Un)reliability of saliency methods by Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T. Schütt, Sven Dähne, Dumitru Erhan, Been Kim
- https://arxiv.org/pdf/1711.0867
- Review of failures for methods extracting most important pixels for prediction
(2018) Classifier-agnostic saliency map extraction by Konrad Zolna, Krzysztof J. Geras, Kyunghyun Cho
- https://arxiv.org/pdf/1805.08249
- Code: https://github.com/kondiz/casme
(2018) A Benchmark for Interpretability Methods in Deep Neural Networks by Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, Been Kim
- https://arxiv.org/pdf/1806.10758
(2018) The Building Blocks of Interpretability by Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, Alexander Mordvintsev
- https://dx.doi.org/10.23915/distill.00010
- Has some embeded links to notebooks
- Uses Lucid library https://github.com/tensorflow/lucid
(2018) Hierarchical interpretations for neural network predictions by Chandan Singh, W. James Murdoch, Bin Yu
- https://arxiv.org/pdf/1806.05337
- Code: https://github.com/csinva/hierarchical_dnn_interpretations
(2018) iNNvestigate neural networks! by Maximilian Alber, Sebastian Lapuschkin, Philipp Seegerer, Miriam Hägele, Kristof T. Schütt, Grégoire Montavon, Wojciech Samek, Klaus-Robert Müller, Sven Dähne, Pieter-Jan Kindermans
- https://arxiv.org/pdf/1808.04260
- Code: https://github.com/albermax/innvestigate
(2018) YASENN: Explaining Neural Networks via Partitioning Activation Sequences by Yaroslav Zharov, Denis Korzhenkov, Pavel Shvechikov, Alexander Tuzhilin
- https://arxiv.org/pdf/1811.02783
(2019) Attention is not Explanation by Sarthak Jain, Byron C. Wallace
- https://arxiv.org/pdf/1902.10186
(2019) Attention Interpretability Across NLP Tasks by Shikhar Vashishth, Shyam Upadhyay, Gaurav Singh Tomar, Manaal Faruqui
- https://arxiv.org/pdf/1909.11218
(2019) GRACE: Generating Concise and Informative Contrastive Sample to Explain Neural Network Model’s Prediction by Thai Le, Suhang Wang, Dongwon Lee
- https://arxiv.org/pdf/1911.02042
- Code: https://github.com/lethaiq/GRACE_KDD20

Extracting Interpretable Models From Complex Ones

(2017) Extracting Automata from Recurrent Neural Networks Using Queries and Counterexamples by Gail Weiss, Yoav Goldberg, Eran Yahav
- https://arxiv.org/pdf/1711.09576
(2017) Distilling a Neural Network Into a Soft Decision Tree by Nicholas Frosst, Geoffrey Hinton
- https://arxiv.org/pdf/1711.09784
(2017) Detecting Bias in Black-Box Models Using Transparent Model Distillation by Sarah Tan, Rich Caruana, Giles Hooker, Yin Lou
- http://www.aies-conference.com/2018/contents/papers/main/AIES_2018_paper_96.pdf