XXAI: Extending Explainable AI Beyond Deep Models and Classifiers

Bolei Zhou is an Assistant Professor with the Information Engineering Department at the Chinese University of Hong Kong. He received his PhD in computer science at the Massachusetts Institute of Technology. His research is on machine perception and decision, with a focus on visual scene understanding and interpretable AI systems. He received the MIT Tech Review’s Innovators under 35 in Asia-Pacific award, Facebook Fellowship, Microsoft Research Asia Fellowship, MIT Greater China Fellowship, and his research was featured in media outlets such as TechCrunch, Quartz, and MIT News. More about his research is at http://bzhou.ie.cuhk.edu.hk.

Osbert Bastani is a research assistant professor at the Department of Computer and Information Science at the University of Pennsylvania. He is a member of the PRECISE and PRiML centers. Previously, he completed my Ph.D. at Stanford advised by Alex Aiken, and spent a year as a postdoc at MIT working with Armando Solar-Lezama.

Grégoire Montavon received a Masters degree in Communication Systems from École Polytechnique Fédérale de Lausanne in 2009 and a Ph.D. degree in Machine Learning from the Technische Universität Berlin in 2013. He is currently a Research Associate in the Machine Learning Group at TU Berlin. His research interests include interpretable machine learning and deep neural networks.

Scott Lundberg is a senior researcher at Microsoft Research. Before joining Microsoft, I did my Ph.D. studies at the Paul G. Allen School of Computer Science & Engineering of the University of Washington working with Su-In Lee. My work focuses on explainable artificial intelligence and its application to problems in medicine and healthcare. This has led to the development of broadly applicable methods and tools for interpreting complex machine learning models that are now used in banking, logistics, sports, manufacturing, cloud services, economics, and many other areas.

Zeynep Akata is a professor of Computer Science within the Cluster of Excellence Machine Learning at the University of Tübingen. After completing her PhD at the INRIA Rhone Alpes with Prof Cordelia Schmid (2014), she worked as a post-doctoral researcher at the Max Planck Institute for Informatics with Prof Bernt Schiele (2014-17) and at University of California Berkeley with Prof Trevor Darrell (2016-17). Before moving to Tübingen in October 2019, she was an assistant professor at the University of Amsterdam with Prof Max Welling (2017-19). She received a Lise-Meitner Award for Excellent Women in Computer Science from Max Planck Society in 2014, a young scientist honour from the Werner-von-Siemens-Ring foundation in 2019 and an ERC-2019 Starting Grant from the European Commission. Her research interests include multimodal learning and explainable AI.

Sepp Hochreiter is director of the Institute for Machine Learning at the Johannes Kepler University of Linz after having led the Institute of Bioinformatics from 2006 to 2018. In 2017 he became the head of the Linz Institute of Technology (LIT) AI Lab which focuses on advancing research on artificial intelligence. Previously, he was at the Technical University of Berlin, at the University of Colorado at Boulder, and at the Technical University of Munich. Sepp Hochreiter has made numerous contributions in the fields of machine learning, deep learning and bioinformatics. He developed the long short-term memory (LSTM) for which the first results were reported in his diploma thesis in 1991. In addition to his research contributions, Sepp Hochreiter is broadly active within his field: he launched the Bioinformatics Working Group at the Austrian Computer Society; he is founding board member of different bioinformatics start-up companies; he was program chair of the conference Bioinformatics Research and Development; he is a conference chair of the conference Critical Assessment of Massive Data Analysis (CAMDA); and he is editor, program committee member, and reviewer for international journals and conferences. As a faculty member at Johannes Kepler Linz, he founded the Bachelors Program in Bioinformatics, which is a cross-border, double-degree study program together with the University of South-Bohemia in České Budějovice (Budweis), Czech Republic. He also established the Masters Program in Bioinformatics, where he is still the acting dean of both studies.

Ribana Roscher received the Dipl.-Ing. and Ph.D. degrees in geodesy from the University of Bonn, Germany, in 2008 and 2012, respectively. Until 2015, she was a Postdoctoral Researcher with the University of Bonn, the Julius-Kuehn Institute, Siebeldingen, Germany, Freie Universitaet Berlin, Germany, and the Humboldt Innovation, Berlin. In 2015, she was a Visiting Researcher with the Fields Institute, Toronto, Canada. She is currently an Assistant Professor of remote sensing with the Institute of Geodesy and Geoinformation, University of Bonn. From 2019 to 2020, she was an Interims Professor of semantic technologies with the Institute of Computer Science, University of Osnabrueck, Germany. Her research include pattern recognition and machine learning for remote sensing, especially applications from agricultural and environmental sciences.

Adrian Weller is a principal research fellow in machine learning at the University of Cambridge. He has broad interests across machine learning and artificial intelligence (AI), their applications, and their implications for society, including: scalability, reliability, interpretability, fairness, privacy, ethics, safety and finance. Adrian is Programme Director for AI at The Alan Turing Institute (national institute for data science and AI), where he is also a Turing Fellow leading work on safe and ethical AI. He is a principal research fellow at the Leverhulme Centre for the Future of Intelligence (CFI) leading their Trust and Transparency project; the David MacKay Newton research fellow at Darwin College; and an advisor to the Centre for Science and Policy (CSaP), and the Centre for the Study of Existential Risk (CSER). Adrian serves on the boards of several organizations, including the Centre for Data Ethics and Innovation (CDEI). Previously, Adrian held senior positions in finance. He continues to be an angel investor and advisor.

Tree-based machine learning models are popular nonlinear predictive models, yet comparatively little attention has been paid to explaining their predictions. In this talk I will explain how to improve their interpretability through the combination of many local game-theoretic explanations. I'll show how combining many high-quality local explanations allows us to represent global structure while retaining local faithfulness to the original model. This will enable us to identify high-magnitude but low-frequency nonlinear mortality risk factors in the US population, to highlight distinct population subgroups with shared risk characteristics, and to identify nonlinear interaction effects among risk factors for chronic kidney disease, and to monitor a machine learning model deployed in a hospital by identifying which features are degrading the model’s performance over time.

Recent progress in deep generative models such as Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) has enabled synthesizing photo-realistic images, such as faces and scenes. However, it remains much less explored on what has been learned inside the deep representations learned from synthesizing images. In this talk, I will present some of our recent progress in interpreting the semantics in the latent space of the GANs, as well as reversing real images back into the latent space. Identifying these semantics not only allows us to better understand the internal mechanism in generative models, but also facilitates versatile real image editing applications.

Unsupervised models such as clustering or anomaly detection are routinely used for data discovery and summarization. To gain maximum insight from the data, we also need to explain which input features (e.g. pixels) support the cluster assignments and the anomaly detections.—So far, XAI has mainly focused on supervised models.—In this talk, a novel systematic approach to explain various unsupervised models is presented. The approach is based on finding, without retraining, neural network equivalents of these models. Their predictions can then be readily explained using common XAI procedures developed for neural networks.

An agent who interacts with a wide population of other agents needs to be aware that there may be variations in their understanding of the world. Furthermore, the machinery which they use to perceive may be inherently different, as is the case between humans and machines. In this work, we present both an image reference game between a speaker and a population of listeners where reasoning about the concepts other agents can comprehend is necessary and a model formulation with this capability. We focus on reasoning about the conceptual understanding of others, as well as adapting to novel gameplay partners and dealing with differences in perceptual machinery. Our experiments on three benchmark image/attribute datasets suggest that our learner indeed encodes information directly pertaining to the understanding of other agents, and that leveraging this information is crucial for maximizing gameplay performance.

Assigning credit for a received reward to previously performed actions is one of the central tasks in reinforcement learning. Credit assignment often uses world models, either in a forward or in a backward view. In a forward view, the future return is estimated by replacing the environment through a model or by rolling out sequences until episode end. A backward view either learns a backward model or performs a backward analysis of a forward model that predicts or models the return of an episode. Our method RUDDER performs a backward analysis to construct a reward redistribution to credit those actions that caused a reward. Its extension Align-RUDDER learns a reward redistribution from few demonstrations. An optimal reward redistribution has zero expected future reward and, therefore, immediately credits actions for all they will cause. XAI aims at credit assignment, too, when asking what caused a model to produce a particular output given an input. Even further, XAI wants to know how and why a policy solved a task, why an agent is better than humans, why a decision was made. Humans best comprehend a strategy of an agent if all its actions are immediately evaluated and do not have hidden consequences in the future. Reward redistributions learned by RUDDER and Align-RUDDER help to understand task-solving strategies of both humans and machines.

For some time now, machine learning methods have been indispensable in many application areas. Especially with the recent development of neural networks, these methods are increasingly used in the sciences to obtain scientific outcomes from observational or simulated data. Besides a high accuracy, a desired goal is to learn explainable models. In order to reach this goal and obtain explanation, knowledge from the respective domain is necessary, which can be integrated into the model or applied post-hoc. This talk focuses on explainable machine learning approaches which are used to tackle common challenges in the sciences such as the provision of reliable and scientific consistent results. It will show that recent advances in machine learning to enhance transparency, interpretability, and explainability are helpful in overcoming these challenges.

Explainable machine learning offers the potential to provide stakeholders with insights into model behavior, yet there is little understanding of how organizations use these methods in practice. In this talk, we discuss recent research exploring how organizations view and use explainability. We find that the majority of deployments are not for end-users but rather for machine learning engineers, who use explainability to debug the model. There is thus a gap between explainability in practice and the goal of external transparency since explanations are primarily serving internal stakeholders. Providing useful external explanations requires careful consideration of the needs of stakeholders, including end-users, regulators, and domain experts. Despite this need, little work has been done to facilitate inter-stakeholder conversation around explainable machine learning. To help address this gap, we report findings from a closed-door, day-long workshop between academics, industry experts, legal scholars, and policymakers to develop a shared language around explainability and to understand the current shortcomings of and potential solutions for deploying explainable machine learning in the service of external transparency goals.

Structured control policies such as decision trees, finite-state machines, and programs have a number of advantages over more traditional models: they are easier for humans to understand and debug, they generalize more robustly to novel environments, and they are easier to formally verify. However, learning these kinds of models has proven to be challenging. I will describe recent progress learning structured policies, along with evidence demonstrating their benefits.