No matter whether you are managing your own funds or clients' funds, as long as you are managing assets, every investment decision is of great significance. Friends who do technical analysis or fundamental analysis are well aware of the details of every decision, but friends who model through machine learning and deep learning may be very upset, because the model that directly produces decision signals may be a black box, and it is difficult to understand why the model will produce some kind of signal. Even many conservative private equity funds put the interpretability of the model in the risk control beforehand. In fact, the interpretability of the model is easy to realize, and the difficulty lies in whether the researchers have in-depth thinking and understanding of the model.
introduce
Great changes have taken place in the field of machine learning in the past decade. Starting from a purely academic and research field, we have seen the wide application of machine learning in various fields, such as retail, science and technology, medical care, science and so on. In the 2 1 century, the important goals of data science and machine learning have changed to solving practical problems and completing complex tasks automatically, which makes our life easier, not just doing experiments and publishing papers in the laboratory. Model tools for machine learning, statistics or deep learning have basically taken shape. New models such as capsule network are constantly being proposed, but it will take several years for these new models to be adopted by the industry. So in industry, the main focus of data science or machine learning is more on application than theory. It is very important that these models can be effectively applied to correct data to solve practical problems.
Machine learning model is essentially an algorithm, which tries to learn potential patterns and relationships from data, rather than establishing immutable rules through code. Therefore, there will always be a series of challenges in explaining how a model works in business. In some fields, especially in finance, such as insurance and banking, data scientists usually have to use more traditional and simpler machine learning models (linear models or decision trees) in the end. The reason is that the interpretability of the model is very important for enterprises to explain every decision of the model. However, this usually leads to a sacrifice of performance. Complex models, such as ensemble learning and neural networks, usually show better and more accurate performance (because real relationships are rarely linearly divided in nature). However, we can't explain the decision of the model properly in the end. In order to solve and discuss these gaps, in this paper, I will focus on the importance and necessity of model interpretability.
motive
As a data scientist who works in an enterprise and often guides others, I find that data science is still often regarded as a black box, which can provide what people want with "magic" or "alchemy". However, the cruel reality is that real-world projects rarely succeed without a reasonable and sufficient explanation of the model. Today, data scientists build models and provide solutions for enterprises. Enterprises may not know the complex details of model implementation, but they have to rely on these models to make decisions. They do have the right to ask such questions: "How can I trust your model?" Or "How does your model make decisions"? Answering these questions is something that data science practitioners and researchers have been trying for many years.
Data scientists know that there is a trade-off between model interpretability and model performance. Here, the model performance is not the running time or execution performance, but the accuracy of the model in forecasting. There are several models (including simple linear models and even tree-based models), whose prediction principles are easy to understand intuitively, but they need to sacrifice model performance because their results have high deviation or variance (under-fitting: linear models) or are easy to over-fit (tree-based models). Recently, more complex models that have developed rapidly, such as ensemble model and deep learning, usually produce better prediction performance, but they are regarded as black-box models because it is difficult to explain how these models really make decisions.
Although some people say that it is enough to know that the model has good performance, why should we know the principle behind it? However, as human beings, most decisions are based on logic and reasoning. Therefore, the decision-making mode of artificial intelligence (AI) will undoubtedly be doubted. In many practical situations, biased models may have real negative effects. This includes the prediction of potential crimes, judicial sentencing, credit scoring, fraud detection, health assessment, loans, autonomous driving and so on. The understanding and interpretation of the model is the most important. Cathy O' Neil, a data scientist and writer, also emphasized this point in her famous book The Weapon of Mathematical Destruction.
Kate Crawford, a famous scholar and writer, talked about the influence of deviation in machine learning and its influence on society in NIPS 20 17.
Interested readers can also take a look at her famous article "The White Man Problem of Artificial Intelligence" published in The New York Times, which shows us cases of machine learning applications, including image classification, crime risk prediction, service availability and so on. These applications are extremely unfriendly to blacks. If we want to use machine learning to solve these problems, all these real-world scenarios are telling us how important model interpretation is.
In the past year, I have seen the need for model interpretation while solving industrial problems. I am also writing my new book "Practical Machine Learning" in Python 2. During this period, I had the opportunity to interact with outstanding employees in DataScience.com, who were very aware of the necessity and importance of human interpretability in the machine learning model. They have also been actively studying solutions and developed the popular python framework Skater. Later, we will study the skater in depth and make some practical model explanations in this series of articles.
Understanding model interpretation
Machine learning (especially deep learning) has only been widely adopted in recent years. Therefore, model interpretation as a concept is still mainly theoretical and subjective.
Any machine learning model has a response function, which attempts to map and explain the relationship and pattern between self (input) variables and dependent (target or response) variables.
Model interpretation attempts to understand and explain these decisions made by the response function. The key to model interpretation lies in transparency and how easy it is for people to understand model decisions. The three most important aspects of model interpretation are explained as follows.
What are the leading factors of model prediction? We should find out the interaction between features to understand which features may be important in the decision-making strategy of the model. This ensures the fairness of the model.
Why does the model make a specific decision? We should also be able to verify why some key features will drive the model to make some decisions during the forecasting process. This ensures the reliability of the model.
How can we trust the model prediction? We should be able to evaluate and verify any data point and how the model makes decisions. For the direct stakeholders, this should be provable and easy to understand, that is, the model runs as expected. This ensures the transparency of the model.
Interpretability means that people (including non-experts in the field of machine learning) can understand the choices (how, why and what) made by the model in its decision-making process.
When comparing models, apart from the performance of the model, if the decision of one model is easier to be understood by human beings than that of another model, it is said that this model is more interpretable than the other model.
Importance of model interpretation
When solving machine learning problems, data scientists usually tend to pay attention to the performance indicators of the model, such as accuracy, precision and recall (undoubtedly, this is very important! )。 However, measurement can only explain part of the prediction decision of the model. As time goes by, the model concept drifts due to various factors in the environment, and the performance may change. Therefore, it is extremely important to understand what makes the model make some decisions.
Some of us may think that this model is very good, so why do we need to study it further? It must be remembered that when solving data science problems in the real world, in order to convince enterprises of your model predictions and decisions, they will keep asking "Why should I trust your model?" This is very reasonable. If a person has cancer or diabetes, if a person may pose a risk to society, or even lose customers, will you be satisfied with a model that only makes predictions and decisions? Maybe the other one will be better. If we can know more about the decision-making process of the model (why and how), we may like it better. This provides us with more transparency, explains why the models make some decisions and may make mistakes in some cases, and helps us to build a certain degree of trust in these machine learning models over time.
The point of this part is that it is time to stop treating the machine learning model as a black box, not just trying and analyzing the data, but also analyzing how the model makes decisions. In fact, some key steps towards this path are made by the famous paper "Why should I trust you?"? (Explain the prediction of any classifier) ",the concept of LIME (locally interpretable model agnostic explanation) was introduced by MT Ribeiro, S. Singh and C. Guestrin in SIGKDD 20 16.
They mentioned some important points worth remembering in their papers.
However, understanding the reasons behind the prediction is very important for evaluating trust. It is very important to trust a new model if you plan to act according to the forecast, or choose whether to deploy it. Whether humans directly use machine learning classifiers as tools or deploy models in other products, there is still a crucial problem: if users don't trust models or forecasts, they won't use them.
This is what we have discussed many times in this paper, and it is also one of the key factors that determine the success of data science projects in the industry. This promotes the necessity and importance of model interpretation.
Model interpretation method standard
There are specific standards for the classification of model interpretation methods. Christoph Molnar's Explanatory Machine Learning, An Explanatory Guide to Making Black Box Models mentioned a good guiding standard:
Intrinsic or afterwards? Intrinsic explanation is about using machine learning model, which is essentially explanatory (like linear model, parametric model or tree-based model). Post-event interpretation refers to selecting and training a black box model (integrated method or neural network) and applying interpretable methods (feature importance, partial dependence graph) after training. In our series of articles, we will pay more attention to the methods that can be explained by ex post model.
Is it aimed at a certain model or general purpose? Model-specific interpretation tools are very specific to the inherent model interpretation methods, which are completely dependent on the capabilities and characteristics of each model. This can be coefficient, p value, AIC score related to regression model, rules of decision tree, etc. The general model interpretation method still depends on the analysis of the model afterwards, and can be used in any machine learning model. It usually operates by analyzing input and output pairs of features. By definition, these methods cannot access any model interior, such as weights, constraints or assumptions.
Local or global? This classification of interpretation will discuss whether the interpretation method is to explain a single prediction or the whole model behavior. Or is the range somewhere in between? We will discuss more local and global explanations as soon as possible.
This is not a complete set of standards to classify interpretable methods, because it is still a new field, but it can be a good standard for comparing and contrasting various methods.
Scope of model interpretation
How do we define the scope and boundaries of interpretation? Some useful aspects can be transparency, fairness and reliability of the model. Local or global model interpretation is a clear way to define the scope of model interpretation.
Global interpretation
This is to try to understand "how does the model make predictions?" And "How does model subset affect model decision?" . In order to understand and explain the whole model immediately, we need to explain it globally. Global interpretability refers to the ability to explain and understand model decisions based on the conditional interaction between dependent variables (response variables) and independent features (predictive features) on a complete data set. Trying to understand the interaction and importance of features is usually a good step to understand the global interpretation. Of course, when trying to analyze interactions, it becomes very difficult to visualize features after more than two or three dimensions. Therefore, it is helpful for global interpretation to often look at modular components and feature subsets that may affect the prediction of global knowledge model. Complete model structure knowledge, assumptions and constraints are necessary for global interpretation.
Local interpretation
This is to understand "why does the model make specific decisions for a single instance?" And "Why does the model make specific decisions for a group of instances?" . For local interpretability, we don't care about the internal structure or assumptions of the model, we regard it as a black box. In order to understand the prediction decision of a single data point, we pay special attention to this data point and look at the local sub-region in the feature space near this point, trying to understand the model decision of this point according to this local region. The local data distribution and feature space may behave completely differently and provide a more accurate explanation than the global explanation. The locally interpretable model agnostic interpretation (LIME) framework is a good method, which can be used for the local explanation of model agnosticity. We can use global explanation and local explanation to explain the model decision of a group of examples.
Model transparency
This is to understand "what is the process of creating models from algorithms and features?" . As we all know, typical machine learning models are all about using algorithms to build representations of data features and map inputs to potential outputs (responses). The transparency of the model can try to understand more technical details of how the model is built and the factors that affect its decision. This can be the weight of neural network, the weight of CNN filter, the coefficient of linear model, the segmentation of nodes and decision trees. However, because enterprises may not be very familiar with these technical details, it is helpful to show the transparency of the model by trying to explain the model decision with unknowable local and global interpretation methods.
conclusion
For real-world machine learning projects, model interpretability is very important. Let's try to study human interpretable machine learning, let us open the black box of machine learning model and help people increase their trust in model decision-making.
Write at the end: model interpretation is very important, which can deepen our confidence in the model, especially in the financial market. When the model continues to retreat, confidence is more important than gold. Previously, a private equity team used the StockRanker strategy, but it was too late to make a firm offer. Later, the theoretical principle of the model, the details of each step and the processing flow were clarified, and finally a reliable firm offer was made. It is normal for the firm to encounter retracement in the early stage, not because the model failed to predict, but because the net strategic value began to rise after the early retracement. Therefore, it is very important that the machine learning model can be explained and the model can also be explained, but researchers need to make more efforts.