Many people feel that they have participated in so many trainings and have accumulated a lot of experience in actual work, but why are my modeling results not ideal? Today we will talk about the methods and ideas of predictive modeling in data analysis, and use small cases to help you understand possible misunderstandings and biases in methodology, so that you can carry out modeling and analysis more easily.
What is predictive modeling?
When it comes to predictive modeling, friends who don’t know much about it may find it inscrutable. In fact, prediction modeling is everywhere in life. For example, when we wake up in the morning and see that the weather is cloudy, we predict that it may rain today. This is a simple prediction. In other words, when a doctor sees a patient with various symptoms, the doctor predicts what percentage of the patient is likely to have a cold and what percentage of the disease is the patient. This is also a prediction.
What are the core and methods of predictive modeling?
There are many methods of predictive modeling. Traditional methods include regression analysis, such as linear regression, polynomial regression, logistic regression, etc.; penalty models, such as ridge regression, Lasso regression, etc. Modeling methods based on machine learning, such as neural networks, support vector machines (SVM), decision trees, random forests, boosted trees and other methods, without exception, provide many choices for data analysts.
But no matter what kind of predictive modeling is done, it is essentially to find the best model that can truly restore the actual situation and discover a real law. But in most cases, perhaps the most important problem we face is: How to judge whether the model is good or not? Is there overfitting? That is to say, the most critical core issue of the prediction model is whether it can achieve the optimal balance of variation and deviation.
What is the equilibrium of variation and deviation? In layman's terms, the model must not only perform well in the training set data used for modeling, but also perform better if the data collected by a different group of people is changed. For example, if the same medicine is taken by the first group of people, the effectiveness can reach 80%, which seems to be very good, but if another group of people takes the medicine, the effectiveness of the medicine may be only 30%. This shows that the effect is not that good. The real effect should be like this: if (theoretically) we do countless samplings and sample a hundred people each time, the effect should be very stable in these countless samplings, and the effect of each sampling will be very good. The so-called good effect means that the model prediction effect is good, the so-called stable means that the number of variations is small, and the so-called instability means that the number of variations is large.
It sounds easy to understand, but in practice we cannot achieve countless samplings. In most cases, there is only one sampling. So, how to use one sampling to evaluate the stability of the model? This requires the use of cross-validation methods.
What is cross-validation? For example, let's say we divide the crowd into several parts, keep some of them as training data, and then keep one part as verification data, and then extract some of the others to use as training data. In short, every piece of data is taken out for verification, which is called cross-validation. All the data for verification are not data used for modeling, so the results are more convincing.
For example, if there are two points, it is very easy to build an optimal model. This straight line is an optimal model (the deviation is small, in fact, its deviation here is 0).
As can be seen from the figure, in this data, the green line obviously performs not as well as the red line (large deviation). However, when adding a verification data, if the blue one is the verification data, then you will find that the red one may not necessarily be as good as the green one. So in this case, the red line has a small deviation but a large variation. Because you only seem to have the best fitting effect in the training data, but if you switch to another batch of data (validation data), you will find that the fitting effect immediately becomes worse. It seems that the deviation is very small, but the variation is too large, so this is not what we are pursuing. What we are really pursuing is a variation that must be small and the deviation must be small, or at least find it between the two. A compromise, which is what predictive modeling is all about. How to find such a compromise, generally use cross-validation. Therefore, to build a good prediction model, whether it is a neural network, decision tree, SVM, etc., we are actually looking for such a compromise point.
From a practical point of view, predictive modeling is mainly about determining two points.
- Determine which variables influence the results . Because prediction must find variables to predict. For example, if there are a hundred variables, do all these hundred variables contribute to the prediction? So you should first determine which indicators contribute to it.
- What is the relationship between these contributing indicators and the results? ?
In fact, in most cases, many of us focus on the first one, which variable affects the result, but in fact, few people pay attention to the actual relationship between these variables and the result, especially in traditional prediction models. , if you don’t pay attention to these, you will find that many constructed models are actually invalid. In this regard, machine learning methods will be better, because machine learning can help you find an optimal dividing point regardless of whether the variables are linear or nonlinear.
However, traditional models are slightly inferior in this regard, because most of the traditional models are linear models and cannot directly handle nonlinearity. Traditional models, such as linear regression, logistic regression, etc., generally only pursue small deviations, but methods such as Lasso regression and elastic network not only pursue small deviations, but also pursue small variations, which can be between the two. Find an equilibrium, so these methods are often very popular now because it takes into account the balance of deviation and variation.
So, are traditional predictive modeling methods obsolete? the answer is negative.
Neural networks, support vector machines, decision trees, random forests, boosted trees and other methods are indeed cutting-edge methods that many people like to pursue, but traditional modeling will not be eliminated because of this. It actually still has its own advantages.
First of all, as long as the variables are transformed appropriately, traditional linear models can also make nonlinear predictions. Traditional modeling methods mainly use some linear models, most of which can be attributed to generalized linear models, mainly making some linear distinctions. Machine learning methods may have more advantages in nonlinear distinction. But this does not mean that traditional models must not be able to make these nonlinear distinctions. Of course, for building traditional prediction models, if you simply put the variables directly into the software, you will never be able to make a non-linear distinction, because the model itself is linear, which has been defined. However, with some exploration and transformation of these variables, a non-linear distinction can still be achieved.
Secondly, the traditional regression model is very intuitive. In the traditional regression model, we can see very clear parameter estimates, P values and other indicators that everyone is concerned about. If you want to list the model, it is very simple. You can directly List all the models.
In contrast, for machine learning, especially deep learning, etc., it is almost impossible to list a model (of course, this does not mean that these models do not have formulas, it is just that they are too complicated to list). So in fact it is more like a black box process. The so-called black box can be imagined as a machine. You put meat in here and sausage comes out there. But you don’t know how to process it in the middle. Maybe sometimes you want to understand the intermediate processing process, and you will find that only the machine can understand the things inside. The machine knows very well how to implement the intermediate steps step by step, but we can't understand it because it is too complicated. Therefore, we often only see a black box process, so we cannot give a clear formula.
Sometimes we don't care about the formula and only care about the prediction effect, but sometimes when you have to list the model (for example, you need to apply for a patent), if you can't describe it clearly, it will be difficult to apply for the patent. At this time, you may often need a clear formula to help.
To sum up, there is no such thing as saying whether a method is outdated. A method is not outdated or outdated, only whether it is suitable or not. It's just that some methods have some applicable conditions when applied. As for when to use which method, it mostly depends on the actual situation. For example, machine learning often has greater advantages when there are many variables.
Next, we use two small cases to actually look at the comparative application of traditional methods and machine learning.
Logistic return VS neural network
Case: A study analyzed the influencing factors of hypertension (binary variable, yes or no), and investigated gender, age, and 4 serological indicators, represented by A, B, C, and D respectively. The researcher plans to build a prediction model using the three variables age, B, and C. For the same set of data, we use logistic regression and neural network to model respectively, and then look at the final results.
First, we use logistic regression to build a prediction model and directly incorporate these three continuous variables into the model. The results show that the area under the ROC curve is 0.6023, which is actually a relatively low result, indicating that the model that may be constructed does not have much practical significance.
Then, we used two-layer neural network technology to directly incorporate these three variables (based on the results of the training set), and then also obtained a ROC curve, which was finally found to be around 0.86.
Judging from the results, you may feel that neural networks are better than logistic regression predictions. In fact, this conclusion is not very reliable. why? This has a lot to do with the analyst. If you put the data directly into the software, as the traditional regression method mentioned above is a linear model, the software will of course perform linearly. But machine learning can find the best fitting effect and the best prediction effect in the end. In fact, most machine learning is a process of continuous trial and error, constantly comparing the results and constantly adjusting. Just like we continue to try and make mistakes, we can slowly achieve a better result in the process of trial and error.
Taking the data just now, if we re-analyze it as a statistician, it seems that these three indicators are indeed meaningless, so when we directly include these three continuous data in the analysis, it seems that there is really no statistics. It has scientific significance, so the ROC curve is relatively low.
However, what do you find if you take a closer look? Because the three variables are not linearly related, the results will naturally have no statistical significance. If you use classification, you can see it very clearly.
Therefore, if you put these variables that are not linearly related directly into logistic regression to build a linear model, you can imagine that the prediction results obtained are definitely not ideal.
If we do not put the original variables in, but re-enter the classification results as variables into the model, and still use logistic regression to re-operate, you will see that the ROC curve immediately becomes 0.86, which is much higher than the 0.6 just now.
why? Because we have found its law, the variables are not a linear relationship, but a non-linear one. After such a non-linear classification, you will find that this is actually its real relationship. And once you find the relationship, you will find that the model results will become much more ideal, and the prediction effect will be much better.
Therefore, when using traditional regression models to build predictive models, we must have more analytical experience. Otherwise, you may conclude that "traditional models are really not as good as machine learning and other new technologies, and are not as good as machines." Study well" is such an illusion. In fact, in many cases, the most important thing in building a model is not the method, but the analyst himself. Many times the problems we find are not problems with the model itself, but more that we have not discovered their real laws.
Logistic return VS decision tree
Similarly, let’s compare logistic regression and decision trees.
Still the same case above. Take the variable age as an example. If a decision tree is used, age is directly included in the model as a continuous variable, and the decision tree will automatically divide it into three categories.
After exploration, we found that it is more reasonable to divide it into three categories. Finally, we found that the results of our own exploration were similar to those of the decision tree.
Of course, this kind of analysis does require some experience accumulation on the part of the analyst. When you have done enough data analysis work, you will naturally have experience in such modeling and analysis. Therefore, data analysts need to have the ability to clearly distinguish models and determine whether they are right or wrong. Otherwise, if they blindly pursue so-called advanced methods and are divorced from reality, just to pursue the fitting effect on data, it will be meaningless. of. All models must be based on the actual situation of the research, and finally must return to actual analysis work and serve actual work.
based on JMP interactive predictive modeling
What is interaction? In general, the interaction in data analysis means that the computer can respond at any time to the content that the operator adjusts at any time. Traditional software generally writes a command and then outputs the result. If you want to get other results, you need to write another command, which is a command-to-result model.
But JMP can not only go from commands to results, but also from results to commands. These two are real interactions. Any operations on the data will be directly reflected in the interactive results, and conversely, any adjustments to the results will also be directly reflected in the data table. In addition, JMP not only provides various platforms for traditional regression analysis, such as logistic regression, but also includes currently commonly used machine learning platforms such as neural networks, random forests, decision trees, boosting trees, etc., and can provide interactive visualization This method, without programming, can meet the various needs of academia and industry for basic analysis, exploratory data analysis and predictive modeling.
JMP Starter options for Predictive Modeling.. From JMP Pro VIEW>JMP STARTER.JMP Starter options for Predictive Modeling.. From JMP Pro VIEW>JMP STARTER.JMP Starter options for Predictive Modeling.. From JMP Pro VIEW>JMP STARTER.JMP Starter options for Predictive Modeling.. From JMP Pro VIEW>JMP STARTER.JMP Starter options for Predictive Modeling.. From JMP Pro VIEW>JMP STARTER.
If you want to know more about how JMP interactive predictive modeling can better serve your actual work and how to better establish scientific data analysis thinking, Welcome to download Experience it for yourself with a one-day free trial of JMP.
