Machine learning plays an important role in an era where AI is becoming more and more popular. Most people see completed products or concepts. The method behind them is actually the key. Machine learning also distinguishes two different categories in terms of algorithms, namely supervised learning and unsupervised learning . Common machine learning methods are classified as follows:
supervised learning
|
unsupervised learning
|
decision tree
|
clustering
|
neural network
|
self-organizing map
|
random forest
|
Correlation Analysis
|
regression analysis
|
singular value decomposition
|
K nearest neighbor
|
|
Naive Bayes
|
|
exist supervised learning Medium, general use Flagged Data sets for training, supervised learning is mainly used for Classify and establish regression relationships ; rather than supervised learning use Unlabeled data The purpose of training is to discover some structures or patterns from the data. Most of them are used in clustering and Dimensionality reduction superior.
Understanding machine learning: the neural network approach
Today I will introduce the neural network that is often heard in supervised learning and has a long history of development. The neural network used in JMP is ANN (artificial neural network) artificial neural network method , compared with the human body, it is the concept of neurons. Neurons include cell bodies, dendrites, and axons. The main function of dendrites is to receive signals from other neurons. These signals will be transmitted to dendrites through synapses, and dendrites will The received signals are integrated and passed to the neuron cell body for further processing. The axon is mainly responsible for transmitting the output signal of the neuron. When a neuron is activated, it transmits signals to other neurons through the axon. The synapse is the point of contact between two neurons. Through it, signals can be transmitted between neurons through neurotransmitters, and each Each synapse has a specific strength equivalent to the weight of the neural network. (figure 1)
figure 1
In the neural network (Figure 1-2), the perceptron is an important component, which includes input, weighted sum, activation function (Activation functions), and output, where output = activation function ( ∑ i ( w i × x i ) + b ) , wi is the weight, xi is the input value, and b is the bias. The following is a schematic diagram of a single-layer neural network, and the simple multiple linear regression expression when the activation function is a linear function (Figure 2).
Figure 2: Simple multiple linear regression expression when the activation function is a linear function
3 kind JMP The activation function used within
- TanH:
: The hyperbolic tangent function is an S-shaped function, between -1 and 1. It is the centralized and scaled version of the logistic function, where x is a linear combination of input variables (Figure 3), and its output is centralized, When Y is a categorical variable or a nonlinear relationship, it can be used as an activation function, which is also a commonly used activation function. The gradient of the extreme values -1 and 1 at both ends is close to 0, which may slow down the learning speed.
Figure 3: Linear combination
- Linear: Also known as the identity function, it does not go through the variable conversion process (Figure 4), that is to say: The variables show a linear relationship, and the usual usage scenario is to reduce the dimension of the input variables. .
Figure 4:
- Gaussian: Gaussian function:
, it belongs to the Radial Basis Function, which is a nonlinear function and is applicable when the reaction plane is a Gaussian plane (Figure 5). The disadvantage is that the weight update speed is slow, which affects the efficiency and effect of training. Compared with Tanh, The frequency is lower.
Figure 5-1:
Figure 5-2:
Next, we will explain how to set up the neural network in JMP: (Figure 6)
Figure 6
The initial Validation Method is used to evaluate the model explanation ability of the training set and validation set. Two cross-validation methods, KFold and Holdback, are given, because KFold uses all the data for verification and provides multiple estimates, which helps the stability of the model evaluation. , biased results can be avoided for small samples, the execution time is longer, and the Holdback execution time is short, but data verification will have relatively high variability.
The Number of Tours below represents the number of times to restart the model training process to prevent the model from falling into the local optimal value instead of the global optimal value. In terms of hidden layer settings, JMP Standard Edition provides single-layer hidden layer settings. JMP Pro provides a dual-layer hidden layer setting. Using more hidden layers is usually to explore complex non-linear relationships when the data structure is complex or the data is hierarchical and sequential, such as variables of different types and numerous data sources; The purpose of setting multiple nodes is to learn more patterns. When there are more input variables, the nodes that need to be set need to be improved.
For overfitting, JMP provides Penalty Method as follows:
(Squared): Used when it is assumed that most of the input variables have an impact on the model.
(Absolute) or
(Weight Decay): Used when there are many input variables, but only some of them are expected to affect the model.
JMP also provides options for variable conversion, such as Transform Covariates, which converts input variables into approximately normal distributions. The benefits of conversion can make training converge faster and avoid gradient disappearance. 1 problems, etc. Similarly, for situations where the output variable has a skewed distribution, thick tail, etc., you can use the Robust Fit option to avoid the influence of outliers.
The process of fitting the model
Based on the above description, briefly describe the process of fitting the model:
Use the likelihood function plus the penalty function, and use the quasi-Newton iteration method, also known as BFGS. 2 Method, find the point where the loss function plus the penalty function (if considered) reaches the minimum (Figure 7) to adjust the parameter weights. Generally speaking, the minimum point can be found by using cross-validation about 20-50 times.
Figure 7
When JMP is fitting the model, there is also a Boosting method to choose from. The learning rates are between 0 and 1 and the number of models to be fitted can be adjusted. It is added up by a series of small models. To form the final model, each small model is fitted based on the error of the previous model, taking into account the number of fitted models and whether the overall model interpretability can no longer be improved after cross-validation. For small models, Generally, only a single hidden layer and one or two nodes are set, which has the advantage of improving the accuracy of the model. (Figure 8)
The setting of the learning rate and the number of models is generally that the closer the learning rate is to 1, the smaller the number of models should be. The more complex the small model is set using the Boosting method, the meaning of the bonus will be lost. JMP does not allow the setting of two layers. Hidden layer combined with Boosting method.
Figure 8
Lithography / Etching case sharing
Examples cited below from the 2022 JMP Discovery Summit 3 , Sidewall defects sometimes occur during the lithography/etching stage (Figure 9). It is necessary to use a scanning electron microscope SEM (Scanning Electron Microscope) to see whether the line width is within the specification. It is usually difficult to catch the abnormality because it is difficult to measure it exactly. The location of the detected defect. Therefore, open source software is used to first convert the images into image data indicators, and then JMP is used for analysis and modeling. At this time, the corresponding Y is a categorical variable of good or bad, and X corresponds to each image data indicator (Figure 10) ( For indicator description, please refer to the last link),
Figure 9
Figure 10
The first step is to select the Y and X variables (Figure 11)
Figure 11
In the second step, KFold is selected to avoid small sample estimation bias, a single hidden layer and TanH activation function are selected to convert the relevant input variables into variables (Figure 12)
Figure 12
The third step is to determine indicators such as error resolution, prediction defect accuracy, and Rsquare (Figure 13)
The fourth step is to find the parameters that have a significant impact on the weight based on the prediction profiler (Figure 14)
(Figure 14)
The fifth step is to store the prediction formula and view the weights (Figure 15)
(Figure 15)
Finally, we summarize some rules of thumb for setting up JMP on neural networks, as follows:
- Linear activation functions are usually paired with input variables, while output variables are usually paired with nonlinear activation functions, especially when there are too many variables and you want to reduce the dimension.
- When more nodes may lead to overfitting, keeping the number of nodes in the hidden layer less than or equal to the number of input variables can be used as a starting set point.
- Small neural network models have better computing speed and better stability. If the prediction accuracy is similar to that of large models, small models should be the main ones.
- When the number of nodes or hidden layers in the hidden layer of the input variable is too large, unnecessary complexity may be introduced, leading to model overfitting. Cross-validation methods can be used to confirm and adjust settings.
- Eliminating outliers as much as possible can avoid estimation bias and make predictions more efficient.
- The balancing performance of the data improves the prediction accuracy. For example, when Y is a categorical variable, the number of each category should be as consistent as possible.
Neural network, as a common supervised learning method, in this article we summarize the main components and training process of neural network, including perceptron, activation function and model configuration. welcome Try JMP for free for 30 days , develop different AI learning models for your company.
Recommended reading:
References
- Use Case: JMP in the Building of an Image Analysis Pipeline (2022-US-30MP-1096)
