To the JMP Japan community:
Could you please tell me about cross-validation methods for data with a small number of samples?
Number of cases: 44, positive: 23, negative: 21.
Because there were many extracted factors, two metrics were selected based on the data reliability (ICC between two raters) and the ROC-AUC value, and a logistic analysis was performed. Both factors were significant for the results, and the combination of the two produced a reasonable result of ROC-AUC>0.85. The reviewer pointed out that cross-validation was necessary to prevent overfitting, and because there were few cases, we decided to choose LOOCV (leave one out of the box method).
When I run Analyze > Generalized Regression > Lasso > LOOCV, the following results are displayed. I have never used this method before, so I'm not sure how to interpret the results. Could someone with more knowledge please explain?
By the way, the results of the binomial logistic are as follows.
Thank you for your cooperation.
Lasso regression (pronounced "lasso" or "ras-") is one of a group of regression methods broadly called penalized regression or regularized regression. Other types of penalized regression (regularized regression) include ridge regression and elastic net regression.
In standard logistic regression, regression coefficients (parameters) that maximize the log likelihood are found. The estimation method for finding parameter values that maximize the log likelihood is called maximum likelihood estimation. The larger the log likelihood calculated from the training data, the better the fit to the training data. Therefore, maximum likelihood estimation can be said to be an estimation method for finding regression coefficients (parameters) that best fit the training data.
Maximum likelihood estimation in logistic regression has good mathematical properties (consistency, validity, asymptotic normality) when data is large.
Penalized regression (regularized regression) is a method for avoiding overfitting and improving prediction accuracy by using the negative log-likelihood plus a penalty term as the objective function. In the case of Lasso regression, the penalty term is the sum of the absolute values of the parameter estimates excluding the intercept multiplied by a constant (the penalty parameter). Setting this penalty parameter to zero (i.e., weakening the penalty) results in normal maximum likelihood estimation, while setting the penalty parameter to infinity (i.e., strengthening the penalty) results in all parameter estimates excluding the intercept becoming zero.
In the screenshot report you posted, the penalty parameters increase as you move to the left, and decrease as you move to the right.
The "negative log-likelihood" is the log-likelihood with the opposite sign. The "scaled negative log-likelihood" is the average negative log-likelihood per observation (row). A larger log-likelihood indicates a better fit to the data, while a smaller negative log-likelihood with the opposite sign indicates a better fit to the data.
When calculating parameter estimates in maximum likelihood estimation or penalized regression, the negative log-likelihood for the training data is used. In the plot on the right of the report you submitted, not only the negative log-likelihood for the training data is plotted, but also the negative log-likelihood for the validation data. The more complex the model is (by loosening the penalties, increasing the number of explanatory variables, etc.), the smaller the negative log-likelihood for the training data becomes. However, this is not the case for the negative log-likelihood for the validation data.
Generally, if you have a fairly large amount of data, a simple model, and specify the true model, the negative log-likelihoods of the training data and validation data will be the same, as shown in the following figure.
However, in situations where there is little data, even if the true model is fitted, the negative log-likelihood of the validation data will be larger than the negative log-likelihood of the training data, as shown in the following figure, and the plot of the validation data will often be U-shaped.
A larger negative log-likelihood for the validation data than for the training data suggests that the model is overfitted to the training data, meaning that it fits the training data well but the validation data poorly. A U-shape for the validation data indicates that the more complex the model, the worse the fit to the validation data.
■The confidence interval for AUC can provide a rough estimate of how uncertain the AUC estimate is. If you fit a logistic regression model using "Generalized Regression," you can obtain the confidence interval for AUC using the following steps:
(1) Click the red triangle button in the "Logistic Regression" outline.
(2) From the menu that appears, select [Save Columns] → [Save Prediction Formula] .
This operation creates a column of probability prediction formulas in the data table.
(3) From the pull-down menu, select [Analysis (A)] → [Predictive Model] → [Compare Models] .
(4) In the "Select Columns" list box on the left side of the invoked launch window, select the two columns containing the probability prediction values ( "Probability (Column Name = Event Name)" and "Probability (Column Name = Non-Event Name)") and click the [Y, Predictor] button.
(5) Click the [OK] button.
(6) Click the red triangle button in the "Model Comparison" outline and select "AUC Comparison."
■If you are asked to "calculate the AUC based on probability predictions obtained using the leave-one-and-leave-one method," JMP does not provide a GUI for such a process. To do this in JMP, you must program using a scripting language called JMP Scripting Language (JSL).
Below is a sample program. It is just an example.
This will clear the row attributes of the data table, so please prepare only the data table you want to analyze in advance. When doing so, please remove any rows with excluded attributes.
In the following example, rows are removed one by one, a logistic regression model is fitted, and the predicted probability for each row is calculated.The ROC curve and AUC are then calculated based on the predicted probability.
The following examples are just examples.
Also, for this to work, you will need to change the column names and event data values accordingly. Note that by default, all rows with missing values are removed before the analysis (the analysis is performed using listwise removal).
Names Default To Here(1);
dt = Current Data Table();
eta = J(NRow(dt),1,.);
For(i = 1, i <= NRow(dt), i++,
dt << Clear Row States; // 行属性のクリア
Row State(i)= Excluded State(1); // 第i行に除外属性を付与
// ロジスティックモデルのあてはめ
plat = dt << Fit Model(
<
<INVISIBLE></INVISIBLE>
<DIV>
</DIV>
<DIV>■In any case, if the explanatory variables are two-variable logistic regression models with n = 50 (event and non-event variables are about n = 25), even if the true model is fitted, the estimation error will be large, and it will be difficult to determine whether the two-variable model is a good model (for example, would it be better to use other explanatory variables? Would it be better to add other explanatory variables?). It will also be quite difficult to determine whether the estimated model will be practical as a real predictive model in the future. Even if internal validation can be confirmed using the above methods and other methods, I believe that judgments will have to be made conservatively and cautiously.</DIV>
<DIV>
</DIV>
<DIV>■TRIPOD is a guideline for building predictive models in the medical field, but I did not pay attention to the TRIPOD guidelines when writing the answer above. I think there are many other things that should be checked when building a predictive model.</DIV>
<DIV>
</DIV>
