Report results of all models from all tours in artificial neural networks

Question

Hello,

I am trying to use ANN with 10-fold cross validation on a 400-point set. I am using 10 tours and 10 models in boosting. I want JMP to report model performance measures of all 100 models (10-fold * 10 tours). Is there an easy way to do that?

Also is there a way to tell JMP to try different combinations of nodes and report all their results instead of running one at a time?

Thank you,

Mina.

Dan_Obermiller · Answer

I am travelling for the next few weeks, but if you post your code, I am sure that someone would be willing to help.

But why do you need the fit statistics for each fold? Isn't the goal to find the best neural network model? If so, JMP will do it automatically for you. Again, that is the purpose of k-fold.

As for the confidence intervals, neural networks are typically fit with very large datasets which makes the traditional statistical inference non-informative. In other words, everything typically shows up as statistically significant because the sample size is so large. Confidence intervals on the weights would not be terribly useful since you have the hidden nodes and the boosting would make that difficult. A confidence interval on the predicted values would probably best be handled by bootstrapping rather than trying to make any distributional assumptions.

Mina · Answer

Thank you Dan!

Can you help me with the coding please? I tried but it does not work (this is the first time for me to code in JMP).

Also is there a way to get the 95% confidence limits from neural networks from JMP as it does for linear regression?

Dan_Obermiller · Answer

A few questions. You said you are using 10 tours. Do you mean that you are using 10 models for the boosting? Regardless, I hope my suggestions below will point you in the right direction.

In order to get the fit statistics for each fold, I think you will need to do some scripting. First, you will need to create a validation column that defines each of the fold. Then fit the Boosted ANN model with fold #1 held back, save the results (whichever results you want). Then the script can increment the hold back set to be fold #2, etc. Yes, this would be tedious, but remember that the purpose of k-fold is to have JMP do the heavy lifting for you. JMP should pick the best performing result, you don't need to inspect each fold.

Finally, your last question. What do you mean by "trying different combination of nodes"? Are you referring to changing the number of hidden nodes and you want to compare performance of models with 2 nodes, 3 nodes, 4 nodes, etc.? If so, again, I think that will need to be handled by a script.