Modeling performance in Neural NetworkHi,
1. In many(some ?) cases, modeling performance of JMP NN is better than that of others(i.e. python).
Yesterday, I tested and compared the NN results of JMP and python with their data (a few other data sets) in a semiconductor-related company. Based on R-square, python's results were 70%-80%, but JMP's was 95%-99%.
Why is JMP's analysis better ? In what cases would this result ?
2. Related question. Often, validation performance is better than training performance. In what cases would this result ?
When there is no clearly important factor ? When the number of data is not large ? When there are many same values in data ? When the number of nodes is large ?
I am wondering how to interpret this. Please help me. Thanks.
Re: Modeling performance in Neural NetworkFor (1), I guess it won't be an easy comparison. I guess there are countless NN packages in python. Which one did you use? What setting? Also, what R-square are you comparing? If JMP partitions data, do you use the same partition to try out in python? etc.
For (2), I just tried our TireTread data, Y is Abrasion. And the result is opposite of what you observed. Here is what I have done.
- Use the embedded Neural script, but remove all Y's, except Abrasion.
- Default settings, Go.
- Right click Training R-sqr, Bootstrap, check Fractional Weights, a random seed 55555.
- Right click Validation R-sqr, Bootstrap, check Fractional Weights, same random seed 55555.
- Put bootstrapped R-sqr's side by side in the same table. Create a new column with formula: ValidationRSqr > TrainingRSqr. The mean of the column is less than 0.5. That means validation R2 is worse than the training one on average.
Re: Modeling performance in Neural NetworkHi Peng,
The customer's question is a bit vague, but in many cases JMP's results have been excellent, so I estimate they asked this question. (Q2) As in your comment, if I check using the bootstrap function, the probability that the Training R-Square value is greater than the Validation R-Square value is much higher.
Thanks
