One of the dangers in machine learning is overfitting. John Elder’s target shuffling provides an very intuitive assessment of the model. There are many references out there. Below is just one of them. http://www.kdnuggets.com/2014/02/3-ways-to-test-accuracy-your-predictive-models.html. It will be good to have that capability in JMP.
Xiaojie Zhou zhou.x.1@pg.com - Sharon Woods Innovation Center, Cincinnati
@SamGardner It looks like this is being investigated by Development should we change the status?
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />Simulate will be available in JMP Pro 13.0.
This will be possible using the Simulate feature in JMP Pro by following these easy steps. To do target shuffling on a target named :Y, for example:
1) Right-click on Y and select New Formula Column -> Random -> Sample Without Replacement. This creates a new formula column called Shufflethat randomly shuffles the rows of Y.
2) Run the model on the original response column.
3) Find the model diagnostic you are interested in (e.g. R Square, AIC, Misclassification Rate, etc.) and right click on it in the report.
4) In the popup that appears, select the Simulate option, near the bottom of the list of options.
5) In the Simulate dialog that appears select Y for “Column to Switch Out” and Shufflefor “Column to Switch In”.. Then click OK. If you want, prior to clicking OK you can optionally set the random seed if you want the results to be reproducible and or change the number of samples to something other than 2500 if you wish. A smaller number of samples could be desired if the model fit will take a long time. You can also just go with the defaults.
Simulate will then do the target shuffling for you and create a new table of the shuffled R Squares or other diagnostic you chose to evaluate using target shuffling. Click the Distribution Script in the new table and you can compare the value in the report to the distribution of 'shuffled' R Squares.
I will point out that while John Elder is a strong advocate of target shuffling, Target Shuffling is really intended as a way to prevent you from using a model that has overfit the data. Using models chosen using proper validation practices in combination with the overfit-preventing algorithms in JMP's Neural, GenReg, PLS, and Partition platforms tend to make Target Shuffling less necessary than it would be with other general purpose statistical software like R, SPSS, or Statistica for which Target Shuffling is an essential component of the model evaluation process.
