Degree of confidence in decision tree prediction

Question

Hi,

A customer in a forensic analysis situation has built models in order to classify unknown samples. They have measured variables for a series of samples of known origin. They use these to build a model that predicts the origin for any sample based on the measured variables.

They have built some useful partition and neural net models. The models tell them the most likely origin for a sample and the probability that it belongs to that class.

The question the customer has is: what is our degree of confidence in that probability?

They think bootstrapping seems like a solution to this. However, that is not possible for NNs or DTs. You can build a bootstrap forest and specify that you don't want it to do the random variable selection. This creates a series of probabilities and then averages them all. But there is no easy way to get at the distribution or qauntiles for the probabilities predicted by the individual trees. Not without breaking down the prediction formula manually or with some scripting.

A simpler solution would be to rebuild the models using logistic regression or PLS-DA and then you can do the 1-click bootstrapping. But the customer rather likes their NN and DT models. Plus the PLS-DA model gives probability predictions < 0 and > 1, which is not very reassuring.

Are there any elegant solutions that I am missing?

Thanks,

Phil

Phil_Kay · Answer

Thanks for the scripting solution, Don.

I wanted to make sure there wasn't a point-and-click solution that I was missing. Seems like I am not.

Thanks again,

Phil

DonMcCormack · Answer

Phil,

You can bootstrap with any of the predictive models in JMP, it just takes a bit of JSL. Basically, loop over the number of time you want to resample, rerunning the platform each time. At the end of each loop, you'll have to determine whether an observation from the test group is correctly IDed or not. At the end of the bootstrap, you can total the number of times and observation was correctly IDed. I think it would be easiest if you used a validation column. In general, the code would look like this:

For(i =1, i<=nLoops, i++,

//Reshuffle validation column

//Run analysis platform

//Collect info for correct/incorrect classification for observations in test set at step i

);

While it's not a true confidence interval (not sure you can even get one for most predictive modeling techniques) I think it's close enough in spirit to be useful.

Don