237369_en-US

liliconcom

PLS with categorical response

Need to do a JMP Pro demo next week for a customer interested in PLS with categorical (binomial) response.  Does anyone have a good dataset to demo this?  All the PLS datasets I've found so far involve numeric responses.

 

Thanks,

Mark

Re: PLS with categorical response

Update to this thread.  I've noticed an issue with PLS when modeling categorical responses where the prediction formula saved to the datatable does not have the Response Probability column property.  As a result, it is not possible to compare the PLS model with other classification models in Model Comparison.  There may be a specific reason for this, but I am not yet aware what it might be.  I'll update this thread with what I learn from Technical Support regarding this issue. 

Re: PLS with categorical responsefair enough Re: PLS with categorical response

The ROC curve actually contains both error types that you can see on a misclassification rate, it is just expressed differently. The shape of the ROC curve allows you to determine which error type (if either) is a "better" error to make.

The problem with misclassifications is that misclassifications are not very sensitive. In other words, it does not take into account the probabilities of an event. Consider two models: model 1 and model 2. For a given condition, Model 1 predicts a 95% probability of the event occurring while Model 2 predicts 55%. For that condition both models would categorize that observation the same way, but one is much more "certain" of the result than the other. Information criteria such as AICc and BIC are more sensitive model metrics. 

 

I'm not saying that I would not use misclassification, but two models that have the same misclassification rate are not necessarily equal.

Re: PLS with categorical response

Just another opinion: I know ROC curves are a nice way to compare models. However for binomial responses I like to look more at the misclassification rates and especially how homogeneous the errors are distributed to error 1 vs error 2. Why? a model which predicts one level perfectly and the other almost because of more or less rare events it might still look better than another model looking at the ROC curve alone, I believe (@statisticians, please correct me if I'm wrong, am just a mathematician ).

That said you result in a model which just says all predictions are resulting in one level and gets only some error 1 predictions. There might be a model which predicts more errors in an absolute sense but they are predicting both levels kind of good. This model might be not as good as the other model from an ROC curve or as well the pure misclassification rate, but I would still choose it because it is probably closer to the reality.

 

Re: PLS with categorical response

@Dan_Obermiller
I agree with your snark but the issue would be if they have to write a custom script to do it, it doesn't give as compelling a reason to adopt JMP Pro in addition to Python.

I'm currently looking at the Discriminant platform as a potential option which has the ROC diagnostic that customer is used to seeing but trying to understand the pros/cons between that and the PLS platform within Fit Model. Adding additional complexity, the customer also wants to include cross terms which Discriminant does not support but PLS-DA does. Oof...

Re: PLS with categorical response

Well of course you can do it in Python because it is a programming language. By the same token, you can do it in JMP via scripting. Okay, now that I have gotten my "snarkiness" out of the way...

 

I was hoping that you could save the prediction formula from PLS-DA and get the ROC curve through the Model Comparison platform. Unfortunately, it looks like the format for the PLS-DA prediction formula does not work well with the Model Comparison platform. It looks like the ROC curve would need to be built via scripting.

Re: PLS with categorical response

Thank you all, very helpful! The customer (NeuroVision) is also asking if it's possible to get a ROC diagnostic curve from PLS? Seems like it's not an option, but is there a round about way to do it and also what is the reasoning why it is not an option for PLS w/ categorical responses?  Apparently they are using Python to get this type of a result and would like JMP Pro to do something similar. 

Re: PLS with categorical responseactually, sorry i just realized that dataset is not binomial, there's multiple categories.Re: PLS with categorical response

Any of the example discriminant analysis datasets in the sample library would also work.

The "Owl Diet" dataset is simple to understand and work through the concepts of PLS-DA.

Re: PLS with categorical responseI wish I had been involved in writing the PSL book, but it's purely the product of Marie Gaudard and Ian Cox.Re: PLS with categorical response

Curious if someone can help me with explaining the applications for PLS vs. Gen Reg (with validation)?  Why would one use PLS over Gen Reg?  Or vice versa?  

Re: PLS with categorical response

Please remember to mention the excellent book about PLS using JMP by Mia and Ian!

Re: PLS with categorical response

Hi Mark,

 

Just adding an example that I have used from a science/engineering industry situation.

 

This is from a forensic analytical chemistry lab. The company's branded products are sold all over the world. Sometimes counterfit product is sold as their's. When they have customer complaints they need to establish quickly if it is their product or a counterfit. Traditionally this is done by testing a sample with many different chemical tests over days. This costs a lot in lab time and means customer problems take too long to be resolved.

 

The solution is to use data analytics on a fast chemical analysis that generates a lot of data. 

 

In the attached, the RT columns represent the peak area at different retention times from chromatograms. Each peak is a different chemical entity. And the area is proportional to the amount of that entity in the sample.

 

I show how different visual and modelling techniques allow you to understand the differences between the 3 sources and how you could determine what the source is for an unknown. PLS would be one of the options.

 

Hope that helps.

Phil

Re: PLS with categorical response

In the attached zip file is a journal comparing PCA and PLS . Within there is an Amour tiger data set having a categorical response. 

There is some description involved and some other examples too. I guess this should help with what you are looking for. You have to check if all works in JMP 15 as I have it not checked.since some time