Prediction Profiler and desirability functionsHi All,
JMP Pro 16.1.0.
I'm using the Prediction Profiler to work through some verification of model tuning and was wondering if there is a way to "trick" the desirability functions into doing something they're not really designed to do. With the standard setting for desirability, you have the option to Maximize, Match Target, Minimize or None.
During the model tuning process, I record the R^2 values for training and validation, as well as the difference in between the two. One of the things I try to look for when model tuning are parameter settings that not only maximize R^2 for BOTH training and validation, but minimizes the difference between the two. For example, it's better for me if I have a model that has an R^2 training of 0.4 and R^2 validation of 0.35 than it is to have a R^2 training of 0.7 and an R^2 validation of 0.25. I.e., the closer the training and validation sets are to each other, even if smaller tend to perform better for my situations.
The problem is that with the Prediction Profiler, If you set both training and validation R^2 to maximize and minimize the difference, the profiler suggests settings that result in training R^2 of 0.6 and validation R^2 of 0.2 or so. Not what I'm after.
What I'd really like the Profiler to do is to try and converge on training and validation R^2 that are as close to each other as possible, while also being as large as possible, in order to minimize the difference while keeping large R^2 values. Unfortunately, changing things like the importance (0 to 1) or the goal don't really achieve what I'm after.
If anyone knows a trick on how to get the Profiler to do something like this, it would be great to know. Or, if it's not possible, it might be a really nice new feature to be able to direct the Profiler to try and converge two of the responses while maximizing (or minimizing) them.
Thanks!,
DS
AnalyticsRe: Prediction Profiler and desirability functionsYou are using the Profiler and Desirability Functions correctly. It is not wrong to use desiriability = (0,1), but you might try using values like 1 for maximize and 2 for difference and see if that gets you closer to what you want. The definition of the functions (limits, importance) is key to telling the profiler what you are after.
That said, it is not guaranteed that you will get what you want!
Re: Prediction Profiler and desirability functionsI am guessing that you don't care whether the R2 difference is 0.01 or 0.1, but you gradually care more about it as it increases above some threshold. You could try making the R2 difference a formula that is flat when below a certain value and then gradually decreases as it goes above that limit, or you could set up the desirability function to describe this. I often do the latter with the T2 metric in multivariate models. You can see an example of this by skipping to about 16:30 in the video linked below.
Optimizing Multivariate Models: From Data Collection to Optimal Conditions and Scoring Models (2021-US-45MP-927)
Re: Prediction Profiler and desirability functionsYou can also change the shape of the graph of the desirability to indicate that a larger difference is a bigger penalty. It does not have to be linear with a middle value in the center.
Note: you can control click (windows) or command click (macOS) on the plot of the function graph and change values quickly while you try different schemes. I recommend NOT changing the desirability, but only change the response levels Move the center value towards one end to make it non-linear.
Re: Prediction Profiler and desirability functionsHi @markbailey and @ih ,
@markbailey , your first reply was one that will "make it work". Unfortunately, it's not an automated optimization process, so it doesn't iteratively find the optimal conditions. Nonetheless, it's certainly easy enough to manually change the relative importance of each response.
Utilizing a mix of your suggestion and @ih 's suggestion is proving to be more along the lines of what I think ultimately I'd like it to do. There's just a lot of manual adjustment of the desirability functions -- either changing the numbers by hand, or moving the little boxes in the graph around. It would be nice to have a more automated and slightly less subjective method of obtaining my end goal.
@ih you are partially right that I don't care if the R2 Diff is 0.01 or 0.1, however, if I have an R2 Diff of 0.1 and my training R2 is 0.8 and my validation R2 is 0.7, I'd prefer that to a R2 Diff of 0.01 if my training is at 0.31 and my validation is at 0.3. In practice, my data sets tend to behave more like the latter rather than the former -- I wish I could get my two R2's to be as high as 0.8 and 0.7!
Mixing the two suggestions gives me a lot to play with, just wish is was a little less subjective.
Thanks for the feedback!,
DS
