nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

SVEM ADD IN FOR SUPPORT VECTOR MACHINE

frankderuyck

I have a DOE results I would like to model with Support Vector Machine (there are strong non linear effects that can't be estimated with a polynomial model) How do I install the SVEM add in?

Accepted answers

Victor_G

Hi @frankderuyck,

Some questions before answering yours :

Why would you like to use SVEM for SVM ? Support Vector Machines is a robust ML algorithm, that can be used for small-sized datasets like DoE. More infos about algorithm's robustness on LinkedIn.
There are also many ways to assess overfitting for SVM without using validation set, by checking the number of support vectors, and the values of penalty (cost) and curvature (gamma) if you have fine-tuned your SVM hyperparameters with a Tuning design... I am thinking about writing a blog post about assessing and preventing overfitting for robust algorithms (SVM, Random Forest) in the absence of validation (in DoE datasets for example).
Did you also consider other algorithms ? SVM is a robust algorithm for analysis of small-size datasets, but Random Forest can also provide good results and may be a good benchmark to evaluate and compare performances. More infos on Random Forests on LinkedIn.
How many levels do you have for your factors ? ML algorithms can only be useful if you have enough levels for each of your factors to enable a smooth interpolation. More than 3 levels may be a good starting point.

Concerning the use of SVEM, you can access it through different ways :

In the platform "Make Validation Column", you have the possibility to create an Autovalidation table : Launch the Make Validation Column Platform (jmp.com) This will duplicate the rows and create anti-correlated weights (between training and validation) column that you can use as "Frequency" in the platform launch. You can then simulate prediction results by varying the weights between training and validation (right-click a panel with validation results, click on Simulate, and specify weights column as the column to switch in and out).
You can also download the addin SVEM, that will do the same things as the autovalidation table : Re-Thinking the Design and Analysis of Experiments? (2021-EU-30MP-776) - JMP User Community
You can then reproduce the same steps as before and simulate results to assess model's robustness.

I would however like to emphasize that cross-validation (K-folds or SVEM) is not properly a validation method, but more a "debugging" method to assess robustness of algorithm's performances on various training and validation sets. The main difficulty after using a cross-validation technique is how to make sense of the results and how to proceed next:
Average the individual models trained on various folds ? Create an ensemble with these models ? Keep the best model trained on a fold (I wouldn't recommend this option) ? Train the model on whole dataset once the robustness has been checked with the cross-validation ? ... There is no definitive answer to this question and this is related to the dataset, modelisation complexity, performances improvement, user's practices, ...

More infos here with some episodes of Cassie Kozyrkov on the topic of cross-validation: https://youtu.be/zqD0lQy_w40?si=41jfBs83Q0TIlusr

https://youtu.be/by59hsYO0Lg?si=XeZ33j2e4jKQPaRA

Hope this answer will help you,

All comments

Victor_G

Hi @frankderuyck,

Some questions before answering yours :

Why would you like to use SVEM for SVM ? Support Vector Machines is a robust ML algorithm, that can be used for small-sized datasets like DoE. More infos about algorithm's robustness on LinkedIn.
There are also many ways to assess overfitting for SVM without using validation set, by checking the number of support vectors, and the values of penalty (cost) and curvature (gamma) if you have fine-tuned your SVM hyperparameters with a Tuning design... I am thinking about writing a blog post about assessing and preventing overfitting for robust algorithms (SVM, Random Forest) in the absence of validation (in DoE datasets for example).
Did you also consider other algorithms ? SVM is a robust algorithm for analysis of small-size datasets, but Random Forest can also provide good results and may be a good benchmark to evaluate and compare performances. More infos on Random Forests on LinkedIn.
How many levels do you have for your factors ? ML algorithms can only be useful if you have enough levels for each of your factors to enable a smooth interpolation. More than 3 levels may be a good starting point.

Concerning the use of SVEM, you can access it through different ways :

In the platform "Make Validation Column", you have the possibility to create an Autovalidation table : Launch the Make Validation Column Platform (jmp.com) This will duplicate the rows and create anti-correlated weights (between training and validation) column that you can use as "Frequency" in the platform launch. You can then simulate prediction results by varying the weights between training and validation (right-click a panel with validation results, click on Simulate, and specify weights column as the column to switch in and out).
You can also download the addin SVEM, that will do the same things as the autovalidation table : Re-Thinking the Design and Analysis of Experiments? (2021-EU-30MP-776) - JMP User Community
You can then reproduce the same steps as before and simulate results to assess model's robustness.

More infos here with some episodes of Cassie Kozyrkov on the topic of cross-validation: https://youtu.be/zqD0lQy_w40?si=41jfBs83Q0TIlusr

https://youtu.be/by59hsYO0Lg?si=XeZ33j2e4jKQPaRA

Hope this answer will help you,

frankderuyck

Hi Victor, this tuning design works great, I want to model a guality response Y as a function of three Y FPCA's from NIR spectra (FDA). I tried neural network but even with 3 TanH nodes profiler shows to my opinion too high and irregular ncurvature. SVE after tuning does a nice job, I support vector machine learning!

Victor_G

Hi @frankderuyck,

Great to know !
I use Neural Networks as the last modeling option possible, there are many classical models and ML algorithms more simple that perform equally well (and sometimes better). These other options are also faster to compute, don't require extended knowledge, tests and validation to finetune them (unlike NNs where you have to specify the architecture, with the number of layers, type of functions, boosting or not, ... or using the Neural Network Tuning AddIn) and are easier to interpret (in terms of response profiles and relative factors importance).
On tabular data, if the only model able to perform well is Neural Networks, there are good chances that you might have other problems, in terms of data quality, representativeness, missing values/outliers, repeatability/reproducibility, etc...

Tree-based models provide a good performances benchmark and are often a safe and reliable option, see the article "Why do tree-based models still outperform deep learning on tabular data?" (2022) https://arxiv.org/abs/2207.08815

In your screenshot, it seems the optimal SVM model found with the Tuning design is quite interesting :

It uses 10 support vectors, so less than half of the points are used to build the "boundaries"/vectors used in SVM model, which may be a good sign of low complexity and good generalization properties,
The curvature value (gamma) is quite low, so the response profile and boundaries may not be over-complex and highly curved (so again, low complexity and good generalization properties),
The penalty (cost value) seems quite high, but with a low-size dataset this is often the case in practice, as any misclassified/mispredicted point will greatly increase the misclassification rate/RMSE : 1 misclassified point out of 25 = 4% misclassification rate, compared to higher dataset size where one misclassified point would represent less than 1% misclassification rate. In some situations, it can be interesting to choose a model with a slightly larger error (RMSE/Misclassification rate), but with a lot less support vectors : you degrade your predictive performances on the training set in favor of a less complex model that could be able to generalize better on new data.

You can use the table generated by the Tuning design, with the performance metrics, to better assess the relative importance and contribution of the two hyperparameters on the performance metrics (Misclassification rate for classification, or RASE and R² for regression).

As always, it is recommended to validate your model with new validation points, to make sure your model doesn't overfit.

frankderuyck

Guess tree based models like random forest & XGboost require a large data set unless you can use SVEM for small sets, correct?

frankderuyck

Hi Victor, I tried to use this for mixture results, got a nice model but optimization with the profiler did not work because the component su did not add up to 1, was almost 3.. The same happens in other ML models, PLS and even Gen Reg

Victor_G

Hi Franck,

Have you checked that your mixture factors have the right column properties :

"Mixture" with low and high individual factors values
"Design Role" with role set as Mixture
"Factor Changes" set at Easy (or other type if you have a split-plot/restricted randomization scenario)

Here is an example on Donev Mixture Data with factor CuSO4 :

When launching the Profiler from Gaussian Process (or any other ML model), the mixture constraint is respected and enforced (mixture factors always sum up to 1):

Hope this answer may solve your problem,

frankderuyck

OK Victor, thanks!