SVM Confidence Interval Generation

CentroidError56

Is there any way to generate confidence intervals for an SVM model?

Victor_G

Hi @CentroidError56 

Sorry that you didn't get a response so far. I have searched in litterature, and it seems the easiest option to get close to what you would like is to calculate bootstrap interval (there doesn't seem to be confidence intervals with SVM).

One could imagine calculating intervals like in SVEM Generalized regression, so the process would be :

1. Create a new table with fractionnally weighted bootstraps through an autovalidation table.
2. Run the SVM analysis with "Valid Weight" as Freq role and "Valid Set" as Validation role.
3. Once the model is obtained, save column prediction formula.
4. Create permutations of the valid weights values (open the column formula, and check/uncheck the evaluation of the formula to force the re-evaluation of the formula to generate new weights), and re-run the SVM analysis.
5. Save new column prediction formula and repeat the process (100-200 times like in SVEM).
6. Once all column prediction formula have been saved in the table, create a new column which is the mean of all previous resample formulas, and a new column which is the StdError of your predicted response by calculating Std Dev of the individual resample formulas.
7. Then, you can use the Profiler, and use these two newly created columns (mean and stddev of the resample formulas), specify "Expand Intermediate Formula" and click OK when JMP asks if StdError column can be used to create intervals.

As you can imagine, there are no easy ways to do this by only using clicks (the process is quite tedious to repeat), so I would recommend try scripting it (but I am clearly not a scripter). As an example, I tested it on the Diabetes dataset with an RBF SVM with cost=0,265 and gamma=0,09 and only repeating the process 4 times (so 5 resample formulas, file attached) :

Once my mean and StdError columns have been created from these 5 resample formulas, I can use the Profiler with these two columns :

 

Maybe someone from the Community or JMP staff could help designing this JSL script ?

You can also in the meantime add this need for intervals in the JMP Wish List. 

 

Hope this answer may help you,

Diabetes Autovalidation.jmp