Setting random seeds to make analyses repeatable<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />Happy Friday, all,
A JMP Pro evaluator asked me how to make models like Neural networks and Random Forests reproducible in JMP Pro. Because these methods use a random seed to establish starting conditions, it's not enough to save the script -- we need to specify a seed in the script, using the "set random seed" option, like this:
Partition(
Y( :chas ),
X( :crim, :zn, :indus, :mvalue, :nox, :rooms, :age, :distance, :radial ),
Validation( :TVT ),
Method( Bootstrap Forest ),
Portion Bootstrap( 1 ),
Number Terms( 2 ),
Number Trees( 12 ),
Set Random Seed (9876),
Go
);
A search in the Scripting Index shows a number of platforms that support this option -- Neural and Partition, among them.
I showed him how to set the seed, but the evaluator complained -- rightly so, I think -- that it should be easier to do this. There's no way to set the seed from the launch dialog, you can only save the script retrospectively and add the syntax to set a random seed. If you made a model using point-and-click, there's no way to repeat the analysis to get exactly the same results.
Does anybody have a solution or good workaround? The best I have so far is that you can issue the "Random Reset()" JSL function with a seed number before every point-and-click analysis, but this seems pretty unworkable.
Cheers,
Jordan
Re: Setting random seeds to make analyses repeatableAn updated version of the add-in has been posted. This one also works in neural - thanks Jordan! The instructions have been updated to reflect this. Should work in most platforms - run prior to launching the platform.
Re: Setting random seeds to make analyses repeatableYes, on the surface it seems like it would be straightforward. It certainly would be good to have a more robust version. Know any good scripters who could make this happen : )
Re: Setting random seeds to make analyses repeatableThanks all for feedback. This is a more complex topic than I realized.
Here's a note from John Sall, which he recently appended to the Defect referenced above by Dan V:
There are two random number generators, and Random Reset and Random Seed go to the two different ones. Any random number generator will not be consistent when the platform is multithreaded and the random numbers are done in the threads. A number of platforms have an Unthreaded option to disable threading, which will make the random reset work better.
Mia, I noticed that the Random Seed() function works for Neural, but not Random Reset(). The reverse seems to be true for Partition. Perhaps the add-in could be made more effective by setting both seeds simultaneously.
Jordan
Re: Setting random seeds to make analyses repeatableFYI - the add-in has been uploaded to the file exchange. Am working on instructions now. It doesn't work across all platforms (i.e. Neural), but seems to do the job with Partition.
Re: Setting random seeds to make analyses repeatableHi Jordan,
There is an add-in for resetting the random seed, developed by Brady and/or Sue Walsh a year or two ago for academic purposes (to allow professors and students to obtain the same results). Its been informally tested and seems to do the job. I'll post this on the file exchange.
Mia
Re: Setting random seeds to make analyses repeatableHello Jordan -- I've submitted this to the JMP Beta list. There is already an open defect. Your suggestions and customer have been added to the thread: http://sww.sas.com/cgi-bin/quick_browse?defectid=S0898202
Re: Setting random seeds to make analyses repeatableOr how about an JMP Add-in which sets the random seed to the same start value by clicking on it?
Regards,
Martin
Re: Setting random seeds to make analyses repeatableHi Jordan,
Not sure if this helps, but Partial Least Squares does give the ability to set the random seed through a drop down red hot spot in the model launch. Maybe something that can be added to the other analysis platforms.
Best,
Bill
