nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Not getting same results in neural network run from menu vs run from script.

jwalk

I am trying to run a script that automates screening the eight different penalty modes in Neural Network and outputting the results into the current table. I have been troubleshooting why I am not getting different results from the different Fits in the script. This led me to look at the results for each individual penalty mode when the model was set up from the Neural Menu vs the script. The script was made by saving the analysis script to the Data Table and then running it. So the results should be identical.

Set up of model from Analyze>Predictive Modeling>Neural

Initial run of modelRedo Analysis #1Redo Analysis #2

If I relaunch the model and select all the same settings, I get the same result as the Initial Result above. If I run from a script, I get the same results as from Redo Analysis.

Accepted answers

All comments

jwalk

Also, the following scripts all give the same result: It's as if neural network models run from a script only use one of the penalty methods

Neural(
	Y( :Target ),
	X( :Name( "0-4_Mean" ), :Name( "4-8_Mean" ), :Name( "8-12_Mean" ) ),
	Informative Missing( 0 ),
	Validation Method( "Holdback", 0.3333 ),
	Set Random Seed( 999 ),
	Fit(
		NTanH( 3 ),
		Robust Fit( 1 ),
		Penalty Method("Squared"),
		Number of Tours( 3 )
	)
)

Neural(
	Y( :Target ),
	X( :Name( "0-4_Mean" ), :Name( "4-8_Mean" ), :Name( "8-12_Mean" ) ),
	Informative Missing( 0 ),
	Validation Method( "Holdback", 0.3333 ),
	Set Random Seed( 999 ),
	Fit(
		NTanH( 3 ),
		Robust Fit( 1 ),
		Penalty Method("Absolute"),
		Number of Tours( 3 )
	)
)

Neural(
	Y( :Target ),
	X( :Name( "0-4_Mean" ), :Name( "4-8_Mean" ), :Name( "8-12_Mean" ) ),
	Informative Missing( 0 ),
	Validation Method( "Holdback", 0.3333 ),
	Set Random Seed( 999 ),
	Fit(
		NTanH( 3 ),
		Robust Fit( 1 ),
		Penalty Method("Weight Decay"),
		Number of Tours( 3 )
	)
)

Neural(
	Y( :Target ),
	X( :Name( "0-4_Mean" ), :Name( "4-8_Mean" ), :Name( "8-12_Mean" ) ),
	Informative Missing( 0 ),
	Validation Method( "Holdback", 0.3333 ),
	Set Random Seed( 999 ),
	Fit(
		NTanH( 3 ),
		Robust Fit( 1 ),
		Penalty Method("NoPenalty"),
		Number of Tours( 3 )
	)
)

Byron_JMP

In your scripts you have this line:

Validation Method( "Holdback", 0.3333 )

And that's your problem. Each time you run this you're getting a different random holback set.

Since you're using JMP Pro, set up a Validation column and use that.

1. get to control which rows are in the holdback, like when you make a stratified subset.

2. use the same rows for the hold back for each version of the model.

jwalk

My problem is that I'm not getting different results when running from a script using a fixed seed and different penalty methods.

jwalk

My problem started when I was trying to run 2 instances of JMP, each running a script with 4 NN runs. Instance 1 was running seeded models using Squared, Robust Squared, Absolute and Robust Absolute penalties. Instance 2 was running the exact same script (in a differently named file) except the 4 models were running Weight Decay, Robust Weight Decay, No Penaly and Robust No Penalty NNs.The runs take ~ 20 hours to complete and the script saves the formulas to the data table at the end. What I found was that both instances returned exactly the same results. There were 4 different results in each instance, but they were exactly the same. I tried various combinations to try to find out what was going on. I made every variable unique between the two scripts. I used a text box to set the Data Table to a variable in each instance. No matter what I did, I could not generate different results between the two instances.

If I run each model from the menu, I get eight different results. If I run in a script, I do not.