915439_en-US

liliconcom

Scripting Help Requested - Making a simulator for Type 1 errors in equivalence testing

Hi All,

 

I'm trying to write a script that will generate a random dataset (n=3 reference and test measurements) and then run a simulation with some number (1000 or whatever).  For each set I'll run an equivalence test and I'd like the output to be a report/table with the frequency of events where equivalence was demonstrated.  The goal of this tool will be to give a probability of type 1 errors given a certain equivalence margin.

I've landed on the following code but this is not working and I'm pretty much stuck.  Any direction/help that the group can provide would be much appreciated.

Thanks!

Chris

// Parameters
numSimulations = 10;
nPerGroup = 3;
refMean = 99.1;
testMean = 98.8;
sd = 0.1;
equivMargin = 0.3;
type1Errors = 0;

// Critical t-value for 90% CI with df = 2n - 2
df = 2 * nPerGroup - 2;
tCrit = StudentTQuantile(0.950, df);

// Create summary table
summaryTable = New Table("Simulation Summary",
    New Column("Simulation", Numeric),
    New Column("CI_Lower", Numeric),
    New Column("CI_Upper", Numeric),
    New Column("Equivalence_Concluded", Character),
    New Column("Mean_Diff", Numeric)
);

// Loop through simulations
For(sim = 1, sim <= numSimulations, sim++,

    // Create new data table
    dt = New Table("Sim" || Char(sim),
        Add Rows(nPerGroup * 2),
        New Column("Group", Character),
        New Column("Value", Numeric)
    );

    // Simulate Reference group
    For(i = 1, i <= nPerGroup, i++,
        dt:Group[i] = "Reference";
        dt:Value[i] = Random Normal(refMean, sd);
    );

    // Simulate Test group
    For(i = nPerGroup + 1, i <= nPerGroup * 2, i++,
        dt:Group[i] = "Test";
        dt:Value[i] = Random Normal(testMean, sd);
    );

    // Extract values
    refVals = dt << Get As Matrix( :Value )[1::nPerGroup];
    testVals = dt << Get As Matrix( :Value )[nPerGroup+1::nPerGroup*2];
    meanDiff = Mean(testVals) - Mean(refVals);

    // Sample variances
    varRef = Variance(refVals);
    varTest = Variance(testVals);

    // Pooled variance
    pooledVar = ((nPerGroup - 1) * varRef + (nPerGroup - 1) * varTest) / df;
    pooledSD = Sqrt(pooledVar);

    // Standard error of difference
    seDiff = pooledSD * Sqrt(2 / nPerGroup);

    // 90% CI
    ciLower = meanDiff - tCrit * seDiff;
    ciUpper = meanDiff + tCrit * seDiff;

    // Check equivalence
    eqResult = "No";
    If(ciLower > -equivMargin & ciUpper < equivMargin,
        type1Errors++;
        eqResult = "Yes";
    );

    // Log results
    summaryTable << Add Rows(1);
    summaryTable:Simulation[sim] = sim;
    summaryTable:CI_Lower[sim] = ciLower;
    summaryTable:CI_Upper[sim] = ciUpper;
    summaryTable:Equivalence_Concluded[sim] = eqResult;
    summaryTable:Mean_Diff[sim] = meanDiff;

    // Close the table to avoid clutter
    dt << Close Window;
);

// Show results
New Window("Type I Error Summary",
    Text Box("Total Simulations: " || Char(numSimulations)),
    Text Box("Type I Errors: " || Char(type1Errors)),
    Text Box("Estimated Type I Error Rate: " || Char(Round(type1Errors / numSimulations * 100, 2)) || "%")
);

Automation and ScriptingRe: Scripting Help Requested - Making a simulator for Type 1 errors in equivalence testing

Which part are you stuck with?

Re: Scripting Help Requested - Making a simulator for Type 1 errors in equivalence testing

H Jarmo,

At the moment the code breaks trying to extract the critical T values from the equivalence report.  But there are other issues as well.  I've had it in a state where it will run the simulation and equivalence tests but not be able to get values out of the report.  

I've tried throwing some of this at AI as well but I think some of the syntax and coding conventions might be incorrect.

 

Chris

Re: Scripting Help Requested - Making a simulator for Type 1 errors in equivalence testing

I would suggest these edits:

There's no StudentTQuantile() function.  That should be t Quantile().

Variance() is not a built-in function.  Define it in your script (or just square the Std Dev() instead).

refVals = dt << Get As Matrix( :Value )[1::nPerGroup] should be more like refVals = dt:Value[Where( :Group == "Reference" )];

Same for testVals =

Set the visibility for dt to "Private" to speed things up greatly.

Variance = Function( {Values},
	Return( Std Dev( Values ) ^ 2 )
);
// Parameters
numSimulations = 1000;
nPerGroup = 3;
refMean = 99.1;
testMean = 98.8;
sd = 0.1;
equivMargin = 0.3;
type1Errors = 0;

// Critical t-value for 90% CI with df = 2n - 2
df = 2 * nPerGroup - 2;
tCrit = t Quantile( 0.950, df );

// Create summary table
summaryTable = New Table( "Simulation Summary",
	New Column( "Simulation", Numeric ),
	New Column( "CI_Lower", Numeric ),
	New Column( "CI_Upper", Numeric ),
	New Column( "Equivalence_Concluded", Character ),
	New Column( "Mean_Diff", Numeric )
);

// Loop through simulations
For( sim = 1, sim <= numSimulations, sim++, 

    // Create new data table
	dt = New Table( "Sim" || Char( sim ), "Private", Add Rows( nPerGroup * 2 ), New Column( "Group", Character ), New Column( "Value", Numeric ) );

    // Simulate Reference group
	For( i = 1, i <= nPerGroup, i++,
		dt:Group[i] = "Reference";
		dt:Value[i] = Random Normal( refMean, sd );
	);

    // Simulate Test group
	For( i = nPerGroup + 1, i <= nPerGroup * 2, i++,
		dt:Group[i] = "Test";
		dt:Value[i] = Random Normal( testMean, sd );
	);

    // Extract values
	refVals = dt:Value[Where( :Group == "Reference" )];
	testVals = dt:Value[Where( :Group == "Test" )];
	meanDiff = Mean( testVals ) - Mean( refVals );

    // Sample variances
	varRef = Variance( refVals );
	varTest = Variance( testVals );

    // Pooled variance
	pooledVar = ((nPerGroup - 1) * varRef + (nPerGroup - 1) * varTest) / df;
	pooledSD = Sqrt( pooledVar );

    // Standard error of difference
	seDiff = pooledSD * Sqrt( 2 / nPerGroup );

    // 90% CI
	ciLower = meanDiff - tCrit * seDiff;
	ciUpper = meanDiff + tCrit * seDiff;

    // Check equivalence
	eqResult = "No";
	If( ciLower > -equivMargin & ciUpper < equivMargin,
		type1Errors++;
		eqResult = "Yes";
	);

    // Log results
	summaryTable << Add Rows( 1 );
	summaryTable:Simulation[sim] = sim;
	summaryTable:CI_Lower[sim] = ciLower;
	summaryTable:CI_Upper[sim] = ciUpper;
	summaryTable:Equivalence_Concluded[sim] = eqResult;
	summaryTable:Mean_Diff[sim] = meanDiff;

    // Close the table to avoid clutter
	dt << Close Window;
);

// Show results
New Window( "Type I Error Summary",
	Text Box( "Total Simulations: " || Char( numSimulations ) ),
	Text Box( "Type I Errors: " || Char( type1Errors ) ),
	Text Box( "Estimated Type I Error Rate: " || Char( Round( type1Errors / numSimulations * 100, 2 ) ) || "%" )
);

 

Re: Scripting Help Requested - Making a simulator for Type 1 errors in equivalence testing

Jarmo,

 

Thanks for the suggestion and explanation.  This definitely works!

I am going to work through the code to try to internalize how everything works...Appreciate your help!

Chris

Re: Scripting Help Requested - Making a simulator for Type 1 errors in equivalence testing

The original code quite strongly looks like to be created by LLM (due to obvious errors they make if you know some JSL). 

 

Working version has already been provided but here is how I would approach this. This is slightly more complicated version using matrices (generally they are faster) and I haven't verified the calculations

Names Default To Here(1);

numSimulations = 1000;
nPerGroup = 3;
refMean = 99.1;
testMean = 98.8;
sd = 0.1;
equivMargin = 0.3;
type1Errors = 0;

df = 2 * nPerGroup - 2;
tCrit = t Quantile(0.950, df);


dt = New Table("Simulation Summary",
    New Column("Simulation", Numeric),
    New Column("CI_Lower", Numeric),
    New Column("CI_Upper", Numeric),
    New Column("Equivalence_Concluded", Character),
    New Column("Mean_Diff", Numeric),
    Invisible
);


For(sim = 1, sim <= numSimulations, sim++,
	m_ref = J(nperGroup, 1, Random Normal(refMean, sd));
	m_test = J(nperGroup, 1, Random Normal(testMean, sd));

	avgdiff = Mean(m_test) - Mean(m_ref);
	
	var_ref = Std Dev(m_ref)^2;
	var_test = Std Dev(m_test)^2;

    pooledVar = ((nPerGroup - 1) * var_ref + (nPerGroup - 1) * var_test) / df;
    pooledSD = Sqrt(pooledVar);

    seDiff = pooledSD * Sqrt(2 / nPerGroup);

    ciLower = avgdiff - tCrit * seDiff;
    ciUpper = avgdiff + tCrit * seDiff;

    eqResult = "No";
    If(ciLower > -equivMargin & ciUpper < equivMargin,
        type1Errors++;
        eqResult = "Yes";
    );

	dt << Add Rows(
		{Simulation = sim, CI_Lower = ciLower, CI_Upper = ciUpper, Equivalence_Concluded = eqResult, Mean_Diff = avgdiff}
	);
);


dt << Show Window(1);

nw = New Window("Type I Error Summary",
    Text Box("Total Simulations: " || Char(numSimulations)),
    Text Box("Type I Errors: " || Char(type1Errors)),
    Text Box("Estimated Type I Error Rate: " || Char(Round(type1Errors / numSimulations * 100, 2)) || "%")
);