nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

sync mode?

matlag

Hi all,

I am struggling with the async parts of JMP/JSL. Now I understand you can use "Run Formulas()" to force evaluation of a formula before using it, but I have similar issues with other functions:

When the value is used, it may be incorrect. This is very dangerous!

Is there a "force sync mode" in which JSL goes instruction by instruction and systematically wait for one to be completed before moving to the next?

Example of my current issues, I extract values from a selection of rows (selRows) from one table (dt) to fill a row in another table (sumdt):

            thisCol = Column(sumdt, thisName||"_Q10");
            valueSet = Column(dt, colList[i])[selRows];
            thisValue = Quantile(0.10, valueSet);
            thisCol[] = thisValue;


            thisCol = Column(sumdt, thisName||"_Q25");
            thisValue = Quantile (0.25, valueSet);
            thisCol[] = thisValue;


            thisCol = Column(sumdt, thisName||"_Q50");
            thisValue = Quantile (0.50, valueSet);
            thisCol[] = thisValue;


            thisCol = Column(sumdt, thisName||"_Q75");
            thisValue = Quantile (0.75, valueSet);
            thisCol[] = thisValue;


            thisCol = Column(sumdt, thisName||"_Q90");
            thisValue = Quantile (0.90, valueSet);
            thisCol[] = thisValue;

This works SOMETIMES! But sometimes, all data in the sumdt column are just the same value, and sometimes, nothing comes out at all.

I know some recommand the wait(0) command, but I read someone saying it is not a guarantee either.

So how do I make sure the calcuations get completed??

Thanks!

Accepted answers

David_Burnham

It's a lot easier to diagnose if you can provide some sample code that we can run that replicates the problem. Without that I've written my own version. It contains all your code but I have provided the table source and destination (sumdt). The only difference is that you have the notation thisCol[] which implies you are using a For Each Row loop. I didn't understand the context of that so I simply used a For loop:

dt = Open("$SAMPLE_DATA/Big Class.jmp");
colList = dt << Get Column Names(string,numeric);
N = NItems(colList);
sumdt = new table("sumdt",
	Add Rows(N),
	new column("name", character),
	new column("Q10"),
	new column("Q25"),
	new column("Q50"),
	new column("Q75"),
	new column("Q90"),
);	
selRows = dt << Get Rows Where(:sex=="F");
For (i=1,i<=N,i++,

	thisName = colList[i];
	Column(sumdt,"name")[i] = thisName;

	thisCol = Column(sumdt, "Q10");
	valueSet = Column(dt, colList[i])[selRows];
	thisValue = Quantile(0.10, valueSet);
	thisCol[i] = thisValue;


	thisCol = Column(sumdt, "Q25");
	thisValue = Quantile (0.25, valueSet);
	thisCol[i] = thisValue;


	thisCol = Column(sumdt, "Q50");
	thisValue = Quantile (0.50, valueSet);
	thisCol[i] = thisValue;


	thisCol = Column(sumdt, "Q75");
	thisValue = Quantile (0.75, valueSet);
	thisCol[i] = thisValue;


	thisCol = Column(sumdt, "Q90");
	thisValue = Quantile (0.90, valueSet);
	thisCol[i] = thisValue;
);

This code works fine without any synchronisation issues. You might want to compare the code and see what is different.

All comments

MikeD_Anderson

You could try using a loop. You could incorporate a step into the loop that copies the value to a matrix or an Associative Array before the next loop starts. That would probably get things to follow the cadence you want.

Best,

M

David_Burnham

dt = Open("$SAMPLE_DATA/Big Class.jmp");
colList = dt << Get Column Names(string,numeric);
N = NItems(colList);
sumdt = new table("sumdt",
	Add Rows(N),
	new column("name", character),
	new column("Q10"),
	new column("Q25"),
	new column("Q50"),
	new column("Q75"),
	new column("Q90"),
);	
selRows = dt << Get Rows Where(:sex=="F");
For (i=1,i<=N,i++,

	thisName = colList[i];
	Column(sumdt,"name")[i] = thisName;

	thisCol = Column(sumdt, "Q10");
	valueSet = Column(dt, colList[i])[selRows];
	thisValue = Quantile(0.10, valueSet);
	thisCol[i] = thisValue;


	thisCol = Column(sumdt, "Q25");
	thisValue = Quantile (0.25, valueSet);
	thisCol[i] = thisValue;


	thisCol = Column(sumdt, "Q50");
	thisValue = Quantile (0.50, valueSet);
	thisCol[i] = thisValue;


	thisCol = Column(sumdt, "Q75");
	thisValue = Quantile (0.75, valueSet);
	thisCol[i] = thisValue;


	thisCol = Column(sumdt, "Q90");
	thisValue = Quantile (0.90, valueSet);
	thisCol[i] = thisValue;
);

This code works fine without any synchronisation issues. You might want to compare the code and see what is different.

David_Burnham

In my last reply I used a small table which would not necessarily highlight any synchronisation issues. If there is a delay it's going to be the calculation of the quantiles. The code below calculates a quantile for 10 million data points. The function runs synchronously as you would expect i.e. the show does not get executed until the value of q has been established. If there was a synchronisation issue then we'd see the value of q be shown as zero.

vals = J(10000000,1,Random Normal());
q = 0;
q = Quantile(0.5,vals);
show(q);

As a general observation though, I would use Wait(0) if I wanted to ensure synchronisation - this normally only relates to I/O type issues. There are only rare instances where Wait does't work - and that's because execution is running in a separate thread.

matlag

Hi again,

Thanks for your answer and insight.

Unfortunately, I still have the same issues: missing values, duplicate values in set of output (from Q10 to Mean, StdDev missing).

I tried multiple things. I made a subscript on the side to check a few of the abnormal values "manually" and they came out fine.

I added many "wait" statements.

I'm quite out of ideas!

Here is the script (almost all of it, removed some verification for existence of columns, etc., it's big enough):

Names Default To Here(1);

dt = Current Data Table();

colNames = dt << Get Column Names (String);

Summarize(itemsList = by("Item"));

binsList = {"1", "2", "3", "4", "5", "6", "7", "8", "9", "A", "B", "C", "D"};

colNamesSum = sumdt << Get Column Names (String);


For (k=1, k <= Nitems(itemsList), k++,
	itemID = itemsList[k];
	dt << Select All Rows << Exclude(0);
	dt << Select Where (dt:Item == itemID);
	itemRows = dt << Get Selected Rows;
		
	prodID = Right(Column (dt, "proc_id")[itemRows[1]],4);

	lotID = Regex(Column(dt, "lot_id")[itemRows[1]], "\d{6}");
	show(lotID,itemID);
	
	checkDup = sumdt << Get Rows Where (:Lot == lotID & :Item == itemID);
	If(NRows(checkDup) == 0,
		newIt = 1;
		show(newIt);
		sumdt << AddRows(1);

		Column(sumdt, "Test Date")[] = Column(dt, "finish_t")[itemRows[1]];
		Column(sumdt, "Item")[] = itemID;
		Column(sumdt, "N ")[] = NRow(itemRows);
		Column(sumdt, "Product")[] = prodId;
		Column(sumdt, "Lot")[] = lotID;
		
		dt << Invert Row Selection << Exclude(1);
	
		Current Data Table(dt);
		Summarize(bins = by(:SBIN_NUM), countTable = count());
		
		dt << Select All Rows << Exclude(0);
		Current Data Table(sumdt);
	
		For (i=1, i <= Nitems(binsList), i++,
			binPos = Contains(bins, binsList[i]);
			Column(sumdt, "Bin"||binsList[i])[] = 
				If(binPos == 0, 0, countTable[binPos]);
		);
		
		colNamesSum = sumdt << Get Column Names (String);
		
		For (i=1, i<= Nitems(colList), i++,
			thisName = colNames[colList[i]];
			show(thisName);
			valueSet = Column(dt, colList[i])[itemRows];
			
			thisValueQ10 = Quantile(0.10, valueSet);
			wait(0);
			thisValueQ25 = Quantile (0.25, valueSet);
			wait(0);
			thisValueQ50 = Quantile (0.50, valueSet);
			wait(0);
			thisValueQ75 = Quantile (0.75, valueSet);
			wait(0);
			thisValueQ90 = Quantile (0.90, valueSet);
			wait(0);
			thisValueMean = Mean (valueSet);
			wait(0);
			thisValueStdDev = Std Dev (valueSet);
			wait(0);
			show(thisValueStdDev);
			
			thisColQ10 = Column(sumdt, thisName||"_Q10");
			thisColQ25 = Column(sumdt, thisName||"_Q25");
			thisColQ50 = Column(sumdt, thisName||"_Q50");
			thisColQ75 = Column(sumdt, thisName||"_Q75");
			thisColQ90 = Column(sumdt, thisName||"_Q90");
			thisColMean = Column(sumdt, thisName||"_Mean");
			thisColStdDev = Column(sumdt, thisName||"_StdDev");

			thisColQ10[] = thisValueQ10;
			thisColQ25[] = thisValueQ25;
			thisColQ50[] = thisValueQ50;
			thisColQ75[] = thisValueQ75;
			thisColQ90[] = thisValueQ90;
			thisColMean[] = thisValueMean;
			thisColStdDev[] = thisValueStdDev;
		);
		show("Item completed"),
		newIt = 0;
		show(newIt);
		show("Item already exists in summary table!");
		New Window("WARNING",
			<< modal,
			Text Box("Item "||lotId||"-"||itemID||" already in the table"),
			Text Box("Skipping this item"),
			Button Box("OK")
		);
	);
);
show("Lot completed");
sumdt << Save();

Craige_Hales

remove the current data table(dt); and use an explicit dt where needed. the current data datable can change if you click on a table while the JSL is running.
don't use column[ /* empty brackets with no subscript*/ ] unless you set a current row. The empty [] means "the current row". Empty brackets usually are not needed. Your example probably needs an explicit row subscript.

matlag

I fixed both, and it didn't change my problem.

I no longer believe it is a race condition. It is quite systematic and happening at the same place on the same sets of data. I tried to shuffle around the items order or reduce the scope to one item a time, put the std dev as the first calculation, change the elements order, and I still have the same exact numbers wrong or missing at the output.

Are there known bugs on 12.1 statistical functions?