Faster way to read/write a cell

Question

Hello

I have a big data file (250k rows by 2k columns) which contains production data from electronic  Integrated Circuit devices (ICs).

And I have a jsl script which i use to evaluate the results and look for patterns in the data, but in such a big data table it takes hours to complete a run.

To read/write a cell I was using dt:MeasureX[Row] but that instruction seems to be not execution efficient, so i switched to Column( dt, "MeasureX" )[intRow] and this one is like 20% faster to execute.

Are there tricks to improve the execution time? like stopping the table window refresh while the script is executing...

Thanks

uday_guntupalli · Answer

@oscarmforero,
             Since your data is numeric. It will be singnificantly faster to try and perform most of your actions on a matrix .

dt = Current Data Table();

Mat = dt << Get As Matrix; // fetches all numeric data as a matrix

Once you have the matrix, you can use the loc() for selection based on logical conditions. Here is a more detailed example:

dt = Open( "$SAMPLE_DATA/Big Class.jmp" );

// Get as Matrix 
Mat = dt << Get as Matrix; // Note Matrix will only be 3 columns

// Selecting data in data table 
SR = dt << Select Where(:height <= 65) << Get Selected Rows; 
dt[SR,"height"] = 65; 
dt << Clear Select;

// Selecting data in matrix 
SR1 = Loc(Mat[0,2] <= 65); 
Mat[SR1,2] = 65;

oscarmforero · Answer

Hi Mark For example, I have the below piece of code (which is the one that takes the longer to execute) where I use loops to read cells and to compare their values agains their spec limits. If the cell value is outside the limits, then read and overwrite the value of different cells. numDataColumns = Current Data Table() << get column names( Numeric ); main_table = Current Data Table();
//Loop all columns with parametric data For( intCol = 1, intCol <= N Items( numDataColumns ), intCol++, Spec_Lims = Column( numDataColumns[intCol] ) << Get Property( "Spec Limits" ); Spec_USL_num = Try( Spec_Lims["USL"], . ); Spec_LSL_num = Try( Spec_Lims["LSL"], . ); USL_exist = Is Missing( Spec_USL_num ) == 0; LSL_exist = Is Missing( Spec_LSL_num ) == 0; //Loop all the selected rows (fails) and count the number of parametric fails and executed tests. For( intRow = 1, intRow <= N Rows( Data Table( main_table ) ), intRow++, If( Selected( Row State( Data Table( main_table ), intRow ) ) > 0, If( Is Missing( Column( main_table, numDataColumns[intCol] )[intRow] ) == 0, //Only check for cells with data //If both spec limits are available If( (USL_exist == 1 & LSL_exist == 1) & (Column( main_table, numDataColumns[intCol] )[intRow] > Spec_USL_num | Column( main_table, numDataColumns[intCol] )[intRow] < Spec_LSL_num), Column( main_table, "parFailCount" )[intRow] = Column( main_table, "parFailCount" )[intRow] + 1;
//log names of failing tests Column( main_table, "failListNames" )[intRow] = Column( main_table, "failListNames" )[intRow] || Char( numDataColumns[intCol] ) || ","; ); //Considered only as executed tests when any of the limits are available. If( USL_exist == 1 | LSL_exist == 1, Column( main_table, "parExecutedTests" )[intRow] = Column( main_table, "parExecutedTests" )[intRow] + 1 ); ) ) ); );

Mark_Bailey · Answer

There are many approaches to improving the performance of a script but the method depends on the actions in the script. Can you describe what the script does to "evaluate the results and look for patterns in the data?"