What is the proper/efficient way to filter data + use of dynamic column names?

Question

I am currently working on a script that filters the data table using Get Rows Where Then I use the rows to get the values in a list. Below is part of my script: rows = main_dt << Get Rows Where( As Column(G_TEMP_COL_STR) == M_TEMP & As Column(G_WAFER_COL_STR) == M_WAFER );

values_list = main_dt:(Eval(G_VALUE_COL_STR))[rows]; I'd like to know: * Is there a better way of doing this? * Why is As Column() slow compared to directly using :?* I am using As Column because I made it so that I can assign specific columns on a GUI

David_Burnham · Answer

The example given by @txnelson can be simplified (IMO) by also using the EvalInsert function: this allows variable substitution based on the caret (^) symbol.  Additionally the notation \[ ...]\ can be used so that quotes can be used without having to use escape sequences.  So ultimately it's possible to write:

Eval(Parse(EvalInsert("\[

rows = main_dt << Get Rows Where(
		:^G_TEMP_COL_STR^ == "^M_TEMP^" &
		:^G_WAFER_COL_STR^ == "^M_WAFER^"
	);

]")));

This style of code could also be used to reference the column instead of

values_list = main_dt:Eval(G_VALUE_COL_STR)[rows]

But I'm not sure why you can't just write:

values_list = column(main_dt,G_VALUE_COL_STR)[rows];

TheRealYeti · Answer

Thanks @txnelson ! Yes, it is being called like 50 or more times and that depends on the number of combinations of wafer, temperature and vcc (added). I got the same performance using the code above. In fact I also tried doing

main_dt:Eval(G_TEMP_COL_STR) == M_TEMP

but it is still noticeably slower compared to doing

:temperature = M_TEMP

I think it must be the evaluation string that is slowing it down.

I did "lazy/lousy" workaround, I kept the

Get Rows Where( :temperature == M_TEMP ...)

And renamed the column I selected in the GUI to temperature to match it.

As Column(dlg_ret[2]) << Set Name( "temperature" );

It works fast as expected but renames the column names.

txnelson · Answer

You are not providing enough information for me to tell you if there is a better way.  "Get Rows Where" is very efficient, unless you are using it a lot.  It does a complete pass through the data to get its results.  If you are doing this a lot, it might be better to use a different method that reduces the number of full data scans.

You stated that using As Column() is slower than using a direct reference to a Column Name.  If that is what you have determined, you could change your code to the following.  It will pass to JSL a hardcoded version of your Get Rows Where().

G_TEMP_COL_STR="Height";
M_TEMP =72;
G_WAFER_COL_STR = "WAFER_NUM";
M_WAFER =12;
G_VALUE_COL_STR = "Values";

Eval(Parse("rows = main_dt << Get Rows Where( :" || 
        G_TEMP_COL_STR || " == \!"" || Char(M_TEMP) || "\!" & :" ||
        G_WAFER_COL_STR || " == " || Char(M_WAFER) || " );
        values_list = main_dt:" || G_VALUE_COL_STR || "[rows];" ));