How to make Uppercase, Regex and Contain function efficient?I have a script to that takes user input and scanned through the entire table and tag the user input in the new column. I have tables with >10M rows and this script is taking few minutes. How can I make it faster? Thanks in advance.
// Capture the start time
start_time = Tick Seconds();
// --- Setup & guards ---
dt = Current Data Table();
If( Is Empty( dt ), Throw( "No active data table." ));
colNames = dt << Get Column Names("String");
Case sensitive = 0;
// Open a dialog to select exactly one column
nw = New Window("User Inputs",
<<MODAL></MODAL>
Basic Data Analysis and ModelingRe: How to make Uppercase, Regex and Contain function efficient?What is the script supposed to do?
Re: How to make Uppercase, Regex and Contain function efficient?It creates a “Suspected Column1” column and marks “Yes” when the value in Column1 matches the user input. I have few additional steps that summarizes which user inputs were found and which were missing. But Those steps are efficient so I did not included them.
Re: How to make Uppercase, Regex and Contain function efficient?In your example table you are comparing numeric values to characters. What types of checks are you trying to do:
- You say match but the code is saying contain
- Code is also saying contain WITH data modifications
- The example table you provided has numeric values, but you are comparing them to characters
- Edit: are all the values unique?
Edit: the code is looking for "exact matches" as it utilizes Contains with Associative Array. Biggest issue here is the amount of data and comparison between numbers and characters. The check can be most likely optimized with some evaluations but depending on the real issue, there might be plenty other optimizations to be done.
