599896_en-US

liliconcom

I want to convert a large string to matrix and do it fast

I have data I retrieve from a database which is in a pipe-delimited string format. For example "70|71|69|78".

They are really long strings and I have a large data set of them.

What I need to do is convert the strings to a matrix ( vector really ) and do some operations on the resulting vector ( Max, 95th percentile etc )

Here's how I'm doing it today :

ExampleStr = "70|71|71|72|72|72|72|72|74|74|74|74|74|74|74|74|74|74|75|75|75|75|75|75|75|75|75|75|75|75|75|75|75|75|75|75|75|75|75|76|75|75|76|76|76|75|75|76|76|75|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|77|76|76|76|76|76|76|76|76|76|76|76|76|77|76|76|76|76|76|76|77|76|77|76|77|76|76|77|76|76|76|76|76|76|76|76|77|76|76|76|76|77|77|77|77|76|76|77|77|76|76|76|76|77|76|77|76|76|76|76|76|77|76|76|76|77|77|77|76|76|77|76|76|76|77|76|76|76|76|76|76|77|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|77|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|76|77|76|76|76|76|76|76|76|76|76|77|77|77|77|77|77|77|77|76|77|77|76|76|77|76|76|77|76|77|76|77|76|77|77|77|76|76|76|77|77|76|76|77|76|76|76|76|76|76|76|77|76|76|76|76|76|76|76|77|77|77|76|77|77|76|76|77|76|77|76|77|77|77|77|77|76|76|77|77|77|77|77|77|76|76|77|77|77|77|77|77|76|77|77|77|77|77|77|77|77|77|77|77|76|77|77|77|77|76|77|77|76|76|76|77|77|77|77|77|77|77|77|77|76|76|77|77|77|77|77|77|77|77|77|77|76|76|76|77|77|77|77|77|77|76|77|77|77|76|77|77|77|77|76|76|77|77|77|77|77|77|77|77|77|77|76|77|77|77|77|76|77|77|77|77|77|76|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|76|77|77|76|77|77|77|77|76|77|76|77|77|77|77|77|77|77|77|77|77|76|77|77|77|77|77|77|77|77|77|76|77|77|77|77|77|76|77|77|77|77|77|77|77|77|77|77|77|77|76|77|77|77|77|77|77|77|76|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|76|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|77|76|77|77|77|77|77|77|77|77|77|77|77|77|77|78|78|78|78|78|78|78|78|78|78|77|78|78|77|78|77|77|77|78|78|78|78|78|78|77|78|77|78|78|78|78|78|78|77|78|78|78|77|78|77|78|78|78|77|78|77|78|78|78|78|78|77|78|78|78|78|78|78|78|78|78|77|78|78|78|77|78|78|78|77|77|78|78|78|78|78|78|78|78|78|78|78|78|78|77|77|78|77|77|78|77|78|78|78|78|78|78|78|78|77|77|78|78|78|77|77|78|78|78|78|78|78|78|78|78|78|78|78|77|78|78|78|78|78|78|77|78|78|78|78|77|78|78|78|78|77|77|78|78|78|78|78|78";
  t1 = HPTime();
  V = parse("["||substitute(ExampleStr,"|",",")||"]") ;
  t2 = HPTime();
  show((t2-t1)/1000); // takes about 0.9 milliSec

So I replace the pipes with commas, then make the string look like a matrix by prefixing and suffixing matrix brackets. That part is quick.

Where the bulk of the time goes is in the Parse() call.

For this example with 775 values in the pipe-delimited string, it takes about 0.9 mSec to do the above code.

That might seem like very little time, but I need it to be at least an order of magnitude less. Because my data table, in my current example, has 762,000 rows of data!

Not every string is 775 values long, some are shorter, and not every iteration takes 0.9 mS, but in total it took almost 10 mins to go through the data table and do what I need to do  which is something like this :

 

dt << new column("P95_Value");
foreachrow(
 V = parse("["||substitute(:StrColumn,"|",",")||"]") ;
 :P95_Value = VQuantile(V,0.95); 
);

The VQuantile() call is not a problem, it is very fast. The bulk of the time goes on the parse() line.

I just wondered is there any smarter way to process this string and turn it into a vector, ideally in more like 0.09 milliSec or better !

 

Re: I want to convert a large string to matrix and do it fast

I think it is not a smart way, but a little faster. It takes about 3 minutes on my computer when the data has 762,000 rows.

Names Default To Here( 1 );

dt = Current Data Table();
t1 = HP Time();

//Subset data table
subset = dt << Subset(
 All rows,
 Selected columns only( 0 ),
 columns( :StrColumn )
);
subset << show window(0);
subset << Begin Data Update;

//Text to Column
subset << Text to Columns( columns( :StrColumn ), Delimiters( "|" ) );

// Change Character to Numeric
subset << delete column(:StrColumn);
col_ref_list = subset << Get Column Reference(); 
For Each( {col, index}, col_ref_list, col << Data Type( Numeric, Format( "Best", 12 ) ) );

//Get 95% Percentile
mt = subset << get as matrix();
P95_value_mt = Transpose( V Quantile( Transpose( mt ), 0.95 ) );

close(subset,nosave);

//Set Value to Column
dt << New Column( "P95_Value" );
:P95_Value << set values( P95_value_mt );

dt << End Data Update;

t2 = HP Time();
Show( (t2 - t1) / 1000000 );