VConcat many matrices together

tsl

Concatenating matrices in a loop slows down with loop iterations.

For example this pseudocode:

OutputM = [];
for( i=1, i<=N, i++,
&nbsp; // do some calculations, generate a matrix of results, call it,&nbsp; ThisM
&nbsp; OutputM&nbsp; |/ = ThisM
);

If I do the above, then as N increases, the VConcat line gets slower and slower with every iteration.

Note that each iteration produces a matrix with same number of columns, but variable number of rows.

It's much faster to do this :

OutputList = {}; // will be a list of matrices
for( i=1, i<=N, i++,
&nbsp; // do some calculations, generate a matrix of results, call it,&nbsp; ThisM
  insertinto(OutputList, ThisM)
);

Now we want to concat together the list of matrices we've generated.

The question is how ?

//This doesn't work:
VConcat(OutputMList);
// Neither does this :
VConat(Eval List(OutputMList);
// Nor this:
VConcat(Eval list(Eval(OutputMList);
// I know I can concat multiple matrices, for example this works:
VConat(OutputMList[1],OutputMList[2], OutputMList[3]);
// But the desire is to generalize this

  

 

 

 

Craige_Hales

If you are concatenating 50 or more, you might not want to use a loop to concatenate them one at a time. If you already have a list of mats to concat, you can turn the list into a vconcat function like this:

	Substitute Into( OutputList, {}, Expr( vconcat ) ); // convert {a,b,c} -> vconcat(a,b,c)
	result2 = Eval( OutputList ); // evaluate the expression

The eval can be 10X faster than the loop:

 

 

JSL to make the graph:

dt = New Table( "timecat", New Column( "nmats" ), New Column( "timeloop" ), New Column( "timeeval" ) );
dt << Graph Builder( Size( 534, 456 ), Show Control Panel( 0 ),
	Variables( X( :nmats ), Y( :timeloop ), Y( :timeeval, Position( 1 ) ) ),
	Elements( Points( X, Y( 1 ), Y( 2 ), Legend( 3 ) ) ),
	SendToReport( Dispatch( {}, "timeloop", ScaleBox,
		{Scale( "Log" ), Format( "Engineering SI", 9, 0 ), Min( 10 ), Max( 1e7 ), Inc( 1 ), Minor Ticks( 1 ),
		Add Ref Line( 1e5, "Solid", "Black", "1/10 second", 1 ), Add Ref Line( 1e6, "Solid", "Black", "one second", 1 ),
		Label Row( Show Minor Labels( 0 ) )} ),
		Dispatch( {}, "graph title", TextEditBox, {Set Text( "smaller is better" )} ),
		Dispatch( {}, "X title", TextEditBox, {Set Text( "number of vconcats" )} ),
		Dispatch( {}, "Y title", TextEditBox, {Set Text( "microseconds" )} )
	)
);

For( N = 1, N < 550, N = Ceiling( n * 1.2 ), // build a bunch of test sets of bigger and bigger sizes
	OutputList = {}; // will be a list of matrices
	For( i = 1, i <= N + 1, i++, 
  // do some calculations, generate a matrix of results, call it,  ThisM
		ThisM = J( i, 30, i ); // rows, cols, initial value. 
		Insert Into( OutputList, ThisM );
	);
// loop is slower ... blue in graph
	start = HP Time();
	result1 = OutputList[1];
	For( i = 2, i <= N Items( OutputList ), i += 1,
		result1 |/= OutputList[i]; // this is the vconcat into operator
	);
	t1 = HP Time() - start;
// eval is faster ... red in graph
	start = HP Time();
	Substitute Into( OutputList, {}, Expr( vconcat ) ); // convert {a,b,c} -> vconcat(a,b,c)
	result2 = Eval( OutputList ); // evaluate the expression
	t2 = HP Time() - start;
// capture the timings
	dt << addrows( 1 );
	dt:nmats[N Rows( dt )] = N;
	dt:timeloop[N Rows( dt )] = t1;
	dt:timeeval[N Rows( dt )] = t2;
	If( !All( result1 == result2 ),
		Throw( "bug?" ) // they should be identical
	);
	Wait( .1 ); // watch the graph grow
);