Loc Inverse function

Question

Hi all.  I made this JSL function for a specific task, but I was wondering if there is a better way to do this.

What I want is a way is if you have a vector of indices, to make an indicator vector of a specified link where the value of the vector element is 1 for the specified indice and 0 otherwise.  E.G.

[2, 3, 6] --> convert to indicator vector of length 10.

[0, 1, 1, 0, 0, 0, 1, 0, 0, 0]

The use case is for selecting subsets of rows in a data table based on multiple conditions, where for each condition I get a vector of the row numbers that meet the condition, and I want to get the rows that meet all of the conditions.

// Loc Inverse Function
// This function converts a vector of positive integers (locvals) into an
// indicator vector of a specified length, n,  with the values
// v[I] = 1 if I is in locvals, and 0 otherwise
// example
//      Loc Inverse(6, [1,3])
// returns
//      [1, 0, 1, 0, 0, 0]
*/

Loc Inverse = Function( {n, locvals},
	v = J( n, 1, 0 );
	v[locvals] = 1;
	v;
);

So for the use case, for example, if I have three conditions, I get the row numbers in vectors that meet each condition (v1, v2, v3), so I would do this

dt=current data table();
Loc Inverse = Function( {n, locvals},
	v = J( n, 1, 0 );
	v[locvals] = 1;
	v;
);
// ...
// code to get v1, v2, v3
// ...
rowindicator1=Loc Inverse(n rows(dt), v1);
rowindicator2=Loc Inverse(n rows(dt), v2);
rowindicator3=Loc Inverse(n rows(dt), v3);

meet_all_conditions_indicator = rowindicator1 & rowindicator2 & rowindicator3;

rows_that_meet_all_conditions = Loc(meet_all_conditions_indicator);

yusuke_ono · Answer

Thank you very much, Brady-san

I also try For Each like the below, but your matrix method is still faster. I wildly guess that it takes much time to create Associatve Arrays from vectors.

t1 = HP Time();
a = Associative Array(vec[0, 1]);
For Each({i}, 2::N Col(vec),
	a << Intersect(Associative Array( vec[0, i] ));
);
answer = a << Get Keys;
t2 = HP Time();

brady_brady · Answer

Hello Yusuke-san,

Thank you for bringing associative arrays into the discussion. As you mention, <<intersect is a very nice way to find common elements of associative arrays, and for this reason I used associative arrays in my initial approach to this problem. I eventually abandoned them because I could not get them to run as quickly as the matrix-based method.

As an example, below I include code and log output to compare each method using your 30K row by 9 column sample data set. In this case, the associative array method takes about 65x as long to run as the matrix method.

Cheers,

Brady

Names Default To Here( 1 );
/* Sample Data */
n = 30000;
p = 9;
vec = J( n, p, . );
For( i = 1, i <= N Col( vec ), i++,
	vec[0, i] = (1 :: n)` * i
);

t1 = HP Time();
a = {};
For( i = 1, i <= N Col( vec ), i++,
	a[i] = Associative Array( vec[0, i] )
);
For( i = 2, i <= N Items( a ), i++,
	a[1] << Intersect( a[i] ); /* Intersection Set (A1 cap A2 cap ... cap Ap)*/
);
answer = a[1] << Get Keys;
t2 = HP Time();

common = Function( {nr, vList},
	{default local},
	mat = J( nr, 1, 0 );
	For Each( {i}, 1 :: N Items( vList ), mat[vlist[i]]++ );
	Return( Loc( mat, N Items( vList ) ) );
);

t3 = HP Time();
	vecLis = {};
	For Each( {i}, 1 :: N Col( vec ), Insert Into( vecLis, vec[0, i] ) );
	nr = Max( vec );

answer2 = common( nr, vecLis );

t4 = HP Time();

Print( Eval Insert( "AAs: ^(t2-t1)/1e6^ secs; ^answer^." ) );
Print( Eval Insert( "matrix: ^(t4-t3)/1e6^ secs; ^answer2^." ) );

yusuke_ono · Answer

Although I might misunderstand the question, Associative Array can get the intersection set of two sets.

Names Default To Here(1);
/* Sample Data */
n = 30000;
p = 9;
vec = J(n, p, .);
For(i = 1, i <= NCol(vec), i++,
vec[0, i] = (1::n)`*i;
);

/* From Here */
a={};
For(i = 1, i<=NCol(vec), i++,
a[i] = Associative Array(vec[0,i]);
);
For(i = 2, i <= N Items(a), i++,
a[1] << Intersect(a[i]); /* Intersection Set (A1 cap A2 cap ... cap Ap)*/
);
answer = a[1] <<Get Keys;
Show(Matrix(answer));