List Intersection PerformanceHere’s a
question for all you JSL wizards.
I have text
data that I need to group in different ways.
style="">
My table looks like this:
style="font-weight: bold; text-decoration: underline; font-family: monospace;">My
Value Category 1
Category 2 Category 3
style="font-family: monospace;">
AA
1
1
0
style="font-family: monospace;">
BB
0
1
0
style="font-family: monospace;">
CC
1
0
0
style="font-family: monospace;">
DD
0
1
0
style="font-family: monospace;">
EE
1
0
0
style="font-family: monospace;">
FF
0
0
0
style="font-family: monospace;">
AA
1
1
0
style="font-family: monospace;">
BB
0
1
0
style="font-family: monospace;">
GG
0
0
0
style="font-family: monospace;">
HH
0
0
1
style="font-family: monospace;">
MM
0
1
1
The category
columns allow me to group the data in different ways.
style=""> The
categories for this example come from the
following lists:
style="font-size: 10pt; font-family: "Courier New"; color: black;">Category
1
style="font-size: 10pt; font-family: "Courier New"; color: navy;">=
style="font-size: 10pt; font-family: "Courier New"; color: black;">
{
style="font-size: 10pt; font-family: "Courier New"; color: purple;">"AA"
style="font-size: 10pt; font-family: "Courier New"; color: navy;">,
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: purple;">"CC"
style="font-size: 10pt; font-family: "Courier New"; color: navy;">,
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: purple;">"EE"
style="font-size: 10pt; font-family: "Courier New"; color: black;">}
style="font-size: 10pt; font-family: "Courier New"; color: navy;">;
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: black;">Category
2
style="font-size: 10pt; font-family: "Courier New"; color: navy;">=
style="font-size: 10pt; font-family: "Courier New"; color: black;">
{
style="font-size: 10pt; font-family: "Courier New"; color: purple;">"AA"
style="font-size: 10pt; font-family: "Courier New"; color: navy;">,
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: purple;">"BB"
style="font-size: 10pt; font-family: "Courier New"; color: navy;">,
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: purple;">"DD"
style="font-size: 10pt; font-family: "Courier New"; color: navy;">,
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: purple;">"MM"
style="font-size: 10pt; font-family: "Courier New"; color: black;">}
style="font-size: 10pt; font-family: "Courier New"; color: navy;">;
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; line-height: 115%; font-family: "Courier New"; color: black;">Category
3
style="font-size: 10pt; line-height: 115%; font-family: "Courier New"; color: navy;">=
style="font-size: 10pt; line-height: 115%; font-family: "Courier New"; color: black;">
{
style="font-size: 10pt; line-height: 115%; font-family: "Courier New"; color: purple;">"KK"
style="font-size: 10pt; line-height: 115%; font-family: "Courier New"; color: navy;">,
style="font-size: 10pt; line-height: 115%; font-family: "Courier New"; color: black;">
style="font-size: 10pt; line-height: 115%; font-family: "Courier New"; color: purple;">"MM"
style="font-size: 10pt; line-height: 115%; font-family: "Courier New"; color: black;">}
style="font-size: 10pt; line-height: 115%; font-family: "Courier New"; color: navy;">;
If you’re
wondering why I don’t just add one category column,
it’s because there can be
overlap in the category lists.
Thus I
need a separate column for each category.
If a value
in the MyValue
column matches a
value in a category list, then the category column is 1, otherwise it
is 0.
This allows me to rapidly perform separate analyses of the data
matching each category.
My problem
is one of performance when calculating the category columns.
style=""> It’s
not bad for small tables (10,000 rows, 3
categories, small category lists).
style="">
However it gets bogged down
for larger datasets (300,000 rows, 200
categories, large category lists).
style=""> Some
of the category lists can be quite large (2000 values).
Basically I
need a fast way to do some sort of list intersection between the data
in the MyValue
column and the data in each
category list.
This is a very simple
example but it illustrates the concept.
Here’s the
logic I’m using for one category.
style=""> The
ability to find matches between a list and a column using
style="">SELECT WHERE
is very powerful.
If it can be sped up somehow
I’d love to know how!
style="font-size: 10pt; font-family: "Courier New"; color: black;">one_category
style="font-size: 10pt; font-family: "Courier New"; color: navy;">=
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: purple;">"Category
1"
style="font-size: 10pt; font-family: "Courier New"; color: navy;">;
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: black;">category_list
style="font-size: 10pt; font-family: "Courier New"; color: navy;">=
style="font-size: 10pt; font-family: "Courier New"; color: black;">
{
style="font-size: 10pt; font-family: "Courier New"; color: purple;">"AA"
style="font-size: 10pt; font-family: "Courier New"; color: navy;">,
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: purple;">"CC"
style="font-size: 10pt; font-family: "Courier New"; color: navy;">,
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: purple;">"EE"
style="font-size: 10pt; font-family: "Courier New"; color: black;">}
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: black;">nr
style="font-size: 10pt; font-family: "Courier New"; color: navy;">=
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: rgb(0, 0, 221);">nrows
style="font-size: 10pt; font-family: "Courier New"; color: black;">(
style="font-size: 10pt; font-family: "Courier New"; color: black;">dt)
style="font-size: 10pt; font-family: "Courier New"; color: navy;">;
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: black;">col_zeroes
style="font-size: 10pt; font-family: "Courier New"; color: navy;">=
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: rgb(0, 0, 221);">j
style="font-size: 10pt; font-family: "Courier New"; color: black;">(
style="font-size: 10pt; font-family: "Courier New"; color: black;">nr
style="font-size: 10pt; font-family: "Courier New"; color: navy;">,
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: teal;">1
style="font-size: 10pt; font-family: "Courier New"; color: navy;">,
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: teal;">0
style="font-size: 10pt; font-family: "Courier New"; color: black;">)
style="font-size: 10pt; font-family: "Courier New"; color: navy;">;
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: green;">//
Add the category column
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: black;">dt
style="font-size: 10pt; font-family: "Courier New"; color: navy;"><<
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: rgb(0, 0, 221);">new
column
style="font-size: 10pt; font-family: "Courier New"; color: black;">(
style="font-size: 10pt; font-family: "Courier New"; color: black;">one_category
style="font-size: 10pt; font-family: "Courier New"; color: navy;">,
style="font-size: 10pt; font-family: "Courier New"; color: black;">
numeric)
style="font-size: 10pt; font-family: "Courier New"; color: navy;">;
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: rgb(0, 0, 221);">column
style="font-size: 10pt; font-family: "Courier New"; color: black;">(
style="font-size: 10pt; font-family: "Courier New"; color: black;">dt
style="font-size: 10pt; font-family: "Courier New"; color: navy;">,
style="font-size: 10pt; font-family: "Courier New"; color: black;">
one_category)
style="font-size: 10pt; font-family: "Courier New"; color: navy;"><<
style="font-size: 10pt; font-family: "Courier New"; color: black;">
hide(
style="font-size: 10pt; font-family: "Courier New"; color: teal;">1
style="font-size: 10pt; font-family: "Courier New"; color: black;">)
style="font-size: 10pt; font-family: "Courier New"; color: navy;">;
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: green;">//
Set the entire column to be
zeroes initially
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: rgb(0, 0, 221);">column
style="font-size: 10pt; font-family: "Courier New"; color: black;">(
style="font-size: 10pt; font-family: "Courier New"; color: black;">dt
style="font-size: 10pt; font-family: "Courier New"; color: navy;">,
style="font-size: 10pt; font-family: "Courier New"; color: black;">
one_category)
style="font-size: 10pt; font-family: "Courier New"; color: navy;"><<
style="font-size: 10pt; font-family: "Courier New"; color: black;">
values(col_zeroes[
style="font-size: 10pt; font-family: "Courier New"; color: teal;">0
style="font-size: 10pt; font-family: "Courier New"; color: navy;">,
style="font-size: 10pt; font-family: "Courier New"; color: teal;">1
style="font-size: 10pt; font-family: "Courier New"; color: black;">])
style="font-size: 10pt; font-family: "Courier New"; color: navy;">;
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: green;">//
Select all rows with values
matching this category
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: black;">dt
style="font-size: 10pt; font-family: "Courier New"; color: navy;"><<
style="font-size: 10pt; font-family: "Courier New"; color: black;">
select where(
style="font-size: 10pt; font-family: "Courier New"; color: rgb(0, 0, 221);">contains
style="font-size: 10pt; font-family: "Courier New"; color: black;">(
style="font-size: 10pt; font-family: "Courier New"; color: black;">category_list
style="font-size: 10pt; font-family: "Courier New"; color: navy;">,
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: navy;">:
style="font-size: 10pt; font-family: "Courier New"; color: black;">My
Value))
style="font-size: 10pt; font-family: "Courier New"; color: navy;">;
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: green;">//
Get a list of the selected
rows
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: black;">one_cat_rows
style="font-size: 10pt; font-family: "Courier New"; color: navy;">=
style="font-size: 10pt; font-family: "Courier New"; color: black;">
dt
style="font-size: 10pt; font-family: "Courier New"; color: navy;"><<
style="font-size: 10pt; font-family: "Courier New"; color: black;">
get selected
rows()
style="font-size: 10pt; font-family: "Courier New"; color: navy;">;
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: green;">//
Set all matching rows to 1 for
this category
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: rgb(0, 0, 221);">if
style="font-size: 10pt; font-family: "Courier New"; color: black;">
(
style="font-size: 10pt; font-family: "Courier New"; color: rgb(0, 0, 221);">nrows
style="font-size: 10pt; font-family: "Courier New"; color: black;">(
style="font-size: 10pt; font-family: "Courier New"; color: black;">one_cat_rows)
style="font-size: 10pt; font-family: "Courier New"; color: navy;">>
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: teal;">0
style="font-size: 10pt; font-family: "Courier New"; color: navy;">,
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="">
style="font-size: 10pt; font-family: "Courier New"; color: rgb(0, 0, 221);">column
style="font-size: 10pt; font-family: "Courier New"; color: black;">(
style="font-size: 10pt; font-family: "Courier New"; color: black;">dt
style="font-size: 10pt; font-family: "Courier New"; color: navy;">,
style="font-size: 10pt; font-family: "Courier New"; color: black;">
one_category)[one_cat_rows]
style="font-size: 10pt; font-family: "Courier New"; color: navy;">=
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; font-family: "Courier New"; color: teal;">1
style="font-size: 10pt; font-family: "Courier New"; color: navy;">;
style="font-size: 10pt; font-family: "Courier New"; color: black;">
style="font-size: 10pt; line-height: 115%; font-family: "Courier New"; color: black;">)
style="font-size: 10pt; line-height: 115%; font-family: "Courier New"; color: navy;">;
