Scripting and very large files

Question

I have a script that imports many files and concatenates them.

This file can be >41 million rows and be in the 10's of Gigabits (I have 100's of GB free space and 128GB memory : Using JMP16.2)

I then have scripts to select rows and columns and such.  My question really is not about scripting per se, but why if I run each script in order versus having a script that includes all the scripts, JMP "hangs".

E.g., Everything appears to work in that if I run each JSL in order, it works

1.  Concat_files.JSL

2.  Change_Test_Instance_Name.JSL

3.  Summarize_Test_Instance.JSL

vs

a Script that runs each of the above for me...

E.g.,

Run_all_scripts.JSL

include("ADDIN_HOME(my_scripts)\Concat_files.jsl");

include(("ADDIN_HOME(my_scripts)\Change_Test_Instance_Name.JSL");

include("ADDIN_HOME(my_scripts)\Summarize_Test_Instance.JSL.jsl");

This just hangs my system....  I have the log open and I can tell it is stuck in the next script (Log shows a show statement like show("Entering Change_Test_Instance name");

I have tried using wait(); statements between each include (and within each of the scripts), but that still does not work.  Is there anything else I should try?

justvince · Answer

Sorry for the late reply(busy with work).

Thanks for the ideas.  I have never used the debugger.

I will look at the namespace idea as well.  I have also found that if I have the main DT open (it has been concatenated) and then run the script with include statement it also runs as fast as when I run the script as is (no include).  So something about the main data table being in memory for a while vs concatenating then immediately running the include script.

Craige_Hales · Answer

You might be able to run the code with the JSL debugger/profiler, or just instrument it with finer-grained time_start..time_end like you are doing. My guess is the code has an N^2 behavior (4X time for doubling the data) for some reason with the includes. Maybe a namespace issue with some name that collides with a name in the data table.

One place that could be bad is

dt_summary << select where( Contains( Test_names, :Test_Instance ) );

but it is hard to know without timing it. If Test_names might have 1e6 items and dt_summary might have 1e6 rows, on the average 500,000 test_names would be examined on each of the 1e6 rows. Or maybe there is only 10 or so and it isn't an issue. You might want to rewrite it, approximately like this

// untested. combine two loops and use an aa rather than a list for lookup
Test_names = AssociativeArray();
For Each Row( 
	If( ( Contains( dt_summary:Test_Instance, "CHAR_char" )),
		Test_names[ dt_summary:Test_Instance ] = 1 
	)
);
dt_summary << select where(  Test_names<<Contains( :Test_Instance ) );

The associative array will be much faster if the list has several hundred items.

If you want to use the JSL profiler, don't do the 3 hour run for your first profiling experience.Try a small program, using an include, and make sure you play with these buttons:

Something in the toolbox

A new window opens, the clock is for timing JSL.

The tooltip for the icon says "run profiler".

you might be able to use the blue pause button or wait for it to end, then examine the hot spots:

tabs for sources
tabs for sources

justvince · Answer

Looks like I pasted same log....

Here is when Running alone...  200s or about 3.33minutes

N Rows(dt) = 41929065;
"In Reduce_Test_Instance";
As Date(Today()) = 15Feb2022:10:11:57;
"Reducing Test Instances Ended";
As Date(Today()) = 15Feb2022:10:15:17;
start_time - As Date(Today()) = -200;