52689_en-US

liliconcom

JMP Script Error with large table only: Maximum Length of Arrays?

Hello JMP community,

 

Using the script below, which works great for tables with ~500 columns, I ran into a weird problem when using it on a large table with 20,000 columns: essentially, the script appears to complete normally, producing the output table as desired but the actual values are pretty much random (as verified manually).

 

Any clue on what may be happening? I have a hunch that this might be due to a memory allocation error due to exceeding the maximum length of the arrays I use to collect the data.

 

Note: I cannot share the large table I'm working on because of the confidential nature of the data

Thanks,

Names Default to Here (1);

dt = Current Data Table ();

m = {};
m1 = [];
m2 = [];
m3 = [];
m4 = [];

For (i = 12 , i <= N Col (dt),i++,

  mv = Multivariate(
    Y( :BIOMARKER, column(i) ),
    Estimation Method( "Row-wise" ),
    Scatterplot Matrix(
     Density Ellipses( 1 ),
     Shaded Ellipses( 0 ),
     Ellipse Color( 3 )
    ),
    Spearman's ρ( 1 ),
    Kendall's τ( 1 )
   );
   
   mvr = mv << report;
   insert into (m, Column Name (i));
   m1 = m1 || ((mvr [Outline Box (1)] [Number Col Box (1)])<< get) [1];
   m2 = m2 || ((mvr [Outline Box (1)] [Number Col Box (2)])<< get) [1];
   m3 = m3 || ((mvr [Outline Box (1)] [Number Col Box (3)])<< get) [1];
   m4 = m4 || ((mvr [Outline Box (1)] [Number Col Box (4)])<< get) [1];
   mv << close window;
  );
New Table ("Non-Parametric Correlation",
 New column ("Symbol", character, set values (m)),
 New column ("Spearman Rho", numeric, set values (m1)),
 New column ("Spearman p", numeric, set values (m2)),
 New column ("Kendall Tau", numeric, set values (m3)),
 New column ("Kendal p", numeric, set values (m4)),
);

Re: JMP Script Error with large table only: Maximum Length of Arrays?

Try this:

Names Default To Here(1);

dt = Open("$SAMPLE_DATA/Body Measurements.jmp");
dt_out = New Table("Non-Parametric Correlation",
    Add Rows(N Col(dt)),
    New Column("Symbol", character),
    New Column("Spearman Rho", numeric),
    New Column("Spearman p", numeric),
    New Column("Kendall Tau", numeric),
    New Column("Kendal p", numeric),
    Invisible
);
r = 1;
For(i = 2, i <= N Col(dt), i++,
    m = [];
    mv = dt << Multivariate(
        Y(:Mass, Column(i)),
        Estimation Method("Row-wise"),
        Scatterplot Matrix(0),
        Spearman's ρ(1),
        Kendall's τ(1)
    );   
    mob = Report(mv)[Outline Box(1)];
    Column(dt_out, 1)[r] = Column(dt, i) << get name;
    For(c = 2, c <= 5, c++,
        Column(dt_out, c)[r] = (mob[Number Col Box(c - 1)] << get)[1];
    );
    mv << close window;
    r++;
);
dt_out << show window(1);

Re: JMP Script Error with large table only: Maximum Length of Arrays?

Thanks for an elegant solution to my problem.

 

I just tested it on my data (after replacing the data table and fixed variable) and it returned an error message (debugger): Invalid Argument at the line

dt_out<SPAN class="token punctuation">[</SPAN>r<SPAN class="token punctuation">,</SPAN> <SPAN class="token number">1</SPAN><SPAN class="token punctuation">]</SPAN> <SPAN class="token operator">=</SPAN> <SPAN class="token function">Column Name</SPAN><SPAN class="token punctuation">(</SPAN>i<SPAN class="token punctuation">)</SPAN><SPAN class="token punctuation">;</SPAN>

I guess this command or syntax that is not compatible with JMP 11.1: what would be the best work around?

Re: JMP Script Error with large table only: Maximum Length of Arrays?

Not sure what's happening. It may work better if incrementally populating the output table instead of collecting all data in big lists/matrices. 

 

An example (JMP 13):

Names Default To Here(1);

dt = Open("$SAMPLE_DATA/Body Measurements.jmp");
dt_out = New Table("Non-Parametric Correlation",
    Add Rows(N Col(dt)),
    New Column("Symbol", character),
    New Column("Spearman Rho", numeric),
    New Column("Spearman p", numeric),
    New Column("Kendall Tau", numeric),
    New Column("Kendal p", numeric),
    Invisible
);
r = 1;
For(i = 2, i <= N Col(dt), i++,
    m = [];
    mv = dt << Multivariate(
        Y(:Mass, Column(i)),
        Estimation Method("Row-wise"),
        Scatterplot Matrix(0),
        Spearman's ρ(1),
        Kendall's τ(1)
    );
   
    mob = Report(mv)[Outline Box(1)];
    m ||= mob[Number Col Box(1)] << get as matrix;
    m ||= mob[Number Col Box(2)] << get as matrix;
    m ||= mob[Number Col Box(3)] << get as matrix;
    m ||= mob[Number Col Box(4)] << get as matrix;
    mv << close window;
    dt_out[r, 1] = Column Name(i);
    dt_out[r, 2 :: 5] = m;
    r++;
);
dt_out << show window(1);

  

Re: JMP Script Error with large table only: Maximum Length of Arrays?

Adding some detail, most of the JMP Platforms, (i.e. Distribution, Partition, etc.) use multiple cores in processing.  JSL code is single processor, but the Multivarate platform will use all of the processors it can get a hold of.

Re: JMP Script Error with large table only: Maximum Length of Arrays?

Hi Uday,

 

Thanks.

 

I just tried running the same script with no other application running in the background and it worked fine. So it looks like it might have been some kind of memory overflow even though according to the Task Manager (Windows 10) I had 2 GB RAM free.

Re: JMP Script Error with large table only: Maximum Length of Arrays?

@sornasst, 

  I don't know how much help this would be, but can you share your system RAM details here ? I don't think JMP runs on multi core, so assuming it is running on single core, is you RAM able to support this ?