521957_en-US

liliconcom

Efficient Way via JSL to Hide and/or Exclude a Large Number of Columns

For a genomics examples @Chris_Kirchberg has developed (see https://community.jmp.com/t5/Technical-Reference-Library/C40-mRNA-Expression-Cattle-Follicle-Development/ta-p/519216), part of the workflow is to identify columns that should be excluded from analysis due to data quality issues.  He and I are working on a script to do this efficiently.  The example has 28000 columns and about 9000 of those columns need to be hidden and excluded.  We are doing this with a loop in JSL right now, but it is rather slow (takes several seconds)

// collist is a list of column references
For( i = 1, i <= N Items( collist ), i++,
 collist[i] << Hide( 1 ) << Exclude( 1 )
);

It seems there ought to be faster way to do this.  Any suggestions to make this faster?  

Re: Efficient Way via JSL to Hide and/or Exclude a Large Number of Columns

Follow-up:  I shared this issue with development, and several improvements have been made to speed up this type of operation.  Looping through columns to hide/exclude will still see a slow down as the number of columns increase, but some new JSL messages for hiding and excluding columns will be available, and the action recorder will use those where it is appropriate. This should be available in JMP18 development builds and the 18 Early Adopter version.  

From the Jira issue (JMP-13445)

Fixed by adding new Hide Columns and Exclude Columns commands on the data table. They can be called like this:

<SPAN class="">dt << Hide Columns; // hides selected columns </SPAN>

dt << Hide Columns(1|0); // hides or unhides columns

dt << Hide Columns( COLUMN_LIST ); // hides listed columns

dt << Hide Columns( 1|0, COLUMN_LIST ); // hides or unhides listed columns

Exclude Columns uses the same syntax.

Re: Efficient Way via JSL to Hide and/or Exclude a Large Number of Columns

 Quick update: This seems to work on the Mac if I included it in a larger script, but does not work on windows.  Is there something special I need to do for windows version of JMP  to have the Main Menu () call work within a larger script?

Re: Efficient Way via JSL to Hide and/or Exclude a Large Number of Columns

Main Menu( "Cols:Hide/Unhide" );

Tried this again but copy and pasted it into my script and now it works. Sun of a Gunderson! Why would it not work and now it does? The only thing I can think of is that it works if I run only that line with columns selected, but when put in a script that selects columns then the next line is this, it does not work.

Re: Efficient Way via JSL to Hide and/or Exclude a Large Number of Columns

@brady_brady , for some reason Main Menu("Cols:Hide/Unhide") does not work on Windows 10 JMP 17 EA9. Any reason why this might be? At least it does not work for me.

Re: Efficient Way via JSL to Hide and/or Exclude a Large Number of Columns

The main menu commands are really fast, and I don't view them as hacks, but it would be nice if JSL could run the same code that the Main Menu commands do, via message. Alternatively, there could be an expansion of Main Message syntax in the scripting guide, and standardization across platforms in the case where the main menus don't look the same between Mac and PC.

Re: Efficient Way via JSL to Hide and/or Exclude a Large Number of Columns

It is part of a workflow to process genomic expression data.  Often there are many columns in the analysis data table (where the column corresponds to a particular genomic expression variable) where there is little to no expression, so you would like to identify those columns and then not use them in further analysis.  Hiding and Excluding is a convenient way to not display those columns in subsequent analysis platform dialogs.  While we could also just delete the columns or subset the columns we want to use, there is some value to keeping the excluded/hidden columns in the data table.

Re: Efficient Way via JSL to Hide and/or Exclude a Large Number of Columns

I also tried using for each() and it was still very slow

for each({c, ii}, collist,
c << Hide(1) << Exclude(1)
)

and that was equally slow.  

Using the Main Menu command works super fast (thanks @yuichi_katsumur !), but it feels like a hack.  I appreciate the suggestions and help on this.  

Re: Efficient Way via JSL to Hide and/or Exclude a Large Number of Columns

Hi Sam,

How about following script? It is much faster.

dt << clear column selection();
dt << Select Columns( collist );
Wait( 0 );
Main Menu( "Cols:Hide/Unhide" );
Main Menu( "Cols:Exclude/Unexclude" );
dt << clear column selection();

Re: Efficient Way via JSL to Hide and/or Exclude a Large Number of Columns

I tried a few tricks, none of them worked.

-Enclosing the work in 

dt << Begin Data Update;
//work here...
dt << End Data Update;

-Making data table invisible

-Making data table private

What's the purpose of hiding and excluding the columns in this application? Maybe there's a workaround.

Re: Efficient Way via JSL to Hide and/or Exclude a Large Number of Columns

I also tried 

collist << Hide(1);
collist << Exclude(1);

and that is also very slow.