Select column from effect summary

Question

Hello everyone:

I have a huge dataset with 1000+ columns.  I am doing a regression model using Fit model function.  The effect summary gave me 171 columns out of 6700+ which are the best for my model.  My question is how can I choose those 171 easily without picking them one by one? please see the picture below for more details.

Thanks,

Anwar

jules · Answer

Hi @AlotaibiA, That seems to be a bit more tricky. And, I just discovered that Right Click > Copy Columns copies not just the selected columns, but ALL the columns in that table. So, here's what I would do (until we figure out a more elegant solution): rather than right-click select columns, right-click > make into data table. This will make a table of those columns (with their statistics). The columns you selected before will still be selected (as rows). Click on the "Source" column so that you have selected just the column names, and now you can use a standard edit > copy to copy just those columns to your clipboard. Now, as for selecting those columns in your original table, I do not have a simple interactive solution handy. But, we can script our way around this until we figure out one. With your original table in the foreground, you can use the following script: colNames = words(Get Clipboard(),"\!r"); colList = {}; for(i=1,i<=length(colNames),i++, insertInto(colList, parse(":Name(\!"" || colNames[i] || "\!")" ) ) ); current data table() << clear column selection; current data table() << Select Columns( colList ); This will take your clipboard (with those column names), and assemble a list of column references, and then send a message to your current table to select those columns. It's not the most elegant solution, but it's something to start! I hope this helps until we can come up with something simple. @jules

AlotaibiA · Answer

Thanks! How can I select them to subset them from the dataset?

Mark_Zwald · Answer

Another option is to use the stepwise personality under Fit Model to identify significant effects but you would be guaranteed in this case to end up with an overfit model because you have more model effects than rows of data (6700 model effects and 4000 rows of data).  Stepwise is primarily effective when you have relatively few model effects compared to the sample size of data.

In your particular case, you would need to use the Partial Least Squares modeling option under multivariate if your response is numeric, or the PLS option under Fit Model if your response is categorical (this option requires JMP Pro).

Another option (also requires JMP Pro) is to create a Fit Model using Training/Validation/Test sets which can help mitigate overfitting.