Strategies for removing unimportant terms in Text Explorer Analysis

Question

I'm curious what kind of strategies people are using for deciding how to remove unimportant terms from a Text Explorer analysis.

I am working with a customer and they want to reduce the number of terms to something more manageable, and something that explains more of the percent variation when the do latent sematic analyis (the Cum Percent reported under the singluar value).

They are interatively doing LSA then RotatedTopic Analysis for the top x% of the Singular Value Cum Percent (they want to get to 70%), then removing terms.

How do we remove terms in a smart way?

I suggested they use the absolute value of each term for each topic and pick the lowest scores (cause those words aren't contributing to ANY topic).

They were also using Gen Reg to model topics against an outcome variable to find topics not important to that outcome, and removing top terms from zero-ed topics (with the LASSO). I'm less sure about this approach.

Any other thoughts or ideas out there? Should i post this to the public community?

jiancao · Answer

If the goal is to screen out the importatnt terms that are relevant to an outcome variable, you might just run the GenReg for variable selection directly with the DTM columns.  Since it is a large regression it will take a while to finish. For the example of Toronto Casino data, I ran the Lasso with 1,787 binary DTM columns. It took about 20 minutes to complete. I found 483 active (non-zero) terms of them 203 are  statistically significant (p-value<0.05).

A more visual approach is to use the Coloring/By Column Value option in the Word Cloud. Coloring the words by the outcome variable let you see which words are associated with outcomes. The ones with mixed color are not or less associated with the outcome.

Bill_Worley · Answer

Hi Laura,

One of the things I remembered from way back was the suggestion to generate the SVD Scatterplot Matrix and to use that plot as tool to help find stop words from both the term and pharse list.  Select some phrases and it will also highlight the terms.  You can then "Show Text" to see whether or not you want to keep the terms/phrases or add them as stop words.

Best,

Bill

Laura_Higgins · Answer

Great suggestions! I am specifically looking for something after doing all the term and phrase curation - something that is an analytic technique beyond "I think this word might be important". I love Craige's suggestion!

We have ANALYTICAL techniques for finding top terms, so what about bottom terms?