Strategies for removing unimportant terms in Text Explorer AnalysisI'm curious what kind of strategies people are using for deciding how to remove unimportant terms from a Text Explorer analysis.
I am working with a customer and they want to reduce the number of terms to something more manageable, and something that explains more of the percent variation when the do latent sematic analyis (the Cum Percent reported under the singluar value).
They are interatively doing LSA then RotatedTopic Analysis for the top x% of the Singular Value Cum Percent (they want to get to 70%), then removing terms.
How do we remove terms in a smart way?
I suggested they use the absolute value of each term for each topic and pick the lowest scores (cause those words aren't contributing to ANY topic).
They were also using Gen Reg to model topics against an outcome variable to find topics not important to that outcome, and removing top terms from zero-ed topics (with the LASSO). I'm less sure about this approach.
Any other thoughts or ideas out there? Should i post this to the public community?
Re: Strategies for removing unimportant terms in Text Explorer Analysistry making stop words out of all the terms that have more occurrences than there are documents. When I did that for the project Gutenberg books, it made the languages separate out better because the English boilerplate text was gone. At least that's how I interpreted the results.
Re: Strategies for removing unimportant terms in Text Explorer AnalysisThere are many guiding principles and rules but there are many exceptions. The rules only get you so far. A meaningful analysis requires a domain expert to spend considerable time examining the term list (and later the phrase list) to discover meaningless terms and stop them (ala Craige). Yes, I mean curating the list one term at a time.
So this analysis is not a fully automated process. Tokenizing, terming, and phrasing parameters help but they are a sledghammer, not a scalpel. All the experts that I talked to while writing the course for JMP told me that the analyst must spend this time on the list. They were using SAS Text Miner, a much more powerful analysis system than JMP, and still could not avoid this work.
Text Explorer provides wonderful tools for the analyst. Tools, not a solution.
I should add that it really helps to define questions to be answered by the analysis. This set of questions will serve you well when you ask, "Should I keep this term?" or "How would I use this term?"
Re: Strategies for removing unimportant terms in Text Explorer AnalysisAdding support to @markbailey. Aside from ditching common connecting words, etc. as @Craige_Hales suggests, this an extremely manual process. A good book to get the broad strokes that Mark mentions is Miner's Practical Text Mining - but it's a lot of trial and error any way you slice it.
MRe: Strategies for removing unimportant terms in Text Explorer AnalysisGreat suggestions! I am specifically looking for something after doing all the term and phrase curation - something that is an analytic technique beyond "I think this word might be important". I love Craige's suggestion!
We have ANALYTICAL techniques for finding top terms, so what about bottom terms?
Re: Strategies for removing unimportant terms in Text Explorer AnalysisHi Laura,
One of the things I remembered from way back was the suggestion to generate the SVD Scatterplot Matrix and to use that plot as tool to help find stop words from both the term and pharse list. Select some phrases and it will also highlight the terms. You can then "Show Text" to see whether or not you want to keep the terms/phrases or add them as stop words.
Best,
Bill
Re: Strategies for removing unimportant terms in Text Explorer AnalysisIf the goal is to screen out the importatnt terms that are relevant to an outcome variable, you might just run the GenReg for variable selection directly with the DTM columns. Since it is a large regression it will take a while to finish. For the example of Toronto Casino data, I ran the Lasso with 1,787 binary DTM columns. It took about 20 minutes to complete. I found 483 active (non-zero) terms of them 203 are statistically significant (p-value<0.05).
A more visual approach is to use the Coloring/By Column Value option in the Word Cloud. Coloring the words by the outcome variable let you see which words are associated with outcomes. The ones with mixed color are not or less associated with the outcome.
