221693_en-US

liliconcom

LCA in Clustering vs. LCA in Text Explorer

For users who do not have JMP Pro, can they save the DTM and use the LCA under the Clustering platforms to group similar documents in the same way they use LCA in the Text Explorer platform?  Are there any additional considerations they need to think about?

 

There is a note in the documentation (screenshot below) that says the results do not exactly match but I get fairly different results even when I set the random seed (see last 2 scripts in data set attached).  Can someone explain further what is meant by “specific structure of the document term matrix”? and whether or not you have tried running the same data through both platforms?

 

 

 

Thanks,

Wendy

@brady_brady @phersh 

Re: LCA in Clustering vs. LCA in Text ExplorerSomeone asked this when Text Explorer was first launched. I believe the response was that LCA in Text Explorer is optimized for working with the DTM, while the LCA platform is a more general tool. The guidance was that when working with text we should use the LCA in Text Explorer.

Best,

M
Re: LCA in Clustering vs. LCA in Text Explorer

Thanks for the quick response, Mike!

I would love more detail as to what the differences are if you or others can provide it.  Why are the # of clusters different and why would one platform suggest a different likely cluster for some documents than the other?

 

 

 

Re: LCA in Clustering vs. LCA in Text ExplorerI believe it was Chris Gotwald that made the comment… But from the documentation (Multivariate Methods Book, Page 322 – in the v15 docs):

“Note: The LCA algorithm that is used in the Text Explorer platform takes advantage of the specific structure of the document term matrix. For this reason, the LCA results in the Text Explorer platform do not exactly match the results in the Latent Class Analysis platform.”

Hope that helps.

Best,

M
Re: LCA in Clustering vs. LCA in Text Explorer

The DTM is a high dimension matrix and therefore very sparse. The size and sparsity pose problems for LCA platform. The TE platform uses a special SVD for the high dimension spare case.

 

You do not get exactly the same results from the two LCA implemenations but they are close.

 

Text data should be analyzed in TE in JMP Pro when possible.

 

I would try the LCA platform in JMP, but the DTM might pose too big a problem.

 

(What Mike said.)

Re: LCA in Clustering vs. LCA in Text Explorer

FYI: As a result of this conversation the documentation for JMP 15.1 will replace "specific structure" with "sparsity". Hopefully that's a little more descriptive.