Need help: Grouping Categorical values, finding optimal number of categories and exceptions

miguello

Folks, 

 

I have a problem that I'm hoping somebody would help me find a solution for, or at least nudge me in a right direction.

 

So, let's say we have N tools. They have multiple atributes, let's say 10. This might be exact model number, color, bells and whistles, modifications etc. 

They also have hundreds of parameters. Some might be the same across all the tools, some might depend on a combination of attributes or a single attribute.

And I need to manage all those many hundreds of parameters. These parameters might change and be different from what it should be and I know each parameter at any point of time.

Now, the approach is following: break all the tools into few big groups, let's call them categories, that are defined by combinations of attributes. Let's say all tools of model A and blue color are one category, Model A red color are second category and Model B is third category, regardless of the color.

Now, not all the parameters are going to be the same within one category. Some specific parameter might be different between Model A red color green bell and Model A red color yellow bell. So we find the most common combination and call it "default" for that category. All those that are different we need to explain and put some kind of comment for them that explains why the parameter is different.

 

Eventually, we will have two unknown numbers: number of categories we need to  break it into and number of exception comments we need to fill for parameters that do not match that of "default".

 

If we set one category - it's going to be easy to maintain one category, but we're going to have too many exceptions from the "default" to be manageable. If we create a category for almost each combination of attributes, we won't have to manage too many exceptions from "default", but we'll have trouble managing that many categories.

The problem is based on the set of data, find the optimal number of categories, where it both has reasonable amount of categories with reasonable amount of exceptions.

 

Any ideas on how to approach solving this problem?

 

Thanks, 

M.

gzmorgan0

miguello,

 

This is reminiscent of a profile analysis or K Nearest Neighbor. Create a table where each row is an entity and other than the entity column, there is a column for each attribute. I suggest there be a code for the attribute, like 1=red and 2=green, etc., then according to your hypothetical example, the column for bell will have these values.  I suggest you specify labels for the values.

 

Then you can define one or more metric and weights.  For example, it might be most important that certain specifications must match: group A must be high capacity sprayer (> some cubic units) and group C low capacity(< some cubic units) and group B is other. To refine or subdefine the groups you might use:

• Latent Class Analysis - note the categories do not need to be coded, but coding can simplify other analyses.
• a Multiple Correspondence Analysis, which would show likely attribute pairings
• k nearest neighbor methods, see Scripting Index Functions: KDE (statistical) and KDTable (matrix).

That said, often groups are defined over time by their results. Since it is getting close to midterm U.S. elections, I'll use it as an example. Predictions are often made on classes. These classes are defined/refined by age, sex, education, city vs. rural, which state, religion, church attendance, # of times voted in a midterm, in other words a mix of social and spatial demographics but also behavior (# times voted, church attendance).

 

Regarding the entity mismatch, that will need a little extra care. Suppose group A entities are closely matched for configuration, except older entities do not have 3 in situ sensors and the newer entities have the performance read-out from the sensors. Mismatch will need to be calculated by available data. Continuing the scenario, suppose there are 100 metrics for all group A tools but the newer ones have the 3 extra, or 103. Many JMP platforms easily handle missing values, but how you handle this depends upon which analysis/criteria you might use to identify a mismatch. 

 

I like the JMP virtual join.  If I were doing this analysis, I might create my table of enity attributes to define the categories and then virtually join it with the table of other variables. This also makes it easy if the other variables have multiple values (rows) per run.

 

These are just crumbs. Good luck with this interesting task.