How many DFs are required to test X number of main effects - JMP Fit Model

travis_alexande

I have been asked to analyze multiple main effects and their various interactions, but don't think I can do so as I achieve Lost DFs error messages when I use the Fit Model analysis. In case it helps, here the details of the data. Main effects (size): Location (4), Cultivar (4), Region (2, 2 Locations grouped per region), and Year (3). I only get results/appear to have enough DF when I run any two main effects maximum, but never with an interaction.

Question 1) Do I have a degree of freedom issue in terms of not being able to analyze more than two main effects, hoping to look at all four?

Question 2) If I was to run only two main effects, it is correct to rerun the model when a main effect is not found to be significant? For example, I ran Year Location and found neither to be significant, so then I reran the the model removing Year as I am interested in Location and again found Location to be non-significant, but with a lower p-value.

Appreciate any advice

martindemel

Hi travis.alexander0!

all I answer below is based on having no data to take a short look at it. I say this as there are different possibilities why you may or may not find more main effects to be significant.

So first of all, I guess you are using Standard least squares as modelling algorithm and the model resulting effect tests, rigth? If so, there are mainly two reasons why you will have a main effect shown as not significant. Either it is not significant, means it has no significant effect on the Response, or there are colinearities and e.g. Location (which is not significant in all three models is e.g. in correlation with either Year or Cultivar.

Again, without having the full Setup for the data I just can guess. But that's what I would do first before modelling:

1. Open the Distribution Plattform and put all factors and Response variables as Y. Then click on any histogram bar e.g. of Location and see where the selected Distribution lays and how it differs from the Overall Distribution for all of the other variables. Do you see any differences in the distributions of the selected and Overall distributions? Or Patterns ,like one Location  has lower values than the other Location and so on. This will allow you in a very easy way to see if there is any suspicious going on here which could result in significant behaviour (even if it is not statistically proven yet).

2. Take a look at the Fit Y by X. I guess Location, Cultivar and Regional are categorical, may also year. Use your Response as Y and the others as X. You will get an Anova for the categorical X (in case your Y is continuous, otherwise contingency Analysis) Either way you will see if there is a significant effect of one X on one Y. Thsi can help to see if there may be something significant what is not significant in your model. Then it is likely due to some interaction effect.

3. Take a look at Fit Model. Fit an RSM model, so all interaction and quadratic effects for all of your X variables. As they probably are all categorical you will not Need the quadratic effects and may get a warning. Not do not run a Standard least squares but stepwise regression. Use p-value threshold as stopping criterion, and leave the rest as Default. Now check what effects are significant. Do they differ from your previous model?

Make the model with the selected effects only and run it again with Standard least squares. Do you still get a DoF issue?

In Addition to the above you may look at the lack of fit test in your models, usually in the Report below variance Analysis. Is the F-probability significant? If so it tells you that the model is missing an efffect. So either there is an interaction missing or some other variable you have not used or captured. But the model will not reflect your data good enough. You could take a look at the actual by predicted plot, the residuals plots as well on the profiler. to get a better understanding of the behaviour of your factors based on the model.

So using visuals from the Graph Builder/Distribution or other platforms will provide you some General understanding on the data, the modelling plattforms will help you to find a good model but can only work on what you provide it. There are sophisticated models which could do that work for you based on the algorithm, and if you have JMP Pro you could use them (e.g. bootstrap forest, generalized Regression ...).

Another aspect I didn't ask so far is about your data. How many observations (/rows) do you have? Are there missing data in it? do you have (potential) outliers. Is the data from a designed Experiment or from a measurement or gathered sales data?

All this could lead to a slightly different Approach I would recommend as each of them have other challenges to overcome and to take into account,

Hope this helps to make the next step,

Martin