Sample size requirment for machine learning

Question

Does JMP  have sample size requirment for machine learning  method?

JMP have many machine learning method to fit suitable model, how many samples should we get if we want to fit model by machine learning method?
Or is there a rule for the value that samples size divide by factors size? 
Such as sample size at least 10 times categorical data type factor size.

Thank you for any suggestion.

Phil_Kay · Accepted Answer

Hi Eric,

This is another one where I don't think it is possible to give a definitive answer.

First of all, not everyone agrees on the same definition of "machine learning".

For me, an important part of machine learning is holdback validation. I tell users that you probably need at least 100 rows to use holdback validation.

You could probably use holdback with fewer rows, if you only had 2 or 3 predictor variables. But with such a small number of predictors, machine learning is probably not needed.

I don' think that you can easily say how many rows based on the number of predictors. Because the required number of rows will also be a function of the complexity of behaviors in the system, which you don't know until you fit models.

Of course there are other validation methods that you can use to apply a machine learning approach with fewer rows (k-fold, SVEM).

It might be useful to take a look at this demo case study. The full dataset has 4608 rows. There are only 5 predictors but they are multi-level categorical, so it is quite a complex system. You can take random samples to see how well machine learning methods compare with traditional statistical modelling. I have not carried out rigorous studies but I was surprised to find that the ML seems to do better than standard models with quite small samples (~150 rows).

I hope that helps,

Phil

Mark_Bailey · Accepted Answer

I concur with @Phil_Kay. Remember that ML and AI methods are intended for predictive models, not explanatory models. Inference is not a goal, so sample size is not a consideration. Also, practically all cases involve Big Data, so inference is also moot. Use the metrics developed to compare the performance of prediction with cross-validation. For example, AICc is probably the best in more than one sense.

malcolm_moore1 · Accepted Answer

Of course I agree with what Phil and Mark say. One additional thought, if by Machine Learning your customer actually means a multiple regression model (with or without higher order terms) then they could use the evaluate design option from the DOE menu which will give power estimates for the model terms providing they can indicate the true error variation. Otherwise follow the guidance from Phil and Mark.