nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

best model

maryam_nourmand

Hello
my question is how can i find best parametric model that fits very well on my dataset?
i want find a parametric model that can predict good my response

Accepted answers

All comments

Victor_G

Hello @maryam_nourmand,

Your question concerns a broad topic, and there may be (a lot) of questions to adress and answer before answering this question:

How is the data collected ? Observational study, experimental design, ... ? Representativeness and completeness of the dataset ? An Exploratory Data Analysis may be helpful to detect some patterns and possible pitfalls regarding the assumptions in regression models, like multicollinearity which may require adapted model like PLS or pre-processing steps like PCA.
Objective of the model(s) ? Causal explanations, prediction & optimization, or both (also linked to the available dataset and collection method) ?
Validation strategy/feature selection ? How to ensure the model(s) created has the right level of complexity and still has good predictive performance for example ? Do you assess the model performances and robustness through a "standard" Machine Learning validation strategy (with cross-validation or train/validation/test splits), or through a "statistically-oriented" approach, based on likelihood, information criteria (AICc, BIC), p-values ... ? Note that the model complexity should also be directly limited by the data collection : if you have factors with 3 different levels for example, you won't be able to fit higher terms than 2nd order terms.
Evaluation/selection metrics and threshold ? How do you evaluate the models ? What is the selection process/criterion : do you select the ones with the best predictive results with the selected metric, or do you select all models which have a better performance than a benchmark model or a naive model, ... ? How do you finally test the model ?

Some of these questions and answers were already described in previous posts :

https://community.jmp.com/t5/Discussions/Statistical-Significance/m-p/765928/highlight/true#M94573

https://community.jmp.com/t5/Discussions/Analysis-of-split-plot-design-with-full-factorial-vs-RSM/m-p/770579/highlight/true#M95183

Creating, comparing and selecting model(s) require evaluation metrics linked to your objective and thresholds/citeria to select one or several models. If you simply want the best predictive model, you could :

Create a model with a standard ML validation strategy (cross-validation for example) or a strategy able to control the model's complexity,
Use one or several metrics linked to predictive accuracy, like RMSE, MAPE, ...
Compare models based on the metric(s) and domain expertise : which one(s) is/are the most appropriate/relevant for your topic and which ones have the best performances,
Choose to estimate individual predictions with the selected model(s) to see how/where they differ, and/or to use a combined model to average out the prediction errors.
Test the model in "real" situation/production environment.

You might be interested by these ressources as well :

Which Model When?

Specifying and Fitting Models

Building and Understanding Predictive Models

Building Predictive Models for Correlated and High Dimensional Data

Building Better Predictive Models Using JMP Pro - Model Screening (it might help screening parametric and Machine Learning models options and compare them simultaneously)

Predictive Modeling

STIPS Module 7: Predictive Modeling and Text Mining

What Model When? (and Which Modeling Type?) - (2023-US-PO-1509)

Choosing Models in JMP with Model Selection Criteria - (2023-US-30MP-1456)

Data Mining and Predictive Modeling

I hope this first discussion starter will help you,

maryam_nourmand

If I want to explain my goal more precisely:

I have an initial dataset related to cancer patient data. I want to find a suitable parametric statistical model that best fits my dataset. Using this model, I aim to simulate data in order to apply a shift in the model's intercept. Ultimately, I want to see how quickly my pre-existing machine learning model can detect this shift in a control chart (i.e., obtaining the ARL). For this purpose, I need the best-fitting model on my data so that I can simulate data from it.

Victor_G

Hi @maryam_nourmand,

Ok, then you'll be more likely in a Data Mining approach, with a fixed dataset of observational data where you try to fit an acceptable predictive model. Using validation columns (with stratification on your features) will help validate (to avoid overfitting) and test your model on production data. In terms of modeling strategy, you'll very likely use stepwise approaches to select features, and Generalized Regression approaches with validation column method.

As an example, I used the Cancer_Data dataset from Kaggle to predict if there is a benign or malign cancer based on individual characteristics : https://www.kaggle.com/datasets/erdemtaha/cancer-data?resource=download

After a first exploratory data analysis focussed mainly on distributions and correlations between features, I created a validation column formula (stratification on the features) with 70/20/10 ratios for training/validation/test sets, and then used a Generalized regression model with validation column method, and all main effects features and 2-features interactions terms entered as possible terms in the model. I choose an adaptative Elastic Net as features are strongly correlated, but there might be other options as well, like PLS or PCA pre-processing.

You can then save the formula of this model, and use the Prediction Profiler to create simulations, to assess impact of features effects on the response, and possibly estimate the effect of increasing noise on the predicted response.

I hope these few options may help you for your topic,

Cancer_Data.jmp

dlehman1

Your response triggers me to ask: what do you mean by a "parametric statistical model?" I usually think of machine learning models as non-parametric, so are you excluding such models here. It is unclear since you say you have a pre-existing machine learning model. Are you wanting to compare a parametric and non-parametric model? If you use the model screening platform, you can build a number of both types of predictive models. Using whatever you find to be the "best fitting," you can then save the prediction formula in order to do simulations. I'm not entirely sure what you mean by a "shift in the model's intercept" but I think you could just put an additive disturbance into the formula to generate the simulated data.

maryam_nourmand

Thank you for your response.

But I think for simulation, it is better to go to the section `save columns->save simulation formula` because it contains a formula for simulation that I can use for writing code as well, right?

And thank you for introducing the dataset.

But do you have a dataset related to a treatment process that includes multiple stages? I mean, after the first treatment, the disease relapses, and the second treatment is performed, and the treatment information such as drug dose, etc., is recorded. If you have such a dataset, I would appreciate it if you could share it.

maryam_nourmand

Yes, by "parametric model" I do not mean machine learning models.

My goal in finding the best parametric model is to simulate and generate data with a larger quantity than my initial dataset. I want to use more data for my machine learning model, but I want the simulated data to closely resemble and be similar to my initial dataset.