How do I conduct a global sensitivity analysis in JMP?

Question

Hello. First post to the forum for me... I am a research ecologist with a wildlife conservation emphasis. Not the typical JMP user, I'm guessing. I have developed and tested a complex individual-based simulation model of bird population dynamics. I am currently at the point in model analysis, where a global, variance-based sensitivity analysis would be appropriate. I am wondering if JMP has the tools to complete such an analysis.

While I have used JMP for years for basic statistical analyses and graphical exploration of data, I am new to using the different profiler tools and the Design of Experiments platform. It appears to me that these tools were developed to meet needs in very different disciplines (e.g., Six Sigma, clinical trails in medicine). Consequently, some of the terminology is quite foreign to me compared to standards from my field (see attached paper by Cariboni et al. (2007). In ecological modeling, sensitivity analysis is often described as the process of systematically varying parameter values across a model's parameter space to understanding the degree to which different parameters affect key model outcomes and identify parameters that are most influential.

I can clearly see elements of this type of analysis in the profilers and DOE platforms, but I haven't been able to figure out which platform/tools I might use for the type of sensitivity analysis typically applied in ecological modeling (see additional attachments). Specifically, how might I obtain the type of parameter sensitivity outputs that are typical of sensitivity analyses?

I've attached 4 papers to this post to illustrate what I am talking about. I am hoping that I am just working in a different analysis tradition than most JMP users and that existing tools can meet my needs. In particular, I'd love to be able to perform the 2-step analysis process of the attached paper by Song et al. (2012), that first applies a screening design (Campologo et al. 2007) to limit the number of parameters that would be included in a global, variance-based sensitivity analysis (Saltelli et al. 2010).

I'd be grateful for help that anyone can provide to help bridge the distance between sensitivity analyses in ecological modeling and DOE/profiler type analyses that can be performed in JMP.

Sincerely,

Casey Lott

American Bird Conservancy

(208) 629-8705

Cariboni et al 2007.pdf

song_et_al_2012_Ecol_Mod_3PG.pdf

Campolongo et al 2007.pdf

saltelli-et-al-2010.pdf

clott · Answer

Thank you, @Ian. I am hoping that this question generates some discussion. After re-reading the papers I attached to my message, I think I could probably state my interest and ask a few questions more specifically:

First, I'd like to take the two step approach toward sensitivity analysis that is illustrated in the Song et al. paper that is attached to my original post. This means that I'd like to start by using the modified Morris method that is published in Campolongo et al. 2007 to calculate the Elementary Effects (EE) Sensitivity Index with the objective of "factor fixing", that is, finding which parameters have a minimal effect on model outputs so that they can either be: 1) removed from the model via model simplification or 2) fixed at a value anywhere in their range and ignored in subsequent analyses. Can anyone point me to the feature in JMP that I might use to conduct this kind of analysis? If this index (the EE sensitivity index) can be calculated in JMP, I'd like to be able to look up the statistical details to see if it is being calculated following Morris 1991, or the updated methods presented in Campolongo et al. 2007, or some other set of methods where I can reference an original paper AND a specific page number or section of JMP's documentation. In particular, I'd be interested in calculating the EE sensitivity index for both single factors and groups of factors. If JMP does something similar, but uses a different method, I'd love some help finding my way to it. Again, I'm very new to the platforms/tools in JMP that may do this kind of thing.

Second, I'd like to run a more detailed variance-based sensitivity analysis, using the Main Effect and Total Effect Sensitivity Indices that have been presented in many papers by Saltelli and colleagues and most recently updated, as far as I can tell, in Saltelli et al. 2010 (which I also attached to my original post). I looked at the link Ian sent for Assess Variable Importance and then went to the statistical details link inside that text. It appears that JMP is using some form of these indices, since this text references Saltelli 2002. I'd be curious if any of the existing JMP platforms/tools follow the updated methods that are described in detail in Saltelli et al. 2010. Again, if anyone can help me navigate JMPs documentation to figure this out, I'd be grateful.

Ultimately, I'd like to be able to perform a sensitivity analysis for a peer reviewed paper and cite the appropriate original papers for statistical details (e.g., Campolongo et al. 2007, Saltelli et al. 2010). I'm not sure if this is possible or if it would be more appropriate just to cite JMP's documentation for existing platforms/tools. If this is the case, I'd like to be sure that I'm using the right ones and that they are at least analogous, if not exactly the same, to the approaches outlined in the two papers listed above (and provided in my original post).

Thanks in advance for help that anyone can provide on this.

ian_jmp · Answer

Unfortunately, I've not had the time to do your question justice.

I think it's fair to say that JMP nicely supports a sequential approach to experimentation (whether or not the response has noise), even though the relatively recent Definitive Screening Designs disrupt this paradigm a little.

Scanning Song et al, I see they mention the Sobol method. In case it helps, I wanted to make you aware that Assess Variable Importance in JMP's Profiler uses this approach.