Hi everyone, I’m currently working with a Design of Experiments (DOE) created in JMP, and I’m interested in evaluating the quality of the randomized run order. Although JMP automatically randomizes the run sequence during design generation, I would like to objectively assess how effective that randomization is. In particular, I’d like to know whether JMP provides built‑in functionality or established workflows to: * detect trends, systematic patterns, or drift across the run sequence, * identify clustering or non‑uniform distribution of factor levels, * perform formal statistical tests of randomness (e.g., runs tests, autocorrelation or independence tests), * or otherwise quantify how closely the run order approximates true randomization. If JMP does not offer a direct method for this type of assessment, are there recommended JSL scripts, add‑ins, or best‑practice approaches for evaluating randomness in DOE execution order? Additionally, I am interested in whether JMP provides tools to verify that the integrity and statistical properties of the experimental design are preserved after manually reordering the runs. Thank you in advance for your insights. BayesKnight

nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Verifying Random Run Order and Design Validity in JMP DOE

BayesKnight

Hi everyone,

I’m currently working with a Design of Experiments (DOE) created in JMP, and I’m interested in evaluating the quality of the randomized run order. Although JMP automatically randomizes the run sequence during design generation, I would like to objectively assess how effective that randomization is.

In particular, I’d like to know whether JMP provides built‑in functionality or established workflows to:

detect trends, systematic patterns, or drift across the run sequence,
identify clustering or non‑uniform distribution of factor levels,
perform formal statistical tests of randomness (e.g., runs tests, autocorrelation or independence tests),
or otherwise quantify how closely the run order approximates true randomization.

If JMP does not offer a direct method for this type of assessment, are there recommended JSL scripts, add‑ins, or best‑practice approaches for evaluating randomness in DOE execution order?

Additionally, I am interested in whether JMP provides tools to verify that the integrity and statistical properties of the experimental design are preserved after manually reordering the runs.

Thank you in advance for your insights.

BayesKnight

Accepted answers

All comments

Victor_G

Hi @BayesKnight,

Welcome in the Community !

I see at least two options that can help you detect response trends due to a possible lack of randomization or due to a time sensitivity in the options of Model Fit:

Graphical option: Plot Residual by row: This plot can help you detect patterns that result from the row ordering of the observations.
Statistical testing option: Durbin-Watson Test: Statistic to test whether the residuals have first-order autocorrelation. Only appropriate if the rows are in time order (experiments done in the same order as in the data table).

You might have also other graphical or statistical options looking at Control charts (assess whether the repartition of factors levels follow a trend or not thanks to control charts and Westgard rules ) and Time Series analysis options. You can also check through Hierarchical clustering that the clustering of points is not imbalanced or creates ordered groups of rows. Finally, you can also check if there are significant correlations between factors and row order using the platform Multivariate.

The evaluation of design through the platform Evaluate Design does not take into consideration the ordering of the rows in the different evaluations. If you suspect a possible Time-Trend when running your design and would like to make the repartition of factors levels robust to any time trend, you can check the following posts:

https://community.jmp.com/t5/R-D-Blog/How-to-create-an-experiment-design-that-is-robust-to-a-linear/ba-p/30138

Covariates in defined order in custom design

Hope this answer may help you,

BayesKnight

Hi @Victor_G,

Thank you very much for the warm welcome and for the information provided. I would like to clarify that my primary interest is in assessing the quality of the randomized run order before the experiment is executed. Once responses are collected, it becomes too late to address potential issues caused by non‑random sequencing. Is there a way to evaluate it?

Dan_Obermiller

From your original post, you wanted to "quantify how closely the run order approximates true randomization".

You tell me what true randomization looks like and I could probably find a way to quantify how close I am to it.

Ultimately, randomization of the DOE runs is like an insurance policy against unknown/unexpected error sources that could be related to time or the order that the experiments are conducted. Those error sources, if present, would manifest themselves in the response data that is collected which is why @Victor_G provided that insight.

As long as a reputable random number generator is used (which JMP does have reputable random number generators), any random pattern is typically appropriate. Specific situations may indicate that it is NOT, but you would need to know those specific situations.

For example, suppose a piece of equipment will always make a mistake on the fourth item that is produced. In that situation, a valid random pattern just might put one of the factors at the high setting every fourth time. For an 8 run design, that would be very plausible. But that would certainly influence the results. You would not know that until conducting the analysis and then, most importantly, when VERIFYING the results. I have seen something similar to this happen in the real world in spite of a "valid" random pattern.

Finally, the statistical analysis and properties that you ask about assume that the error terms from the model are independent and identically distributed. Manually reordering the runs of a DOE should not affect that. In fact, you do not even need to randomize, if you know that each run is truly independent. Most people randomize to act as an insurance policy against those unknown error sources (as mentioned above). If manually moving design runs affects the independence, then your design (and analysis) should take those time features into account.

I hope this information helps. And remember, all experiments should be verified and even the sequence 1, 2, 3, 4, 5 is a possible random pattern!

statman

I offer a different perspective. There are plenty of "techniques" to evaluate the effectiveness of the experimental strategy, post experiment. I suggest spending more time on planning. I believe randomization in experimentation is a techniques to prevent some unidentified, untested factor (I'll call this noise) from being confounded with a factor in the experiment. I think we can be more effective. So where does that noise effect go? Do you want to know about that noise effect? Using randomization prevents assignment (Shewhart). I want to know about the noise. What noise is significant? How do I introduce noise into the experiment? How to be robust to the noise. I agree with G.E.P. Box:

"Block what you can, randomize what you cannot".

By this he means if you can identify the noise, confounding the noise with the block is more effective than randomization. It is assignable. For noise you can't identify (not sure why you can't), randomize.

I suggest a couple of papers:

Youden, W.J., Randomization and Experimentation, Technometrics, Vol. 14, No. 1, February 1972

Hwan, Marilyn (2000), “To Randomize or Not to Randomize, That is the Question”, ASQ Statistics Division Newsletter, Vol. 17, No. 1, 26

SDF1

Hi @BayesKnight ,

In addition to what has already been well-stated by others, I would offer a couple other thoughts to consider.

1. How do you plan to quantify the DOE order BEFORE you run the experiment? If the runs depend on the sate of something (sample state, equipment, or oven temp, or something that you aren't taking into account in the DOE), then this will show up in the data, but only AFTER you do all the experimental runs, and can be evaluated as previously discussed.

2. If you really want to test out the randomness of a DOE before, you could generate hundreds or thousands of DOEs and if you have runs 1 through 20, you can gather the statistics on how frequently each unique run is picked. If you generate enough DOEs, you shouldn't see any pattern in the selection process. But then, which DOE do you choose? Which one is the most random? You could set up a JSL script to do this and it shouldn't take very long. You should of course be very careful about how you define your DOE so that you don't have any hidden factors that can mess it up. Even if you did this analysis, it still would never be able to account for any hidden variable that does influence the run order.

3. After the DOE is done, there's several ways to evaluate the randomness of the runs, as mentioned already (the residual by row is a great place to start). I would also point out, you could add a column that is "run order" and include that in the analysis to determine if there is any statistically important dependence of the results on the run order. You could even include a "null factor" (read up on autovalidation in JMP -- there is a JSL code available, but it is not a built-in feature) that is a completely random factor, and any factor that comes in as statistically less than or equally important than this null factor can be ignored.

Pre-planning is key, and often ends up constituting a larger percentage of your discussion time. A well planned out DOE will be less time consuming to analyze than a poorly planned DOE, and you will gain more insights faster as well. In this regard, consider things like: Is there reason to think that order matters, and if "yes", why; where is it coming from; and how can we account for it in our design? What other factors might affect the results and can we account for them? Where does the noise in our signal come from? How can we minimize the noise and/or take it into account in the DOE?

Hope this helps!,

BayesKnight

Hi @Dan_Obermiller ,

Thank you for your detailed response.

You're absolutely right. There is no such thing as a “perfect” or “true” randomization. Still, some randomizations can be better than others in the sense that they avoid undesirable patterns (for example long runs of identical factor levels, periodic structure, etc.).

That is why I was hoping to explore whether JMP offers tools to evaluate these aspects objectively, rather than relying purely on trust in the random number generator or on visual inspection.

I’m looking for additional ways to evaluate the quality of the run order, particularly in situations where subtle time‑related risks or equipment behaviors may influence results.

Thanks again for your insights. They are very helpful.

BayesKnight

Hi @SDF1

Thanks for the reply.

I completely agree that the Define/Plan phase of a DOE is critical, that is precisely why I raised this question. I want to start with the strongest possible design and ensure the randomized run order is robust before execution.

I recognize there may be hidden variables we cannot foresee, but my main focus is:

how to generate a robust randomized run order
how to evaluate its quality before running the experiment.

Thanks again for the perspective.

MRB3855

Hi @BayesKnight : I've been following this with some interest...and, if I'm honest, some confusion. It seems to me that what you are asking is; is the "random" run order (generated by JMP) truly random? As @Dan_Obermiller said, "As long as a reputable random number generator is used (which JMP does have reputable random number generators), any random pattern is typically appropriate." Of course, even "reputable" random number generators are largely pseudorandom number generators. And, as such, they typically use an algorithm to generate a sequence of random numbers that have the properties of a random sequence. So, the answer is "yes".

One simple way to randomize the run order would be to generate a Uniform(0,1) number for each run. Then reorder the runs by that U(0,1).

My question to you though is this; what prompted your question?

BayesKnight

Hi @statman

Thank you for your response and for sharing the reference papers.

I agree that there are many ways to evaluate a design, but as you mentioned, most of them are used after the experiment is finished.

My goal is to avoid finding problems only at the end, when it is too late to correct them.

I am not questioning the purpose of randomization or blocking. I just want to detect any possible issues as early as I can. In particular, I want to check beforehand that the random order does not accidentally introduce patterns or trends that could weaken the design.

Thanks again for your helpful perspective.

SDF1

Hi @BayesKnight ,

JMP will generate a robust randomized run order for your DOE, you don't have to worry (or even shouldn't worry) about that. JMP has been an industry leader in this field for more than 35 years. They certainly wouldn't still be around if their software was not reliable.

Even if you did create some way to quantify and evaluate the run order BEFORE you did the DOE, all you'd really end up doing is testing JMP's randomization algorithm to determine if it's "truly" random. As @MRB3855 mentioned, no (digital) random number generator is truly random. Take your favorite music app and test out their "random" play function. In reality, their "random" play is restricted because they prioritize playing each song in the playlist once before "re-randomizing" the list. That's not a truly random order.

I also agree with @MRB3855 , what has prompted you to bring up this question of evaluating the randomness BEFORE even running the DOE in the first place? Why do you think that JMP doesn't provide a robust random run order?

If you have any historical data that suggests this is the case, then I'd first spend more time on planning the DOE and use blocking or some other form to account for it. I would not double-guess JMP and make the assumption something is wrong on their end -- it's much more likely the DOE wasn't planned out right in the first place.

rcast15

Hi @BayesKnight. I agree with @MRB3855 that this is quite an interesting thread to follow. There are a ton of good well thought out answers here that offer different perspectives. It does seem that in response to every answer you are still not satisfied and as pointed out it seems that you just want to know if there is a way to evaluate the "quality of the randomness." I do believe that random is random is random, and sometimes to the naked eye if we see patterns we might not think something is random even when it is.

I am not aware of any accepted statistical technique that exists to evaluate the quality of randomness aside from just looking at the order of your factor levels and seeing if they are "jumbled up enough." Maybe the answer is to do just that and to keep randomizing until you get an order that looks most jumbled? One could argue (I probably would as a statistician) that this is biasing the run order and potentially removing randomness.

One silly solution could be to plot your factors on the y-axis, your run order on the x-axis and use a heatmap with factor level (-1, 0, +1) as color. I guess this could help you visually assess if there are any patterns.