In the Profiler for constrained mixture DOEs, I often find that it isn't just the other factor settings that change when I change one factor's x-axis setting. The width of one or more of the x-axis potential factor ranges no longer goes from (for example) 0 to 1 as designed, but now from 0.2 to 0.3. In my opinion, this is generally caused by the severe constraints the customers made me design into the DOE when we set it up. Do you have any more tactful, clarifying, and/or truthful ways of answering the "Why does it do that?" question?
Most traditional DOEs are a balance of resources (time/money/runs) vs Power of the design. I am always trying to talk my engineers and chemists into just one or two more runs because it will increase the Power of the design. However, when I tried to make that argument with a mixture DOE, I discovered that increasing the number of runs did not change the Power. Or seemed to not change the Power. I just tried it in Custom Design with 4 Mixture factors unconstrained (Interactions 2, Scheffe Cubic) of 20 runs, 22 runs, and 30 runs. Unlike my memory, the increase in sample size does seem to slightly increase the main effect power, but not the Power for any second order terms. The change from 20 runs to 30 runs makes the main effect power go from 0.073 to 0.15 or 0.25. From a traditional DOE perspective, this does not seem like a huge improvement in Power, certainly not worth it for such a huge increase in runs (20 runs to 30 runs!). If not Power, then what should be the evaluative focus of mixture DOEs? If it is Power, should I reset my Power expectations to something much lower than I would for more traditional DOEs?
@diball , it's a great series of questions you are asking here - and I believe it can be consolidated well to:
"Can we meaningfully interpret the design diagnostics (Power, Uniformity, Orthogonality) like we can for traditional DOE's?"
"If not, then why not?" and lastly,
"If not, then what metrics (diagnostics) should we use?"
@wendytseng could you kindly point us in the right direction?
thanks!
-@PatrickGiuliano
If the predicted optimal formula from the mixture DoE resides on the boundary, will a follow up augment design with expanded design space around the optimal setting help? @wendytseng
A question about Power for Mixture Desgins came up at the July 17 session. @diball We found this JMP Systems Engineer's discussion about power:
Here was the question:
Using the factors from the attached table and the default 21 trial experiment, the power calculation is shown below. Is there a good explanation for why the power is so low?
Mixture designs have low power because the effects are highly correlated. More correlation == lower power. Ultimately, your interest with a mixture design is response estimation and not factor significance. Non significant factors (main effects - i.e., the mixture component) can't be removed from the model, otherwise you don't have a mixture design (i.e., it breaks the math). What would likely happen to a factor that has no effect on the response is that it gets set to 0. While this may appear to be the same thing from a practical standpoint, it isn't mathematically. You can use a similar argument for retaining non-significant when fitting non-mixture data with LASSO (or Elastic net or Ridge). The focus is on response estimation and throwing out non-significant factors may actually make estimation worse (i.e., inflate the prediction variance).
And @Dan_Obermiller added:
Power really does not matter in a mixture experiment because you are not testing to see if the main effect coefficient is 0. You actually would test to see if the main effect coefficients are different from the overall average of the data. That is why you would not set a coefficient to 0. It would actually be stating that the coefficient has an effect.
And @Mark_Bailey added:
For future reference... Mixtures are often constrained by limited proportions of individual components and limited combinations of proportions. These constraints dramatically increase the correlations of the estimates, which further exacerbates the low power. It can look really scary! But as @DonMcCormack and @Dan_Obermiller have explained, we really don't care about power anyway.
@PatrickGiuliano the diagnostic are interpreted the same way. That said, those involving power or parameter estimation are unimportant relative to prediction variance diagnostics. See what @gail_massari posted above regarding low power. @yanrongshi, given the mixture components are constrained to add up to a specific amount, moving outside of the design space is not possible (i.e., it would cause the amount to exceed the permitted total amount). The optimal must reside inside the mixture component design space. If you are on the boundary of a non mixture component, you can certainly move outside of design space of the non-mixture components.
@diball When one locks a mixture factor (let's say to 60%), the lengths of the other mixture factor profile ranges decrease to reflect the fact that they are now constrained collectively to 40% (1 - 60%). Here is a 3-minute video that I made in case it is helpful. (Let me know if I am not understanding your question of course.)
@diball Take a look at the comments by @DonMcCormack , @Mark_Bailey and @Dan_Obermiller on power. I think the punch line is: don't worry about power for mixture experiments because it is going to be low given the nature of them (factors are highly correlated).
Excellent discussion thank you @gail_massari and @wendytseng and @DonMcCormack for your contributions here.
First I like to Part 2 of this Mastering Series by @wendytseng for continuity in going back and referencing this material: Designing Mixture Experiments - Part 2 - JMP User Community
I am confused by @Dan_Obermiller's comment, specifically the parts that I have underlined:
"Power really does not matter in a mixture experiment because you are not testing to see if the main effect coefficient is 0. You actually would test to see if the main effect coefficients are different from the overall average of the data. That is why you would not set a coefficient to 0. It would actually be stating that the coefficient has an effect".
When we are testing Power, are we testing that the main effect coefficient is zero? My impression was that we were always testing whether or not the effect was significant (statistically) given a signal = 2 (minus 1 to + 1 in coded factor space when anticipated coefficient is set to 1) and noise = RMSE = 1 (by default).
How does this square up with testing to see if the main effect coefficient is zero?
Under what circumstances are we using statistical power to test whether the main effects are different from the overall average of the data?
@PatrickGiuliano, let me try to clarify my comments.
Power is the probability of finding a real effect of a certain size given the sample size, error, and confidence. In the traditional modeling, an effect would manifest as a nonzero coefficient. However, in the Scheffe mixture models this is not the case. For a 3 component case, the traditional model of Y = b0 + b1*X1 + b2*X2 + b3*X3 cannot be fit due to the constraint of X1+X2+X3 = 1. So, the model is reparameterized. Y = b0(X1+X2+X3) + b1*X1 + b2*X2 + b3*X3. Simplifying results in Y = (b0+b1)X1 + (b0+b2)X2 + (b0+b3)X3. So I would expect the Scheffe coefficient to be the overall average. If a term had "an effect", I would need to see a value different than the overall average, b0.
To see this, consider this simple example:
Clearly, none of these components have an effect. But when I fit the model, these are the parameter estimates:
The coefficients are clearly not zero.
To further illustrate, add one more overall centroid run with a response of 59 so that there is some error for statistical testing. Nothing should be significant, but look at the resulting summary of fit and parameter estimates:
Notice that the parameter estimates are still the overall mean of the response. JMP says they are significant because the test compares against 0. But these main effects should not be significant. None of these components have an effect.
Note that my comments only refer to the main effects of a mixture model. Testing on the cross products (more accurately, nonlinear blending terms) is appropriate.
I hope this helps.
Hello Dan_Obermiller This little toy example is definitely very helpful! Thanks for clarifying: Let me see if I can reiterate some points and please let me know if I am astray anywhere and I will correct them. I am doing this to reclarify for myself and provide a point of reference in the future for when I may return to this post, but also for other community members:
Given the inherent mixture constraint: X1+X2+X3 = 1, then X1 = X2 = X3 which = 1/3
If this the reparametrized mixture model for three factors: Y = b0(X1+X2+X3) + b1*X1 + b2*X2 + b3*X3,
then substituting 1/3 in for each of the X's, we get: Y = b0*(1/3 + 1/3 + 1/3) + (1/3)*(b1 + b2 + b3) = b0 + (1/3)*(b1 + b2 + b3)
So Y = b0 + average(b1,b2,b3)
b0 which is typically the intercept coefficient (aka parameter estimate) in this case represents the grand mean
"average(b1,b2,b3)" is the Sheffe coefficient (aka sheffe parameter estimate).
What we care about is whether the Sheffe coefficient is statistically significantly different than the grand mean, b0, NOT whether it is statistically significantly different from 0 (where the test of whether it is different from zero is what we are doing in the standard OLS hypothesis test on the regression slope - which is what is shown in the Parameter Estimates section). Since we care about whether the Sheffe coefficient is substantively different than the grand mean, and since the Power calculations in some way do not consider this grand mean (because they are in-effect looking at the test of whether the parameters are different from 0 given n, SE, Conf.), Power calculations are essentially meaningless for us.
@DonMcCormack Regarding the question of predicted optimal mixture component is on the boundary, for example, my first mixture DoE design have one of the constrains of 0.3<X1<0.7 based on prior knowledge. The results come out with a optimal setting at X1=0.7. A confirmation run at X1=0.7 shows good result. In this case, for further optimization, will an augment design moving the constrain to 0.6<X1<0.8 or somewhere around 0.7 help? Any other suggestions?
@yanrongshi it's hard to say whether changing the constraint would result in better results since you would both need to change to one or more other mixture factors and extrapolate outside of the current experimental region. Typically, constraints are set due to know or posited infeasibilities of mixture settings (e.g., X1 > 0.7 would lead to chemical instability). If you have process knowledge suggesting increasing X1 makes sense, then you could running a design that either moves outside of the current feasible region or overlaps with it. Rather than use JMP's Augment Design feature, I'd try designing a new experiment. This way, you don't have to worry about the previous constraints causing issues with the design generation. You could then combine the old data with the new and analyze everything together.
@DonMcCormack @Dan_Obermiller @Mark_Bailey (or others out there):
Can you comment on which design diagnostics are helpful in comparing 2 different mixture designs? Let's say I specify the same model but want to compare a 20-run mixture DOE to a 30-run mixture DOE, are there any diagnostics that can guide me in terms of determining trade-off between adding runs and what I can learn?
I believe that most, if not all mixture designs are focused on prediction. I like the Fraction of Design Space plot and the prediction variance profiler. It is not a single-number summary, but there is a bit of "art" in creating a good mixture design. You can look at power, but only on the nonlinear blending terms (i.e. interactions).
@Dan_Obermiller Thanks for the quick response. The Fraction of Design Space plot is intuitive (first screen shot below from Compare Designs) but the Average Variance of Prediction values from the individual design evaluation outputs (second screenshot below) are not. The 30-run design (on the right) has a lower value than the 40-run design (on the left). Is this something you can comment on?
Honestly, I can't explain it. I would not have expected the 40 run design to have a higher average variance of the prediction. And, according to the FDS plot, it shouldn't. I have run through this scenario several times and have never seen results like this.
I noticed that for the FDS plot, you used Compare Designs. The average variance of the prediction came from different instances of Custom Design. Did you use the same models for both designs? Did the Compare Designs platform use the same designs that you used for these individual Custom Designs? Are there any constraints that may have been added? I notice that everything looks like you had an unconstrained space with a model including 3-way interactions. But the last run of the 40-run design is rather strange for that situation. Could you provide more details on the experimental setup?
I had 4 mixture factors and I specified the same scheffe cubic model for both designs. No constraints. Only difference is one was 30-runs (default) and the other was 40-runs (user specified).
I also just noticed that Compare Designs also does not allow one to specify a scheffe cubic model (just a special cubic).
I don't see an option for uploading the JMP tables so below are the scripts to generate the 2 designs.
New Table( "Mixture Design 4 Factors, Scheffe Cubic, 40 runs", Add Rows( 40 ), New Table Variable( "Design", "Custom Design" ), New Table Variable( "Criterion", "D Optimal" ), New Script( "Model", Fit Model( Effects( :X1 & Mixture, :X2 & Mixture, :X3 & Mixture, :X4 & Mixture, :X1 * :X2, :X1 * :X3, :X1 * :X4, :X2 * :X3, :X2 * :X4, :X3 * :X4, :X1 * :X2 * :X3, :X1 * :X2 * :X4, :X1 * :X3 * :X4, :X2 * :X3 * :X4, Scheffe Cubic( X1, X2 ), Scheffe Cubic( X1, X3 ), Scheffe Cubic( X2, X3 ), Scheffe Cubic( X1, X4 ), Scheffe Cubic( X2, X4 ), Scheffe Cubic( X3, X4 ) ), Y( :Y ), No Intercept( 1 ) ) ), New Script( "Evaluate Design", DOE( Evaluate Design, X( :X1, :X2, :X3, :X4 ) ) ), New Script( "Generalized Regression", Fit Model( Y( :Y ), Effects( :X1 & Mixture, :X2 & Mixture, :X3 & Mixture, :X4 & Mixture, :X1 * :X2, :X1 * :X3, :X1 * :X4, :X2 * :X3, :X2 * :X4, :X3 * :X4, :X1 * :X2 * :X3, :X1 * :X2 * :X4, :X1 * :X3 * :X4, :X2 * :X3 * :X4, Scheffe Cubic( X1, X2 ), Scheffe Cubic( X1, X3 ), Scheffe Cubic( X2, X3 ), Scheffe Cubic( X1, X4 ), Scheffe Cubic( X2, X4 ), Scheffe Cubic( X3, X4 ) ), No Intercept( 1 ), Personality( Generalized Regression ), Generalized Distribution( Normal ), Run( Fit( Estimation Method( Forward Selection( Initial Displayed Solution( Biggest in Green Zone ) ) ), Validation Method( AICc ), Enforce Heredity ) ) ) ), New Script( "DOE Dialog", DOE( Custom Design, {Add Response( Maximize, "Y", ., ., . ), Add Factor( Mixture, 0, 1, "X1", 0 ), Add Factor( Mixture, 0, 1, "X2", 0 ), Add Factor( Mixture, 0, 1, "X3", 0 ), Add Factor( Mixture, 0, 1, "X4", 0 ), Set Random Seed( 1738999858 ), Number of Starts( 403 ), Add Term( {1, 1} ), Add Term( {2, 1} ), Add Term( {3, 1} ), Add Term( {4, 1} ), Add Term( {1, 1}, {2, 1} ), Add Term( {1, 1}, {3, 1} ), Add Term( {1, 1}, {4, 1} ), Add Term( {2, 1}, {3, 1} ), Add Term( {2, 1}, {4, 1} ), Add Term( {3, 1}, {4, 1} ), Add Term( {1, 1}, {2, 1}, {3, 1} ), Add Term( {1, 1}, {2, 1}, {4, 1} ), Add Term( {1, 1}, {3, 1}, {4, 1} ), Add Term( {2, 1}, {3, 1}, {4, 1} ), Add Term( {1, 1}, {2, 1} ), Add Term( {1, 1}, {3, 1} ), Add Term( {2, 1}, {3, 1} ), Add Term( {1, 1}, {4, 1} ), Add Term( {2, 1}, {4, 1} ), Add Term( {3, 1}, {4, 1} ), Set Sample Size( 40 ), Simulate Responses( 0 ), Save X Matrix( 0 ), Make Design, Set Run Order( Randomize ), Make Table} ) ), New Column( "X1", Numeric, "Continuous", Format( "Best", 12 ), Set Property( "Mixture", {0, 1, 1} ), Set Property( "Design Role", DesignRole( Mixture ) ), Set Property( "Factor Changes", Easy ), Set Values( [0, 0.3, 0.700000000000001, 0.730430906076223, 0, 0, 0, 0, 0.333333336572919, 0, 0.737656135098273, 0.3, 0.279256271431449, 0, 0, 0.333333331193037, 0, 0.333333341063914, 0.258949675746752, 0.700000000000004, 0, 0.240823992538088, 0, 0, 0, 0.7, 1, 0.700000000000008, 0.333333329569012, 0, 0, 0.3, 0, 0, 0, 1, 0.333333331288508, 0, 0, 0.333333336466388] ), Set Display Width( 100 ) ), New Column( "X2", Numeric, "Continuous", Format( "Best", 12 ), Set Property( "Mixture", {0, 1, 1} ), Set Property( "Design Role", DesignRole( Mixture ) ), Set Property( "Factor Changes", Easy ), Set Values( [0.75618876556416, 0, 0, 0.269569093923777, 0, 0.286447740062051, 0.705518040050005, 1, 0, 0.3, 0, 0, 0.720743728568551, 0, 0.333333330430823, 0.333333331193039, 0.723291462364164, 0.333333329468043, 0, 0.3, 1, 0, 0, 0.3, 0, 0, 0, 0, 0, 0.700000000000001, 0, 0.7, 0, 0.300000000000001, 0.33333333227439, 0, 0.333333337422987, 0, 0, 0.333333327067223] ), Set Display Width( 100 ) ), New Column( "X3", Numeric, "Continuous", Format( "Best", 12 ), Set Property( "Mixture", {0, 1, 1} ), Set Property( "Design Role", DesignRole( Mixture ) ), Set Property( "Factor Changes", Easy ), Set Values( [0, 0, 0, 0, 0.300000000000001, 0, 0, 0, 0.33333333171354, 0, 0.262343864901727, 0.700000000000002, 0, 0.273396724345484, 0.333333339138354, 0, 0.276708537635836, 0.333333329468045, 0, 0, 0, 0.759176007461917, 0.700000000000001, 0.7, 0, 0.300000000000001, 0, 0, 0.333333335215495, 0.300000000000001, 1, 0, 0, 0.7, 0.333333335451219, 0, 0, 0.700000000000001, 1, 0.333333336466391] ), Set Display Width( 100 ) ), New Column( "X4", Numeric, "Continuous", Format( "Best", 12 ), Set Property( "Mixture", {0, 1, 1} ), Set Property( "Design Role", DesignRole( Mixture ) ), Set Property( "Factor Changes", Easy ), Set Values( [0.243811234435841, 0.700000000000003, 0.3, 0, 0.7, 0.713552259937952, 0.294481959949996, 0, 0.33333333171354, 0.700000000000002, 0, 0, 0, 0.726603275654518, 0.333333330430823, 0.333333337613926, 0, 0, 0.741050324253249, 0, 0, 0, 0.3, 0, 1, 0, 0, 0.3, 0.333333335215494, 0, 0, 0, 1, 0, 0.333333332274392, 0, 0.333333331288506, 0.3, 0, 0] ), Set Display Width( 100 ) ), New Column( "Y", Numeric, "Continuous", Format( "Best", 12 ), Set Property( "Response Limits", {Goal( Maximize ), Lower( . ), Upper( . ), Importance( . )} ), Set Values( [., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., .] ), Set Display Width( 100 ) ) )
New Table( "Mixture Design 4 Factors, Scheffe Cubic, 30 runs", Add Rows( 30 ), New Table Variable( "Design", "Custom Design" ), New Table Variable( "Criterion", "D Optimal" ), New Script( "Model", Fit Model( Effects( :X1 & Mixture, :X2 & Mixture, :X3 & Mixture, :X4 & Mixture, :X1 * :X2, :X1 * :X3, :X1 * :X4, :X2 * :X3, :X2 * :X4, :X3 * :X4, :X1 * :X2 * :X3, :X1 * :X2 * :X4, :X1 * :X3 * :X4, :X2 * :X3 * :X4, Scheffe Cubic( X1, X2 ), Scheffe Cubic( X1, X3 ), Scheffe Cubic( X2, X3 ), Scheffe Cubic( X1, X4 ), Scheffe Cubic( X2, X4 ), Scheffe Cubic( X3, X4 ) ), Y( :Y ), No Intercept( 1 ) ) ), New Script( "Evaluate Design", DOE( Evaluate Design, X( :X1, :X2, :X3, :X4 ) ) ), New Script( "Generalized Regression", Fit Model( Y( :Y ), Effects( :X1 & Mixture, :X2 & Mixture, :X3 & Mixture, :X4 & Mixture, :X1 * :X2, :X1 * :X3, :X1 * :X4, :X2 * :X3, :X2 * :X4, :X3 * :X4, :X1 * :X2 * :X3, :X1 * :X2 * :X4, :X1 * :X3 * :X4, :X2 * :X3 * :X4, Scheffe Cubic( X1, X2 ), Scheffe Cubic( X1, X3 ), Scheffe Cubic( X2, X3 ), Scheffe Cubic( X1, X4 ), Scheffe Cubic( X2, X4 ), Scheffe Cubic( X3, X4 ) ), No Intercept( 1 ), Personality( Generalized Regression ), Generalized Distribution( Normal ), Run( Fit( Estimation Method( Forward Selection( Initial Displayed Solution( Biggest in Green Zone ) ) ), Validation Method( AICc ), Enforce Heredity ) ) ) ), New Script( "DOE Dialog", DOE( Custom Design, {Add Response( Maximize, "Y", ., ., . ), Add Factor( Mixture, 0, 1, "X1", 0 ), Add Factor( Mixture, 0, 1, "X2", 0 ), Add Factor( Mixture, 0, 1, "X3", 0 ), Add Factor( Mixture, 0, 1, "X4", 0 ), Set Random Seed( 2038461407 ), Number of Starts( 497 ), Add Term( {1, 1} ), Add Term( {2, 1} ), Add Term( {3, 1} ), Add Term( {4, 1} ), Add Term( {1, 1}, {2, 1} ), Add Term( {1, 1}, {3, 1} ), Add Term( {1, 1}, {4, 1} ), Add Term( {2, 1}, {3, 1} ), Add Term( {2, 1}, {4, 1} ), Add Term( {3, 1}, {4, 1} ), Add Term( {1, 1}, {2, 1}, {3, 1} ), Add Term( {1, 1}, {2, 1}, {4, 1} ), Add Term( {1, 1}, {3, 1}, {4, 1} ), Add Term( {2, 1}, {3, 1}, {4, 1} ), Add Term( {1, 1}, {2, 1} ), Add Term( {1, 1}, {3, 1} ), Add Term( {2, 1}, {3, 1} ), Add Term( {1, 1}, {4, 1} ), Add Term( {2, 1}, {4, 1} ), Add Term( {3, 1}, {4, 1} ), Set Sample Size( 30 ), Simulate Responses( 0 ), Save X Matrix( 0 ), Make Design} ) ), New Column( "X1", Numeric, "Continuous", Format( "Best", 12 ), Set Property( "Mixture", {0, 1, 1} ), Set Property( "Design Role", DesignRole( Mixture ) ), Set Property( "Factor Changes", Easy ), Set Values( [0, 0, 0, 0.7434800714054, 0.333333330708985, 0, 0, 0, 0.249749752574731, 0, 0, 0.3, 0, 0.333333332375124, 0.333451117338599, 0.333239324710344, 0, 0.333333334942452, 0, 0.741048403090733, 0.7, 0.7, 0, 0.731693360893765, 0, 0, 1, 0, 0.293144492690121, 0.3] ), Set Display Width( 100 ) ), New Column( "X2", Numeric, "Continuous", Format( "Best", 12 ), Set Property( "Mixture", {0, 1, 1} ), Set Property( "Design Role", DesignRole( Mixture ) ), Set Property( "Factor Changes", Easy ), Set Values( [0.745622895216678, 0, 0, 0.25651992859461, 0.333333334645507, 0.286863403799343, 0.333333335537305, 0.333333331350374, 0, 0.722673370024493, 0, 0, 0.7, 0.333333332375124, 0.333451117338599, 0.333239324710344, 1, 0, 0, 0, 0, 0.3, 0.3, 0, 0, 0, 0, 0.262745312275326, 0, 0.700000000000012] ), Set Display Width( 100 ) ), New Column( "X3", Numeric, "Continuous", Format( "Best", 12 ), Set Property( "Mixture", {0, 1, 1} ), Set Property( "Design Role", DesignRole( Mixture ) ), Set Property( "Factor Changes", Easy ), Set Values( [0.254377104783325, 1, 1, 0, 0, 0.71313659620066, 0.333333335537305, 0.333333331350374, 0, 0, 0, 0, 0.3, 0, 0.333097765322802, 0.333521350579312, 0, 0.333333330115096, 0.238247826420103, 0, 0, 0, 0, 0.268306639106238, 0.3, 0.711070148703948, 0, 0, 0.706855507309879, 0] ), Set Display Width( 100 ) ), New Column( "X4", Numeric, "Continuous", Format( "Best", 12 ), Set Property( "Mixture", {0, 1, 1} ), Set Property( "Design Role", DesignRole( Mixture ) ), Set Property( "Factor Changes", Easy ), Set Values( [0, 0, 0, 0, 0.333333334645508, 0, 0.33333332892539, 0.333333337299251, 0.750250247425272, 0.277326629975507, 1, 0.7, 0, 0.333333335249753, 0, 0, 0, 0.333333334942457, 0.761752173579898, 0.258951596909271, 0.3, 0, 0.7, 0, 0.7, 0.288929851296052, 0, 0.737254687724675, 0, 0] ), Set Display Width( 100 ) ), New Column( "Y", Numeric, "Continuous", Format( "Best", 12 ), Set Property( "Response Limits", {Goal( Maximize ), Lower( . ), Upper( . ), Importance( . )} ), Set Values( [., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., ., .] ), Set Display Width( 100 ) ) )
I did not run the script, but I believe you found the problem. The individual Custom Designs are evaluating a Scheffe Cubic model. The Compare Designs platform does not allow that, so you are only comparing those designs on a Scheffe SPECIAL Cubic Model.
That being said, I cannot reproduce your results. I ran your scripts and the 40 run design had a lower average variance of prediction.
I generated 5 different 30-run designs and they all had a higher average variance of prediction. I generated 5 different 40-run designs and they all had a lower average variance of prediction than the lowest 30-run design.
I suspect there was some other anomaly or something going on for the example you included as the 40-run design should certainly have a lower prediction variance.
