Poll-- What to do when there is Noise in Input Factors for DOE

Question

I'm looking for comments on how to handle the following problem, either in terms of general advice or specific "how to's" in JMP.  Any and all comments welcome.

We have a client who does air conditioning design.  He wants to do a DOE, using two Temperatures as continuous input factors, measuring refrigerant Mass Flow Rate as the response.

They can design a DOE (with constraints) easily enough.  Now comes time to run the test.  When they ask for (e.g.) T1=100 deg and T2=150 deg, the operator will try to set those values as closely as possible, but there will be error.  Perhaps actual T1=98 deg and actual T2=157 deg.  While they can't set the requested values exactly, they can measure the setpoints precisely, and they can measure the response.

Of course this means they can't do true replicates (at least without a lot of testing).

So the question for the SE community is:  How do you recommend that clients address this problem?

Here are a few ways I've seen in the past.  I'm interested if you use these, or other methods, to address the problem:

1)  Proceed with the test as planned, and have the test operator record the actual temperatures (and mass flow rate) for each requested setpoint.  When finished, average the replicate temperatures at each replicate setpoint, then use these (slightly different than requested) average temperatures in the data table instead of the requested temperatures.  This of course combines the temperature error into the response variation.

2)  Use more sophisticated mathematical means to account for the temperature variation, i.e. use the actual measured temperatures at each test point.  This means not having any true estimate of response variation.

3)  Rethink the problem.  T1 and T2 are actually Responses that are set by adjusting Dial 1 and Dial 2.  Design the experiment around Dial 1 and Dial 2, use T1 and T2 and Mass Flow Rate as responses.

4)

5)

6)

Your thoughts?  If enough people reply with enough responses, I'll compile a final post with the results.

Thanks!

Daniel_Valente · Answer

Jerry --

I think run the experiment at the points that you have in the design -- T1 and T2 at 100/150 and then get an estimate of SD from those actual values you were able get; i.e. how close was the operator able to get the temps after the first complete experiment.

The Profiler has the ability to use the Simulator to the look at making the output robust to factor variation. The "Stochastic Optimization.jmp" sample data table shows this off quite well. If you fit the model and then optimize desirability you get a max yield setting that shows high temp and short reaction time are required to get the highest yield.

But then if you turn the simulator on an put in a SD of 1 for the temp and .03 for the time, which represents how close the operators can hold the temp and control the time of the reaction, you get a different story. When you run this simulation, you can see that the defect rate is pretty high for these settings (when desirability is optimized for highest yield).

Now you can run a simulation experiment and find out the settings, which, when taking into account factor variation (that you measure in the experiment), will yield a response that is robust to this factor variation and still has an appropriately low defect rate. You can find the simualtion experiment in the simulator LIRT (simulation experiment, I used the full-factorial setting here and then ran the GP script in the generated data table). You can maximize disireability on the simulation results (factor setting, which yield the lowest defect rate and then transfer the factor settings back to the original profiler by clicking the link in the report). In this case, running the experiment at a slightly lower temperature for a long time yields a much lower defect rate.

The point is, we know you are going to have variation in the input factors; the operator is only going to be able to be so precise in how tight they can control the factor levels. But because you can use the Profiler and Simulator/Simulation Experiment, you can still find settings, which maximize yield (or match a target), which minimze defect rate robust to the variation you might expect when you actually start running this thing in production.

Stochastic Optimization.jmp

louv · Answer

Jerry,

A very common scenario for me in the chemistry laboratory. We used CTB's (constant temperature baths) and have run hundreds of designs. Since we were often varying the temperature along with factors like solvent volume, reactant stoichiometries etc the actual temperature versus the setpoint temperature were different due to viscosity effects etc. We always just used setpoint temperature as the factor and recorded the actual contents temperature. When modeling the setpoint temperature was used. The key here of course is that the range of temperature investigated is wide enough to be within the noise of the temperature factor. Of course the CTB were calibrated before the design. If one uses the actual temperature then I would argue that the measurement system for temperature also has noise in it. Therefore we always used setpoint temperature as the factor and for the modeling. The actual temperature was recorded to gain an understanding of the impact of the concentration and viscosity etc.

PS The actual measured temperature variation can then be cascaded through the use of the simulator.

MikeD_Anderson · Answer

Maybe taking a step back here would help.  It sounds like there is a GRR element here.  If the operator can't hit a particular temperature precisely, what is the variability in hitting a given temperature?  Also, what is the normal tolerance around a given temperature?

My reasoning is that, if the tolerance and variances allow, maybe 98 deg is equivalent to 100 deg for their process.  If 2 or 3 degrees isn't important then don't sweat it.  Sometimes it can be, but with the AC manufacturers I've worked with it probably won't matter.  Just run it as written.

M