Transcript |
| Hello Chris Gotwalt here. |
| Today, we're going to be |
| constructing the history of |
| graphic paradoxes and oh wait, |
| wrong topic. Actually we're going |
| to be talking about candidate |
| set designs, tailoring DOE |
| constraints to the problem. |
| So industrial experimentation |
| for product and process |
| improvement has a long history |
| with many threads that I admit I |
| only know a tiny sliver |
| of. The idea of using observation |
| for product and process |
| innovation is as old as humanity |
| itself. It received renewed |
| focus during the Renaissance and |
| Scientific Revolution. During the |
| subsequent Industrial |
| Revolution, science and industry |
| began to operate more and more |
| in lockstep. In the early 20th |
| century, Edison's lab was an |
| industrial innovation on a |
| factory scale, but it was done |
| to my knowledge, outside of |
| modern experimental traditions. |
| Not long after RA Fisher |
| introduced concepts like |
| blocking and randomization, his |
| associate and then son in law, |
| George Box, developed what is |
| now probably the dominant |
| paradigm in design of |
| experiments, with the most |
| popular book being Statistics |
| for Experimenters by Box, |
| Hunter and Hunter. |
| The method described in Box, Hunter |
| and Hunter are what I call the |
| taxonomical approach to design. |
| So suppose you have a product |
| or process you want to improve. |
| You think through the things you |
| can change. The knobs can turn |
| like temperature, pressure, |
| time, ingredients you can use or |
| processing methods that you can |
| use. These these things become |
| your factors. Then you think |
| about whether they are |
| continuous or nominal, and if |
| they are nominal, how many |
| levels they take or the range |
| you're willing to vary them. If a |
| factor is continuous, then you |
| figure out the name of the |
| design that most easily matches |
| up to the problem and resources |
| that you...that fits your budget. |
| That design will have... |
| will have a name like a Box |
| Behnken design, a fractional |
| factorial, or a central |
| composite signs, or possibly |
| something like a Taguchi array. There will |
| be restrictions on the numbers |
| of runs, the level...the numbers |
| of levels of categorical |
| factors, and so on, so there |
| will be some shoehorning the |
| problem at hand into the design |
| that you can find. For example, |
| factors in the BHH |
| approach, Box Hunter and Hunter |
| approach, often need to be |
| whittled down to two or three |
| unique values or levels. |
| Despite its limitations, the |
| taxonomical approach has been |
| fantastically successful. |
| Over time, of course, some |
| people have asked if we could |
| still do better. |
| And by better we mean to ask |
| ourselves, how do we design our |
| study to obtain the highest |
| quality information pertinent to |
| the goals of the improvement |
| project? This line of |
| questioning lead ultimately to |
| optimal design. Optimal design is |
| an academic research area. It was |
| started in parallel with the Box |
| school in the '50s and '60s, but |
| for various reasons remained out |
| of the mainstream of industrial |
| experimentations, until the |
| custom designer and JMP. |
| The philosophy of the custom |
| designer is that you describe |
| the problem to the software. It |
| then returns you the best design |
| for your budgeted number of |
| runs. You start out by declaring |
| your responses along with their |
| goals, like minimize, maximize, |
| or match target, and then you |
| describe the kinds of factors |
| you have, continuous, categorical |
| mixture, etc. Categorical |
| factors can have any number of |
| levels. You give it a model that |
| you want to fit to the resulting |
| data. The model assumes at least |
| squares analysis and consists of |
| main effects and interactions in |
| polynomial terms. The custom |
| designer make some default |
| assumptions about the nature |
| of your goal, such as whether |
| you're interested in screening |
| or prediction, which is |
| reflected in the optimality |
| criterion that is used. The |
| defaults can be overridden |
| with a red triangle menu |
| option if you are wanting to |
| do something different from |
| what the software intends. |
| The workflow in most |
| applications is to set up |
| the model. |
| Then you choose your budget, |
| click make design. Once that |
| happens, JMP uses a mixed, |
| continuous and categorical |
| optimization algorithm, solving |
| for the number of factors times |
| the number of rows terms. |
| Then you get your design data |
| table with everything you need |
| except the response data. This |
| is a great workflow as the |
| factors are able to be varied |
| independent from one another. |
| What if you can't? What if |
| there are constraints? What |
| if the value of some factors |
| determine the possible ranges |
| of other factors? |
| Well then you can do....then |
| you can define some factor |
| constraints or use it |
| disallowed combinations |
| filter. |
| Unfortunately, while these |
| are powerful tools for |
| constraining experimental |
| regions, it can still be very |
| difficult to characterize |
| constraints using these. |
| Brad Jones' DOE team, Ryan Lekivetz, |
| Joseph Morgan and Caleb |
| King have added an |
| extraordinarily useful new |
| feature that makes handling |
| constraints vastly easier in |
| JMP 16. These are called |
| candidate or covariate runs. |
| What you can do is, off on your |
| own, create a table of all |
| possible combinations of |
| factor settings that you want |
| the custom designer to |
| consider. Then load them up |
| here and those will be the |
| only combinations of factor |
| settings that the designer |
| will... |
| will look at. The original |
| table, which I call a |
| candidate table, is like a |
| menu factor settings for |
| the custom designer. |
| This gives JMP users an |
| incredible level of control over |
| their designs. What I'm going to |
| do today is go over several |
| examples to show how you can use |
| this to make the custom |
| designer fulfill its potential |
| as a tool that tailors the |
| design to the problem at hand. |
| Before I do that, I'm going to |
| get off topic for a moment and |
| point out that in the JMP Pro |
| version of the custom designer, |
| there's now a capability that |
| allows you to declare limits of |
| detection at design time. |
| If you want a non missing values |
| for the limits here the custom |
| designer will add a column |
| property that informs the |
| generalized regression platform |
| of the detection limits and it |
| will then automatically get the |
| analysis correct. This leads to |
| dramatically higher power to |
| detect effects and much lower |
| bias in predictions, but that's |
| a topic for another talk. |
| Here are a bunch of applications |
| that I can think of for the |
| candidate set designer. The |
| simplest is when ranges of a |
| continuous factor depend on the |
| level of one or more categorical |
| factors. Another example is when |
| we can't control the range of |
| factors completely |
| independently, but the |
| constraints are hard to write |
| down. There are two methods we |
| can use for this. One is using |
| historical process data as a |
| candidate set, and then the |
| other one is what I call filter |
| designs where you create...design |
| a giant initial data set using |
| random numbers or a space |
| filling design and then use row |
| selections in scatter plots |
| to pick off the points that |
| don't satisfy the constraints. |
| There's also the ability to |
| really highly customize mixture |
| problems, especially situations |
| where you've got multilayer |
| mixturing. This isn't |
| something that I'm going to be |
| able to talk about today, but in |
| the future this is something |
| that you should be looking to be |
| able to do with this candidate |
| set designer. You can also do |
| nonlinear constraints with the |
| filtering method, the same |
| ways you can do other kinds of |
| constraints. It's it's very |
| simple and I'll have a quick |
| example at the very end |
| illustrating this. |
| So let's consider our first |
| example. Suppose you want to |
| match a target response in an |
| investigation of two factors. |
| One is equipped...an equipment |
| supplier, of which there are two |
| levels and the other one is the |
| temperature of the device. The |
| two different suppliers have |
| different ranges of operating |
| temperatures. Supplier A's is more |
| narrow of the two, going from |
| 150 to 170 degrees Celsius. |
| But it's controllable to a |
| finer level of resolution of |
| about 5 degrees. Supplier B |
| has a wider operating range |
| going from 140 to 180 degrees |
| Celsius, but is only |
| controllable to 10 degrees |
| Celsius. Suppose we want to do |
| a 12 run design to find the |
| optimal combination of these |
| two factors. |
| We enumerate all possible |
| combinations of the two |
| factors in 10 runs in the |
| table here, just creating |
| this manually ourselves. |
| So here's the five possible |
| values of machine type A's |
| temperature settings. And then |
| down here are the five possible |
| values of Type B's temperature |
| settings. We want the best |
| design in 12 runs, which exceeds |
| the number of rows in the |
| candidate table. This isn't a |
| problem in theory, but I |
| recommend creating a copy of the |
| candidate set just in case so |
| that the number of runs that |
| your candidate table has exceeds |
| the number that you're looking |
| for in the design. |
| Then we go to the custom |
| designer. |
| Push select covariate |
| factors button. |
| Select the columns that we want |
| loaded as candidate design |
| factors. Now the candidate |
| design is loaded and shown. |
| Let's add the interaction effect, |
| as well as the quadratic effect |
| of temperature. Now we're at the |
| final step before creating the |
| design. I want to explain the |
| two options you see in the |
| design generation outline node. |
| The first one, which will force |
| in all the rows that are |
| selected in the original table |
| or in the listing of the |
| candidates in the custom |
| designer. So if you have |
| checkpoints that are unlikely to |
| be favored by the optimality |
| criterion and want to force them |
| into into the design, you can |
| use this option. It's a little |
| like taking those same rows and |
| creating an augmented design |
| based on just them, except that |
| you are controlling the possible |
| combinations of the factors in |
| the additional rows. |
| The second option, which I'm |
| checking here on purpose, allows |
| the candidate rows to be chosen |
| more than once. This will give |
| you optimally chosen |
| replications and is probably a |
| good idea if you're about to run |
| a physical experiment. If, on |
| the other hand, you are using an |
| optimal subset of rows to find |
| to try in a fancy new machine |
| learning algorithm like SVEM, a |
| topic of one of my other talks |
| at the March Discovery |
| Conference. You would not want |
| to check this option if that was |
| the case. Basically, if you |
| don't have all of your response |
| values already, I would check |
| this box and if you already have |
| the response values, then don't. |
| Reset the sample size to 12 and |
| click make design. The candidate |
| design in all its glory will |
| appear just like any other |
| design made by the custom |
| designer. As we see in the |
| middle JMP window, JMP also |
| selects the rows in the original |
| table chosen by the candidate |
| design algorithm. Note that 10 |
| not 12 rows were selected. |
| On the right we see the new |
| design table, the rightmost |
| column in the table indicates |
| the row of origin for that |
| run. Notice that original rows |
| 11 and 15 were chosen twice |
| and are replicates. |
| Here is a histogram view of the |
| design. You can see that the |
| different values of temperature |
| were chosen by the candidate set |
| algorithm for different machine |
| types. Overall, this design is |
| nicely balanced, but we don't |
| have 3 levels of temperature in |
| machine type A. Fortunately, |
| we can select the rows we |
| want forced into the design |
| to ensure that we have 3 |
| levels of temperature for |
| both machine types. |
| Just select the row you want |
| forced into the design in the |
| covariate table. Check include |
| all selected covariant rows into |
| the design option. |
| And then if you go through |
| all of that, you will see |
| that now both levels of |
| machine have at least three |
| levels of temperature in the |
| design. So the first design |
| we created is on the left |
| and the new design forcing |
| there to be 3 levels of |
| machine type A's |
| temperature settings is over |
| here to the right. |
| My second example is based on a |
| real data set from a |
| metallurgical manufacturing |
| process. The company wants to |
| control the amount of shrinkage |
| during the sintering step. They |
| have a lot of historical data |
| and have applied machine |
| learning models to predict |
| shrinkage and so have some idea |
| what the key factors are. |
| However, to actually optimize |
| the process, you should really |
| do a designed experiment. |
| As Laura Castro-Schilo |
| once pointed... |
| As Laura Castro-Schilo once |
| told me, causality is a |
| property not of the data, but |
| if the data generating |
| mechanism, and as George Box |
| says on the inside cover of |
| Statistics for Experimenters, |
| to find out what happens when |
| you change something, it is |
| necessary to change it. |
| Now, although we can't use the |
| historical data to prove |
| causality, there is |
| essential information about |
| what combinations of factors |
| are possible that we can use |
| in the design. |
| We first have to separate the |
| columns in the table that |
| represent controllable factors |
| from the ones that are more |
| passive sensor measurements |
| or drive quantities that |
| cannot be controlled directly. |
| A glance at the scatter plot of |
| the potential continuous factors |
| indicates that there are |
| implicit constraints that could |
| be difficult to characterize as |
| linear constraints or disallowed |
| combinations. However, these |
| represent a sample of the |
| possible combinations that |
| can be used with the |
| candidate designer quite |
| easily. |
| To do this, we bring up the |
| custom designer. Set up the |
| response. I like to load up some |
| covariate factors. Select the |
| columns that we can control as |
| factor...DOE factors and |
| click OK. Now we've got them |
| loaded. Let's set up a quadratic |
| response surface model as our |
| base model. Then select all of |
| the model terms except the |
| intercept. Then do a control |
| plus right click and convert |
| all those terms into if |
| possible effects. This, in |
| combination with response |
| surface model chosen, means |
| that we will be creating a |
| Bayesian I-optimal candidate |
| set design. |
| Check the box that allows for |
| optimally chosen replicates. |
| Enter the sample size. |
| It then creates the design |
| for us. If we look at the |
| distribution of the factors, |
| we see that it is tried hard |
| to pursue greater balance. |
| On the left, we have a |
| scatterplot matrix of the |
| continuous factors from the |
| original data and on the right |
| is the hundred row design. We can see |
| that in the sintering |
| temperature, we have some |
| potential outliers at 1220. |
| One would want to make sure that |
| those are real values. In |
| general, you're going to need to |
| make sure that the input |
| candidate set it's clear of |
| outliars and of missing values |
| before using it as a candidate |
| set design. In my talk with Ron |
| Kennet this...in the March |
| 2021 Discovery conference, |
| I briefly demo how you can use |
| the outlier and missing value |
| screening platforms to remove |
| the outliers and replace the |
| missing values so that you could |
| use them at a subsequent |
| stage like this. |
| Now suppose we have a problem |
| similar to the first example, |
| where there are two machine |
| types, but now we have |
| temperature and pressure as |
| factors, and we know that |
| temperature and pressure cannot |
| vary independently and that the |
| nature of that dependence |
| changes between machine types. |
| We can create an initial space |
| filling design and use the |
| data filter to remove the |
| infeasible combinations of |
| factors setting separately for |
| each machine type. Then we can |
| use the candidate set designer |
| to find the most efficient |
| design for this situation. |
| So now I've been through this, |
| so now I've created my space |
| filling design. It's got 1,000 |
| runs and I can bring up the |
| global data filter on it and |
| use it to shave off different |
| combinations of temperature |
| and pressure so that we can |
| have separate constraints by |
| machine type. |
| So I use the Lasso |
| tool to cut off |
| a corner in machine B. |
| And I go back and I cut off |
| another corner in machine B so |
| machine B is the machine that has |
| kind of a wider operating region |
| in temperature and pressure. |
| Then we switch over to machine |
| A. And we're just going to use |
| the Lasso tool |
| to shave off |
| the points that are outside its |
| operating region. And we see |
| that its operating region is a |
| lot narrower than Machine A's. |
| And here's our combined design. |
| From there we can load that back |
| up into the custom designer. |
| Put an RSM model there, then |
| set our number of runs to 32, |
| allowing coviariate rows to be |
| repeated. And it'll crank |
| through. Once it's done that, it |
| selects all the points that were |
| chosen by the candidate set |
| designer. And here we can see |
| the points that were chosen. |
| They've been highlighted and the |
| original set of candidate points |
| that were not selected are are |
| are gray. |
| We can bring up the new design |
| in Fit Y by X and we can |
| see a scatterplot where we see |
| that the |
| the the machine A design points |
| are in red. They're in the interior |
| of the space, and then the Type |
| B runs are in blue. It had the |
| wider operating region and |
| that's how we see these points |
| out here, further out for it. So |
| we have quickly achieved a |
| design with linear constraints |
| that change with a categorical |
| factor without going the |
| annoying process of deriving the |
| linear combination coefficients. |
| We've simply used basic JMP 101 |
| visualization and filtering |
| tools. This idea generalizes |
| to other nonlinear |
| constraints and other complex |
| situations fairly easily. |
| So now we're going to use |
| filtering and multivariate to |
| set up a very unique new type of |
| design that I assure you you |
| have never seen before. |
| Go to the |
| Lasso tool. We're going |
| to cut out a very unusual |
| constraint. |
| And we're going to invert |
| selection. |
| We're going to delete those rows. |
| Then we can speed this up a |
| little bit. We can go through |
| and do the same thing |
| for other combinations of X1 |
| and the other variables. |
| Carving out a very |
| unusual shaped candidate set. |
| We can load this up into |
| the custom designer. Same |
| thing as before. Bring our |
| columns in as covariates, |
| set up a design with all... |
| all high order interactions made |
| if possible, with a hundred |
| runs. And now we see |
| our design for this very |
| unusual constrained region |
| that is optimal given these |
| constraints. |
| So I'll leave you with this |
| image. I'm very excited to |
| hear what you were able to do |
| with the new candidate set |
| designer. Hats off to the DOE |
| team for adding this |
| surprisingly useful and |
| flexible new feature. Thank |
| you. |
