Transcript |
| Hello, my name is Chris Gotwalt. My co-presenter Hadley |
| Myers and I are presenting an add-in for obtaining improved |
| confidence intervals on sums or linear combinations of variance |
| components. This is part of a series of talks we have given as |
| we work on improving and evaluating several approaches. |
| Obtaining confidence intervals on sums of variance components |
| is important in quality because it provides an uncertainty |
| assessment on the repeatability plus the reproducibility of our |
| measurement system. The problem is that when we ask for a 95% |
| confidence interval, there are approximations involved and the |
| actual interval coverage can be as low as 80%. |
| In our previous studies, we found that two methods have |
| improved coverage rates, parametric bootstrapping and |
| Satterthwaite intervals, but it was still less than 95% in small |
| samples. The earlier version of the add-in implemented the |
| parametric bootstrap as a stopgap and Elizabeth Claassen |
| implemented the Satterthwaite intervals in the Fit Mixed |
| platform natively in JMP Pro 16. I want to stop here to give |
| Elizabeth Claassen credit for making interval estimation of |
| linear combinations of variance components so much easier and |
| JMP Pro 16. She also greatly extended the Mixed Model output, |
| which has made this presentation vastly easier. I'm also hoping |
| that this presentation will serve as an inspiration to |
| others to check these new Save commands out so they can get |
| more from JMP Pro’s Mixed Model capabilities. Now we're going to |
| combine the two approaches using a technique called Bootstrap |
| Interval Calibration that was introduced by Loh in a 1991 |
| Statistica Sinica article. Bootstrap Calibration is a very |
| general procedure for improving the coverage of confidence |
| intervals that can be applied to almost any parametric |
| statistical model. I'm going to introduce the basic idea of |
| Bootstrap Interval Calibration in the simplest terms that I |
| can, and hand the mic over to Hadley, who's going to demo the |
| add-in and discuss our simulation results. To make this |
| simple, let's make it specific. Consider a very small nested |
| Gauge R&R-type study where we want to estimate the total |
| variation. We collect the data and run a nested variance |
| components model with an Operator effect, a Part within |
| Operator effect, and a residual effect. The software reports a |
| Satterthwaite-based interval on the total. It's well known that |
| this is an approximation that assumes a “large” |
| amount of data is present in order for the actual coverage of |
| the interval to be close to 95%. |
| In small samples, the actual coverage, the probability that the |
| interval procedure generates intervals that actually contain |
| the true value of the estimated quantity, will tend to be less |
| than 95%. Thing is the actual interval coverage is a |
| complicated function of the design, true values of |
| the functions, and a long list of other assumptions that are hard |
| or impossible to verify. What we can do though is used the fitted |
| model and their parameters to do a parametric bootstrap. When we |
| do this, we know the true value of the quantity we are |
| estimating because we were simulating using that value. |
| We can do the simulation thousands of times. We apply the |
| same model fitting process to all the simulated samples. We |
| can collect the intervals from JMP and calculate how often |
| they contain the generating value of the quantity that you |
| were interested in. In this case we were interested in the sum of all |
| the variance components, so the true value is 4.515. |
| Suppose we took our original data set, took the estimates, |
| use the Save Simulation Formula that is comes from Fit Mixed, |
| and generated a large number of new data sets, and applied the |
| same model fitting process that we applied here to each of them, |
| and we collected up all of the confidence intervals that were |
| reported around the total. After having done this, suppose |
| that that...the estimated coverage, the estimated number |
| of times that these intervals actually contained the truth, |
| turned out to be 88%. |
| So we wanted that 95% interval, but the Bootstrap |
| procedure is telling us that the actual coverage is closer to |
| 88%. Now we can play a little game and we can repeat the |
| Parametric Bootstrap using a 99% interval this time. So we go |
| through that process, we redo all the bootstrap intervals and |
| when we did the 99% interval we get an actual coverage of |
| approximately 98%. Now suppose we did this game over and over |
| again until we found an alpha |
| with actual coverage approximately 95%. So in this |
| case, suppose we did that and we ended up with finding that 97.6% |
| when we asked for a 97.6% interval, we actually got |
| something like a 95% coverage. |
| Then what we can do is set 1 minus alpha to 0.976 using the Fit |
| Model launch dialogue, set alpha option and will get an |
| interval that has been Bootstrap Calibrated to have |
| approximate coverage 95%. This is still an approximation. There |
| is still a simulation component to it, as well as a deeper |
| underlying approximation that is extraordinarily hard to analyze, |
| but it can be made easy to use, and this is where Hadley comes |
| in. Now I'm going to hand it over to him and he will demo the |
| add-in and go over the simulations that he did that |
| show that we are able to get better coverage rates than |
| before by applying Bootstrap |
| Calibration to Satterthwaite intervals on linear combinations |
| of variance components. Take it away Hadley. |
| Thank you very much, Chris, and hello to everyone watching online |
| wherever you are. So I'm going to start out by showing you how the add-in |
| works and how you can use it to calculate Bootstrap Calibrated |
| confidence limits for random components in Mixed Models in |
| JMP Pro 16. |
| And from there we'll take a step back. We'll see how the add-in |
| makes these calculations and I'll highlight some of the additions |
| to Mixed Models in JMP Pro 16 that allow it to do that. |
| From there, I'll show you the results of some simulation |
| studies to give you an idea about how accurate this |
| interval estimation method is, the Bootstrap Calibration |
| method, and how it compares to some of the other methods for |
| calculating confidence limits, as well as the situations where it's |
| more or less accurate and some of the limitations and things you |
| should be aware of if you're going to be applying it. |
| We’ll discuss possibilities for improvements in future work |
| just briefly, and from there, I'll conclude by showing you the |
| new MSA Designer, Measurement Systems Analysis Designer, |
| available from the DOE menu in JMP Pro 16 so that you can quickly |
| and easily design and analyze your own MSA Gauge R&R |
| studies. So let's start out by looking at this data set. |
| This is one that I pulled from the sample data files. |
| I'm going to run this Fit Mix script here that I've saved. So |
| what we've got here are our random estimates, estimates |
| for a random components. |
| Now. |
| it could be that you want to, for some reason, calculate an |
| intermediate total, for example Operator and Part nested with |
| Operator, or the three of these, |
| you know, Operator and residual. |
| So to calculate those is very simple, we simply add these |
| estimates, but what's not so simple is to determine those |
| confidence limits. |
| There's a new feature in Mixed Models that's been added in 16. |
| The linear combination of |
| The linear combination of variance components feature |
| right here, and so what you can do is you can click that. |
| You can choose the combination of variance components that you're interested in, |
| and you can press done. So now we have an estimate for those. |
| Components as well as their |
| confidence limits. So, |
| what I'm going to do now is I'm going to take this one step |
| further and I'm going to calculate the Bootstrap |
| Calibrated Satterthwaite estimates and I'm going to do that by |
| going to my add-ins and clicking the Bootstrap Calibrated |
| confidence intervals there. So from here we can estimate the |
| number of simulations. |
| 2500 is a |
| recommended number to the default number. It's also the |
| default number in some of the other simulation platforms and |
| in JMP Pro. I'm going to choose |
| this one. But one thing to note is that it takes some |
| time to be able to do this, and so in the interest of time |
| what I'm going to do is I'm going to stop it early. |
| And here we have our |
| calibrated intervals, calibrated upper and lower confidence |
| limits added to the report. |
| So let's take a step back and see what happened there. |
| I'm going to go ahead and add this again. |
| Now, one thing that the add-in |
| does, as soon as you run it, |
| is it adds |
| this simulation formula to the data table, so you can see the |
| simulation formula here. |
| When the add-in is closed, |
| the simulation formula disappears. |
| The simulation formula there takes advantage of another |
| feature that's been added to... |
| to the Mixed Models platform in JMP Pro, and that is the Save |
| Simulation Formula feature |
| here. So what this would allow you to do is to save the |
| simulation formula and then to |
| use that, for example, to simulate |
| these values here. So, we can swap out our “Y” with our |
| new simulation formula, |
| and go ahead and run that. So when you run the add-in, this is |
| all done in the background. But this is how the add-in goes |
| about calculating these intervals. So I'm going to stop |
| this early, once again in the interest of time. |
| And now we see here the samples |
| estimated for each. |
| simulation. And so how the add-in works is |
| it takes all of these. |
| And it calculates new estimates for the upper and |
| lower Satterthwaite intervals from this |
| estimate and this standard error, swapping out different |
| values for alpha. So what we're aiming for 0.05, right? |
| So that we get 95% upper and lower limits, and what it |
| does is it finds |
| an alpha value that results in 95% coverage, that is 95% hits |
| and 5% misses, |
| swaps that in, that's how you get your calibrated intervals. |
| So I hope you enjoyed seeing that. I hope you find it useful. |
| We've done some simulation studies and what we found out |
| is that |
| the intervals, which you can see here for four operators |
| and 12 days as our random components, |
| we've achieved misses of about |
| 7%, so a 92.8 hit ratio. Now this is better than all of the |
| others, including this, so the linear combination, which is |
| simply the standard Satterthwaite interval |
| calculated on the combination of linear components, as well as the |
| Bootstrap quantiles, the bias-corrected intervals in the |
| bias-corrected and accelerated intervals, but as you'll see |
| these intervals improve, all of them, |
| as you increase your number of Operators from |
| 4 to 8 and the number of Days from 12 to 24. So increasing the |
| levels of these random components |
| result in much better, much more |
| accurate estimates for the confidence limits, and so much so |
| that we now have |
| a method here |
| that is equivalent, |
| just, to an |
| alpha value of .05. |
| So. this improvement in performance of course, comes at |
| a cost, and one of those costs is the length of the intervals. |
| And so you can see here, |
| that with our |
| Bootstrap calibrated, well with all of our intervals in fact, |
| that when we have increasing number of Operators, |
| that the length of the interval is much more |
| bundled closer to 0 than it is when you've got smaller |
| number of Operators. You can see that this tails out much |
| further, so that's this blue area here. That's true for |
| all of them, but it's especially true for the |
| Bootstrap Calibrated interval. You can see this |
| long tail here. |
| On average, you're going to get longer lengths using this |
| method, but you have a more accurate method. |
| Exploring that a little bit deeper, you can see here |
| that this increase in length is true for four Operators, as |
| well as eight Operators, and it is significant. |
| Statistically significant. |
| The other thing that I looked at, |
| is the effect of adding repetitions, so the difference |
| between two repetitions and five repetitions, and what you'll see |
| here is that there really is no |
| difference. So looking across |
| the different sets of combinations from |
| four Operators and two reps to four Operators and five reps, about 6 |
| measurements total versus 3 measurements, we really don't |
| gain anything. All of these are equivalent to each other. |
| So that's something to be aware of, that you see improvements in |
| accuracy when increasing the number of Operators, and you |
| don't see improvements when increasing the number of |
| repetitions. |
| One thing that I'd like to mention as a |
| possibility to improve upon these results is the |
| Fractional Random Weight Bootstrap, which we would have |
| liked to have been able to implement for this in time for |
| this conference. We weren't able to do that, to take this |
| and to apply it to random variance components, and so we |
| hope to be able to do that in future work and perhaps even |
| see an improvement upon the Bootstrap Calibrated interval. |
| And then the other thing that I'd like to highlight before I |
| go is the new |
| MSA designer that's been added to JMP 16, and so |
| from here what we can do is we can very quickly |
| create our own design in order to be able to |
| perform our own MSA or |
| Gauge R&R analysis. And so let's see, I'll do this with three |
| Operators and Five parts. |
| I'll label these |
| A, B and C. |
| And we'll do one repetition of each. So that's two |
| measurements total. |
| So here we've got a table with our |
| design. What I can do is I can press this button to very |
| quickly send that to the different operators, have |
| them fill out their parts, send that back to me. |
| And then I can add |
| those results together. |
| So I'll just sort this because I've got another table over here |
| where I've done this ahead of time. So I'll just add these |
| values over there. And now from the scripts within the table we |
| can quickly and easily do our own Measurement Systems Analysis |
| and Gauge R&R. So I hope you found this useful. I hope you |
| continue to enjoy the talks at this conference. Thank you very |
| much for listening. |
