Informal Survey: Do your customers use Levey-Jennings Control Charts?Hey SEs, happy Friday.
I've spoken with several customers recently who use Levey-Jennings Control Charts for their process data. I try to talk them out of it -- because of the potential for missing important signals -- with mixed success.
How prevalent is the use of these charts, outside the intended purpose of tracking measurement system accuracy in a clinical chemistry lab? Do you have customers who use Levey-Jennings? If so, what industries are they in, and what reasons do they give for using them?
Thanks for any input.
Jordan
Re: Informal Survey: Do your customers use Levey-Jennings Control Charts?The LJ charts are very useful for situations where there are known reasons to expect shifts in the data and you do not want to be oversensitive. In pharma this is often the case. For example, there may be variation in a process due to input material variation, and a single input batch might be used for multiple production runs. That input variation is uncontrollable and is a natural part of the process. Using the I-MR chart will underestimate the total natural variation in the data and make the control chart limits too narrow, causing too many out of control signals that are not really important. At Eli Lilly and Elanco, we wholesale switched from I-MR charts to Levey-Jennings plots because we we wasting too much time investigating control chart signals that did not matter.
Ultimately it comes down to risk management. Control charts are "economic" tool, to allow you to help manage the risks with a process. The cost and risk of false alarms has to be considered in the overall risk.
Example: This process has "natural variation" due to input batch effects and random variation for each batch. The I-MR chart shows several points outside the 3-sigma limits, indicating a process that is not in control.
vs the LJ plot, where all the points are within the wider limits.
Also, keep in mind one of the reasons I-MR charts were originally so popular...you can use a paper-based table easily to calculate limits, since you only really need to add and subtract values in a tabular form to calculate the overall mean, the average moving range, and then finally use the appropriate value to multiple the average moving range by to get the estimate of sigma. In the era before handheld calculators, this was very useful.
Re: Informal Survey: Do your customers use Levey-Jennings Control Charts?WOW SAM!!!! I am surprised to see you replaced your charts instead of fixing the charts for the type of data you have. But Jordan and I have seen this behavior and that’s why he asked the question. The following shouldn't be taken as criticism, I want to stimulate discussion about this topic.
It seems to me that by applying Levey-Jennings charts to every process, you are not using control charts to detect changes in the underlying distribution of the process but are using them as reaction limits or disposition limits instead. The sample standard deviation is never the right statistic to use to estimate sigma when you want to determine if the distribution has changed.
How would you even calculate the risk of a false alarm?
If you want to determine if the process distribution has changed, and you have independent data with one source of variation, then use a short-term estimate of sigma. If you have a batch process with two sources of variation (within and between batches) and collect all the data from the batch at one time, use an Xbar-S chart with Xbar limits based on MRbar, JMP makes this easy with the three-way chart. If you have a batch process and collect data from the batches over time, use a short-run chart. JMP 18 will have an automated way of doing short runs charts, but in the meantime, they are not hard to implement in CCB. If you have autocorrelated data, there are ways of expanding the control limits while still using a short-term estimate of sigma, or modeling the correlation to get to an independent plotting statistic.
Between variation should always be judged by within variation! Anything else shouldn’t be called process control.
Re: Informal Survey: Do your customers use Levey-Jennings Control Charts?I think that I need to chime in on how things "used to be" at my previous employer, which means it likely still is! We never went as far as Sam of switching strictly to LJ, but it was used in several different areas. Plus, by responding, I think I can learn something here.
Here is the scenario: a batch process that makes products according to a product wheel. The product wheel says that we need to make 40 consecutive batches of product X (a very popular product). Once a batch starts, it is difficult to make adjustments to the reaction as it is occurring and it could violate SOP. Therefore, once the batch is completed, properties are measured and placed on to a control chart. This can then be used to make adjustments for the next batch. Is this short-run SPC? Most likely. Is it SQC rather than SPC? Perhaps. But you only get one data point per batch. Thus, an ImR chart makes sense. However, autocorrelation is often a reality, which makes the moving range a horrible estimate for variability. Therefore, LJ charts! Yes, I could also use EWMA or something that handles autocorrelation, but remember who is running the process and making the adjustments: plant operators who are likely not familiar with the mathematics involved in EWMA. There is inherent mistrust because it may not make sense to them. This also lends itself to LJ.
So, Di, there may be within batch variability, but it cannot be easily measured. We are controlling based on batch-to-batch variability. I fail to see what is wrong with using the standard deviation to estimate that batch-to-batch variability. Educate me! Especially on this statement that you made: "The sample standard deviation is never the right statistic to use to estimate sigma when you want to determine if the distribution has changed." If you have a stable process, it seems like a logical choice to me. Might be rough when first starting, but once you are stable, I think it would be okay.
Re: Informal Survey: Do your customers use Levey-Jennings Control Charts?Thanks @SamGardner, thanks @Di_Michelson. Sam, I wanted to respond to your last point, about SPC using hand calculations vs software. Leave aside for the moment the issue of batch-to-batch variation.
While it’s true that I-MR charts are easily drawn by hand, I think there is a common misconception that we need to fight. When I ask customers why they use Levey Jennings, they often tell me that it’s better to use s (the sample standard deviation statistic) for estimation of sigma. Commonly they think that the old-school I-MR chart is optimized for hand calculation, and if you have software, why wouldn’t you use s instead of MR?
The problem with using s, of course, is that it is inflated by any special causes that may be present. More so than the MR-based estimate. Using a short-term sigma estimate is a more effective way to separate common cause variation from special causes. SPC 101, right? But this point is lost on many practitioners.
I use this (admittedly ridiculous) example to illustrate the problem to customers. Randomly generated normal data with a sigma of 5, and there’s a dramatic 20 point (4 sigma) shift right in the middle. The IMR chart has a relatively accurate sigma estimate of 5.7, and correctly shows the process as out-of-control:
While the Levey Jennings chart uses s=11.5 to estimate sigma, quite inflated compared to the true value of 5. And it incorrectly shows the process as in control.
And let’s stipulate that it’s fine to use s to estimate sigma for fixed external limits based on well-behaved historical data. When using fixed limits, I don’t think that is really a Levey-Jennings chart in spirit. The situation that Di and I are objecting to is using s calculated on the chart data to draw limits.
All this said: I agree completely that a control chart is an economic tool, a guide for making decisions. Customers can choose whatever limits they desire. I’m not arguing that we should be slavish about doing things the "proper" way. But I do think it’s important for users to understand the implications of their choices, so they can weigh the risks appropriately.
Re: Informal Survey: Do your customers use Levey-Jennings Control Charts?The key issue is determining the appropriate measure of variance to use in the chart. In the example I provided, the true variance of the process has multiple components, both short term and longer term, and that characterizes the natural variability of the process. In addition, the reason for using LJ vs IMR was the sensitivity of the control chart and the cost associated with a out of control signal. In pharma, an OOS signal typically means a formal investigation, involving hours of work by more than one person. If the chart is overly sensitive, then you will be wasting a lot of time investigating natural variation, taking time away from focusing on other important matters such as process improvement. I have experienced this personally.
Note that this kind of "batching" variation also comes from analytical testing. Often, several batches of material will be produced. and then those batches will be tested at the same time in the same analytical run. If the measurement has a large intermediate precision and the run-to-run variation is large relative to the total variation, then then you will also see many short term shifts in the data.
The process where there is only a single random source of variation that occurs from batch to batch is not the only type of process, and the control chart needs to be appropriate for the process and the decision making process that the control chart is used with.
That being said, I do like the IMR chart, because of it's sensitivity and simplicity. Sometimes the MR chart is as informative as the individual chart.
Re: Informal Survey: Do your customers use Levey-Jennings Control Charts?Thanks, Dan and Sam!
Dan, I think your scenario might be short run SPC. If you adjust the target (and maybe estimate of sigma) when you change products, then yes. My example of short run SPC is when you have a furnace that deposits both 600 and 1000 angstroms of nitride on a substrate. The centerline and limits are different for both processes, but you want to control the furnace, so you standardize the plotting statistic for each process. When a new 300A process comes online, you have to establish the constants used to standardize that process.
I don't have a good answer for using the sample standard deviation rather than the moving range to estimate sigma under autocorrelation. Neither is the best method. But the current best method (modeling the correlation and using EWMA charts on the residuals) is not possible without control systems and not very interpretable even if you have the systems in place.
That leads me back to saying you're not trying to detect changes in the process distribution, you're trying to do SQC instead. And there's nothing wrong with that! But then, just make up some engineering limits that make more sense than using the sample standard deviation. I feel that using a statistical procedure gives you false security that you are doing SPC.
To Sam's point, when you have multiple sources of variation in a process, neither the X-MR chart nor L-J is appropriate. So control those sources separately instead of using one chart. I'm thinking of a process where you want one point on a chart per batch, but you have multiple observations in that batch. In my past life. we often charted three sources of variation: plot the batch average with limits based on MR of the averages (controls batch to batch variation), plot the range of the wafer averages with limits based on Rbar (controls wafer to wafer variation within a batch), plot the average standard deviation of the wafers with limits based on the MR of the within wafer std devs (controls within wafer variation).
Why is the sample standard deviation never the right statistic to do process control? The answer is in your next sentence: sure it is when you know your process is stable, but the purpose of SPC is to tell you if you have a stable process. As Jordan's extreme example shows, the sample standard deviation won't show that.
I've been looking up the history of Levey-Jennings charts. It's interesting that their original paper simply describes the application of XBar-R chart with subgroup size = 2 in clinical chemistry. Use of MR to estimate sigma for individuals had not yet been realized. Henry and Segalove published what JMP calls a Levey-Jennings chart using s to estimate sigma right around the same time that Tippett published X-MR charts. Westgard calls these charts Levey-Jennings charts, which is where JMP got it from.
In our SPC course, we have a demo like Jordan's, but with data more like Sam's. We teach that if you are charting gauge (or assay or method) behavior using measurements of a known standard and you know the process is stable, then ok, go ahead and use s to estimate sigma. Run the standard alongside your clinical samples as a check. (I still get stuck on how do you know the process is stable without doing SPC.)
Thanks again! This is exactly the type of discussion I was hoping for.
Re: Informal Survey: Do your customers use Levey-Jennings Control Charts?Another control chart that is used when there is a "wandering mean" is the EWMA chart, and the smoothing level can be chosen to adiust the sensitivity to detect shifts in the process and manage the risks and cost associated with the control chart.
I remember reading many years ago about one major semiconductor manufacturer that preferred EWMA charts over individuals charts, because of the reasons I described earlier.
Re: Informal Survey: Do your customers use Levey-Jennings Control Charts?In the case of individual observations (the only case where I think LJ might make sense) and a non-stable process, I agree that the standard deviation may not be a good estimate of the variability. But I would argue that MR-bar would not be useful either. Which measure handles small process changes better? I'm not really sure. In the scenario with individual observations and autocorrelation, I would rather "roll the dice" that the standard deviation makes more sense as an estimate than MR-bar. As Sam said earlier, it is how are you going to estimate variance and where do you want to place your risk.
Re: Informal Survey: Do your customers use Levey-Jennings Control Charts?Oh, I disagree! I think you can see in Sam's graph (let's assume it's a non-stable process instead of a batch process) that the MRbar is finding real signals and s is not.
Autocorrelation is just hard to deal with, in all sorts of statistical methods.
I'm a big fan of controlling the variation that you want to control on separate charts, and combining data with multiple sources of variation on the same chart doesn't make sense to me.
Again, thanks for this discussion, it's really helpful!
