This article explains, using an example, how to use JMP to calculate " intermediate precision" and " repeatability," which are used in precision assessments in pharmaceutical development.
The impetus for writing this article was a recently published user case study from Ono Pharmaceutical Co., Ltd. In this case study, there is a comment about using restricted maximum likelihood (REML) to estimate variance components as a method for evaluating intra-laboratory precision and repeatability .
After seeing this case study, I received a request from a user asking for a more detailed explanation of this method, so I decided to write this article.
What are intra-laboratory precision and repeatability?
"Precision" here refers to evaluating the variability between measurements when the same sample (specimen) is measured multiple times.
The pharmaceutical guideline "ICH-Q2 Analytical Method Validation" provides the following explanation:
- Intra-laboratory precision : Precision when measurements are taken within the same facility on different test dates, by different testers, instruments, and equipment.
- Repeatability : Accuracy when measuring under the same conditions over a short period of time
So how does calculating these accuracies relate to estimating variance components using restricted maximum likelihood (REML)? Let's explain using a concrete example.
Accuracy evaluation example
The accuracy of measuring the % Recovery of a certain tablet is evaluated.
factor
- Analyst (a, b)
- Test date (Day: 1, 2)
- Machine (Instrument: 1,2)
There are 2 × 2 × 2 = 8 combinations of these, and each pattern is measured four times. Therefore, the total number of measurement data is 32 rows (= 8 patterns × 4 times).
Modeling
- Analyst"Test date" and "equipment" are all included in the model as random effects.
- The variance components for each factor are estimated, and the inter-laboratory reproducibility (RSD%) and repeatability (RSD%) are calculated from the results.
- Here we apply a main effects model (interactions are not taken into account).
The measurement data was organized and the spike recovery rate was plotted for each factor level. Furthermore, the average values for each level were connected with a line, allowing the trend for each factor to be intuitively understood.
*This data is attached to this blog (sample_validation.jmp).
Looking at the graph in the upper right, the variability of the plotted points for each level is roughly the same.However, when comparing the average values, the slope of the line for Day is steeper than for Analyst and Instrument, suggesting that there is a large variability due to the date.
To calculate inter-subject precision and repeatability in JMP, you can primarily use the Variable/Attribute Gauge Chart and Model Fitting platforms.
Use the "Quantitative/Attribute Gauge Chart" platform
This platform is used to examine variability in measurements of continuous variables.
- From [Analyze] > [Quality and Process] > [Variable/Attribute Gauge Chart] , specify [Y, Response Variable] and [X, Group Variable] as shown below, and set the model type to "Main Effects" since we are fitting a main effects model.
First, the "%Recovery Variability Chart" is displayed. This is a plot of the "%Recovery values" for each Analyst, Day, and Instrument. The bars at each level show the minimum, maximum, and average values.
- Selecting Variance Components from the red triangle button to the left of the report "% Recovery Variables Gage Study" displays the "Variance Components" report.
- Additionally, right-click in the displayed report area and select Columns > Coefficient of Variation .
This is the "Variance Components" report. It shows which factors are causing the variability in the measurement data, and the variance components for each component indicate the amount of variance caused by that factor.
The total variance component (=4.263) represents the overall variance and corresponds to the intra-cell repeatability. This overall variance component is broken down into Analyst, Day, Instrument, and Cell. Within-cell refers to the variability that cannot be explained by Analyst, Day, or Instrument, and corresponds to the error in repeated measurements, i.e., repeatability.
The repeatability and reproducibility that are actually reported are calculated by calculating the square root of the corresponding variance component (= standard deviation), dividing that value by the overall mean, and converting it to a percentage, which is RSD % (relative standard deviation) .
The "coefficient of variation" displayed in the report is the calculated RSD %, and the desired precision is as follows:
Intermediate repeatability (RSD %): 2.224%
Repeatability (RSD %): 1.886%
This means that the range of variation is 2.2% and 1.9%, respectively, of the mean. Although both values are small, intra-laboratory precision is slightly larger than repeatability because it includes differences between days and instruments.
Using the Fit Model platform
What is the concept behind the variance components we calculated earlier? This is where the random effects model comes into play.
Analyst, test date, and equipment are treated as random effects. Random effects are effects that are set to capture the variability of levels randomly selected from the population .
For example, when we consider analysts, we consider that the two people who will be performing the analysis, Person A and Person B, were randomly selected from the entire group of analysts. Therefore, this is useful for evaluating how much the measurement results will vary if a different analyst is selected.
(On the other hand, there are fixed effects in addition to random effects. If you treat the analyst as a fixed effect, you will be evaluating the actual difference in the measurements between specific person A and person B.)
A model that includes random effects is called a random effects model . In the Fit Model platform, set the following:
- Specify "Analyst", "Day", and "Instrument" in "Configure Model Effects".
- With "Analyst," "Day," and "Instrument" selected in "Configure Model Effects," select "Attributes" > "Random Effects ."
⇒ Then, “Analyst & Random Effect”, “Day & Random Effect”, and “Instrument & Random Effect” will be displayed, indicating that each has been set as a random effect.
When you specify random effects in this way, the "Method" in the upper right corner will switch to "REML (recommended). " REML (restricted maximum likelihood) is a method for estimating variance components in models that include random effects, and is a method that produces less bias in variance component estimates than standard maximum likelihood (ML).
The REML Variance Component Estimates report displays the estimated variance components for each random effect, the residuals, and the total. You can confirm that these values match the values displayed in the Variable/Attribute Gauge Chart platform.
Therefore, if you right-click here, you can display the "Coefficient of Variation," and the residual and total coefficients of variation correspond to the RSD % of repeatability and inter-laboratory reproducibility, respectively.
In other words, the "Variance Components" report displayed in the "Variance/Attribute Gauge Chart" can be understood as the variance components estimated by the random effects model specified in the model fit.
summary
In this article, we explained:
- About "internal reproducibility" and "repeatability" in pharmaceutical development.
- These precisions are determined from the variance components estimated using a random effects model.
- In JMP, the "Measurement/Attribute Gauge Chart" platform is used to view the coefficient of variation within cells and totals.
- Similar results can be obtained by specifying factors as random effects in "Fit Model."
This example shows a main effects model, but interactions can also be included in the model as random effects. For example, if there are differences in the proficiency of analysts in using equipment, you can include the interaction between analyst and equipment, "Analyst * Equipment."
by Naohiro Masukawa (JMP Japan)
Naohiro Masukawa - JMP User Community
