nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Equivalence test

CentroidJackal1

Hello

I want to compare two analytical methods that measure the purity of my samples and prove they are equivalent. I know they don't provide the exact same results which means the mean of their difference is not 0. However, I only need to prove that the difference between their results is less than 2%. My data are based on 28 different samples and each sample was analyzed once by each method. We tested the equivalence with a two-sided t-test on the purity difference but we are not sure this test is the most appropriate to prove the concept. Please not that the data is not normally distributed. Can someone please advise on this?

Equivalence test.JPG

Accepted answers

SDF1

Hi @CentroidJackal1 ,

I believe your equivalence test shows that you can't reject the equivalency of the two methods -- to within +/-2 units of 0. In the example that you provide, the average difference is -0.72, and since you're accepting anything within +/-2 units of 0, then that the average difference suggests that the values are equivalent. The red whiskers on the average difference are the confidence bounds on that mean difference.

Something to keep in mind is that this equivalence test is looking at the mean difference of all your 28 measurements. It says nothing about a single measurement. So, I think one should be careful about saying the difference will "always" be less than 2 (at whatever confidence you've decided ahead of time). What really should be discussed is that the average difference is less than +/-2 units from 0. This is illustrated by your distribution where you have at least one data point with a difference of -4. In this particular case, this one (or more) measurement(s) has a difference greater than +/-2 units from 0, and would fail your original description. However, the average of all measurements does not fail the description.

The BA analysis should give you this information as well, as you can estimate the confidence interval on the mean-difference plot and see how many of the measurements fall outside these lines as well as determine if there is any bias in one method or the other. With the BA analysis, you can get a pretty good estimate of how many data points will be outside your CIs and hence give a more reasonable estimate of what you'd like to answer, which is for any given measurement, you have a confidence level of X that the difference is within +/-2 units.

You might also consider doing an ANOVA analysis, where you can again do an equivalence test on the means -- but in this case you would use the actual measurement values, and even though you know the absolute values of A and B might not be the same, you can still do the equivalence test to within +/-2 units. The nice thing here is that you can account for unequal variances if need be -- for example if method A and B don't have similar standard deviations.

With either of the equivalence tests, you're really looking at either the difference in means or the mean difference -- for all your measurements, and that should be made clear. A single measurement can always be an outlier, but if the mean response of each method is equivalent, it should pass the equivalence test.

Good luck!,

All comments

SDF1

Hi @CentroidJackal1 ,

The way you present and propose is definitely one way to evaluate the data. One thing to keep in mind though, is that when you say the difference in the results need to be less than 2%, 2% of which one, method A or method B? This part was not well explained.

Another way to look at the data might be with a mean-difference plot, much like a Bland-Altman analysis. You can read up on it from the link. This is a good way to evaluate if two different methods are essentially equivalent and to see if there's any bias in the data and if it's coming from one method versus another. The BA analysis doesn't require normally distributed data.

Hope this helps!,

Mark_Bailey

There are numerous long-standing standards (e.g., NCCLS, IUPAC, ICH) about method comparison. It can involve a number of aspects of the measurement: accuracy, precision, linearity, limit of detection or quantitation, and so on. You can using Fit Orthogonal command in Bivariate to perform a Deming regression that minimizes the error in X and Y. JMP 17 will also include Passing-Bablok regression and Bland-Altman analysis. In the meantime, see the JMP Help chapter about Measurement System Analysis for more ideas.

CentroidJackal1

Hi @SDF1

Thank you so much for the quick reply. Apologies for the confusion. The purity is expressed as percentage so that's why I put 2% but the difference is the absolute value and I want to prove that there is a certain confidence to say the difference will "always" (for 90%, 95% or 99% of the population) be less than 2. I am not sure the BA analysis will prove that point as it seems like it will only prove that there is a bias between the two methods.

Many thanks,

Jennifer.

CentroidJackal1

Hi @Mark_Bailey

Thanks for your reply. The fit orthogonal is saying that the methods are not equivalent just like the fit line. We know the methods don't give the same results exactly and that there is a bias but we want to prove this bias is not significant. Does that make more sense?

Many thanks,

Jennifer.

Mark_Bailey

If the models are not equivalent, then the bias is significant.

SDF1

Hi @CentroidJackal1 ,

Good luck!,

Mark_Bailey

I should have replied that your equivalence test demonstrates sufficient evidence to reject the null hypothesis that the methods are different by more than 2% at the 95% confidence level.

CentroidJackal1

Thank you very much @SDF1. That was really helpful.

Regards,

Jennifer.

CentroidJackal1

Thank you for clarifying @Mark_Bailey. and thank you very much for your help.

Regards,

Jennifer.

AcceptanceFox47

I’m comparing two analytical methods that assess the purity of my samples and would like to determine whether they are equivalent. I’m aware that they don’t yield identical results—so the mean difference between the two methods isn’t zero—but what matters is whether the difference remains within a 2% tolerance.

Here’s the setup:

I tested 28 different samples, with each sample analyzed once by each method.
I’m interested in verifying if the methods are equivalent within a ±2% margin.
I originally applied a two-sided t-test on the differences in purity, but I’m not confident that this approach is the best suited for establishing equivalence.
The kicker: the data are not normally distributed.

I’d greatly appreciate any guidance on:

Which statistical tests are most appropriate for assessing equivalence when data aren’t normally distributed.
Valid ways to implement such tests under these conditions.
Pitfalls I should be aware of when assessing equivalence with non-normal data.

Any recommendations or advice would be really helpful—thank you!

MRB3855

Hi @AcceptanceFox47 : If the data are approximately log-normal (the natural log of the data is approximately normal) then this can be carried out as a confidence interval for paired data would be carried out (could do it via Matched Pair platform, for example). But the analysis would be carried out on the natural log of the data. The endpoints of the 90% confidence interval on the difference (in log-scale) would then be anti-logged (EXP(.) ) to get a confidence interval on the geometric mean ratio. This final interval would need to be within 0.98 to 1.02 to claim equivalence.

And, if the sample size is large enough (n=28 qualifies) then the Central Limit Theorem ( https://en.wikipedia.org/wiki/Central_limit_theorem#:~:text=In%20probability%20theory%2C%20the%20central,themselves%20are%20not%20normally%20distributed.) says not to worry about normality. Edit: so, carry on as I explained above.