Recently, we held a web seminar for Japanese customers titled "Introduction to Statistical Hypothesis Testing with JMP." Due to the nature of the seminar, we thought that many people working in the medical and pharmaceutical fields would participate, but in fact, many people from the manufacturing industry who develop semiconductors, electrical and electronic parts, and materials also participated.
In this seminar, we explained equivalence testing in JMP. According to a survey of participants after the seminar, cases in which equivalence testing is used include investigating the quality before and after process changes, newly installed equipment, products from different assembly plants, changes in raw materials, and equivalence between product lots .
同等性検定の結果を視覚的に表す「フォレストプロット」"Forest plot" for visually displaying the results of equivalence tests
Additionally, in the survey we received comments that equivalence is often checked in the course of business, and that there are uses for equivalence testing in business, and that the test was useful.
In this blog, we will review and follow up on the seminar content by explaining again an example of equivalence testing for two groups of measurement values in the manufacturing industry. In addition, we will also explain the method for calculating the number of samples required for equivalence testing, as several people have requested.
Equivalence test for differences in two groups of variables
When comparing products before and after a process, you compare two groups of values before and after. To perform an equivalence test in JMP for two groups of metric values, select Equivalence Tests > Means as a report option for the One-Way Analysis under Analyze > Bivariate Relationships.
Up until JMP 17, only equivalence testing based on t-tests was supported, but the latest version, JMP 18, supports equivalence testing based on the non-parametric Wilcoxon test in addition to t-tests.
Here we will introduce examples of equivalence tests based on t-tests and Wilcoxon tests. Sample data used in the examples is also attached to this blog for your reference.
Example of Equivalence Test for Two-Sample t-Test (Parametric)
Example 1: Equivalence of surface hardness of metal parts between old and current processes
- When manufacturing metal parts, the manufacturing process (heat treatment method) was changed, and we would like to investigate whether the surface hardness (HRC*) of the parts can be considered equivalent before (old process) and after (current process) the manufacturing process was changed.
- Taking into consideration the product specifications of the delivery destination, the tolerance range that is considered equivalent is 1.5 (HRC).
- 50 part samples were extracted from each of the old and current processes, and their surface hardness was measured.
*HRC: Rockwell hardness
(See attached sample data "Surface hardness data of metal parts (old process, current process).jmp")
Before conducting the equivalence test, we will provide a report that checks the distribution of surface hardness for the old and current processes, as well as normality and homogeneity of variance.
Looking at the normal quantile plot on the right side of the graph, the plot points for each group roughly follow the reference line, and the slopes of the reference lines (proportional to the standard deviations of the groups) are roughly equal, so it can be said that the assumptions of normality and homogeneity of variance that are required for the t-test are met.
Additionally, the "Mean and Standard Deviation" report shows that the average of the current process is about 0.5 higher than the average of the old process. Taking into account the variability of these averages, we will test whether they can be considered equivalent within the tolerance range.
This is a report of the results when you select "t-test" as the type of test in the Equivalence Test > Means settings and set the margin (difference) to 1.5.
The Forest Plot shows the difference between the "Old Process Mean" and the "Current Process Mean" and a two-sided 90% confidence interval for the difference, which is within the tolerance range of (-1.5, 1.5). Therefore, the part hardness of the old process and the part hardness of the new process can be considered to be essentially the same.
Example of Equivalence Test for Two-Sample Wilcoxon Test (Nonparametric)
Example 2: Equivalence of semiconductor wafer thickness due to differences in materials
- I would like to find out whether the film thickness (nm) of semiconductor wafers made from different materials (A,
can be considered equivalent.
- Based on past knowledge, the tolerance for equivalent is set to 5 nm.
- Thirty wafers each made from material A and material B were extracted as samples.
(See the attached sample data "Wafer thickness data (material A, .jmp")
As in Example 1, let's check the distribution and normality.
In this data, there are some samples in each group with very high film thicknesses, and a normal quantile plot shows that the plot points do not follow the reference line.
In such cases, it may be possible to transform the data to bring it closer to a normal distribution, but with data like this one that has large outliers, simple transformations such as logarithmic transformations may not bring it closer to a normal distribution. Therefore, here we will perform an equivalence test using a nonparametric method that does not assume a distribution.
This is the report when you select "Wilcoxon test" as the test type in the [Equivalence Test] > [Means] settings and set the margin (difference) to 5.
The Forest Plot shows the Hodges-Lehmann estimate (location parameter estimate) of the difference between Material A and Material B and its two-sided 90% confidence interval, which is outside the tolerance range of (-5,5). Therefore, in this case, the thicknesses of Material A and Material B cannot be considered equivalent.
Sample size needed for equivalence testing
In JMP, you can use the Sample Size Explorer to determine power and sample size for a two-sample t-test equivalence test.
Example: Determine the sample size required for a two-sample t-test for equivalence testing in the following case:
- When manufacturing metal parts, the manufacturing process (heat treatment method) was changed, so a test was conducted to verify the equivalence of the surface hardness (HRC) of the parts before (old process) and after (current process) the manufacturing process was changed. The tolerance range used in the test is 1.5 (HRC).
- Based on past knowledge, we assume that the standard deviation between the old and current processes is 2 (HRC).
- The significance level is 5%, the power is 80% or more, and the same number of samples are extracted at each process.
procedure
1. Select Sample Size Explorer > Power > Power for Two Independent Samples of Equivalence .
2. In "Explorer Settings", enter the tolerance (1.5, -1.5) for the upper and lower margins. In the "Alpha" field on the right, enter the significance level. In this example, the significance level is 0.05, so you can leave it at the default setting.
3. Under "Profile", enter 2 as the standard deviation for both Group 1 and Group 2.
4. Enter 0.8 in "Power" on the left.
In this case, the smallest sample size that will give a power of 80% or more will be displayed in "Overall sample size". However, in this example, we want to make the sample sizes of Group 1 and Group 2 equal, so we set the sample size of Group 1 toEdit it to 32. The power will then be slightly higher, at 81.39%.
This means that we need to sample 32 parts from each of the old and new processes.
In the above procedure, the "difference to detect" was left at 0, but you should actually enter the expected difference between the average of the old process and the average of the new process. For example, if you can assume that the average of parts in the old process is 44.5 and the average of the new process is 45.0, enter 0.5 (or -0.5) as the difference to detect.
However, in reality, there are few cases where sufficient knowledge about a new process is known in advance. Therefore, in that case, it may be a good idea to leave the "difference to be detected" at 0 for the time being. However, as can be seen from the shape of the graph of the difference to be detected, the further the value is from 0, the lower the detection power becomes. Therefore, to be on the safe side, it is a good idea to enter a value for the difference to be detected that is slightly larger (or smaller) than 0, or to set the detection power to 90% and calculate the sample size.
In addition, the sample size obtained using this method is calculated solely based on statistical theory, so the final sample size will need to be determined taking into account practical feasibility (such as the cost and time required to draw samples).
In recent JMP version upgrades (JMP 17, JMP 18), the equivalence testing functionality has been significantly expanded. Although not mentioned here, JMP 18 also adds the functionality for equivalence testing for paired tests (t-tests, Wilcoxon tests). We encourage you to try out the equivalence testing functionality in the latest version of JMP and use it in your work.
*The data used in the above example is hypothetical data.
Related blog series
Use of "Equivalence Tests" in the Manufacturing Industry Part 1. Equivalence Tests of Means - JMP User Community
Use of "Equivalence Tests" in the Manufacturing Industry Part 2. Equivalence Tests of Proportions - JMP User Community
by Naohiro Masukawa (JMP Japan)
Naohiro Masukawa - JMP User Community
