Background
Product durability is obviously an important quality attribute for both end users and manufacturers. For end users, the durability of a mobile phone is particularly important. For example, if a phone is dropped on a hard object, the screen may crack or even shatter, rendering the phone unusable. To assess the durability of these screens, the manufacturer put a batch of screens through a variety of tests designed to simulate typical wear and tear by users, such as dropping the phone on a concrete surface.
Materials scientists at a screen manufacturer have been experimenting with two new formulations of aluminosilicate glass. The two formulations were created by modifying the material composition and curing process. For simplicity, we will refer to these two recipes as screen types A and B.
For testing, a sample of 10 screens of each type was developed. Each screen is installed on the same model of mobile phone. The phones were then dropped from a height of 1 meter onto a concrete surface in a controlled and consistent manner. The binary variables “success” (no damage) and “failure” (screen damage) were recorded.
One of the company's goals is to ensure that 97% of the screens produced can withstand a drop from a height of one meter without damage (i.e. the overall success rate).
Task
The engineering team hopes to answer two main questions through the data:
- Is there enough statistical evidence to conclude that each screen type has a success rate of at least 97% when dropped from a height of 1.0 m?
- Is there statistical evidence to conclude that one screen type has a better success rate than the other?
data
Data is stored in a results / frequency table format.
Screen Two screen types ( A , B )
Results Two results (success, failure)
Frequency leads to number of phones that succeed or fail
analyze
Confidence Interval for a Population Proportion
We first summarize the data graphically using the distribution platform. Figure 1 shows the bar graphs, as well as the counts and proportions of each outcome for the two screen types.
Figure 1 Distribution
To create, select Analyze > Distribution. Select Result as the Y variable, Frequency as the Frequency variable, and Screen as the Basis variable.
All 10 phones with screen type A ( 100% ) experienced no screen damage, while one out of 10 phones with screen type B ( 10% ) experienced damage. Since this is just the result of a test of 20 phones, we shouldn't immediately conclude that screen A is generally better than screen B , but rather do a more formal statistical analysis.
Because these results might occur in such a test if, in fact, the overall success rate between the two screen types was the same. Our analysis will allow us to quantify this possibility. We will also be able to estimate the overall success rate for each screen type, along with a measure of statistical uncertainty.
We will begin this analysis by first calculating a 95% confidence interval estimate for the overall success rate for each screen type. A confidence interval estimates a population parameter by providing a range of possible values for a parameter based on only one sample.
Figure 2 shows the confidence intervals based on these data.
Figure 2 Distribution and confidence interval
To create one, select the confidence interval of 0.95 from the red triangle next to the bar graph for each screen type.
The interpretation of these ranges is as follows: We are 95% confident that the overall success rate for screen type A after a 1- meter drop is between 72.2% and 100% , and we are 95% confident that the overall success rate for screen type B after a 1- meter drop is between 59.6% and 98.2% . Confidence intervals quantify the statistical uncertainty in an estimate. Due to the small sample size ( 10 phones per screen type), the uncertainty is quite large, which is reflected in the width of the confidence intervals ( 100 – 72.7 = 27.3 for screen type A and 98.2 – 59.6 = 38.6 for screen type B ).
Technical note: The confidence intervals shown in Figure 2 and the hypothesis tests performed in the next section are based on statistical procedures that use the binomial distribution. Another common method for estimating population proportions is based on the normal distribution. Results using this technique will vary from those shown here.
In order to statistically prove the required 97% overall success rate at a 95% confidence level, the lower limit of the confidence interval needs to exceed 97% , which means that any possible value of our estimate of the success rate must be greater than 97% . In our data, the lower limit for screen A is 72.2% and for screen B it is 59.6% . Clearly, we have not generated statistical evidence to demonstrate a success rate of more than 97% . In fact, the margins of error in these confidence intervals reveal that it is impossible to demonstrate this level of success rate with a 95% confidence level based on testing only 10 phones. Even with a perfect result for screen type A , we can only estimate the overall success rate to be at most 72.7% . We need to test enough phones to produce a confidence interval with a smaller margin of error, such as a 97.5% - 100% confidence interval.
Hypothesis Tests for One Proportion
Since the confidence interval indicates that the desired criterion of 97% overall success rate is not met, there is no need to perform a hypothesis test as it will lead to the same conclusion. We will still perform hypothesis testing to show how to proceed and interpret the output.
We can formulate the hypothesis of interest as:
H 0 : P A ≤ 0.97
H A : P A >0.97
H 0 : P B ≤ 0.97
H0: PB>0.97
Here are the overall success rates for each screen type.
Figure 3 shows the results of the hypothesis tests added to the distribution output.
Figure 3 Distribution and hypothesis testing
To create, select the test probability from the red triangle next to the bar graph for each screen type. Enter 0.97 as the hypothesized probability of success. The probability of selecting a value greater than the hypothesized value (exact one-sided binomial test).
The p -values for both tests are large ( 0.7374 for Screen A and 0.9655 for Screen B ), indicating, as we have already determined, that there is no statistical evidence that the desired 97% success rate is achieved, with a significance level of 0.05 that we chose.
Importantly, our conclusions should not be interpreted as the present analysis producing statistical evidence that screens do not meet standards. In contrast, we have produced no statistical evidence to conclude that they do. Perhaps if more screens were tested we would be able to generate the necessary evidence.
Confidence Intervals and Hypothesis Tests for the Difference Between Two Proportions
Another goal of the drop test is to assess whether there is statistical evidence to conclude that the success rates of the two screen types are different. Notice how much overlap there is in the confidence intervals: [72.2% - 100%] for Screen A and [59.6% - 98.2%] for Screen B. From this we can see that there is not enough statistical evidence to conclude that there is a difference between the success rates of the two screen types. Again, this may be a result of the small number of screens tested; testing more screens might produce the necessary statistical evidence to demonstrate a difference.
Nonetheless, we will demonstrate the types of statistical analyses that are typically performed when comparing two population proportions, because the exercise will ask you to perform these analyses using a larger number of tested phones. Five different statistical techniques will be demonstrated:
- Contingency table analysis
- Hypothesis Tests for Two-Sample Proportions
- Confidence Interval Estimation for the Difference of Two Proportions
- Confidence interval estimates for relative risk
Each analysis will reach the same conclusion: there is not enough statistical evidence to conclude that there is a difference in success rates between the two screen types. The first three hypothesis tests can be written as:
H 0 : P A =P B
HA : PA ≠ PB
or equivalently
H 0 : P A -P B =0
H A : P A -P B ≠ 0
Where PA and PB are the overall success rates of screen types A and B respectively.
Figure 4 is the output from the contingency table analysis.
To create this, select Analyze > Fit Y by X. Put the result in Y , response, the screen in X , factor, and the frequency in frequency. Right-click the Mosaic chart and select Percent Labels under Cell Labels. The red triangles in the contingency table output provide choices for the information displayed in the table cells.
The likelihood ratio and Pearson's p -values showed 0.2303 and 0.3049 , respectively, which, as expected, indicated insufficient statistical evidence to suggest a difference in overall success rates between the two screen types.
Figure 5 contains the results of the two-sample proportion test, which is another way to test the hypothesis. If there are more than two categories (e.g., screen type C , D , etc.), then the chi-square test in the contingency table analysis can be used, whereas only the two-sample proportion test is available when there are only two groups, as is the case here.
To create, select Two-Sample Proportions Test from the top red triangle.
As expected, the p -value for this statistical analysis technique was large ( 0.5338 ), indicating that there was no statistical evidence of a difference in success rates between the two screen types . Also note that the confidence interval for PA - PB (the difference in overall success rates) is [-.018, 0.346] . This is interpreted as: We can estimate with 95% confidence that the overall success rate for screen type A is 0.18 units smaller than the overall success rate for screen type B , up to 0.346 units. That is, we estimate that the overall success rate of screen type A may be smaller or larger than the overall success rate of screen type B. If the interval is completely greater than 0 or less than 0 , there is statistically significant evidence at the chosen confidence level. This confidence interval shows that 0 is a reasonable value for PA - PB . In other words, there was no difference in success rates.
Confidence interval for relative risk
A fourth way to conduct two-sample comparative analysis is by examining the ratio of success rates called relative risks. The hypothesis testing framework for this ratio can be written as: where PA and PB are the overall success rates of screen types A and B , respectively.
H 0 : P A /P B =1
H A : P A /P B ≠ 1
This approach expresses the parameter of interest as the percentage increase or decrease between two success rates rather than the difference between them. This is a useful way to describe the difference between two scales.
Figure 6 shows the results of this analysis.
To create this, select Relative Risk from the red triangle at the top. Select the desired response outcome and the category in the numerator.
The confidence interval [0.904, 1.366] is interpreted as: We can estimate with 95% confidence that the overall success rate of screen type A is between 90.4% and 136.6% of the overall success rate of screen type A. That is to say, the overall success rate of screen type A may be lower than the overall success rate of screen type B , or it may be higher than the overall success rate of screen type B. Statistically significant evidence exists at the chosen confidence level only if the confidence interval estimate of the relative risk is completely greater than 1 or less than 1 . This confidence interval shows that 1 is a reasonable value for P A / P B. In other words, there was no difference in success rates.
Statistical insights
Our analysis showed that there was no statistical evidence in the data to demonstrate the required 97% overall success rate. We also did not generate any statistical evidence to conclude that there was a difference in overall success rates between the two screens. As stated earlier, we should not conclude that our analysis has produced statistical evidence that screens do not meet the required success rates; rather, we have not produced statistical evidence to conclude that they do. Again, we haven't generated statistical evidence to conclude that the two screen types have the same success rate, we just haven't generated statistical evidence to say that they are different. More testing may be able to produce this evidence.
The case in this article is from " Durability of Mobile Phone Screen - Part 1 | JMP "
