How to interpret results of an experiment where they are statistically significant during 2 weeks and are not during 2 other weeks?Hello,
I am analyzing an experiment where test variation had statistically significant results for some outcomes during 2nd and 3rd weeks with a difference of 20-30%, but had insignificant results for the other 2 weeks (1st and 4th) with a difference of ~0%.
How can I address the interpretation of these observations?
What is the appropriate statistical method to choose to check if the difference in observations among 2nd+3rd and 1st+4th are just within random variance?
Can we still make a valid conclusion for this experiment based on the total results?
We have checked any external factors which could influence the results and couldn't find any.
Table version:
| | Trials | Success step 1 | Diff | p-value | Success step 1 which reports about step 3 | Diff | p-value | Success step 3 | Diff | p-value |
| | Test | Control | Test | Control | | | Test | Control | | | Test | Control | | |
| Weeks | | | | | | |
| March 8-14 | 88358 | 89019 | 2585 | 2590 | -0.19% | 0.5 | 1068 | 1038 | +3.6% | 0.2 | 187 | 191 | -1.36 | 0.89 |
| March 15-21 | 88407 | 88324 | 2656 | 2596 | +2.2% | 0.2 | 1102 | 1037 | +6.17% | 0.08 | 200 | 167 | +20% | 0.04 |
| March 22-28 | 90718 | 92135 | 2835 | 2763 | +4% | 0.059 | 1277 | 1198 | +8.26 | 0.02 | 210 | 165 | +30% | 0.006 |
| March 29 - April 4 | 91496 | 91613 | 2506 | 2521 | -0.4 | 0.57 | 1084 | 1139 | -4.7% | 0.25 | 137 | 137 | 0.13% | 0.5 |
| | | | | | | | | | | | | | | |
| Total | 358979 | 361091 | 10582 | 10470 | 1.6% | 0.11 | 4531 | 4412 | +3% | 0.06 | 734 | 660 | +11.8% | 0.018 |
| | | | | | | | | | | | | | | |
| 1st + 4th weeks | 179854 | 180632 | 5091 | 5111 | +0.04 | 0.5 | 2152 | 2177 | -0.72% | 0.8 | 324 | 328 | -0.79% | 0.92 |
| 2nd + 3rd weeks | 179125 | 180459 | 5491 | 5359 | +3.23% | 0.04 | 2379 | 2235 | +7.24% | 0.017 | 410 | 332 | +24% | 0.003 |
AnalyticsData TableDOE
