Hello,
Would you be able please to share your point of view on how to approach estimation of long-term effect of online A/B(MV) tests for a business?
I read a book by Kohavi where he mentions methods to do so like leaving 5% of traffic to control and switching-off a winner after some period of time.
However, in my case my boss insists on analysis of a period after a winner was implemented in Google Analytics.
In this case he proposed to take all possible external factors into consideration.
What do you think of this approach from statistical point of view?
My concern is that it is difficult to estimate external factors such as quality of traffic (FB ads), technical changes on a site, seasonality, etc.
Is it worth to do such an analysis to increase the level of certainty by including external factors (at least which we are aware of)?
