Differences in distribution fitting results between platformsHi, all!
In a recent call I had a customer ask me about difference in fitting results between Fit Distribution in Distribution and doing an equivalent fit in Life Distribution. They used a Mixture of 2 normals as an example. Intuitively, I'm not sure they should give the same results, but I don't have anything to back that up.
You can see what they're talking about looking at the penguins.jmp sample data set. Get a distribution on the Culman Length and fit a 2 Normal Mixture. Then go into Life Distribution set Culman Length to the response and fit a 2 normal mixture using the Mixture option under the LRT. You'll see the results are pretty different.
Thoughts?
M
Re: Differences in distribution fitting results between platformsThis post by @DonMcCormack is very timely to address this topic: Life Distribution â Not Just for Reliability
I've had similar question and found sometimes Distribution and Life Distribution will match exact, and sometimes they will be a little bit off. In the case of Fitting a Normal 2 Mixture in Distribution vs a Mixture of 2 normal distributions in Life, the results are significantly more off indicating those algorithms are not apples to applies. Here is an example where the results exactly match.
Re: Differences in distribution fitting results between platformsThank you both for the great responses! Much appreciated!
Best,
M
Re: Differences in distribution fitting results between platformsHi Peng,
Very interesting. I was playing around with this a bit yesterday... I ran the script below, mixing a normal(10,2) and normal (12,1) with p1 = 0.25. As luck would have it, P1(sample) - was 0.27.
I then journaled some results. As one might expect, the estimation of p1 is crucial to the fit (and difficult). I noticed a couple of interesting things, made moreso by learning of Distribution's specificity vis-a-vis Life Distribution's:
1) Despite Distribution's tuning to optimize fitting of normal mixtures, my impression is that it was outperformed by Life Distribution on this particular mixture--at least in some ways. Both platforms did well on the N(12,1) distribution, which represented the bulk of the data, and Distribution's CIs were tighter for p1, u1, and s1... but as none of these intervals contained the true parameter values, this was somewhat unsatisfying. Life Distribution's point estimates for these were much better, although the CIs were wider (due to the lack of restriction to the normal case, I presume). Of course, when Distribution was supplied p1 as a fixed parameter, it did very well... but we often don't know p1 a priori.
2) Although the relationship between p1 and p2 is completely dependent, it seems that Distribution fits p1 and p2 at least partially independently. Life Distribution fits only p1, leaving the analyst to infer p2 as 1-p1. The result of this is that in Distribution, one cannot simply obtain the upper confidence bound for p2 by subtracting from 1 the lower confidence bound for p1, while this is presumably what one would do ( ?? ) if using Life Distribution's output for a 2-Normal.
3) FWIW, in this particular case, Life Distribution's "starting point method" (or similar, if that's not the exact phrase) option had no effect on the fit. All 3 gave the same estimates.
Comparative report and graph below. As the graph (note: it is centered + scaled, so 0 always represents the true parameter value) shows, in this case the methods reflect a trade-off between point-estimation accuracy and variance: in all cases, the point estimates of Life Distribution were closer to the "truth", but in all cases, Distribution's CIs lied completely within the CIs produced by Life Distribution.
Re: Differences in distribution fitting results between platformsFor this particular data, it is a multimodal problem. In another word, the likelihood surface has multiple local optimum.
In that case different algorithms may end up with different results, as between Distribution and LD. Even with the same algorithm, if starting values are different or random, results might be different, too.
For this penguins data, the two platforms found two different local optimum. And Distribution's result has a better likelihood value.
In general, mixture distribution fitting is a challenging problem. What is in Distribution has been optimized to address so called Gaussian mixture, i.e. all components are Normal. What in Life Distribution is not optimized for any particular type of component, since it is prepared to support mixture of any type of components.
So if one wants to fit a Gaussian mixture with 2 or 3 components, which are what Distribution supports, I highly recommend using Distribution to finish the job. In all other situations, Life Distribution is the best bet. But if there is any doubt about the accuracy or correctness, please contact tech support, and I will look into the issue case by case. I appreciate it.
Re: Differences in distribution fitting results between platformsThanks @peng_liu. I originally had something in my post about this, but took it out because it made the post overly long. In talking with @clay_barker about how the Distribution platform fits Gaussian mixtures, he indicated that he's using Bayesian estimators to circumvent the problems that @peng_liu points to.
I plan to create a follow-up post to talk about mixtures in Life Distribution and will discuss the discrepancy between it and Distribution.
Re: Differences in distribution fitting results between platforms@brady_brady I looked at the data that you sent me and did some digging. My previous comments remain the same. Your simulated data poses the same multi-modal issue. The two platforms converged to different local optimums, and the subsequent analysis is inconclusive about consistency and efficiency of either approaches. Let me show you the multi-modal issue.
The data that you sent me produce the following reports. And in this instance, LD gets slightly better likelihood.
Now look at the following plot. I will explain what it is, after the screenshot.
It is the loglikelihood function's projections to individual parameter's spaces. To get these dots, I create a data table. Each row has a set of parameter values, and in the end I calculate the 2loglikelihood value. The parameter values are uniformly sampled from intervals that include the best estimates from both platforms.
The horizontal reference line is the 2loglikelihood value from Distribution. Now you see there are quite some dots above that line. And the most striking impression is how many of them, and how spread out they are. There are at least two distinct modes. And one mode appears to spread wider than the other. Though I haven't verified, but it is reasonable to assume some optimums has flatter surrounding than some other optimums, which explain why CIs are different.
But in the end, compare the two results using this data should not lead to a definite conclusion about methodologies. If one does concern which result should one use. Then I have to ask, what is the objective of the study. If the objective is to predict the probability at a given place, both results give similar estimates. If the objective is decide the proportion, then I would say it is inconclusive in this case.
