How can income categorized data be normalized?Hello
I have income categorized data that I need to normalize to use it as a continuous variable. Data come from a willingness to pay survey where respondents selected one out of eleven income categories. For each category I have a number of observations.
As a first step I took the midpoint in each category to use it as the continuous variable. Literature suggests that income follows a log normal distribution. I took the ln of the midpoint and tested it. However, the Shapiro-Wilk test indicates that the data is not normally distributed.
Is there any way to normalize this data?
Also, literature (Hsiao, C., 1983. REGRESSION ANALYSIS WITH A CATEGORIZED EXPLANATORY VARIABLE, in: Amemiya, T., Goodman, L.A. (Eds.), Studies in Econometrics, Time Series, and Multivariate Statistics. Academic Press, pp. 93-129. - Hsiao, C., Mountain, D., 1985. Estimating the Short-Run Income Elasticity of Demand for Electricity by Using Cross-Sectional Categorized Data. Journal of the American Statistical Association 80, 259-265) suggests a two-stage method (pseudo-instrumental variable and conditional mean method) but I have not been able to implement this in JMP.
Would it be possible to use the upper and lower bounds of each category plus the number of observations in each category to obtain a normalized data?
My goal is to have two estimates. One is the simple midpoint of each category and another one is the normalized estimates. I would like to compare the normalized estimates with those from the midpoint in order to assess differences between those two methods.
I would appreciate an interactive explanation since I am new to JMP.
Thanks a lot!
Re: How can income categorized data be normalized?You can also fit models with predictors using the Generalized Regression platform in JMP Pro with censored responses.
Re: How can income categorized data be normalized?All real data fails those tests with a large enough sample size. I think the old George Box adage about all models being wrong but some being useful applies just as well to distributional models. A better question to ask about distributional fit is if it is a good enough model to get you a reasonable answer to your question?
What you have is interval censored data. You can fit a distribution, or even models, to this type of data using the Reliability and Survival platforms in JMP. For example, you can use Life Distribution to fit various distributions to your interval censored data and determine which one is the best fit.
Here I generated 300 lognormal(2, 0.5) data and binned the results with bin widths of 3. The following is the counts in each bin.
I fit this in Life Distribution with Low and High as my response columns and Count as the Frequency column.
The data has a very good lognormal fit (as expected) with estimated parameters 1.975 and 0.43.
I also tested generalized gamma, Weibull, and normal fits. I get a comparison table that shows lognormal is the best fit according to AICc and BIC.
From here, you could get pretty fancy determining the value to assign to each count within a bin. For example, you could compute an expected value for the data within each bin using your distributional fit and assign that value to each of your counts for your "normalized" data. An easy way to do that would be to simulate a really large sample from your fitted distribution and take the mean of the data within each bin.
The midpoints pretty much assume a uniform distribution within each bin, which would not be totally realistic. However, I would guess the expected values within a bin using the fitted distribution won't be dramatically different from the midpoints (though I'm not sure what you do about the last open-ended bin). I would be interested to hear how the results turn out.
Re: How can income categorized data be normalized?hello dale_lehman
thanks a lot for your reply. your approach departs from simulated points and then establishes the bins. unfortunately, I don´t have that info. answers from respondents indicated an income category but they did not give a number as their income. also, income ranges were selected based on miminum salary brackets with an open interval at the end. at this point, I am considering using the standardized values (JMP function) of the mid point, Ln, and Log10 even if the normality tests fails.
Re: How can income categorized data be normalized?From what I have read, the lognormal may not be the best for modeling the income distribution (it seems like a gamma distribution is preferred, among others). Also, I think I came across an R program specifically for fitting lognormal distributions to binned data, but I did not write down where that was. But I did a bit of simulation about your question and found some surprises.
In the example file, I simulated 10,000 points from a lognormal distribution. I then binned them using the binning utility (found under Column utilities). The attached script shows the tabulation of the means for each bin. I made that into a data table (an option at the red arrow of the tablulation). In that data table, I calculated the midpoint of each bin and calculated the "error," i.e. the difference between the mean of the bin's lower and upper ranges and the actual mean of the data in each bin. I expected to find a regular pattern where the means in each bin would lie below the average of the bin's end points. But I did not. There does not appear to be any pattern at all. If I did this correctly, it would seem that using the midpoints of the bins does not produce a systematic bias of the original lognormal data.
If anyone finds an error with my approach, please let me know! My intuition suggests that if bins are infinitely narrow, then the midpoints and the means should be the same. At the other extreme, if there is one bin, then the mean of the lognormal values should be less than the midpoint of that bin. If this logic is correct, then my example shows what happens with bins that are relatively narrow compared with the overall range of the data. The systematic bias should be evident when bins become sufficiently wide.
