Have you ever heard of an ethical problem in Boston Housing data?I was suprised that I got the following error message when I tried to use Boston Housing data in Python's scikit-learn (by the load_boston module). I think we should avoid using this data when you demo to users and prospects. I entered in JIRA ticket JMP-16844 that we should remove the Boston Housing sample data table from JMP.I have often naively used this dataset.
`load_boston` has been removed from scikit-learn since version 1.2.
The Boston housing prices dataset has an ethical problem: as
investigated in [1], the authors of this dataset engineered a
non-invertible variable "B" assuming that racial self-segregation had a
positive impact on house prices [2]. Furthermore the goal of the
research that led to the creation of this dataset was to study the
impact of air quality but it did not give adequate demonstration of the
validity of this assumption.
The scikit-learn maintainers therefore strongly discourage the use of
this dataset unless the purpose of the code is to study and educate
about ethical issues in data science and machine learning.
Re: Have you ever heard of an ethical problem in Boston Housing data?In any case, with our goal to "to spread the use and impact of Statistical Discovery within the scientific and engineering communities", we should really be using more relevant examples.
Re: Have you ever heard of an ethical problem in Boston Housing data?Thank you for information and your reply.
It seems that "b", the 3nd column from the right, has more serious ethical problem than "lstat".
Re: Have you ever heard of an ethical problem in Boston Housing data?Thank you for your reply.
I agree. The "fairness" must be very important in machine learning and scientific area.
Re: Have you ever heard of an ethical problem in Boston Housing data?Correction. In the JMP example data table, it is the variable "B", which is related to the percent of the population that is African American. Keep in mind the time in which this study originated, when racial discrimination and prejudice was much more common in the United States. I think also the LSTAT variable is related to race or ethnicity.
While race or ethnicity can be an important demographic variable to use for predictive modeling, this example purposefully defines a variable that implies that one particular racial group determines overall economic status of a geographic region. This example data is used in the Building Better Models with JMP Pro book, but in the 2nd edition (currently in progress) we are looking to replace this example data set.
In my opinion, I think that it is best to avoid using this data in demos and documentation.
Re: Have you ever heard of an ethical problem in Boston Housing data?@yusuke_ono thank you for bringing this matter up. I wanted to let you know that this is being evaluated closely, and development is actively looking across our sample datasets to identify those examples where the source or background of the data would not be appropriate for us to use.
I think this also highlights that we should take care to understand the source and background of the data we use when we present and demo JMP.
Re: Have you ever heard of an ethical problem in Boston Housing data?I agree with what has been said above. One "work-around" for the time being is to delete or hide & exclude those variables in question. Years ago, when I was working at AT&T we faced similar issues with regards to using the US Census demographic data in modeling our customers our attorneys advised us to not use them in scoring models.
