<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />Every so often I take notice of the date field in my demo data sets and realize I'm working with date ranges that are 10+ years old. Usually its a lot of work or at the very least inconvenient to get the updated data or fake it by adding a couple years to the date column. I think it is feasible to script the acquisition of fresh data and format it so it is demo ready. I suspect that several people are already doing this in small fringe projects.
I'm attaching a script that does a pretty fair job of going out and getting a data set. The script adds some relevant calculated columns, and then adds some graph and analysis scripts to the table so that its ready to use.
The data set is the Google Flu Data. The observed phenomena is that the the frequency of flu search terms seems to parallel the CDC's flu surveillance trends. Although the magnitude of the response seems to be off a bit from year to year, the timing seems to line up fairly well.
Background:
http://gking.harvard.edu/files/gking/files/0314policyforumff.pdf
http://www.theatlantic.com/technology/archive/2014/03/in-defense-of-google-flu-trends/359688/
Google Flu Trends
What's in the data set:
The data set as the search trend for several city,state locations, all the states (HHS regions deleted) The data is collected weekly, and there are time formula columns for the year, week, month, flu season (year+6months) and flu season week. There is also a Week over Week change column and a cumulative search trend by year column along with some cluster columns. All the calculations are done in a formula or by the script that pulls the data. The lat and lon for the location are embedded in the script so that it updates the new data automatically, which is convenient for bubble plots, which look great with this data set.
The graphs in the table show the trends in different ways, and also include a non-linear fit of the cumulative data. This the results of the non-linear fit are useful for finding the inflection points for each of the locations by year. This is a way to rank how soon each place got about 50% of its maximum infection during the season (or who got it worst fastest each year.)
There is one little gotcha to look for. The location column has both states and city, states in it. So filter by location type before trying to use location in the map role for graph builder, otherwise GB won't recognize the location as states.
If you're already got a script that does something like this, post it on this thread (please.)
All the best,
Byron
P.S. I added a second script, it goes out and gets the current Freddie Mac Housing Price Index data. The FMHPI demo is probably one of the most used and abused data sets out there. It show cases the power of Clustering to find states with similar HPI trends. In the past its also been relatively complex, requiring some substantial table manipulation steps to get to the final figure. Some people have even written some nice EG workflows to do the data manipulation. In JMP 12, the new Stacked Data option in the Cluster platform makes the whole clustering part of the demo ridiculously easy. (I recommend clustering the whole time range, and then the time since 2000, and then the last 7 or 8 years. Use the row filter to make the subsets, then use GB, Date vs HPI, State as overlay, and cluster as wrap. Add a quick column switcher to toggle between the different clusters.)
This data is usually only about one quarter behind, so its worth getting a refresh if you're going to use it. In the script, the excel file from the Freddie Mac site is opened and formatted to a stacked table that should be ready to go.
