nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Which platforms have data formats that confuse customers?

michael_jmp

Hi all,

At a CAB meeting a couple years ago, one of the CAB members mentioned that sometimes they have trouble knowing what format their data need to be in to be able to use a particular platform. He specifically mentioned Oneway and the need for the data table to be stacked rather than the column format that often appears in textbooks. To address that particular case, we've added an example to the Oneway documentation to show how to get column-formatted data into the stacked form.

The doc team would like to know from you if there are other particularly problematic platforms in terms of the data format that JMP requires. Please reply with those platforms, and we'll do our best to make the doc for those platforms more clear/complete in terms of describing the required data format.

Thanks,
Michael

Accepted answers

michael_jmp

Thanks everyone for your responses. We have a few places where we can focus on adding data format information (probably in the form of examples, like we already have in Oneway). I appreciate you taking the time to give us some feedback.

All comments

Mark_Bailey

I think that this issue is important but I don't have a solution. It is something that users must learn somehow. I taught a custom course to an on-site class yesterday and we were stacking or splitting columns all day long! They have been using Excel up to this point and the examples that I was given were all in Excel workbooks. They like the wide table format so we had to stack to begin with. Later we wanted to fit and save models for different groups of observations, so we had to split the table one way or another.

They got the hang of it it but it is a fundamental change from the "this is how you solve it with Excel" mindset to the "this is how you solve it with JMP" mindset. The idea of including examples that require stacking or splitting in the platform documentation will go along way towards helping with this transition.

MikeD_Anderson

I guess the only thing that you might do (that I could think of besides what @Mark_Bailey suggested) is to add a table or abstract at the front of each platform section - Provide the description in the JMP Starter with what data type and structure gets the best results and what types of analyses are in the reports.

That said, I completely agree with Mark. There is no "Easy Button." Sometimes they just have to learn it. Also, whenever I've encountered issues with something like this it's generally because the user hasn't opened one of the included books and worked an example. "Have you seen Chapter X (pages Y-Z) in the book?" becoming my default question. Which is then followed by "Have you tried this example?" (providing a link to the exact example that solves their problem in the online books.) Some users just don't want to do the leg-work.

Daniel_Valente

I completely agree with you @MikeD_Anderson. A simple Google search of the question you are trying to do in JMP following by "in JMP" or "with JMP JSL" will usually give the user the answer in the first couple of links whether it is a user community post, online tutorial on JMP.com, the doc etc.

That being said, I think the more difficult problem is before the data ever gets into JMP. Getting the data ready to be loaded into a platform is often where the stumbling blocks are.

For example, I just was digitizing all of my car service records and what I really wanted to do is go directly into a JMP table from my PDF scanner app (which does OCR and can generate a single text file of all the scanned documents). The generation of the text file is easy (It's a single click). What I am then left with is data that is rich with the information I need (date of service, mileage, what was performed, where, etc.) but the next step of parsing all of that text and turning it into an analysis JMP-ready data table is really really hard.

So then I am faced with the decision (a decision that many of our users make as well, I am sure). Do I:

1) Try to write a script to prepare this data for analysis?

2) Do I manually parse the text documents and enter the data by hand into a JMP data table (might work if only dealing with a handful of documents)?

3) Do I give up on trying to work with these data because the problem is too hard?

I think that is what we need to help with. The other challanges, i.e. data must be stacked before it goes into this platform, or multiple response must be turned on in the modeling types before going into distribution etc. Those are things that, as Mike says, the user needs to learn and do the leg work to figure out.

michael_jmp

Mark,

Thanks for the response. Do you have an example of the "wide table format" in Excel? I can understand what that would mean for ANOVA data sets, but I'm having a hard time picturing what that means for regression data sets.

Michael

michael_jmp

Thanks Mike. It's hard to figure out what to add to the documentation to help the users who haven't tried to look in there at all.

michael_jmp

Thanks Dan. For this issue, the documentation could possibly have some additional general information on clean up. But, I wonder if a series of white papers on specific case studies of data clean up like the car service records example you shared would make more sense. Each data clean up situation is going to be unique, but perhaps if there were a series of case studies, people could find one that is at least similar to their situation.

MikeD_Anderson

You know... I wonder if a mastering JMP webinar could fill the need in this thread... it's not going to deal with people not searching, but something like "preparing your data for analysis" would be a good topic for all this.

Just thinking out loud,

M

Peter_Bartell

One of the common Excel based type data DOE matrices I've seen, leading to regression modeling, and maybe this is what @Mark_Bailey is alluding too is kind of a nested display for full factorial or other designs that are orthogonal and balanced. You have i rows (each row a factor in the design, with j levels for each ith row, corresponding to levels ) and the data table looks like a nested type display. Then arrayed below the design arrangment are y(i,j,k) for each treatment combination. Multiple response rows when there are multiple responses. This is a mess for transposing, stacking, getting the responses into columns etc. See the PowerPoint presentation I use slide #5 in the "Evaluating Your Measurement Process..." Mastering event for an example.

MSASPCBartell.pptx

jiancao

Two other platforms that customers sometimes are not clear about the required data format are Time Series and Mixed Model. If the raw data contains time series or repeated measurements in multiple columns, then the data needs to be stacked into a tall format.

Nate_Riordan

I find most users coming from a non-analytic background or faux-analytic (see excel) are very heavily confused by the transition. I also have noted that the question is >90% directed at the oneway platform.

There is already a great reference example in the Texas precipitation.xlsx dataset. It presents extremely well given that many users new to analytics are still thinking of putting data tables into presentations rather than structuring data for analysis.

I am intrigued about needing to repeatedly stack and split as this can usually be accomplished using a By group. There are definitely times when a split/stack is needed such as where the absence of something is not recorded in an effort to save storage space so the data needs zeros back in for proper conclusions.

That aside perhaps "preparing your data for analysis" deserves a tip of the day or a spot in the new users welcome kit to highlight the change from cellular to columnar thinking. It may also be worth mentioning the power of By groups are super powerful and I have seen even longer term users not leverage them to expedite an analysis. (Let alone block, weight, freq)

--Nate
>From the BART

Mark_Bailey

One example is repeated measures. The time points are arranged in separate columns. Here is an example that Di and I are working on now: dissolution of drug tablets over time with six tablets from each of two batches.

So we need to stack columns 5 through 120 as Dissolution and name the label column Time. The last column needs to be converted to the numeric data type and given the continuous modeling type. Then we can study the dissolution in Graph Builder or Fit Curve.

This process would not occur to them and seems complicated because Excel does not make the same distinctions.

michael_jmp

Mark, this is great. We could certainly work in the stacking of repeated measures into an example in either Mixed Models or SLS/REML (or both). Thanks!
In the Mixed Models chapter, we have a repeated measures example using Cholesterol Stacked.jmp, which I think is the stacked form of Cholesterol.jmp, so it would be a natural place to add an example that shows how to get from the wide format to the stacked format.

michael_jmp

Thanks Jian. Mark also suggested Mixed Model, so I'll be working on an example of unstacking the data table there. And, I'll make a note about Time Series as well.

michael_jmp

Thanks Peter. That's a good example of a situation that needs stacking.

michael_jmp