Managing Large Data SetsFor some data sets we collect over time, the file size continues to increase and eventually things can slow down a bit. Sometimes splitting into several files is a solution, but then you may want parts of multiple files for a comparison that could be inconvenient.
I've read about Virtual Join but have not leveraged it. Is this an appropriate approach to utilize for this situation? Are there good examples of implementing a JMP solution for this type of problem?
Re: Managing Large Data SetsIf your file is getting larger because of joins with other data tables then virtual joins can help as the data isn't duplicated across rows.
If, however, the data table is getting larger as more data is appended or concatenated, then virtual joins won't really help.
How are your data tables growing?
What size are you ending up with, both in rows and columns but also in size on disk when saved as a .jmp file?
Re: Managing Large Data SetsThe only thing that will speed JMP up is to use the Columns command to compress columns:
Cols -> Utilities -> Compress Selected Columns.
- For numeric data, you won't see a difference. What happens behind the scene is JMP strips out all the unused bits (we rarely need all 64-bits for storing data).
- For categorical data, it turns it into a list check, so future data entry can be more challenging. It also greatly reduces the overall footprint of the data.
Compressing columns can reduce the file size by a magnitude or more! And it absolutely speeds up computation in JMP.
You can read more about it in the JMP documentation.
