In recent years, an increasing number of JMP users are working with so-called big data, such as data collected from sensors every second, manufacturing data with many parameters, and genomic data.
If the data is large, the file size will be large and more disk space will be required.
JMP originally had a function to compress files when saving data tables, but the latest version, JMP 18, added a function to compact character type columns . Using this function can be expected to reduce file size and save memory.
In this blog, we explain how to compact columns in JMP 18, explain the internal mechanisms of compaction, and compare the file sizes when compacting several data sets.
Compact character columns
When discussing this feature, it is important to note that only character type columns can be compacted; numeric type columns cannot be compacted.
The data in the figure below* records real estate prices in Tokyo and has about 400,000 rows and 38 columns. Let's try to compact the "Region" column, which indicates the region.
Compacting certain character type columns
- Select the column "Region" and select Column > Column Info.
- In JMP 18, if the data type is "Character", you can specify the subtype. By default, it is set to "Normal", but you can change it to "Compact" .
In addition to "Region," this data contains various other character strings such as "Type" and "Prefecture." To compact these all at once, it is a good idea to use the "Column Manager," a new feature in JMP 18.
How to compact multiple strings at once using the column manager
- Select Columns > Column Manager (or click the icon in the Columns panel).
- Click the "Type" item at the top of the Column Manager to sort by type (letters, numbers).
- Select all columns whose type is text (such as "ABC"), hold down the Ctrl key and click the item to the right of "Type", and select [Compact] .
Compacted character columns have an icon added to the right of the column name in the Columns panel, as shown below.
Save the data table compacted using the steps above and compare the file size before and after compaction.
By compacting the data, the original 130MB has been reduced to 67.1M, about half the size. Good news for those who are worried about disk space!
Please note that data tables with compacted strings cannot be opened in versions prior to JMP 17. They are only compatible with JMP 18.
How columns are compacted
So how does this compact string work?
When compacted, internally it is stored in two data structures: string storage and row storage.
For example, in the Tokyo data above, there is a string called "Municipality" that indicates the ward or city the property is in, and each different string is stored in string storage and its index is stored in row storage.
Therefore, by compacting columns that contain long characters or duplicate data values, it is expected that the file size will be reduced.
On the other hand, there may be no benefit to compacting a string that identifies data by assigning a unique ID number to each row. For example, if there are 100 rows of patient data and there is an ID column with values (P1, P2, ..., P100) to identify patients, the benefits of compactness will not be obtained because the values are unique and there are no duplicates.
Here are some results from compacting strings on some JMP sample data:
Some of the file sizes are slightly smaller, but some data does not benefit from the compactness of "Probe.jmp".
In reality, even if you compact a file of a few MB and the file size becomes smaller, it will not save disk space, and you will not experience any performance improvement due to the reduction in memory size.
The benefit comes from compacting data that is several tens of MB or larger.
If you have large data with many strings, try to compact it to see if that reduces the file size.
How to always open character columns as compact columns
So far we have shown how to open a JMP data table and compact strings, but there is also a setting to automatically compact strings when opening data in JMP.
From the JMP menu bar, open File > Preferences and in the General category, check "Open character columns as compact columns."
With this setting, text will be compacted not only when you open JMP data tables, but also when you open Excel, csv, txt, and other files in JMP.
*Data source: Kaggle, Japan Real Estate Prices
Japan Real Estate Prices (kaggle.com)
※Related videos (Japanese)
"JMP 18" New features that are useful for business - Data tables, reporting (Japanese) - JMP User Community
by Naohiro Masukawa (JMP Japan)
Naohiro Masukawa - JMP User Community
