Re: What is the max data set size JMP will handle (efficiently), assuming maxed out MacBook Pro specs ?

Question

JMP can handle at data table about half as big as the total amount of physical memory on the machine. So, a 16GB machine can handle a data table an 8GB data table and an 8 GB machine can deal with a 4GB data table. This guideline means that JMP should be able open a data table of this size and do simple and basic analyses on it. There are many analyses, table manipulations and model building platforms that can require significant amounts of memory, depending on the options and the specific data being used. We can’t promise that every analysis or operation can be performed with data tables of this size. The best estimate of the size of a data table is its (uncompressed) size on disk. The size of the data in text (e.g. .csv, .txt) format is a reasonable surrogate also. We can’t provide guidance on how many rows or columns JMP can handle since it’s highly dependent on the combination of the two and the types of data stored. There are a two different forms of “compression” for JMP data tables. Compress Selected Columns a) Select the columns to be affected b) Cols -> Compress Selected Columns This option: • adds a List Check (default order) to character column if the column has less than 255 distinct values. • changes numeric columns to the smallest 1-byte, 2-byte, or 4-byte integer if all values in the column can be stored. Only integer values columns are checked. • For 1-byte integer, the range of numbers that you can store is from -126 to 127. • For 2-byte integer, the range of numbers that you can store is from -32,766 to 32,767. • For 4-byte integer, the range of numbers that you can store is from -2,147,483,646 to 2,147,483,647. • It will not change the columns if they have list check already. This option affects the amount of memory required to hold a data table in memory and, therefore, can allow JMP to work with more data. Compress Tables a) Set option, “Compress File when Saved”, from red triangle menu on data table. This option uses gz compression to make the file smaller on disk only. The data is stored uncompressed in memory. Jeff Perkinson JMP Customer Care Manager SAS Institute Inc. SAS Campus Drive • Cary, North Carolina 27513 • USA Ph: 919-531-6521 E-mail: Jeff.Perkinson@jmp.com Share. Ask. Learn. Connect. - community.jmp.com Connect with me in the JMP User Community. On Jul 17, 2016, at 8:12 AM, Chuck Boiler > wrote: Hi Jeff. I wonder if your group has produced documentation in this area? I hope to see you this week and if you do have access to this kind of content can we talk about it? Blessings of a weekend be upon you. . Chuck Sent from my iPhone, sometimes with voice to text. Please excuse errors. Begin forwarded message: From: Chuck Boiler > Date: July 17, 2016 at 5:08:55 AM PDT To: Laura Higgins > Cc: Louis Valente > Subject: Re: What is the max data set size JMP will handle (efficiently), assuming maxed out MacBook Pro specs ? Thank you Laura. I'll bet we can find someone that has data on this topic in Cary. Let's talk about it this week. Safe travels. Chuck Sent from my iPhone, sometimes with voice to text. Please excuse errors. On Jul 16, 2016, at 12:03 PM, Laura Higgins > wrote: I need help with this. We need to produce a document for XXXXX management. I can explain more next week. I just want to get this on the radar. Maybe one of you have already done something like this? Laura.Higgins@JMP.com while mobile Begin forwarded message: From: Date: July 15, 2016 at 6:39:23 PM PDT To: Laura Higgins > Subject: Re: What is the max data set size JMP will handle (efficiently), assuming maxed out MacBook Pro specs ? Thanks for this Laura, I’m in China just now - so will look to reconnect with you in a couple weeks or so. The over-arching question from our leadership is essentially: “when does JMP break, due to size of data” and “what can we do to maximize JMP analysis effectiveness, i.e. compression, etc.” I’ll circle with you so we (either I or someone else) can produce a reference document, unless you already have such a file / doc. to push to the teams. On Jul 16, 2016, at 5:55 AM, Laura Higgins > wrote: Hi John- There is no architectural limit on JMP data tables for number of rows or columns. It’s a little hard to answer your question because it depends what’s in the columns, and what operation in JMP you’re doing. Numbers take up less space than character data, and you can compress columns (removing unused bits), and also work with compressed data tables. This greatly reduces the JMP file size. Many operations are multithreaded to take advantage of multiple cores. The one thing that will slow JMP down is heavy visual graphing with tons of columns and/or points. I just make up a data table with 1,000,000 rows and 3600-ish columns of 1) numbers uncompressed = 29.13 GB; 2) Compressed columns = 3.5 GB So compressing columns saves almost 10x space! 1 run 16 GM memory and it was SLOW for creating/saving the JMP table initially. Using compressed columns it acted like the smaller data table! Also, from the documentation. Take home -> compressing columns speeds up analysis; compressing the data table does NOT speed up analysis. Compress file when saved Compresses the data table when it is saved. After the data table is saved, a compressed file icon appears next to the data table name in the table panel. See Compress Tables in Enter and Edit Data. Tip: You can also configure JMP to always use GZ compression when saving tables by selecting Preferences > General > Save Data Table Columns GZ Compressed. Note: The Compress file when saved option only decreases the file size. This command does not affect the memory required to analyze the data. To reduce both the file size and memory required for analyzing, use Cols > Utilities > Compress Selected Columns. See Compress Selected Columns in Enter and Edit Data. Character data is harder to predict, because it depends what’s in the column. When you compress character data, it turns it into a checklist (effectively limiting the input options to the existing set of options - not a big deal unless you’re modifying the original data). Not a crystal clear answer, but hope this info help! Cheers! Laura Laura A. Higgins, Ph.D. JMP Sr. Systems Engineer 650.454.0633 | Laura.Higgins@jmp.com | #LauraHigginsPhD | www.jmp.com | SAS Institute Inc.| San Francisco

Byron_JMP · Answer

I've been messing around with the Kaggle data for this challenge:  https://www.kaggle.com/c/bosch-production-line-performance

There are three tables two with about 1000 columns and one with about 2000, all three are 1.18MM rows, and there is a lot of missing data. (3 tables for training and 3 for testing).

With 16 gb of  ram on a mac, I was able to open the files, use compress  columns, and then very very carefully join the tables together. Think about splitting the tables by row or column for some steps.

The entire 4K+ column file and all the rows fit into memory which is convenient. The files load slowly, and save slowly. When I add a formula to score the data it might take a minute for a tree model column to completely evaluate. Working with big files is tedious. Save early, so that when JMP suddenly wants 64 GB of memory and stops dead, you dump it and start over.

Jeff_Perkinson · Answer

I think this is okay to share verbatim:

JMP can handle at data table about half as big as the total amount of physical memory on the machine. So, a 16GB machine can handle a data table can 8GB data table and an 8 GB machine can deal with a 4GB data table.This guideline means that JMP should be able open a data table of this size and do simple and basic analyses on it. There are many analyses, table manipulations and model building platforms that can require significant amounts of memory, depending on the options and the specific data being used. We can’t promise that every analysis or operation can be performed with data tables of this size.The best estimate of the size of a data table is its (uncompressed) size on disk. The size of the data in text (e.g. .csv, .txt) format is a reasonable surrogate also.We can’t provide guidance on how many rows or columns JMP can handle since it’s highly dependent on the combination of the two and the types of data stored.There are a two different forms of “compression” for JMP data tables.Compress Selected Columnsa) Select the columns to be affectedb) Cols -> Compress Selected ColumnsThis option:• adds a List Check (default order) to character column if the column has less than 255 distinct values.• changes numeric columns to the smallest 1-byte, 2-byte, or 4-byte integer if all values in the column can be stored. Only integer values columns are checked.• For 1-byte integer, the range of numbers that you can store is from -126 to 127.• For 2-byte integer, the range of numbers that you can store is from -32,766 to 32,767.• For 4-byte integer, the range of numbers that you can store is from -2,147,483,646 to 2,147,483,647.• It will not change the columns if they have list check already.This option affects the amount of memory required to hold a data table in memory and, therefore, can allow JMP to work with more data.Compress Tablesa) Set option, “Compress File when Saved”,  from red triangle menu on data table.This option uses gz compression to make the file smaller on disk only. The data is stored uncompressed in memory.

Nate_Riordan · Answer

Jeff,

How much of your response can be shared (copy/paste) with customers?