nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

215301_en-US

liliconcom

Speed of file opening/importing

Hi,

We are in the process of creating a data delivery system. The final collection of data will be pushed in to JMP for its analysis. However we are finding that the time taken to open files is drastically different depending on their type. Ideally we would like to import hdf5 files but based on our analysis the speed of import is highly prohibitive. Attached is a graph that demonstrates the speed of import. Obviously we acknowledge that JMP files are the fastest, however there is no way to create JMP files (except in JMP) so this is not an option for our current infrastructure.

Speed Test of JMP.jpg

Re: Speed of file opening/importing

Thanks for your post.

Are you asking for advice on choosing a file format, or are you requesting improvement in the performance of HDF import performance?
Do you have absolute control over the data your system writes out, or are there limited choices?
Are all the comparisons above reading files into JMP?
I'm having trouble reading/interpreting the legends. Can you explain the various formats you've compared?
You indicated a preference for HDF. What advantages does the HDF format give you?
Can you provide any sample HDF files for us to do comparison performance testing here?

Re: Speed of file opening/importing

Wow, we are slow in responding. I work with Kevin on this project, but I didn't realize he'd posted here for help..

We do have control of the output format. We want both efficient file size (for file transfer efficiency) and speed when opening in JMP. The only open-source compressed format we've seen is HDF5, but the JMP HDF5 import seems very limited. We've only found one way to create an HDF5 file that JMP can import directly into a datatable (most HDF5 file structures trigger the "Wizard"), but that format is slow to import. And opening with the Python JMP interface doesn't buy us anything because it still appears to save as CSV, then import that file in JMP.

So...I'd love help get one of these to work (or more!):

Example Python code that generates an HDF5 file that can import directly as a table, and with performance at least as fast as a CSV import where the Column names and datatypes are hardcoded (i.e. no column/row scan). Full disclosure: I asked for this in another thread and @briancorcoran said he didn't have it: https://community.jmp.com/t5/Discussions/Automate-opening-an-hdf5-file/m-p/266708#M51947
A way to define a custom file type (file extension), and the JSL that runs to import that file type. Then I can create one ZIP file, containing both the data in a CSV file, and a separate metadata file with column names and types, which enables the loading to be efficient.
An alternative file type that is dense for network transfer, and performant when opening in JMP. Parquet?

Thanks,

- Bob

Re: Speed of file opening/importing

I realized I have one more idea that I assumed you won't do, but I should mention...