I would like for JMP to be able to open Apache Parquet files and handle them in the same way that HDF5, JSON etc...can be handled. Thanks a million.
Aw man bummer. For what its worth, I think this is important! Parquet files are really efficient in terms of storage and read times, its particularly nice when you have to transfer data from another platform into JMP or out of JMP into another platform.
Some extra information for the curious:
https://www.databricks.com/glossary/what-is-parquet#:~:text=What%20is%20Parquet%3F,handle%20complex%20data%20in%20bulk.
If JMP really wants to do big data properly it should be able to read parquet/feather files and import the directly with appropriate compression and column definitions. There's not enough memory to open a file, then compress. And this issue will only get worse.
100% agree with John's comments above. JMP should be enabled to read parquet and feather directly into a JMP table.
I also would really like the capability to read directly from a parquet file in to a JMP table. I work regularly with 2GB CSVs (130K rows, 2.5K columns) and have a fairly recent i7 with 64GB RAM laptop. Opening one of these files takes around 5 minutes for the data scan and then another 4 and a half minutes to import the data. This is almost 10 minutes of my time just waiting for JMP to open the file.
Our data system can produce a parquet instead of a csv and the same file is less than 200MB and have encoded the correct data types already in it. When I open one of this parquet files with a single line of code (read_parquet) in pandas it can load it to a dataframe in memory in less than 30 seconds. Lately I am very hesitant to open the files in JMP even if this would be my preference. I can open the file in pandas, do a summary get the info I need in half the time it takes to just open it in JMP.
Having this capability would really be beneficial to people out there doing analysis with bigger files, hope it can be enabled in the close future.
Thank you for the additional comments. We will investigate further.
I made an account just to chime in and say the lack of this feature is currently driving users away from jmp in our organization since it is unable to open these files.
Hi all, I have recently developed and put it into the File Exchange -> Add-ins section this "Apache Parquet file importer" add-in: Apache Parquet file importer
It imports interactively .parquet file(s) into corresponding JMP table(s): it works both for a single file and for multiple files (all stores in the same folder)
It requires JMP V17 and a supported Python installation with PyArrow package (and its dependencies).
That's fantastic! Thanks Massimo :thumbs_up:
Hi @MassimoMartucci,
Excellent add-in...this will be super helpful to our team.
The single file uploader worked, but the multiple file button adds the below error to the log...
JMP17
Python:3.9.7
/*:
// New data table// → Data Table( "Untitled" )New Table( "Untitled" );
Invalid argument in Pick File: Directory "c:\tmp\parquet_multiple_files" not found in access or evaluation of 'Pick Directory' , Pick Directory/*###*/("Select directory with Apache Parquet input files","c:\tmp\parquet_multiple_files", Show Files( 1 ))
at line 10 in ~\SAS\JMP\Addins\com.jmp.martucci.parquet\Apache Parquet file importer - multiple files - interactive.jsl
Thank you!
Hi, that issue should be solved now, I am not pointing to that folder anymore. I have re-uploaded add-in, would you give a try? Massimo
@MassimoMartucci works great, thank you for the swift response!
Hi @MassimoMartucci
Is it possible to open parquet files, folders with the Open or some other function Open Function
No, in the current version of software (V17) it is not possible.
@Ryan_Gilmore Please un-archive this request. These file types are much more efficient to use vs csv and should be an integrated compatibility vs an add in. Thanks for considering this request.
Steve
@kevin_lennon , with JMP 18, it is now possible to open Apache Parquet files via the enhanced Python integration. Please see:
https://community.jmp.com/t5/JMPer-Cable/JMP-is-Pythonic-Enhanced-Python-Integration-in-JMP-18/ba-p/723997
(Marking this request as delivered).
Please see my additions to https://community.jmp.com/t5/JMP-Add-Ins/Apache-Parquet-file-importer/ta-p/637823#U637823 where I detail an updated script taking advantage of JMP 18's capabilities.
@MassimoMartucci The Apache Parquet file importer link is not working for me (says "Access Denied"). Can you re-post it, or share somehow?
Thank You very much!
@ts2 That add-in is not updated anymore. Please refer to the most recent add-in in the JMP Marketplace: 'Can Open-er' File Importer
@MassimoMartucci Thanks for responding.
Unfortunately I am required to use JMP 17.2, so the Can Open-er won't work for me.
No worries, I completely understand not supporting or distributing deprecated code.
