Opening Parquet FilesI would like for JMP to be able to open Apache Parquet files and handle them in the same way that HDF5, JSON etc...can be handled. Thanks a million.
Data AccessRe: Opening Parquet FilesNo, in the current version of software (V17) it is not possible.
Re: Opening Parquet FilesHi @MassimoMartucci
Is it possible to open parquet files, folders with the Open or some other function Open Function
Re: Opening Parquet Files@MassimoMartucci works great, thank you for the swift response!
Re: Opening Parquet FilesHi, that issue should be solved now, I am not pointing to that folder anymore. I have re-uploaded add-in, would you give a try? Massimo
Re: Opening Parquet FilesHi @MassimoMartucci,
Excellent add-in...this will be super helpful to our team.
The single file uploader worked, but the multiple file button adds the below error to the log...
JMP17
Python:3.9.7
/*:
// New data table
// → Data Table( "Untitled" )
New Table( "Untitled" );
Invalid argument in Pick File: Directory "c:\tmp\parquet_multiple_files" not found in access or evaluation of 'Pick Directory' , Pick Directory/*###*/("Select directory with Apache Parquet input files",
"c:\tmp\parquet_multiple_files", Show Files( 1 ))
at line 10 in ~\SAS\JMP\Addins\com.jmp.martucci.parquet\Apache Parquet file importer - multiple files - interactive.jsl
Thank you!
Re: Opening Parquet FilesThat's fantastic! Thanks Massimo :thumbs_up:
Re: Opening Parquet FilesHi all, I have recently developed and put it into the File Exchange -> Add-ins section this "Apache Parquet file importer" add-in: Apache Parquet file importer
It imports interactively .parquet file(s) into corresponding JMP table(s): it works both for a single file and for multiple files (all stores in the same folder)
It requires JMP V17 and a supported Python installation with PyArrow package (and its dependencies).
Re: Opening Parquet FilesI made an account just to chime in and say the lack of this feature is currently driving users away from jmp in our organization since it is unable to open these files.
Re: Opening Parquet FilesThank you for the additional comments. We will investigate further.
Re: Opening Parquet FilesI also would really like the capability to read directly from a parquet file in to a JMP table. I work regularly with 2GB CSVs (130K rows, 2.5K columns) and have a fairly recent i7 with 64GB RAM laptop. Opening one of these files takes around 5 minutes for the data scan and then another 4 and a half minutes to import the data. This is almost 10 minutes of my time just waiting for JMP to open the file.
Our data system can produce a parquet instead of a csv and the same file is less than 200MB and have encoded the correct data types already in it. When I open one of this parquet files with a single line of code (read_parquet) in pandas it can load it to a dataframe in memory in less than 30 seconds. Lately I am very hesitant to open the files in JMP even if this would be my preference. I can open the file in pandas, do a summary get the info I need in half the time it takes to just open it in JMP.
Having this capability would really be beneficial to people out there doing analysis with bigger files, hope it can be enabled in the close future.
Re: Opening Parquet Files100% agree with John's comments above. JMP should be enabled to read parquet and feather directly into a JMP table.
Re: Opening Parquet FilesIf JMP really wants to do big data properly it should be able to read parquet/feather files and import the directly with appropriate compression and column definitions. There's not enough memory to open a file, then compress. And this issue will only get worse.
Re: Opening Parquet FilesSome extra information for the curious:
https://www.databricks.com/glossary/what-is-parquet#:~:text=What%20is%20Parquet%3F,handle%20complex%20data%20in%20bulk.
Re: Opening Parquet FilesAw man bummer. For what its worth, I think this is important!
Parquet files are really efficient in terms of storage and read times, its particularly nice when you have to transfer data from another platform into JMP or out of JMP into another platform.
Re: Opening Parquet Files - Status changed to: ArchivedWe are archiving this request. If this is still important please comment with additional details and we will reopen. Thank you!Re: Opening Parquet Files@Ryan_Gilmore Please un-archive this request. These file types are much more efficient to use vs csv and should be an integrated compatibility vs an add in. Thanks for considering this request.
Steve
Re: Opening Parquet Files - Status changed to: AcknowledgedRe: Opening Parquet Files@kevin_lennon , with JMP 18, it is now possible to open Apache Parquet files via the enhanced Python integration. Please see:
https://community.jmp.com/t5/JMPer-Cable/JMP-is-Pythonic-Enhanced-Python-Integration-in-JMP-18/ba-p/723997
Thank you!
(Marking this request as delivered).
Re: Opening Parquet Files - Status changed to: DeliveredRe: Opening Parquet FilesPlease see my additions to https://community.jmp.com/t5/JMP-Add-Ins/Apache-Parquet-file-importer/ta-p/637823#U637823 where I detail an updated script taking advantage of JMP 18's capabilities.
