你好，
我需要將 Parquet 格式的大型資料集（可能有多個 GB）有效率地匯入 JMP。我在這裡看到了 CSV 導入的指導視頻，並使用了 pd.read_parquet 的相同方法。在這種情況下，我擔心的是導入效率會非常低，因為表需要作為 Pandas df 物件和 JMP dt 佔用記憶體。是否有更好的方法可能更適合具有中等規格的 PC？
多謝。
首先也是最重要的 - 您將需要足夠的 RAM，至少是二進位檔案大小的 2 倍，可能是 3 倍或更多，只是為了在記憶體中直接從 Parquet 到 JMP 資料表執行此操作。使用 Python 整合和 PyArrow Python 套件。
import jmp
from jmputils import jpip
jpip('install','pyarrow')
這是 JMP 18 的範例腳本，它直接遍歷 parquet 檔案並使用資料從 Python 建立資料表，所有資料都在記憶體中。
在 JMP 18 中，您需要從 Python 一次遍歷資料和資料表列中的每一列。您可以直接遍歷 Parquet 模式和數據，也可以將 parquet 表轉換為 pandas 資料框。在 $SAMPLE_SCRIPTS/Python/dt2pandas2dt.jsl 和 .py 腳本中可以找到從 pandas 資料框轉換為 jmp.DataTable 物件的範例。
# parquet.py
# Author: Paul R. Nelson
# JMP Statistical Discovery LLC
#
# Description:
# Directly read a pyarrow table and create a JMP datatable.
# The data for this file comes from a repository that is Apache licensed.
# https://github.com/Teradata/kylo/blob/master/samples/sample-data/parquet/userdata1.parquet
#
import jmp
import pyarrow.parquet as pq
ver = jmp.__version__.split('.')
if(ver[0] == '0' and ver[1] < '5'):
print('This version requires JMP 18.0 EA-5 or newer.')
pq_Table = pq.read_table('/Your_path/to/userdata1.parquet')
#print(pq_Table)
print(f'Table Rows: {len(pq_Table)}' )
print(f'Table Shape: {pq_Table.shape}')
print(f'Table Schema:\n{pq_Table.schema}')
print(f'Column Names: {pq_Table.column_names}')
#print(pq_Table.columns)
print(f'Num Columns: {pq_Table.num_columns}')
dt = jmp.DataTable('From Parquet', pq_Table.num_rows)
dt.new_column('first_name', jmp.DataType.Character)
dt.new_column('last_name', jmp.DataType.Character)
dt.new_column('Salary')
# set column properites (widths)
jmp.run_jsl('''
// Change column display width: last_name
Data Table( "From Parquet" ):last_name << Set Display Width( 105 );
// Change column display width: Salary
Data Table( "From Parquet" ):Salary << Set Display Width( 90 );
''')
#Create a data table column from a Python list
dt[0] = pq_Table.column(2).to_pylist()
dt[1] = pq_Table.column(3).to_pylist()
dt[2] = pq_Table.column(10).to_pylist()
# ===================================================================================
# Copyright © 2025 JMP Statistical Discovery LLC, Cary, NC, USA. All rights reserved.
#
# JMP STATISTICAL DISCOVERY LLC ("JMP") PERMITS THE USE OF THIS COMPUTER SOFTWARE
# CODE ("CODE") ON AN AS-IS BASIS AND AUTHORIZES YOU TO USE THE CODE SUBJECT TO
# THE TERMS LISTED HEREIN. BY USING THE CODE, YOU AGREE TO THESE TERMS. YOUR USE
# OF THE CODE IS AT YOUR OWN RISK. JMP MAKES NO REPRESENTATION OR WARRANTY,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO, WARRANTIES OF
# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT, AND TITLE,
# WITH RESPECT TO THE CODE.
#
# You may use the Code solely as part of a software product you currently have
# licensed from JMP, JMP's parent company, SAS Institute Inc. ("SAS US") or one
# of SAS' subsidiaries (together with SAS US, "SAS") or authorized agents (the
# "Software"), and not for any other purpose. The Code is designed to either
# correct an error in the Software or to add functionality to the Software but
# has not necessarily been tested. Accordingly, JMP makes no representation or
# warranty that the Code (1) will operate error-free or (2) will not contain any
# viruses or other applications or executables (including, without limitation,
# any "trap doors," "worms" and "time bombs") that will degrade or infect any
# software product that you license from JMP or any other software or your
# network or systems. JMP is under no obligation to maintain, support, or
# continue to distribute the Code.
#
# Neither JMP nor its licensors shall be liable to you or any third party for any
# general, special, direct, indirect, consequential, incidental, or other damages
# whatsoever arising out of or related to your use or inability to use the Code,
# even if JMP has been advised of the possibility of such damages. Except as
# otherwise provided above, the Code is governed by the same agreement that
# governs the Software. If you do not have an existing agreement with JMP or SAS
# governing the Software, you may not use the Code.
#
# US export laws and regulations apply to the Code and any other JMP-provided
# technology ("Controlled Material"). The Controlled Material originates from the
# United States. Customer agrees to comply with these and other applicable export
# and import laws and regulations, except as prohibited or penalized by law
# ("Trade Law"). Customer warrants that Customer and its users are not: (a)
# prohibited by Trade Law from accessing Controlled Material without US
# government approval; (b) located in or under control of any country or other
# territory subject to general export or trade embargo under Trade Law; or (c)
# engaged in any of the following end-uses: nuclear, chemical or biological
# weapons; nuclear facilities not under International Atomic Energy Agency
# safeguards; missiles or unmanned aerial vehicles capable of long-range use or
# weapons delivery, military training or assistance, military or intelligence
# end-use in Russia or in any country in Country Group D:5 of the United States
# Export Administration Regulations; deep water, Arctic offshore or shale oil or
# gas exploration involving Russia or Russian companies, or Russian energy export
# pipelines. Customer will not import or use any data within the System that is
# subject to the US International Traffic Arms Regulations. United States export
# classification information for JMP software and its affiliates is available at
# jmp.com/export.
#
# JMP and all other JMP Statistical Discovery LLC product or service names are
# registered trademarks or trademarks of SAS Institute Inc. in the USA and other
# countries. ® indicates USA registration. Other brand and product names are
# registered trademarks or trademarks of their respective companies.
#
# ===================================================================================
對於 JMP 19 EA-4+ 來說，情況要好得多。
import jmp
import numpy as np
import pandas as pd
import pyarrow.parquet as pq
#parque sample files from
# https://github.com/Teradata/kylo/tree/master/samples/sample-data/parquet
pq_Table = pq.read_table('/Your_path/to/userdata1.parquet')
pd_table = pq_Table.to_pandas()
#create table in memory from pandas datadata frame.
dt = jmp.from_dataframe(pd_table)
#Same Disclaimer applies.
Python 方法可能比下面的方法更好。但這確實有效，至少對於玩具拼花檔來說是如此。
有一個 Apache Drill 專案可以讀取 parquet 檔案；它想要成為 JDBC（不幸的是 java，而不是 ODBC），而且它的 ODBC 驅動程式（如果仍然存在）可能不是免費的。但它有一個基於 REST 的 API，而 JSL 可以做到這一點。
這篇文章的第 1、2、3 步指明了正確的方向（感謝@robertspierre ） 。不確定如何在 win 上提取檔案； Linux 可以做到這一點，一些視窗工具可能也可以做到這一點。我使用了https://drill.apache.org/download/並選擇「Drill for Hadoop 3 and non-Hadoop environment, direct download」。我還從https://learn.microsoft.com/en-us/java/openjdk/download獲得了 microsoft-jdk-21.0.6-windows-x64.msi。
我將展開的 apache-drill-1.21.2 目錄（來自 tar.gz）留在桌面上並像這樣啟動它：
cd 進入 apache-drill-1.21.2 然後運行 bin/drill-embedded.bat。然後，我對提供的鑲木地板檔案進行了查詢，您不必這樣做，因為...啟動 JMP 並執行此腳本：
fields = Associative Array();
fields["queryType"] = "SQL";
fields["query"] = "SELECT * FROM `dfs`.`C:\Users\c\Desktop\apache-drill-1.21.2\sample-data\nation.parquet`";
s = New HTTP Request( URL( "localhost:8047/query.json" ), Method( "POST" ), JSON( Fields( fields ) ), Headers( {"Accept: application/json"} ) ) <<SEND></SEND>
<P>並得到此表：</P>
<BR />
<P><SPAN class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Craige_Hales_1-1744504821964.png" style="width: 999px;"><SPAN class="lia-inline-image-display-wrapper" image-alt="Craige_Hales_1-1744504821964.png" style="width: 999px;"><SPAN class="lia-inline-image-display-wrapper" image-alt="Craige_Hales_1-1744504821964.png" style="width: 999px;"><SPAN class="lia-inline-image-display-wrapper" image-alt="Craige_Hales_1-1744504821964.png" style="width: 999px;"><SPAN class="lia-inline-image-display-wrapper" image-alt="Craige_Hales_1-1744504821964.png" style="width: 999px;"><SPAN class="lia-inline-image-display-wrapper" image-alt="Craige_Hales_1-1744504821964.png" style="width: 999px;"><span class="lia-inline-image-display-wrapper" image-alt="Craige_Hales_1-1744504821964.png" style="width: 999px;"><img src="https://us.v-cdn.net/6038925/uploads/images/74790iDA8A4CC22DE4CBA3/Craige_Hales_1-1744504821964.png" role="button" title="Craige_Hales_1-1744504821964.png" alt="Craige_Hales_1-1744504821964.png" /></span></SPAN></SPAN></SPAN></SPAN></SPAN></SPAN></P>
<BR />
<P>當您啟動上述鑽取過程時，它將啟動一個處理來自 JSL 的 REST api 請求的 Web 伺服器；它還有一個 Web 介面，但我認為它對於讀取 Parquet 檔案並沒有什麼用。</P>
<P><SPAN class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Craige_Hales_2-1744505945837.png" style="width: 999px;"><SPAN class="lia-inline-image-display-wrapper" image-alt="Craige_Hales_2-1744505945837.png" style="width: 989px;"><SPAN class="lia-inline-image-display-wrapper" image-alt="Craige_Hales_2-1744505945837.png" style="width: 989px;"><SPAN class="lia-inline-image-display-wrapper" image-alt="Craige_Hales_2-1744505945837.png" style="width: 989px;"><SPAN class="lia-inline-image-display-wrapper" image-alt="Craige_Hales_2-1744505945837.png" style="width: 989px;"><SPAN class="lia-inline-image-display-wrapper" image-alt="Craige_Hales_2-1744505945837.png" style="width: 989px;"><span class="lia-inline-image-display-wrapper" image-alt="Craige_Hales_2-1744505945837.png" style="width: 989px;"><img src="https://us.v-cdn.net/6038925/uploads/images/74791i98CF0F2C6E15C248/Craige_Hales_2-1744505945837.png" role="button" title="Craige_Hales_2-1744505945837.png" alt="Craige_Hales_2-1744505945837.png" /></span></SPAN></SPAN></SPAN></SPAN></SPAN></SPAN></P>
<P><A href="https://drill.apache.org/docs/rest-api-introduction/#query" target="_blank" rel="nofollow noopener noreferrer">https://drill.apache.org/docs/rest-api-introduction/#query 可能對 api 有幫助。</A></P>
<BR />
<P>在你對此進行過多思考之前，先閱讀一些有趣的內容：</P>
<P><A href="https://www.starburst.io/blog/the-death-of-apache-drill/" target="_blank" rel="noopener nofollow noreferrer">https://www.starburst.io/blog/the-death-of-apache-drill/</A></P>
<P><A href="https://stackoverflow.com/questions/59754457/mapr-driver-discontinued-for-apache-drill-what-now" target="_blank" rel="noopener nofollow noreferrer">https://stackoverflow.com/questions/59754457/mapr-driver-discontinued-for-apache-drill-what-now</A></P>
<P>但仍有活動：</P>
<P><A href="https://github.com/apache/drill" target="_blank" rel="noopener nofollow noreferrer">https://github.com/apache/drill</A></P>
<BR />我可以在哪裡試用 JMP EA 版本？我的 Parquets 有太多列，無法明確定義每一列，因此使用 from_dataframe() 非常有吸引力。
聯絡 JMP 銷售人員，有一個早期採用者計劃。但我認為，如果您可以透過 MyJMP 入口網站存取 JMP 18，那麼您應該可以存取早期採用者版本。但也許還有額外的保密表格/步驟。
