你好，
我需要高效地将 Parquet 格式的大型数据集（可能高达数 GB）导入 JMP。我在这里观看了 CSV 导入的指导视频，并使用了 pd.read_parquet 函数。我担心的是，导入效率会非常低，因为这些表既需要作为 Pandas df 对象占用内存，又需要作为 JMP dt 对象占用内存。有没有更适合中等配置电脑的更好的方法？
多谢。
首先，也是最重要的一点：您需要足够的内存，至少是二进制文件大小的 2 倍，甚至 3 倍，才能在内存中直接从 Parquet 数据表传输到 JMP 数据表。请使用 Python 集成和 PyArrow Python 包。
import jmp
from jmputils import jpip
jpip('install','pyarrow')
这是 JMP 18 的示例脚本，它直接遍历 parquet 文件并使用数据从 Python 构建数据表，所有数据都在内存中。
在 JMP 18 中，您需要从 Python 中逐列遍历数据和数据表的每一列。您可以直接遍历 Parquet 模式和数据，也可以将 Parquet 表转换为 Pandas 数据框。在 $SAMPLE_SCRIPTS/Python/dt2pandas2dt.jsl 和 .py 脚本中可以找到从 Pandas 数据框转换为 jmp.DataTable 对象的示例。
# parquet.py
# Author: Paul R. Nelson
# JMP Statistical Discovery LLC
#
# Description:
# Directly read a pyarrow table and create a JMP datatable.
# The data for this file comes from a repository that is Apache licensed.
# https://github.com/Teradata/kylo/blob/master/samples/sample-data/parquet/userdata1.parquet
#
import jmp
import pyarrow.parquet as pq
ver = jmp.__version__.split('.')
if(ver[0] == '0' and ver[1] < '5'):
print('This version requires JMP 18.0 EA-5 or newer.')
pq_Table = pq.read_table('/Your_path/to/userdata1.parquet')
#print(pq_Table)
print(f'Table Rows: {len(pq_Table)}' )
print(f'Table Shape: {pq_Table.shape}')
print(f'Table Schema:\n{pq_Table.schema}')
print(f'Column Names: {pq_Table.column_names}')
#print(pq_Table.columns)
print(f'Num Columns: {pq_Table.num_columns}')
dt = jmp.DataTable('From Parquet', pq_Table.num_rows)
dt.new_column('first_name', jmp.DataType.Character)
dt.new_column('last_name', jmp.DataType.Character)
dt.new_column('Salary')
# set column properites (widths)
jmp.run_jsl('''
// Change column display width: last_name
Data Table( "From Parquet" ):last_name << Set Display Width( 105 );
// Change column display width: Salary
Data Table( "From Parquet" ):Salary << Set Display Width( 90 );
''')
#Create a data table column from a Python list
dt[0] = pq_Table.column(2).to_pylist()
dt[1] = pq_Table.column(3).to_pylist()
dt[2] = pq_Table.column(10).to_pylist()
# ===================================================================================
# Copyright © 2025 JMP Statistical Discovery LLC, Cary, NC, USA. All rights reserved.
#
# JMP STATISTICAL DISCOVERY LLC ("JMP") PERMITS THE USE OF THIS COMPUTER SOFTWARE
# CODE ("CODE") ON AN AS-IS BASIS AND AUTHORIZES YOU TO USE THE CODE SUBJECT TO
# THE TERMS LISTED HEREIN. BY USING THE CODE, YOU AGREE TO THESE TERMS. YOUR USE
# OF THE CODE IS AT YOUR OWN RISK. JMP MAKES NO REPRESENTATION OR WARRANTY,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO, WARRANTIES OF
# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT, AND TITLE,
# WITH RESPECT TO THE CODE.
#
# You may use the Code solely as part of a software product you currently have
# licensed from JMP, JMP's parent company, SAS Institute Inc. ("SAS US") or one
# of SAS' subsidiaries (together with SAS US, "SAS") or authorized agents (the
# "Software"), and not for any other purpose. The Code is designed to either
# correct an error in the Software or to add functionality to the Software but
# has not necessarily been tested. Accordingly, JMP makes no representation or
# warranty that the Code (1) will operate error-free or (2) will not contain any
# viruses or other applications or executables (including, without limitation,
# any "trap doors," "worms" and "time bombs") that will degrade or infect any
# software product that you license from JMP or any other software or your
# network or systems. JMP is under no obligation to maintain, support, or
# continue to distribute the Code.
#
# Neither JMP nor its licensors shall be liable to you or any third party for any
# general, special, direct, indirect, consequential, incidental, or other damages
# whatsoever arising out of or related to your use or inability to use the Code,
# even if JMP has been advised of the possibility of such damages. Except as
# otherwise provided above, the Code is governed by the same agreement that
# governs the Software. If you do not have an existing agreement with JMP or SAS
# governing the Software, you may not use the Code.
#
# US export laws and regulations apply to the Code and any other JMP-provided
# technology ("Controlled Material"). The Controlled Material originates from the
# United States. Customer agrees to comply with these and other applicable export
# and import laws and regulations, except as prohibited or penalized by law
# ("Trade Law"). Customer warrants that Customer and its users are not: (a)
# prohibited by Trade Law from accessing Controlled Material without US
# government approval; (b) located in or under control of any country or other
# territory subject to general export or trade embargo under Trade Law; or (c)
# engaged in any of the following end-uses: nuclear, chemical or biological
# weapons; nuclear facilities not under International Atomic Energy Agency
# safeguards; missiles or unmanned aerial vehicles capable of long-range use or
# weapons delivery, military training or assistance, military or intelligence
# end-use in Russia or in any country in Country Group D:5 of the United States
# Export Administration Regulations; deep water, Arctic offshore or shale oil or
# gas exploration involving Russia or Russian companies, or Russian energy export
# pipelines. Customer will not import or use any data within the System that is
# subject to the US International Traffic Arms Regulations. United States export
# classification information for JMP software and its affiliates is available at
# jmp.com/export.
#
# JMP and all other JMP Statistical Discovery LLC product or service names are
# registered trademarks or trademarks of SAS Institute Inc. in the USA and other
# countries. ® indicates USA registration. Other brand and product names are
# registered trademarks or trademarks of their respective companies.
#
# ===================================================================================
对于 JMP 19 EA-4+ 来说，情况要好得多。
import jmp
import numpy as np
import pandas as pd
import pyarrow.parquet as pq
#parque sample files from
# https://github.com/Teradata/kylo/tree/master/samples/sample-data/parquet
pq_Table = pq.read_table('/Your_path/to/userdata1.parquet')
pd_table = pq_Table.to_pandas()
#create table in memory from pandas datadata frame.
dt = jmp.from_dataframe(pd_table)
#Same Disclaimer applies.
Python 方法可能比下面的方法更好。但这确实有效，至少对于一个玩具 Parquet 文件来说是这样。
有一个 Apache Drill 项目可以读取 Parquet 文件；它想成为 JDBC（可惜是 Java，而不是 ODBC），而且它的 ODBC 驱动程序（如果还存在的话）可能不是免费的。但它有一个基于 REST 的 API，JSL 可以做到这一点。
这篇文章的步骤 1、2、3 指明了正确的方向（感谢 @ robertspierre） 。我不确定如何在 Windows 上提取文件；Linux 可以，一些 Windows 工具可能也可以。我使用了https://drill.apache.org/download/并选择了“适用于 Hadoop 3 和非 Hadoop 环境的 Drill，直接下载”。我还从https://learn.microsoft.com/en-us/java/openjdk/download下载了 microsoft-jdk-21.0.6-windows-x64.msi 文件。
我将展开的 apache-drill-1.21.2 目录（来自 tar.gz）留在桌面上并像这样启动它：
进入 apache-drill-1.21.2，然后运行 bin/drill-embedded.bat。然后，我针对提供的 parquet 文件进行了查询，其实你不需要这样做，因为……启动 JMP 并运行以下脚本：
fields = Associative Array();
fields["queryType"] = "SQL";
fields["query"] = "SELECT * FROM `dfs`.`C:\Users\c\Desktop\apache-drill-1.21.2\sample-data\nation.parquet`";
s = New HTTP Request( URL( "localhost:8047/query.json" ), Method( "POST" ), JSON( Fields( fields ) ), Headers( {"Accept: application/json"} ) ) <<SEND></SEND>
<P>并得到此表：</P>
<BR />
<P><SPAN class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Craige_Hales_1-1744504821964.png" style="width: 999px;"><SPAN class="lia-inline-image-display-wrapper" image-alt="Craige_Hales_1-1744504821964.png" style="width: 999px;"><SPAN class="lia-inline-image-display-wrapper" image-alt="Craige_Hales_1-1744504821964.png" style="width: 999px;"><SPAN class="lia-inline-image-display-wrapper" image-alt="Craige_Hales_1-1744504821964.png" style="width: 999px;"><SPAN class="lia-inline-image-display-wrapper" image-alt="Craige_Hales_1-1744504821964.png" style="width: 999px;"><SPAN class="lia-inline-image-display-wrapper" image-alt="Craige_Hales_1-1744504821964.png" style="width: 999px;"><span class="lia-inline-image-display-wrapper" image-alt="Craige_Hales_1-1744504821964.png" style="width: 999px;"><img src="https://us.v-cdn.net/6038925/uploads/images/74790iDA8A4CC22DE4CBA3/Craige_Hales_1-1744504821964.png" role="button" title="Craige_Hales_1-1744504821964.png" alt="Craige_Hales_1-1744504821964.png" /></span></SPAN></SPAN></SPAN></SPAN></SPAN></SPAN></P>
<BR />
<P>当您启动上面的钻取过程时，它会启动一个处理来自 JSL 的 REST api 请求的 Web 服务器；它还有一个 Web 界面，我认为这对于读取 parquet 文件来说并不有趣。</P>
<P><SPAN class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Craige_Hales_2-1744505945837.png" style="width: 999px;"><SPAN class="lia-inline-image-display-wrapper" image-alt="Craige_Hales_2-1744505945837.png" style="width: 989px;"><SPAN class="lia-inline-image-display-wrapper" image-alt="Craige_Hales_2-1744505945837.png" style="width: 989px;"><SPAN class="lia-inline-image-display-wrapper" image-alt="Craige_Hales_2-1744505945837.png" style="width: 989px;"><SPAN class="lia-inline-image-display-wrapper" image-alt="Craige_Hales_2-1744505945837.png" style="width: 989px;"><SPAN class="lia-inline-image-display-wrapper" image-alt="Craige_Hales_2-1744505945837.png" style="width: 989px;"><span class="lia-inline-image-display-wrapper" image-alt="Craige_Hales_2-1744505945837.png" style="width: 989px;"><img src="https://us.v-cdn.net/6038925/uploads/images/74791i98CF0F2C6E15C248/Craige_Hales_2-1744505945837.png" role="button" title="Craige_Hales_2-1744505945837.png" alt="Craige_Hales_2-1744505945837.png" /></span></SPAN></SPAN></SPAN></SPAN></SPAN></SPAN></P>
<P><A href="https://drill.apache.org/docs/rest-api-introduction/#query" target="_blank" rel="nofollow noopener noreferrer">https://drill.apache.org/docs/rest-api-introduction/#query 可能对 api 有帮助。</A></P>
<BR />
<P>在你对此进行过多思考之前，先阅读一些有趣的内容：</P>
<P><A href="https://www.starburst.io/blog/the-death-of-apache-drill/" target="_blank" rel="noopener nofollow noreferrer">https://www.starburst.io/blog/the-death-of-apache-drill/</A></P>
<P><A href="https://stackoverflow.com/questions/59754457/mapr-driver-discontinued-for-apache-drill-what-now" target="_blank" rel="noopener nofollow noreferrer">https://stackoverflow.com/questions/59754457/mapr-driver-discontinued-for-apache-drill-what-now</A></P>
<P>但仍然有活动：</P>
<P><A href="https://github.com/apache/drill" target="_blank" rel="noopener nofollow noreferrer">https://github.com/apache/drill</A></P>
<BR />我可以在哪里试用 JMP EA 版本？我的 Parquet 数据列太多，无法明确定义每一列，所以使用 from_dataframe() 非常诱人。
联系 JMP 销售人员，他们有一个早期采用者计划。不过，我认为如果您可以通过 MyJMP 门户访问 JMP 18，您应该可以访问早期采用者版本。不过，也许还需要填写一份额外的保密表格/步骤。
