700918_en-US

liliconcom

How do I import only the latest files in JMP.

Hi All,

I currently have a script that imports multiple files from a directory using the IMF function with a date and name filter enabled. I import about 4000 files and then clean up the data tables into one data table that is saved. The directory keeps getting updated with new files and I would like to import the new files only so I can append it to the last date table.  

 

1. I was wondering if I can obtain the timestamp at which the import was done from the JMP logs and use that to update the data filter to run only from the last date I imported.   
2. I tried getting the names of the files from the File Name column, create a list  and compare it to the directory, no luck there. 

your help is much appreciated. 

tblRef = Multiple File Import(

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

) << Import Data;

 

tblRef << Set Name(" Data");

 

Automation and ScriptingData AccessData Blending and CleanupRe: How do I import only the latest files in JMP.

You could pick the maximum File Date from the MFI import result table and use that on the next import to limit the files?

Re: How do I import only the latest files in JMP.

How can I pick the maximum File Date? 

 

Is there a way to obtain the timestamp from the logs?

Re: How do I import only the latest files in JMP.

You should have File Date in your data table as you have Set Add File Date Column enabled. Then you can use for example Col Max()

Col Max(:File Date)

Re: How do I import only the latest files in JMP.

It still importing everything. This is what I did:

 

dt=open("C:\Users\");

 

LastImport = dt << Col Max(:"File Date");

maxDate = LastImport[1];

tblRef = Multiple File Import(

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

<

) << Import Data;

 

tblRef << Set Name("Data");

Re: How do I import only the latest files in JMP.

I think you will have to evaluate the datetime value inside multiple file imports settings. Below is an example. See how Eval(EvalExpr()) and Expr() are used ( Insert one expression into another using Eval Insert, Eval Expr, Parse, and Substitute )

Names Default To Here(1);

dt = New Table("12345678_Cycle_123456781_Cycle",
 Add Rows(6),
 Compress File When Saved(1),
 New Column("Cycle",
  Numeric,
  "Continuous",
  Format("Best", 12),
  Set Values([1, 2, 3, 4, 5, 6])
 ),
 New Column("File Date",
  Numeric,
  "Continuous",
  Format("yyyy-mm-ddThh:mm:ss", 20, 0),
  Input Format("yyyy-mm-ddThh:mm:ss", 0),
  Set Values(
   [3741613139.034, 3741613139.034, 3741613139.034, 3783350373.379, 3783350373.379,
   3783350373.379]
  )
 )
);

max_date = Col Max(Column(dt, "File Date"));

mfi = Eval(EvalExpr(Multiple File Import(
 <<SET folder="">),
 <<SET show="" hidden=""></SET></SET>

Re: How do I import only the latest files in JMP.

It worked. Thank you soooo much! 

Re: How do I import only the latest files in JMP.

You might have to add some time to your max time to avoid getting that max file date file couple of times. Depending on how often you get new files, adding a second could be enough

max_date = Col Max(Column(dt, "File Date")) + 1;

but most likely MFI also understands decimals, so you could add 0.01 for example (might be a good idea to test with small amounts of files)