Make IMF function more efficient.I have thousands of file like the attached example (.CSV). Time information is in raw 102 and my measurement data start at raw 242.
Currently I am doing Importing file 2 times, one for Time information and one for test data. After that I use update function to import time information to test data table.
This logic was working with small set of files but it is not efficient on large number of files as IMF for time information is pulling all data after raw 102.
Is there any way I can set last data line in IMF function for time information import? So I don't import extra 141 rows/file.
Basic Data Analysis and ModelingRe: Make IMF function more efficient.To my knowledge you cannot control that with Multiple File Import. Have you tried importing all the data and then cleaning the resulting table in JMP? Or optionally if you are just trying to export few lines from the .csv, you could do with with JSL and Load Text File
Re: Make IMF function more efficient.Hi Jarmo,
I used JSL and below is the script I came up with. It is working fine but now my data set is getting bigger with more testing and this logic is not efficient (particularly importing Time table as I don't know how to only import one row).
Names Default To Here( 1 );
dir=Pick Directory( "Select Data Directory");
// Import multiple files
Time=Multiple File Import(
<<SET folder="">= 1, i--,
If( (Col N Missing( Column( i ) ) / N Row()) == 1,
Data Table( dt1 ) << delete columns( i )
)
);
For( i = N Col( dt2 ), i >= 1, i--,
If( (Col N Missing( Column( i ) ) / N Row()) == 1,
Data Table( dt2 ) << delete columns( i )
)
);
Data Table( dt1 ) :File Name<< Set Display Width( 300 );
Data Table( "Time" ):Column 3 << Set Name( "Test Time" );
// Update data table
Data Table( "Test Data" ) << Update(
With( Data Table( "Time" ) ),
Match Columns( :File Name = :File Name ),
Add Columns from Update Table( :Test Time )
);
// Close Data Table: Time
Close( Data Table( "Time" ), NoSave );
</SET>Re: Make IMF function more efficient.How many files do you have? One import should be enough and then you can just clean your data accordingly within JMP.
Re: Make IMF function more efficient.Currently, I have around 20,000 files, and each day, 1,000 to 1,500 new files are added. The script takes about 7 minutes and 30 seconds to run. Since the time table has unnecessary rows, it ends up importing 2.8 million rows before cleaning them up.
Re: Make IMF function more efficient.Are you just importing the new files or the whole folder with increasing amount of files?
