How to append data to a file without opening it?

Question

I need to append new data to a file on a weekly basis.

Is there a way for me to do so without opening/loading the file in JMP script?

dt = open("my.csv");
dt<< Concatenate(Data Table ("New data") , append to first table);
dt << Save("my.csv", Replace(1));

Thanks!

Craige_Hales · Answer

@Dawn_Lun -- If the CSV file will always be <1E4 rows, it will probably be better to load it as a JMP table, concatenate in JMP, then save the JMP table as CSV. (1E4 is a guess. 1E6 might be fine too.)

But if the CSV file will become large enough that it takes more than a few seconds to open it, you might want to try the concatenation approach. The concatenation approach: save the new (tiny) JMP file in a new, tiny CSV on disk (without headers). Use loadTextFile on the tiny CSV to get a text string. Then use SaveTextFile as @txnelson suggested to concatenate the tiny text CSV string to the bigger CSV file on disk with the 3rd parameter mode("append").

The potential problem with concatenation is corrupting the big CSV file in the future. If the tiny CSV file changes column ordering, the big CSV becomes scrambled, but without any warning.

Another approach: keep all the small daily CSV files in a directory and use Multiple File Import to open them when you need to see the entire set as a single file. MFI will stack the files if the headers match, and MFI is fast. Maybe use the date as part of the name of the tiny CSV files. Make the date in YYYYMMDD format so it sorts well.

If this data is important and this project will run for months, you need to make regular backups as well. The MFI approach means there is only one small CSV file to add to the backup on each cycle.

txnelson · Answer

I am not aware of being able to not open a file and append to it, if you are using JMP tables.  Here is as close as I think you can get using JMP tables.

names default to here(1);
new table("New Data",new column("name", character,set values({"BILLY","JIMMIE"})));
dt = open("$DOCUMENTS\Discussion Group\my.csv",private);
dt<< Concatenate(Data Table ("New data") , append to first table);
dt << Save("$DOCUMENTS\Discussion Group\my.csv", Replace(1));
close( dt, nosave );

However, if you write the JSL to convert your data table "New Table" into a character string, which is a comma separated values character string, you can us

Save Text File(path, character string, mode("append");

and it will append to your .csv file without having to open it