Concatenated Data Disappears when Appended to First Data Table

Question

MichelleG · Answer

At a glance this looks fine to me. There are no funny arguments in the Concatenate function that would be causing the behavior you describe. Have you tried stepping through your loop manually? (That is, evaluate i=1, then the loop contents, then i++, then the loop contents again, etc.) There is no code after your loop, so things must be going wrong on some particular csv file.

mwarren · Answer

I am running a script that opens a single data table initially. It then runs a For loop to open up subsequent data tables from CSVs and concatenate them to the first data table. While I am watching the script run, it works properly and adds all of the rows to the first data table, however at the end of the script it deletes many of the later columns and ends up with only 519 rows. I cannot figure out why the rows after this are getting deleted. Any help would be fantastic! See Code Below:

//Initiates data table to hold all the data
data = New Table( "3DMark_All Loops" );
//Choose folder containing all .zip files
directory = Pick Directory();
subDir = directory || "All Loops/";
Create Directory(subDir); //create a new sub directory
TheFileList = Files In Directory(directory); //zip files
nFiles = N Items(TheFileList); //get number of zip files in directory

For( i = 1, i < 21, i++,

//open the zip folder and create a list of the files inside
za = Open( directory || zipFolder, zip );
dirlist = za << dir;
Show( dirlist ); // see all contents of zipped folder;

//choose the file from the file list called "Monitoring.csv" and save it in the subdirectory "All Loops" as the Loop Number
For(j = 1, j <= N Items( dirlist ), j++,
file = dirlist[j];

If( Left(file, 3) == "Mon",
text = za << read( file );
current = Save Text File( char(subDir) || char(i) || ".csv", text );
);
);

//Open the .csv file that was just created in the subdirectory
dt_temp = Open(current);

//Select Column 1 and Convert Text to Columns using a ; delimiter
Column(dt_temp, 1) << Set Selected;

dt_temp << Text To Columns(
delimiter( ";" ),
columns(1)
);

//Delete column 1 (which is empty)
dt_temp << Delete Columns(1);

//Create a new column called Loop Number
New Column( "Loop Number", Numeric, Continuous );

//Fill in the loop number for all cells of this column
colLengthTemp = N Rows(dt_temp);
LoopNumber = {};
For (s = 1, s <= colLengthTemp, s++,
LoopNumber[s] = i;
);

:"Loop Number" << Set Values(LoopNumber);

//Concatenate this loops data to the master data table
data << Concatenate( dt_temp, Append to first table );

//Close this temporary data table without saving
Close( dt_temp, nosave );

Wait(.001);
);