Web API opening a zip file?Hi All,
I'm working with a customer getting data from a web API that happens to send the data down to the user in a zip format. I've managed to figure out how to change the default from XML in the archive to CSV, but I'm still running into the challenge of getting the data pulled into JMP instead of the downloads folder.
Data is coming from:
The web command will get the data downloaded nicely. But it would only be truly useful to the customer if the data goes from Web API into JMP. @Craige_Hales had some great posts to point me further down the Run Program() and cURL path:
However I'm still getting stuck. Perhaps I need to learn more about the Run Program read statement, or understand cURL options more.
/* generating a start time string for query injection */
starttime = "20160418T07:00-0000";
startdatetime = "&startdatetime=" || starttime;
If( starttime == "",startdatetime = "");
/* generating a end time string for query injection */
endtime = "";
enddatetime = "&enddatetime=" || endtime;
If( endtime == "",enddatetime = "");
/* FILENAME */
FILENAME = "CURR_LMP_GRP";
// CAISO (california energy pricing API)
oasis = "http://oasis.caiso.com/oasisapi/";
query = "GroupZip?groupid="|| FILENAME || startdatetime || "&version=1&resultformat=6";
url = oasis || query;
// download the data
web(url);
// the goal is to directly pull the item into JMP
/* RunProgram to pull the table into JMP (taken from a Craig Hales Post) */
If( HostIs("Windows"),
data = RunProgram(
executable( Convert File Path( "$desktop", windows ) || "curl.exe" ),
options( "-k -s --basic " || url ),
readfunction( "blob" )
);
, // else mac version
data = RunProgram(
executable( "/usr/bin/curl" ),
options( {"-k", "-s", "--basic", url} )
, readfunction( "blob" )
);
);
try( dt = open(data, "text"); error="", error=exception_msg; dt=empty() );
show(dt,error);
There is some potential extensibility of this problem for us as well. Think of the demo power of being able to scrape amazon's web api for customer reviews. Run Program() and a python script could do the trick.
--Nate
Re: Web API opening a zip file?Indeed, avoiding writing to disk before I have a JMP file was indeed my intent. Thanks for the insight!
Re: Web API opening a zip file?The first arg to open is normally a string...the *name* of a file. When it is a blob, open treats it as if it is the *contents* of a file, but there is no extension (.jmp, etc) so the 2nd arg to open is needed ("data", "script", etc). (Internally, open may write the blob to a temporary location, depending on what subsystem will do the open. The source script Open( "/000000002A75C3D0/"... tells me it was NOT written to disk, which is what you are hoping for.)Re: Web API opening a zip file?Thanks again Craige!
Below is a full JSL example with the suggested items implemented.
/* generating a start time string for query injection */
starttime = "20160418T07:00-0000";
startdatetime = "&startdatetime=" || starttime;
If( starttime == "",startdatetime = "");
/* generating a end time string for query injection */
endtime = "";
enddatetime = "&enddatetime=" || endtime;
If( endtime == "",enddatetime = "");
/* FILENAME */
FILENAME = "CURR_LMP_GRP";
// CAISO (california energy pricing API)
oasis = "http://oasis.caiso.com/oasisapi/";
query = "GroupZip?groupid="|| FILENAME || startdatetime || "&version=1&resultformat=6";
url = oasis || query;
//open the zip download which may have multiple files
za = open( url, "zip"); // 2nd zip important!
zaList = za << dir; //get a list of the files in the archive
for( i = n items(zaList), i > 0, i--,
blob = za<<READ></READ>
Fascinating that specifying the format(blob) enables the open command to work but a format(text) will not.
Re: Web API opening a zip file?try(deletefile("$temp/deleteme.zip"));
za = open("$temp/deleteme.zip","zip");
member = za<<WRITE></WRITE>
Open with a blob. use charToBlob() if you don't have a blob.
Re: Web API opening a zip file?The following will create a text file for opening, but I can't help but feel there should be a better way.
Save Text File("$TEMP/" || FILENAME || Char(i) || ".txt",txt);
open("$TEMP/" || FILENAME || Char(i) || ".txt");
--Nate
Re: Web API opening a zip file?Thanks Craig!
That got me almost all the way there. I could have sworn I tried zip as an argument in the open statement without luck (chances are improper case or string vs not), but with your example code I've been able to get the data actually read in!
//open the zip download
za = open( url, "zip"); // 2nd zip important!
zaList = za << dir; //get a list of the files in the archive
for( i = n items(zaList), i > 0, i--,
txt = za<<READ></READ>
My next step is to convert txt into a table.
Re: Web API opening a zip file?try something like this:
za = open( "http://someurl.that.downloads.a.zip", "zip"); // 2nd zip important!
show(za<
txt = za<
za is the ZipArchive object in the scripting index. the url might not end in .zip but must download a zip.
