How to collect data from external source via webservices using JSL ?

Question

Hi everyone,

For a school project, I have to collect data from dbpedia sources through webservice. I read in the scripting guide that I can connect to an external server with sockets. I tried it but the fact is that when I want to view the data retrieved from dbpedia with a Show command, only the header of the response is shown. When I start debugging the code, miraculously I can see in the log window only a part of the response !

Please, is there anyone who could help me solve the problem ?

I attach the JSL script I wrote here.

Best,

essai.jsl

Craige_Hales · Answer

I tried the loop, checking for the "ok" return code.  It takes about 2 seconds to get the data.  The content length may not be quite right from that URL; use it as an estimate, or ignore it, and watch the return codes.  when there is no more data, the returned value looks like this:

{"Recv", "CLOSED:  no data received.", Char To Blob( "", "ascii~hex" )}

item 2 in the list is "ok" until the remote server closes the connection.

(The server's load might change the 2 second result a lot, especially if others are fetching the same data at the same time.)

eduleboss · Answer

Thank you very much ! It worked now.

I did it by concatenating the "Blob To Char" of the received data in a While loop. In fact, I did not succeeded in concatenating the blobs directly : the script stucks at execution. Now, my script runs a bit longer but the expected result is shown.

Any trick for reducing the time it takes to compute the result is welcome.

Thank you again !

Craige_Hales · Answer

You are nearly there!  read the connection again and again until all the data is delivered.  the <<recv returns immediately with any available data.  The length parameter to recv limits the amount returned each time; it doesn't wait until it is all available.

(I ran your code, up through the recv, then ran the recv a second time and the data flowed some more.  you might get it all if you wait several seconds.  if you wait too long, the connection will close automatically.)

The header tells you how much data to expect with Content-Length:

Loop until you have that much or get a bad return code.

You can concatenate the blobs as you get them, then convert to text; if there are multi-byte characters in the binary blobs they might be split across recv commands and not decode correctly at the end of a blob.