344079_en-US

liliconcom

How to read text data with a large file size

When reading about 120GB of text data, there is not enough memory to read it.
Would you please let me know if there is any way to deal with it, such as splitting and reading.

Data TableRe: ファイルサイズの大きいテキストデータの読み込み方法について

LoadTextFile can do it. ( @audrey_shull there are two versions in the scripting index, one is more complete than the other.)

 

Untested code:

blob = loadtextfile("reallyBig.txt", blob( readLength( 1e9 ), readOffsetFromBegin( n * 1e9 ) ) );

It gives the data back as a BLOB, not as text. Blobs can be > 4GB, but JSL strings are limited to < 2GB. 1e9 is almost 1GB. "n" could be 0, 1, 2, ... to step through the file.

The reason it returns a BLOB: multi-byte characters may be split across the 1e9 boundary. If there are only single-byte characters, great! You can probably convert the blob to a string using (again, untested code)

jslString = blobToChar( blob, "utf-8" );

Let us know how this works for you, thanks!