tFileArchive gzip files can't be read in S3

MattE — Sat, 16 Nov 2024 00:18:33 GMT

Hi,

Long shot.

I've got a job which pulls from an API. If i write the JSON response to a file using tFileOutputDelimited and upload to an S3 bucket via tS3Put, i can read the resulting data in Redshift via Spectrum. It works, however, i need to compress these files to save space.

If i add an extra step and use tFileArchive on the JSON file and upload it to S3, Redshift is unable to read the data with the error below.

SQL Error [500310] [XX000]: [Amazon](500310) Invalid operation: Spectrum Scan Error

Details: Spectrum Scan Error code: 15001 context: Error while reading next ION/JSON value: IERR_INVALID_TOKEN. In file 'https://s3.eu-west-1.amazonaw..... [Line 1, Pos 1] query: 2178747 location: dory_util.cpp:1081

process: fetchtask_thread [pid=27919]

As mentioned,

the raw JSON file can be uploaded and read

If i use 7ZIP to create a .gz archive fromt his file and upload via tS3Put the data can be read

The problem only occurs when i introduce tFileArchive into the mix. I've tried both tar.gz and gzip as the archive format in tFileArchive.

Re: tFileArchive gzip files can't be read in S3

MattE — Tue, 06 Apr 2021 14:19:17 GMT

Ok looks like the issue was actually in the config for tS3Put. In the key section i wasn't adding ".gz" onto the end so S3 was seeing it as .json but then couldn't read the gzipped contents. Now fixed

Re: tFileArchive gzip files can't be read in S3

Anonymous — Thu, 08 Apr 2021 08:59:22 GMT

Hello,

Thanks for letting us know this issue has been resolved by yourself.

Best regards

Sabrina

topic Re: tFileArchive gzip files can't be read in S3 in Talend Studio

tFileArchive gzip files can't be read in S3

Re: tFileArchive gzip files can't be read in S3

Re: tFileArchive gzip files can't be read in S3