Fastest way to process S3 files?

PhilHibbs — Sat, 16 Nov 2024 02:14:28 GMT

I'm currently processing a lot of S3 files by downloading them with tS3Get, reading them as sequential files, writing out sequential files, and then uploading those files back to S3 with tS3Put.

Is there a better way - a way of reading the data directly from S3 without going to the local filesystem?

I have a similar process running in a Python script and not in Talend, and I recently rewrote it to use a module called s3fs that lets me treat s3 just like a filesystem so I can open the file, read it, and write directly back to S3.

Is there any way of doing this in Talend?

Re: Fastest way to process S3 files?

akumar2301 — Thu, 04 Jun 2020 14:02:47 GMT

did you try Athena ?

https://help.talend.com/reader/36FNvH4mND6uL3cKKzTU~w/2xTGZdvfowaLAOwQTtkNmA

It seems Athena can read S3 files.

topic Fastest way to process S3 files? in Talend Studio

Fastest way to process S3 files?

Re: Fastest way to process S3 files?