I'm currently processing a lot of S3 files by downloading them with tS3Get, reading them as sequential files, writing out sequential files, and then uploading those files back to S3 with tS3Put.
Is there a better way - a way of reading the data directly from S3 without going to the local filesystem?
I have a similar process running in a Python script and not in Talend, and I recently rewrote it to use a module called s3fs that lets me treat s3 just like a filesystem so I can open the file, read it, and write directly back to S3.
