topic Re: Issue with loading huge file to Amazon S3 in Talend Studio

Issue with loading huge file to Amazon S3

Karuetl — Sat, 16 Nov 2024 08:35:07 GMT

Hello,

I am loading data from teradata to delimited file to S3 AWS . It failed with below error. Kindly help me out how to load big files

tS3Put_1 - Your proposed upload exceeds the maximum allowed size (Service: Amazon S3; Status Code: 400; Error Code: EntityTooLarge; Request ID: D9E800BC064FE9B3)

Re: Issue with loading huge file to Amazon S3

TRF — Thu, 08 Mar 2018 23:01:13 GMT

A single put operation is limited to 5 GB.
If you need to load large objects up to 5 TB, you need to use multipart upload API as explained in documentation ( https://docs.aws.amazon.com/AmazonS3/latest/dev/UploadingObjects.html).
This option is accessible from tS3Put Advanced settings using the Part Size field.
You have to set the part size according to your file size and the following contraints (issued from documentation):
- maximum number of parts per upload 10,000
- part size 5 MB to 5 GB, last part can be < 5 MB
Hope this helps.

Re: Issue with loading huge file to Amazon S3

Karuetl — Fri, 09 Mar 2018 13:46:38 GMT

Talend version is 6.2 , so tS3Put doesnt have feature for multipart upload

Re: Issue with loading huge file to Amazon S3

TRF — Fri, 09 Mar 2018 22:30:45 GMT

You're right, it seems you have to upgrade to 6.3.1 at least to have this functionality.

Re: Issue with loading huge file to Amazon S3

Anonymous — Tue, 14 Jan 2020 14:17:47 GMT

Hello,

We are attempting to use the s3Put component to load a large file. We have Talend 6.3 and have tried a large variety of part sizes in an attempt to get it to work; however, we keep failing with the following error:

The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchUpload; Request ID: [our request id is here])

Can anyone help on this error?