topic Re: Copying large files on S3 in Talend Studio

Copying large files on S3

PhilHibbs — Sat, 16 Nov 2024 03:38:57 GMT

We are still using Talend 6.3 and the tS3Copy component fails when copying a file that is greater than 5GB.

The documentation says that 6.4 supports the "Part size (Mb)" advanced setting, but can anyone confirm that this is correct and not a documentation error? I have an open question ticket with Talend support about this, but hopefully I can get a real-world answer here from someone who is doing large file copies in either Talend 6.4 or 6.5..

Re: Copying large files on S3

Anonymous — Sat, 11 Jan 2020 15:18:32 GMT

Why don't you install v6.4 and test this feature? This parameter takes effect only when you copy an Amazon S3 object that is larger than 5 GB through multiple parts, I have never tried it yet, it makes sense if you can test this feature in your real scenario.

Re: Copying large files on S3

PhilHibbs — Mon, 13 Jan 2020 10:34:16 GMT

We don't have the 6.4 or 6.5 installer, I'm sure I could get it from Talend if I asked nicely.

However, I have resolved the issue by installing 7.2.1, creating a lookalike job, and copypasting the Java code directly into a tJava component in 6.3. Worked first time!

We will be migrating to 7.2.1 in the not too distant future anyway, so at that point we will drop in a working tS3Copy component in place of the tJava.

Re: Copying large files on S3

Anonymous — Tue, 14 Jan 2020 03:19:11 GMT

Great, glad to see you have resolved it.