topic Re: [resolved] Can we process Files From AWS S3 bucket without storing locally? in Talend Studio

[resolved] Can we process Files From AWS S3 bucket without storing locally?

Anonymous — Sat, 16 Nov 2024 11:27:35 GMT

Hi All,
Please could the experts let me know if when we connect to the AWS S3 bucket to process the files from it are they stored somewhere locally on the disk on which the job is run or is there any way to process the file without storing locally.
I am using the below components in the Job and the tS3Get component has the File option where we have to give the local path and the tFileInputDelimited to pick up from that location which we have given in the tS3Get component.
tS3Connection-->tS3List-->tS3Get-->tFileInputDelimited-->tMap-->tOracleOutput
Thanks

Re: [resolved] Can we process Files From AWS S3 bucket without storing locally?

Anonymous — Mon, 27 Oct 2014 08:53:05 GMT

Hi,
So far, there is no support for handling data by air. You have to get your files stored in S3 into local disk first then reading them.
Best regards
Sabrina

Re: [resolved] Can we process Files From AWS S3 bucket without storing locally?

Anonymous — Tue, 28 Oct 2014 18:57:26 GMT

Thanks Sabrina. Appreciate you response. Please could you also let me know if there are any components like tS3Input or tS3Output which can be used for this purpose of processing the files directly from the AWS S3 bucket without storing them to a local directory.
Thanks

Re: [resolved] Can we process Files From AWS S3 bucket without storing locally?

Anonymous — Wed, 29 Oct 2014 05:50:13 GMT

Hi,
Could you please take a look at a jira issue: https://jira.talendforge.org/browse/TBD-929 to see is it what are you looking for?
Best regards
Sabrina

Re: [resolved] Can we process Files From AWS S3 bucket without storing locally?

Anonymous — Wed, 29 Oct 2014 08:03:41 GMT

Hi,
Yes , The component which I am looking for is probably the tS3Input which I am guessing can be used instead of the tFileInputDelimited when reading the files from AWS S3. Does it have the same functionalities like the tFileInputDelimited component?
Thanks

Re: [resolved] Can we process Files From AWS S3 bucket without storing locally?

Anonymous — Wed, 29 Oct 2014 08:09:08 GMT

Hi,
So far, this feature jira issue is in Progress. There is no component reference about this component. We will keep you posted as long as the new component is available in Talend studio.

Best regards
Sabrina

Re: [resolved] Can we process Files From AWS S3 bucket without storing locally?

Anonymous — Wed, 29 Oct 2014 08:16:43 GMT

Thanks Sabrina. Appreciate the quick response.
Thanks

Re: [resolved] Can we process Files From AWS S3 bucket without storing locally?

Anonymous — Thu, 16 May 2019 11:52:56 GMT

Hi Sabrina,

In the link you had shared earlier, I can see that this issue is closed. So, now is there ts3input component in TOS to get files directly from s3?

Thanks,

Alekhya.

Re: [resolved] Can we process Files From AWS S3 bucket without storing locally?

Anonymous — Mon, 03 Jun 2019 10:37:32 GMT

Hello @alekhyak

We have supported for Talend MapReduce with S3 on Amazon EMR.

For more information, please have a look at this tS3Input component reference:https://help.talend.com/reader/Tr~X7jq25879O8pZsAQ1mg/Fc8uZPrpxI8fSCmmnK5qSA

Note: it is only for users who have subscribed to Talend Data Fabric or to any Talend product with Big Data but it is not applicable to Talend Open Studio for Big Data users.

Best regards

Sabrina

Re: [resolved] Can we process Files From AWS S3 bucket without storing locally?

Anonymous — Fri, 27 Mar 2020 12:35:38 GMT

How to extract data from csv file using ts3input or do we have any other way to read the data from csv file available in s3 bucket ?