[resolved] Can we process Files From AWS S3 bucket without storing locally?

[Anonymous]

Hi All,
Please could the experts let me know if when we connect to the AWS S3 bucket to process the files from it are they stored somewhere locally on the disk on which the job is run or is there any way to process the file without storing locally.
I am using the below components in the Job and the tS3Get component has the File option where we have to give the local path and the tFileInputDelimited to pick up from that location which we have given in the tS3Get component.
tS3Connection-->tS3List-->tS3Get-->tFileInputDelimited-->tMap-->tOracleOutput 
Thanks

[Anonymous]

Hi,
So far, there is no support for handling data by air. You have to get your files stored in S3 into local disk first then reading them.
Best regards
Sabrina

[Anonymous]

Thanks Sabrina. Appreciate you response. Please could you also let me know if there are any components like tS3Input or tS3Output which can be used for this purpose of processing the files directly from the AWS S3 bucket without storing them to a local directory.
Thanks

[Anonymous]

Hi,
Could you please take a look at a jira issue: https://jira.talendforge.org/browse/TBD-929 to see is it what are you looking for?
Best regards
Sabrina

[Anonymous]

Hi,
Yes , The component which I am looking for is probably the tS3Input which I am guessing can be used instead of the tFileInputDelimited when reading the files from AWS S3. Does it have the same functionalities like the tFileInputDelimited component?
Thanks

[Anonymous]

Hi,
So far, this feature jira issue is in Progress. There is no component reference about this component. We will keep you posted as long as the new component is available in Talend studio.

Best regards
Sabrina

[Anonymous]

Thanks Sabrina. Appreciate the quick response.
Thanks

[Anonymous]

Hi Sabrina,

In the link you had shared earlier, I can see that this issue is closed. So, now is there ts3input component in TOS to get files directly from s3?

Thanks,

Alekhya.

[Anonymous]

Hello @alekhyak 

We have supported for Talend MapReduce with S3 on Amazon EMR.

For more information, please have a look at this tS3Input component reference:https://help.talend.com/reader/Tr~X7jq25879O8pZsAQ1mg/Fc8uZPrpxI8fSCmmnK5qSA

Note: it is only for users who have subscribed to Talend Data Fabric or to any Talend product with Big Data but it is not applicable to Talend Open Studio for Big Data users.

Best regards

Sabrina

[Anonymous]

How to extract data from csv file using ts3input or do we have any other way to read the data from csv file available in s3 bucket ?