How can we read file from S3 bucket,with out downloading into local?

[manodwhb]

How can we read file from S3 bucket,with out downloading into local  from Talend ?

[Anonymous]

Hi Manohar,

     You will have to do a hop at local server but you can use the below feature if your intention is to do data selection.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

[TRF]

You can use http get request to access s3 bucket's content and keep it in memory instead of writing into a local file.
Maybe you'll have to use multipart get if file length is too large.
Also you'll have to do this by yourself using a tJava, a tJavaFlex or a routine.

[manodwhb]

Thank You @nthampi and @TRF .

My requirement is like below.

I have few files in S3 bucket like below .

1st the main file and which size is 1 GB,2nd file is kind of lookup the side of file is 500 MB and I do have other lookup file,size is 60 GB.

Now I need to extract the data from those files without getting into local and do the lookups and get the required data and load it into Amazon RDS(which PostgreSQL).

Please let me know ,how can be do in talend with best way.

Thanks,

Manohar

[Anonymous]

Hi Manohar,

     One method I would try is to use the files through EMRFS where the file systems are directly tagged to S3. I have not personally tried it myself but I feel it would a good idea to use Big data batch job to do the heavy lifting compared to a Standard job.

      Once the processed data is ready, I can extract that file alone to local and load it to RDS.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved