topic Retrive single csv from S3 bucket in Talend Studio

Retrive single csv from S3 bucket

Anonymous — Tue, 11 Oct 2016 08:07:43 GMT

Hi,
I want to retrive a single csv file from S3 bucket.
example: I have S3 -> project(folder) -> (contains 11 csv files) -> I want to reterive only one csv (customer) from project folder, That needed csv file's name comes with a combination of (randomnumber-timestamp-transactionid-customer.csv) so this cannot given as default one.

ts3connection->ts3list(here i want to filter file name with recent timestamp "3eca-20161011-164913-Customer.csv" customer file)->ts3get()

Thanks
VK

Re: Retrive single csv from S3 bucket

Anonymous — Thu, 13 Oct 2016 06:14:41 GMT

You can use Key Mask to get only the file name that you require. You can mask like *-Customer.csv"

Re: Retrive single csv from S3 bucket

Anonymous — Fri, 14 Oct 2016 06:01:13 GMT

Hi Sankalp
Thanks for your reply,
I missed to add that in my S3 bucket there are 3 files with -Customer.csv
0e4d-20161013-115009-Customer.csv
re3f-20161013-114639-Customer.csv
we4d-20161013-121309-Customer.csv
In this case how I could get the most recent file to my input csv (tfileinputdelimted) and i can see last modified date manually when i open the bucket but how to get the recent dated file.
Thanks
VK

Re: Retrive single csv from S3 bucket

vapukov — Fri, 14 Oct 2016 09:05:31 GMT

If it Job run on Linux (or windows with bash - CygWin or etc), possible to use tSystem component to run shell script:

#!/bin/bash
FILENAME=$(aws s3 ls s3://_BUCKET_/_FOLDER_/ --recursive | sort | tail -n 1 | awk '{print $4}')

SOURCE='s3://_BUCKET_/'$FILENAME
TARGET=‘DESTINATION'

aws s3 cp $SOURCE $TARGET

of course aws command line utilities must be installed and configured
add. or similar in powershell

Re: Retrive single csv from S3 bucket

Anonymous — Fri, 14 Oct 2016 13:24:10 GMT

Not an ideal approach but can you use tS3List to get the list of files in your bucket --> Store it in a file or tHash --> Get the name of the file with latest time stamp using file name (using tJavaRow or tMap) --> Use that file name is tS3Get file.