topic how to check the flag file from S3 folder in Talend Studio

how to check the flag file from S3 folder

Anonymous — Sat, 16 Nov 2024 07:07:27 GMT

Hi Everyone,

I have a scenario in which i am running a job from Talend which is pulling all the files from AWS S3 folder.But before doing so ,i need to check if the Flag file is present there then only i can proceed.

Suppose ,I have files like this:

File_trigger.Flag

abc.csv

bcd.csv

efg.csv

If the file_trigger.flag is present on AWS S3 then only my job starts pulling the files abc.csv ,bcd.csv etc...

I have made my job which is pulling all the files from S3 but now this is a new scenrio. Could you please let me know how to proceed with that?

Regards,

Mohit

Re: how to check the flag file from S3 folder

vapukov — Wed, 05 Dec 2018 20:35:32 GMT

you could install aws cli commandline client, and setup it

after use tSystm with command

 aws s3 ls s3://bucket/File_trigger.Flag

tSystem will return 0 if file exists, and 1 if not, use it in tRunIf trigger as value

((Integer)globalMap.get("tSystem_1_EXIT_VALUE"))>0

Re: how to check the flag file from S3 folder

Anonymous — Thu, 06 Dec 2018 07:36:39 GMT

Hi Vapukov,

AWS CLI command line is installed already.

Still i need to connect to AWS before using tsystem command.I know the connection and other things.

But How tsystem will fetch AWS connection details and trigger the above command from Talend.

regards,

Mohit

Re: how to check the flag file from S3 folder

vapukov — Thu, 06 Dec 2018 08:51:56 GMT

if you have installed aws-cli, you know - it self-contained, it do not need anything for work, you must configure it separate

so in you job:

preJob - connection

Job:

- tSystem

- and after RunIf all other

Re: how to check the flag file from S3 folder

Anonymous — Fri, 07 Dec 2018 07:45:47 GMT

Thanks,I will try it

Re: how to check the flag file from S3 folder

Anonymous — Fri, 07 Dec 2018 10:12:40 GMT

Hi,

Below is another solution for your query. I have setup two sample files in S3 for testing.

Below is the output of the flow with only flag file as output.

Below are the details of each components. I have added tLogrow components for printing and you can remove them in actual code.

Both approaches provide same result but one is employing command line method and other through GUI.

If the reply has helped you, could you please mark the topic as resolved? Kudos are also welcome 🙂

Warm Regards,

Nikhil Thampi

Re: how to check the flag file from S3 folder

vapukov — Fri, 07 Dec 2018 10:30:20 GMT

I exclude this variant because as is it requires double reading for all files (as on picture) and with 1000 files it could be long

but, if using the full Flag file name as a Key prefix in tS3List - it could reduce the number of iteration to 1

unfortunately, right now I have not S3 access and cannot test it, but it could be a best choice if work

Re: how to check the flag file from S3 folder

Anonymous — Fri, 07 Dec 2018 11:33:41 GMT

@vapukov - Perfect idea !

I added the flag file name in the Key prefix and it reduced the number of iterations to 1.

The modified job flow is as below.

Job Output

Warm Regards,

Nikhil Thampi