topic Extracting data from DB Table else keep on checking till we get records in Talend Studio

Extracting data from DB Table else keep on checking till we get records

BA621858188 — Fri, 15 Nov 2024 22:50:54 GMT

Hi All,

I have one scenario where I need to check the records present or not in the table so that to start the next job. If we get records from the query which I am using select * from table where = condition. then proceed further else keep on checking the records after some interval till we get the records.

Can anyone help me with the job design. I know that we need to use mysqlinput component to check the data once but what components I can use for checking again and again the query to get the data.

Thanks in advance

Re: Extracting data from DB Table else keep on checking till we get records

Anonymous — Wed, 08 Jun 2022 23:45:25 GMT

Talend Jobs are meant to be batch and not real-time. How often do you want to check this query? Also, what type of Talend product are using? Are you using the paid version or the free version?

Re: Extracting data from DB Table else keep on checking till we get records

BA621858188 — Thu, 09 Jun 2022 05:25:31 GMT

Hi @Richard Hall

Thank you for your reply. I have to check for every 5 minutes for half an hr(30 mins) then will force completed the job. I am using Talend enterprise version(v7)

Can you tell me the job design which I should follow for the above scenario.

Re: Extracting data from DB Table else keep on checking till we get records

Anonymous — Thu, 09 Jun 2022 15:59:48 GMT

Do you have access to Talend ESB? With Talend ESB you can have a Route continually running and checking that query. When the query returns something, you can initiate a Talend job from the route to carry out the further processing.

Re: Extracting data from DB Table else keep on checking till we get records

BA621858188 — Thu, 09 Jun 2022 16:01:31 GMT

Hi @Richard Hall ,

we dont have Talend ESB. Is there any functionality in Talend Job itself. Kindly help.

Re: Extracting data from DB Table else keep on checking till we get records

Anonymous — Thu, 09 Jun 2022 16:07:02 GMT

It is a bad idea to have a job running continuously. So another way of doing this is to write a job to check the query and process the output if it is found. Then configure that job in TAC (On-Premises) or TMC to run every 5 minutes.

Re: Extracting data from DB Table else keep on checking till we get records

BA621858188 — Thu, 09 Jun 2022 16:51:56 GMT

@Richard Hall In This can I use with loop components as I just want to check for 20-30 minutes only. If still didn't get records then I will just force complete the job.

Re: Extracting data from DB Table else keep on checking till we get records

Anonymous — Thu, 09 Jun 2022 21:43:14 GMT

You could set the tLoop logic to loop so many times and add a tSleep component to wait at the end of each attempt. So if you set the tSleep to wait 300 seconds and then limit the tLoop to 6 iterations, that would do it.

Re: Extracting data from DB Table else keep on checking till we get records

BA621858188 — Sat, 11 Jun 2022 06:46:09 GMT

@Richard Hall Do you have any Job design where you have used tloop and tsleep components.

As per my understanding I need to use the above job design in a separate job. Correct me if I am wrong.