I have created a talend batch Job which is scheduled at windows scheduler and has to run at every 15 min of gap. This talend batch job contains SQL code to create logs and store in redshift table. So it will run 4 times in an hour. In the beginning it was fine, but now sometimes it is generating logs more than 4 times in a table.
Like one log is created at 08:00:00 and next has to be at 08:15:00 and next at 08:30:00 and so on, but now it is like 08:00:00 and then 08:00:11, then moving forward with the same scenario.
Sometimes it goes 5 to 6 hrs in this fashion only and after after it follows the regular criteria, and sometimes, in between for one hour only. But it is happening at regular basis. Can anyone suggest please what can be the issue.?
Could you please share the job screenshot and component screen shot so that we will get more idea about how you are writing the data to this table. I suspect you are calling the job multiple times inadvertently. It could be because of the usage of On Component Ok instead of OnSubJob Ok or having multiple iterations of same code base based on some condition check.
Warm Regards, Nikhil Thampi
Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved
I believe I got the issue. You are using On Component Ok to trigger the log entry to Redshift table instead of OnSubJob Ok process. You need to do a single entry to log table once your main subjob is complete. In this case, you need to connect the first component of your main subjob to first component of logging subjob using On SubJob Ok link.
It will make sure that the logging component is executed only once after the main subjob is complete. Could you please try and let us know the results. On a different note, please always design your job from top to bottom. You are keeping your Redshift log process above the main process. The job design should be like a story book book where the control should flow from left to right and top to bottom. You should not create the flow in opposite order as its confusing for any other developer while doing any review.
Warm Regards, Nikhil Thampi
Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved
