Reusable job to capture Source record count and Target record count
Hi All, I am new to talend and did created many reusable mapplets in informatica.We have around 150+ talend jobs created couple years ago which read from sql server source db and write to hive..All these are missing Source read and target write count capturing. Can you please advise if we can create a reusable joblet to capture this count and plug in this joblet to all 150+ talend jobs. Current design of talend jobs(version 7.1) : tMSSqlInput-->tHDFSOutput-->tHiveLoad
In case you are dealing with huge volume of data, I will suggest you separate extraction an loading into two different jobs
Commercial version offers a simple way to build generic job using the datatype “dynamic”.
On TOS, I thinks we can find a complex way to build something similar
You can use context to handle some configurations
It's what we did in the past case where we were importing data from 150 sources, around 50 different schemas and a total of about 1 billion records per day.
Since you are specifing about count details, I assume the schema for your joblet will remain same for multple tables. In this case, you can pass the other details as parameters to both source and target components (like tablename, query where clause etc.)
Joblet is nothing but a part of the job which you are moving as separate entity either due to its complexity or to make it as a reusable part for your jobs. Please create your jobs and if there are any errors, please share the details of the job along with job flow and other screenshots for further analysis.
Warm Regards, Nikhil Thampi
Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂
Thanks DGM on your suggestion.We were thinking about this single job with dynamic option instead of independent 150+jobs but it may require lot of reg testing after configuring all source sql as context etc...So checking all the options to not to touch existing jobs and just to add on the fly source /target count capture logic
Hi Nikhil ..thanks for your response on this.
Current design is as below.
tMSSqlInput-->tHDFSOutput-->tHiveLoad.
In this case my intention is not to query the source /target again just for counts capturing .l am looking if there is any way to capture using the existing tMSSqlInput and tHiveload componets on the fly (something like flowmeter) in joblet..In any case looks like I need to modify all the 150+ jobs or as suggested by DGM redesign 150+ jobs to single dynamic job..please advise if any other thoughts
