Need Help with Talend Job Design and Development

SThirunagari — Fri, 15 Nov 2024 22:58:40 GMT

Hi All,

Need help with below requirement.

Develop Generic Talend Job which has to be executed for different 16 locations with source as .txt files from each of location and target to be loaded to Azure SQL server with 16 different databases corresponding to each database.

For Eg : Locations - Atlanta, Newyork, Newjersey source files available in respective shared path as below :-

For Atlanta :-

Atlanta1.txt, Atlanta2.txt, Atlanta3.txt, Atlanta4.txt,Atlanta5.txt

For Newyork :- Newyork1.txt, Newyork2.txt, Newyork3.txt, Newyork4.txt, Newyork5.txt

For Philadelphia :- Philadelphia1.txt, Philadelphia2.txt, Philadelphia3.txt, Philadelphia4.txt, Philadelphia5.txt

Target Azure SQL Server :- ABC.SQLSERVER.NET

Databases: Atlanta , Newyork, Philadelphia

Atlanta Tables: Atlanta.dbo.AtlantaTable1;

Atlanta.dbo.AtlantaTable2; Atlanta.dbo.AtlantaTable3; Atlanta.dbo.AtlantaTable4; Atlanta.dbo.AtlantaTable5

NewyorkTables: Newyork.dbo.

NewyorkT

able1;

Newyork.dbo.NewyorkTable2; Newyork.dbo.NewyorkTable3; Newyork.dbo.NewyorkTable4; Newyork.dbo.NewyorkTable5

PhiladelphiaTables:

Philadelphia.dbo.PhiladelphiaTable1; Philadelphia.dbo.PhiladelphiaTable2; Philadelphia.dbo.PhiladelphiaTable3; Philadelphia.dbo.PhiladelphiaTable4;Philadelphia.dbo.PhiladelphiaTable5;

Requirements:-

1) Design a generic job that can be run for all locations , customer does not want set of 16 different jobs to transfer every 10minutes and other set of 16 different jobs for daily once frequency

2) Load each file into corresponding target table.

3) For all the sites , first three files will be loaded to the target at a frequency of every 10 minutes throughout the day and last two files should be loaded once a day

So far I have created the job design by having two seperate configuration files one for 10 minutes and other for once a day with below details, however i am struck at the point of not knowing how to dynamically pass the Location based 10minute config file and Location based 1 hour file at run time

DAILY_10_MINUTE_CONFIG_FILE.txt

File_Name TGT_DB_NAME TGT_SCHEMA TGT_TABLE

Atlant1.txt Atlanta dbo Atlanta1

Atlant2.txt Atlanta dbo Atlanta2

Atlant3.txt Atlanta dbo Atlanta3

DAILY_ONCE_MINUTE_CONFIG_FILE.txt

File_Name TGT_DB_NAME TGT_SCHEMA TGT_TABLE

Atlant4.txt Atlanta dbo Atlanta4

Atlant5.txt Atlanta dbo Atlanta5

I am seeking help if someone could share their valuable thoughts on how to dynamically pass the DAILY_10_MINUTE_CONFIG_FILE.txt

and

DAILY_ONCE_MINUTE_CONFIG_FILE.txt t

o the

generic job to perform above load

NOTE:- The job has to run in parallel for differnt sites for above configuration.

Any Help would be greatly appreciated

Thanks

Sthirunagari

Re: Need Help with Talend Job Design and Development

gjeremy1617088143 — Fri, 29 Apr 2022 08:18:21 GMT

Hi, design a parent job wich read all locations and send informations to a child job wich be a generic job wich could run from each locations,

pass the param you want via context variable in the tRunJob component, you can use a tPartitionner an tCollector to run between 3 and 5 childs job in different threads in parallel.

Re: Need Help with Talend Job Design and Development

SThirunagari — Tue, 03 May 2022 00:38:01 GMT

Hi gjeremy,

Thankyou for your response. I have designed the job and currently testing. However when I build the job and try to run the job from commandline through the jobrun.sh script I am ending up with below issue while trying to unzip the build .zip file. any thoughts or directions for the below error.

error: invalid zip file with overlapped components (possible zip bomb). However the same job my colleague is able to build , unzip and run from commandprompt.

Thanks

Sthirunagari

topic Need Help with Talend Job Design and Development in Talend Studio

Need Help with Talend Job Design and Development

Re: Need Help with Talend Job Design and Development

Re: Need Help with Talend Job Design and Development