topic Re: Setting up a generic job to load snowflake table based on Context Param values in Talend Studio

Setting up a generic job to load snowflake table based on Context Param values

vvazza10 — Sat, 16 Nov 2024 00:17:07 GMT

Hi ! Wanted to check if there is any way to create a generic job that does the following -

Parent job that contains 2 tRunJob components - One to load Employee table and one to load Department table
tRunJob component for Employee having Context Params Employee as target table and Emp as flow param and
tRunJob component for Department having Context Params Department as target table and Dept as flow param
Based on the context param value, corresponding flow in tXMLMap should execute and load to target snowflake table. For example, if Emp is the flow param value, Emp flow in tXMLMap should execute and Employee table in snowflake table should get executed.

Please advise how to approach a scenario like this.

Parent job:

Context Param values that gets passed to child job for Employee:

Child job:

Re: Setting up a generic job to load snowflake table based on Context Param values

vvazza10 — Sat, 17 Apr 2021 12:14:12 GMT

Hi ! Could someone please help? 🙂

Re: Setting up a generic job to load snowflake table based on Context Param values

Anonymous — Mon, 19 Apr 2021 05:10:53 GMT

Because the employee and department table have different structure and tXMLMap don't support different output flow based on variable, so you can't create a generic job flow for different target object. As a workaround, create different subjobs, and use runIf connector to trigger the subjob based on the context variable. For example:

tJava--runIf1--tFileInputXML--tXMLMap--tSnowFlakeOutput(for employee)

--runIf2---tFileInputXML--tXMLMap--tSnowFlakeOutput(for department )

on tJava:

if(context.tableName.equals("employee")){

globalMap.put("runEmployeeFlow",true);

}

if(context.tableName.equals("department")){

globalMap.put("runDepartmentFlow",true);

}

Set the condition of runIf1 as:

(Boolean)globalMap.get("runEmployeeFlow")

Set the condition of runIf2 as:

(Boolean)globalMap.get("runDepartmentFlow")

Hope it helps!

Regards

Shong

Re: Setting up a generic job to load snowflake table based on Context Param values

vvazza10 — Tue, 20 Apr 2021 13:43:13 GMT

Hi @Shicong Hong ! I would want to load both Employee and department in a single load. XML file will contain both Employee and Department information. Running subjobs based on context variable would allow only one dataset, either Employee or Department, to be loaded to Snowflake table.

Tried testing with the below design, but the load seem to take 13-15 minutes to complete for a 2MB XML file

How can i design this better?