Adding new tables in a ongoing Task

[guilherme-matte]

Hello guys

 My doubt is the following. If i already have a task running with CDC, supposedly with 3 tables. Are there any problems in stopping the task, changing the full load parameters (for example reducing the throughput), adding a new 4th table and resuming the task? 

I know that it will happen a full load of the table, but during this time does the CDC stop on the other tables? Also, will the CDC work on this new table that i just added to the Task?

Are any recommendations/issues regarding this practice? Should I do a full reload after adding new tables?

Kind Regards!

[SachinB · Accepted solution]

Hello @guilherme-matte ,

Adding the new table to the existing task by enabling full load for the task is not going to impact the CDC for the other three tables, When you stop the task it's going to store the last saved stream position (SCN/LSN).

When you enable the full load and add the extra table in it, You just need to resume the task, So newly added table will go full load and other three tables will start CDC process from where they have stopped.  4th table will go CDC when full load is completed.

Please revert if you need more information for the same.

Regards,

Sachin B

[guilherme-matte]

Thank you @SachinB , that was exactly what I needed. 

Cheers

[Heinvandenheuvel]

@SachinB >> 4th table will go CDC when full load is completed.

Nitpicking! The CDC for the new table(s) will start BEFORE the first unload, to make sure nothing changed in the rows already loaded during the unload. It's easy to see in the logs:

12:27:11 [SORTER          ]I:  Reload for table Id 1 is requested  (sorter
12:27:11 [SORTER          ]I:  Start collecting changes for table id = 1  
12:27:11 [TASK_MANAGER    ]I:  Start loading table 'ATT_USER'.'TEST' (Id =
12:27:11 [SOURCE_UNLOAD   ]I:  Unload finished for table 'ATT_USER'.'TEST'
12:27:13 [TARGET_LOAD     ]I:  Load finished for table 'ATT_USER'.'TEST' (
12:27:14 [TASK_MANAGER    ]I:  Loading finished for table 'ATT_USER'.'TEST

[ddb]

Hi, I have 56 tasks, and I need to add the same table to all of them. Is there any script to do that or it's still manual process. It seems that it could be easily done looping thru task's JSON files and adding a code. Just wonder if QLIK vendor come up with the solution. 

[Heinvandenheuvel]

>> Hi, I have 56 tasks, and I need to add the same table to all of them.

Wow that's a lot of times for the same table. The most I typically see is 6 or 9:

DEV/QA/PROD , two or three targets. 

Of course the DEV should be done first, making sure the new table does not crash the task for some bizarre reason.

With 56 tasks reading the same table something seems poorly designed unless there are indeed 56 distinct targets.

[You may want to open up you own topic to expand on this, if need be ]

>>  Is there any script to do that or it's still manual process. It seems that it could be easily done looping thru task's JSON files and adding a code.

Indeed, SMOP!

I have both a PERL and PowerShell script to do this. Admittedly I haven't needed to use them in a while.

For the PERL version use -h for help, and -A to add tables. It needs an input and output JSON text file to work.

For the Powershell use -h for help and AddTables_CSV_file

I also have attached an AEM version which can grap the task definition from AEM, merge in the table, import back. - No temp JSON tiles needed. Some login tweaking may well be needed.

hth,

Hein.

[ddb]

Great! thank you much!