Skip to main content
Announcements
UPGRADE ADVISORY for Qlik Replicate 2024.5: Read More
cancel
Showing results for 
Search instead for 
Did you mean: 
guilherme-matte
Partner - Creator
Partner - Creator

Adding new tables in a ongoing Task

Hello guys

 My doubt is the following. If i already have a task running with CDC, supposedly with 3 tables. Are there any problems in stopping the task, changing the full load parameters (for example reducing the throughput), adding a new 4th table and resuming the task? 

I know that it will happen a full load of the table, but during this time does the CDC stop on the other tables? Also, will the CDC work on this new table that i just added to the Task?

Are any recommendations/issues regarding this practice? Should I do a full reload after adding new tables?

Kind Regards!

Labels (3)
1 Solution

Accepted Solutions
SachinB
Support
Support

Hello @guilherme-matte ,

Adding the new table to the existing task by enabling full load for the task is not going to impact the CDC for the other three tables, When you stop the task it's going to store the last saved stream position (SCN/LSN).

When you enable the full load and add the extra table in it, You just need to resume the task, So newly added table will go full load and other three tables will start CDC process from where they have stopped.  4th table will go CDC when full load is completed.

Please revert if you need more information for the same.


Regards,

Sachin B

 

 

View solution in original post

6 Replies
SachinB
Support
Support

Hello @guilherme-matte ,

Adding the new table to the existing task by enabling full load for the task is not going to impact the CDC for the other three tables, When you stop the task it's going to store the last saved stream position (SCN/LSN).

When you enable the full load and add the extra table in it, You just need to resume the task, So newly added table will go full load and other three tables will start CDC process from where they have stopped.  4th table will go CDC when full load is completed.

Please revert if you need more information for the same.


Regards,

Sachin B

 

 

guilherme-matte
Partner - Creator
Partner - Creator
Author

Thank you @SachinB , that was exactly what I needed. 

Cheers

Heinvandenheuvel
Specialist III
Specialist III

@SachinB >> 4th table will go CDC when full load is completed.

Nitpicking! The CDC for the new table(s) will start BEFORE the first unload, to make sure nothing changed in the rows already loaded during the unload. It's easy to see in the logs:

12:27:11 [SORTER          ]I:  Reload for table Id 1 is requested  (sorter
12:27:11 [SORTER          ]I:  Start collecting changes for table id = 1  
12:27:11 [TASK_MANAGER    ]I:  Start loading table 'ATT_USER'.'TEST' (Id =
12:27:11 [SOURCE_UNLOAD   ]I:  Unload finished for table 'ATT_USER'.'TEST'
12:27:13 [TARGET_LOAD     ]I:  Load finished for table 'ATT_USER'.'TEST' (
12:27:14 [TASK_MANAGER    ]I:  Loading finished for table 'ATT_USER'.'TEST
ddb
Contributor II
Contributor II

Hi, I have 56 tasks, and I need to add the same table to all of them. Is there any script to do that or it's still manual process. It seems that it could be easily done looping thru task's JSON files and adding a code. Just wonder if QLIK vendor come up with the solution. 

Heinvandenheuvel
Specialist III
Specialist III

>> Hi, I have 56 tasks, and I need to add the same table to all of them.

Wow that's a lot of times for the same table. The most I typically see is 6 or 9:

DEV/QA/PROD , two or three targets. 

Of course the DEV should be done first, making sure the new table does not crash the task for some bizarre reason.

With 56 tasks reading the same table something seems poorly designed unless there are indeed 56 distinct targets.

[You may want to open up you own topic to expand on this, if need be ]

>>  Is there any script to do that or it's still manual process. It seems that it could be easily done looping thru task's JSON files and adding a code.

Indeed, SMOP!

I have both a PERL and PowerShell script to do this. Admittedly I haven't needed to use them in a while.

For the PERL version use -h for help, and -A to add tables. It needs an input and output JSON text file to work.

For the Powershell use -h for help and AddTables_CSV_file

I also have attached an AEM version which can grap the task definition from AEM, merge in the table, import back. - No temp JSON tiles needed. Some login tweaking may well be needed.

hth,

Hein.

 

ddb
Contributor II
Contributor II

Great! thank you much!