Skip to main content
Announcements
Have questions about Qlik Connect? Join us live on April 10th, at 11 AM ET: SIGN UP NOW
cancel
Showing results for 
Search instead for 
Did you mean: 
asif_baig
Contributor II
Contributor II

REST API Calls to Get Table Level Details of a Real-time (CDC) Task

Hello There,

This is my first post in this community, I might not know the ins and out, but here is my trouble for which I am looking for some advice or workaround.

I am working on a Java API to connect to Replicate and perform certain tasks based on REST API calls.

There are some CDC (Real time) tasks in our Replicate that continuously runs and keep Target tables in syn with changes happening in Source.

Each of these task consists of multiple tables and however I can get Task level details using "GetTaskDetail" call, I am not able to get table level information like which table is currently being replicated and what all tables are already synced.

 

Can anyone let me know if this is something possible with REST or not, if yes and please do guide me where can I start.

 

Thanks!

Labels (1)
  • Other

1 Solution

Accepted Solutions
JitenderR
Employee
Employee

@asif_baig what is the end goal? to ensure data is completely replicated? In that case you will need to rely on response parameter 'last_update_time'. Kindly note that for a task, like replicate, running infinitely it is practically not possible to confirm if ALL data from source is replicated. Even if achievable, it would take a lot of complex scripting or a huge audit framework to be built. So, the way i would script it is - pick the last_udpate_time for a given table, depending on how frequently the table sends data, have a "timer" say 4 hours and save the current and last values of 'last_update_time', If the difference of these two values is more than 4 hours (timer) , then take an action - like send an email, create an incident etc via the script. Ensure to perform proper error handling like, if the task is stopped this parameter might return a static/null value and you don't want to send false alerts. 

Let me know for any additional questions. 

Regards

JR

JitenderR_0-1594416160381.png

 

View solution in original post

5 Replies
Steve_Nguyen
Support
Support

in replicate there is limited api.

you would be looking at AEM API.

there is a :

get_table_statuses
Retrieves the tables statuses of a specific Replicate task for all tables that match the
specified state(s), table schema(s), and table name(s). This is useful for automation
processes, for example, as it allows you to retrieve tables in a certain state (e.g.
suspended) and then peform an operation on them (e.g. AemReloadTable)

Help users find answers! Don't forget to mark a solution that worked for you! If already marked, give it a thumbs up!
asif_baig
Contributor II
Contributor II
Author

Thank you for this, I guess GetTableList and GetTableDetails are added after version 6.5 and later and earlier I was not looking into them.
They are perfect solution for my problem, however going over developer guide I got confused with the "Status" we got in return for a table hooked with a CDC task:

TABLE_QUEUED
TABLE_LOADING
TABLE_COMPLETED
TABLE_CHANGE_PROCESSING
TABLE_ERROR

For a CDC table I am always getting status as TABLE_CHANGE_PROCESSING even if there are not changes in Source and when I check the AEM using browser I can see that replication is complete, here my question arise that for a CDC task will we not have TABLE_COMPLETED status anytime?

Steve_Nguyen
Support
Support

Hello ,

if your task is CDC only, then table would not have complete status, as these task is for cdc.

so table just start reading source and writing to target.

Thanks

Help users find answers! Don't forget to mark a solution that worked for you! If already marked, give it a thumbs up!
asif_baig
Contributor II
Contributor II
Author

@Steve_Nguyen  Thank you for the information.

Could you please let me know how can we confirm if last transaction on source is completely replicated to target Table for a CDC Task?

JitenderR
Employee
Employee

@asif_baig what is the end goal? to ensure data is completely replicated? In that case you will need to rely on response parameter 'last_update_time'. Kindly note that for a task, like replicate, running infinitely it is practically not possible to confirm if ALL data from source is replicated. Even if achievable, it would take a lot of complex scripting or a huge audit framework to be built. So, the way i would script it is - pick the last_udpate_time for a given table, depending on how frequently the table sends data, have a "timer" say 4 hours and save the current and last values of 'last_update_time', If the difference of these two values is more than 4 hours (timer) , then take an action - like send an email, create an incident etc via the script. Ensure to perform proper error handling like, if the task is stopped this parameter might return a static/null value and you don't want to send false alerts. 

Let me know for any additional questions. 

Regards

JR

JitenderR_0-1594416160381.png