topic Re: REST API Calls to Get Table Level Details of a Real-time (CDC) Task in Qlik Replicate

REST API Calls to Get Table Level Details of a Real-time (CDC) Task

asif_baig — Sat, 16 Nov 2024 00:28:31 GMT

Hello There,

This is my first post in this community, I might not know the ins and out, but here is my trouble for which I am looking for some advice or workaround.

I am working on a Java API to connect to Replicate and perform certain tasks based on REST API calls.

There are some CDC (Real time) tasks in our Replicate that continuously runs and keep Target tables in syn with changes happening in Source.

Each of these task consists of multiple tables and however I can get Task level details using "GetTaskDetail" call, I am not able to get table level information like which table is currently being replicated and what all tables are already synced.

Can anyone let me know if this is something possible with REST or not, if yes and please do guide me where can I start.

Thanks!

Re: REST API Calls to Get Table Level Details of a Real-time (CDC) Task

Steve_Nguyen — Tue, 16 Jun 2020 16:03:28 GMT

in replicate there is limited api.

you would be looking at AEM API.

there is a :

get_table_statuses
Retrieves the tables statuses of a specific Replicate task for all tables that match the
specified state(s), table schema(s), and table name(s). This is useful for automation
processes, for example, as it allows you to retrieve tables in a certain state (e.g.
suspended) and then peform an operation on them (e.g. AemReloadTable)

Re: REST API Calls to Get Table Level Details of a Real-time (CDC) Task

asif_baig — Wed, 17 Jun 2020 10:10:28 GMT

Thank you for this, I guess GetTableList and GetTableDetails are added after version 6.5 and later and earlier I was not looking into them.
They are perfect solution for my problem, however going over developer guide I got confused with the "Status" we got in return for a table hooked with a CDC task:

TABLE_QUEUED
TABLE_LOADING
TABLE_COMPLETED
TABLE_CHANGE_PROCESSING
TABLE_ERROR

For a CDC table I am always getting status as TABLE_CHANGE_PROCESSING even if there are not changes in Source and when I check the AEM using browser I can see that replication is complete, here my question arise that for a CDC task will we not have TABLE_COMPLETED status anytime?

Re: REST API Calls to Get Table Level Details of a Real-time (CDC) Task

Steve_Nguyen — Wed, 17 Jun 2020 12:31:39 GMT

Hello ,

if your task is CDC only, then table would not have complete status, as these task is for cdc.

so table just start reading source and writing to target.

Thanks

Re: REST API Calls to Get Table Level Details of a Real-time (CDC) Task

asif_baig — Fri, 19 Jun 2020 12:46:32 GMT

@Steve_Nguyen Thank you for the information.

Could you please let me know how can we confirm if last transaction on source is completely replicated to target Table for a CDC Task?

Re: REST API Calls to Get Table Level Details of a Real-time (CDC) Task

JitenderR — Fri, 10 Jul 2020 21:24:10 GMT

@asif_baig what is the end goal? to ensure data is completely replicated? In that case you will need to rely on response parameter 'last_update_time'. Kindly note that for a task, like replicate, running infinitely it is practically not possible to confirm if ALL data from source is replicated. Even if achievable, it would take a lot of complex scripting or a huge audit framework to be built. So, the way i would script it is - pick the last_udpate_time for a given table, depending on how frequently the table sends data, have a "timer" say 4 hours and save the current and last values of 'last_update_time', If the difference of these two values is more than 4 hours (timer) , then take an action - like send an email, create an incident etc via the script. Ensure to perform proper error handling like, if the task is stopped this parameter might return a static/null value and you don't want to send false alerts.

Let me know for any additional questions.

Regards