Determine the start or end of a full load

[Yves_s]

Hello,

my tasks read on db2 z and write in kafka.

I would like to know the start or the end at full load, is there any way to know one or the other?

Example

display a static message at the start of the full load: "start full load"

Or

show the start by the first REFRESH message and get a static message at the end of the full load: "end full load"

[john_wang]

Hello @Yves_s ,

You can define the Task Events Notification, include "Full Load started", and "Full Load completed", and others, reference link is here . When the corresponding action done, a notification message will be sent to Email, Windows Event Log/Linux event messages, and/or Qlik Replicate console,  It's configurable. Detailed information can be found in Defining notifications .

Hope this helps.

Regards,

John.

[Yves_s]

Thanks, 

but I need to have the information at the level of the consumer who reads the topic.

for example 

I had thought of adding a column with $AR_V_TASK_RELOAD_TIME,which would have allowed me to see when this reload time changed.

But this value only changes when the task reloads and not the table, so when we have several tables in one task, it doesn't work

[john_wang]

Hi @Yves_s ,

You may add an extra column to each message to reflect the local date/time of the Full Load operation, the expression is like:

DATETIME('NOW','+8 hours')

where '+8 hours' is the modifier to change to local time. In this sample it's HK time. You can change it to your timezone.

BTW, the expression will record the applying time in CDC stage  (for Insert/Update/Delete operations) also.

Regards,

John.

[Yves_s]

we are misunderstood, the example of $AR_V_TASK_RELOAD_TIME  was just to show a possibility to determine the start or the end of the full load, which could be recovered from the consumer kafka.

the request is:

how to make so that my consumer kafka, can see the start or the end of a full load?

[john_wang · Accepted solution]

Hi @Yves_s ,

I'm afraid there is no other options, unless you use the operation equal REFRESH, and the current delivery time of each message.

Regards,

John.

[Heinvandenheuvel · Accepted solution]

For this information I think you need to think outside the data stream towards the Replicate API's 

The means executing GETTASKSTATUS or GETTABLESSTATUS and sending the output to Kafka using a dedicated scheduled script / program on the Replicate server, or on any client with a connection to the Enterprise Manager.

On the server you could check it out for starters with simple REPCTL command to see if it begets the data you need.

For example 

repctl -d <your-data-dir> gettaskstatus <your-task-name>

Look for:

"full_load_completed": true,
"full_load_start_time": 1619647500856762,
"full_load_finish_time": 1619647501629008,

or

repctl -d <your-data-dir>  gettablesstatus <your-task-name> 4 -1

Look for (per table):

"fl_start_time": 1620613252872870,
"fl_last_time": 1620613252918666,

For help try for example: repctl -d <your-data-dir>  gettablesstatus ?

This will enumerate all the arguments you can provive and attributes you might get back (sometime 0 values are simply not returned at all!)

More robust solutions would use the Enterprise Manager API - Curl / Python or Powershell - check the documentation in . 

hth,

Hein.