repctl checkpoint "task name" for downstream task (source: log stream task & target : oracle database) is not giving correct timestamp

[dipankar]

repctl checkpoint "task name" for downstream task (source: log stream task & target : oracle database) is not giving correct timestamp. 

D:\bin>repctl getcheckpoint "TASK_INGCDC_02"

command getcheckpoint response:

{

        "checkpoint":   "checkpoint:V1#33512#1271;638506460899807820;20240120221751514712#0#0#*#0#137"

}

[getcheckpoint command] Succeeded

Regards

Dipankar

[sureshkumar]

Hello @dipankar 

Kindly test it with 

repctl -d "data location” getcheckpoint "task_name"

Check whether you are getting the same issue or not.

Regards,

Suresh

[john_wang]

Hello Dipankar @dipankar ,

Thanks for reaching out to Qlik Community!

I guess I did not get the question very well. Would you please elaborate the exact doubt?

Regards,

John.

[SushilKumar]

Hello @dipankar 

First that getting result from Repctl is not Recommend for customer usage. It is Advised in certain cases wherever its required.  There are lot of timestamps which update along with the data replication.

Regards,

Sushil Kumar

[dipankar]

Environment Information :

1. A log stream (upstream) staging task is created which is reading from source endpoint ( oracle ) and storing changes onto the replicate server
2. There are around 20 downstream tasks whose source endpoint is log stream staging task and target endpoint is oracle.
3. Qlik console provides on latency information which doesn’t helps us much as per requirement of our environment.
4. We Need precise timestamp information  to know uptill which point in time data has been committed in the target database.
5. We used repctl checkpoint “task name”  to get the information about the checkpoint.

Problem statement

1. Detailed output of repctl command for downstream tasks shared below

D:\bin>repctl getcheckpoint "TASK_INGCDC_02"

command getcheckpoint response:

{

        "checkpoint":   "checkpoint:V1#33512#1271;638506460899807820;20240120221751514712#0#0#*#0#137"

}

[getcheckpoint command] Succeeded

D:\bin>

D:\bin>repctl getcheckpoint "TASK_INGCDC_03"

command getcheckpoint response:

{

        "checkpoint":   "checkpoint:V1#33483#1271;638506460899807820;20240120221751514712#0#0#*#0#137"

}

[getcheckpoint command] Succeeded

2. We reached out to engineers from qlik professional services (Michael Chiu)  to help us decode output of above repctl command. They reviewed this output and  told us that above output translates to below time stamp

The timestamp is "202401202217515147", which is "2024-Jan-20 22:17:51.514"

3. These downstream tasks have been created couple of days ago and went online (started replicating) yesterday (07^th May ’24 ) so the checkpoint/commit provided for these downstream tasks is too old.

4. Below is response from Michael Chiu to this problem

“I have just double-checked, turns out the "20240120221751514712" is not the timestamp of the last change written to the target, it is the name (based on the creation time) of the sub-directory under the "..\LOG_STREAM\audit_service" parent directory,

that is why it is unchanging.”

Expected solution

1. We want to able to retrieve timestamp information uptill which downstream tasks have committed/written data to the target database.
2. This commit time stamp information is important as we have to take some decision while generating business critical reports.

[SachinB]

Hello @dipankar ,

Thanks for reaching out to Qlik Community!

If you are looking for last timestamp updated record in change table at target endpoint, then you can refer the "[header__] timestamp" for store change table, which give you the clarity when the last record has been updated.

The above example seems to be fetching "Timeline" folder names from Log stream task, this will generate when you resume the task with timestamp.

Hope my understanding is correct here, Else we can discuss on this further through the case.

Regards,

Sachin B

[Heinvandenheuvel]

note: This comment is just there to prevent other helpers from going down an incorrect path.

At first I thought you might need 'connect; ' on you command line to get live data vs static from the SQLite  repositories like :  data\tasks\<yourtask>\task_table.sqlite --> cdc_status. Checking with gettaskstatus you may well see 'STOPPED" while you know the task is running. checkpoint status  might be the exception because in fact you need it when there is no more live server to connect to.  

The Replicate doc shows

1. repctl -d "C:\Program Files\Attunity\Replicate\data” getcheckpoint "dg1task"
2. repctl -d connect; execute ....

No connect on the first command. And indeed, when you try this first command against a running system it works with or without connect. returning the same values. But when you try against a stopped system it works cleanly without connect but with connect, that connect command (of course!) fails. The getcheckpoint still works.

Now for the actual problem. When you execute the getcheckpoint against the original source tasks which feeds the logstream is that result as expected?

Hein.

C:\scripts>repctl -d \Replicate\data connect; getcheckpoint oracle_to_oracle
[connect; command] Succeeded
Failed to connect to localhost:3550: The operation completed successfully.
 (0) (exit status 1003518)
command getcheckpoint response:
{
        "checkpoint":   "checkpoint:V1#229#00000000.0a9af1a1.00000001.0000.00.0000:876.222804.16#0#0#*#0#0"
}
[getcheckpoint command] Succeeded