topic Re: LogStream Staging Stream Position in Qlik Replicate

LogStream Staging Stream Position

Anonymous — Fri, 15 Nov 2024 23:13:10 GMT

Hello,

We are currently testing a number of replication solutions using log stream and a question has arisen in our work around understanding what the stream position indicators denote in the log files. We are replicating from Oracle.

Here is an example excerpt:

Start source from stream position 500;637477859264524360;20210111084947867861 and id 475378. Confirmed target id is 588049, next target id is 588053, last source timestamp 1612189124435211

So far, I have been able to deduct that 500 reflects the LogStream file that is being processed, because there are corresponding numbers in the source capture task that look the same. So as each rollover occurs this number increases.

The far right number of the stream position appears to be some kind of extrapolation of the date (2021XX...)
I do not know what the middle number is. Perhaps the audit sequence?

Last source timestamp looks close to being the Oracle SCN but I am not yet certain.

It would be helpful to understand this information; it does not appear to be documented.

A similar document was written to this that explains how to parse the stream position when reading directly from Oracle. I'm looking for a similar breakdown of this stream position information.

Re: LogStream Staging Stream Position

Bill_Steinagle — Thu, 14 Apr 2022 11:55:38 GMT

Team,

Below is the format to get the SCN for the Oracle Binary Reader and Log Miner with Replicate.

Binary Reader Stream Position

0075.1fafdbb9.00000006.0000.00.0000:116755.160175.16

0075.1fafdbb9 – SCN (6 bytes before 5.2, 8 bytes from 5.2)
00000006 - thread

0000 - serial number inside current SCN
00 - serial of current vector inside REDO event
0000 - serial number in the multiple insert event
116755 - REDO LOG SEQ
160175 - Block Number in REDO log
16 - not used

Log miner stream position

— Example:
0749A16B2C830100000000169AB40002850E00100000
6 bytes - SCN (6 bytes before 5.2, 8 bytes from 5.2) ‘
1 bytes – thread id
3 bytes – transaction sequencer
4 bytes – rbasqn (file sequence number)
4 bytes – rbablk (block index)
2 bytes – rbabyte (offset in block)
2 bytes – context sequence
last 6 last bytes - start SCN of the previous log miner session (6 bytes before 5.2, 8 bytes from 5.2)

Regards,

Qlik Support

Re: LogStream Staging Stream Position

Heinvandenheuvel — Thu, 14 Apr 2022 14:17:19 GMT

Hi @Bill_Steinagle ! You woke up an old topic.

I'll answer one more question from that older topic:

>> Last source timestamp looks close to being the Oracle SCN but I am not yet certain.

Noop. It is a timestamp in Unix Epoch format.

Verify with https://www.epochconverter.com/

If this is your first time visiting that site, be sure to scroll down for lots of programming hints.

1612189124435211 - GMT: Monday, February 1, 2021 14:18:44.435

Hein.

Re: LogStream Staging Stream Position

Michael_Litz — Thu, 14 Apr 2022 17:15:08 GMT

Hi Team,

I wanted to post this link to an article about starting logstream tasks with that last timestamp value:

https://community.qlik.com/t5/Knowledge/Starting-Log-stream-task-with-timestamp/ta-p/1879839

Michael Litz

Re: LogStream Staging Stream Position

lyka — Tue, 26 Apr 2022 21:58:09 GMT

Hello,

Did any of the suggested solution work?

Thanks

Lyka

Re: LogStream Staging Stream Position

brendanodwyer — Wed, 01 Oct 2025 19:24:00 GMT

This works. I created Python function that will parse it out for you:

def convert_oracle_hex_to_decimal_scn(hex_string):
   byte_offsets = {
       "scn": 8,
       "thread_id": 1,
       "transaction_sequencer": 3,
       "file_sequence_number": 4,
       "block_index": 4,
       "context_sequence": 2,
       "start_previous_session": 8
   }
   start_idx = 0
   end_idx = 0
   for field, offset in byte_offsets.items():
       end_idx = end_idx + (offset * 2)
       if field == "start_previous_session":
           hex = hex_string[start_idx:]
       else:
           hex = hex_string[start_idx:end_idx]
       start_idx = end_idx
       print(f"{field}: {int(hex,16)}")