topic Re: Is it possible to enable Store Changes when using Logstream ? in Qlik Replicate

Is it possible to enable Store Changes when using Logstream ?

iti-attunity-sup — Thu, 27 Jun 2024 07:12:54 GMT

Hello support team.

[Info]
Source : Oracle(Logstream)
Target : Snowflake

[Question]
Is it possible to enable Store Changes when using Logstream ?(Is there any limitation ?)

When executed task (enable Store Changes and using Logstream), task was crashed.

If there is no limitation, would you modify task won't crash.

Best Regards.

Re: Is it possible to enable Store Changes when using Logstream ?

john_wang — Thu, 27 Jun 2024 11:17:28 GMT

Hello @iti-attunity-sup ,

I don't believe this is an appropriate usage, as it introduces unnecessary overhead to storage and resources without providing any benefit to downstream tasks.

Hope this helps.

John.

Re: Is it possible to enable Store Changes when using Logstream ?

aarun_arasu — Thu, 27 Jun 2024 11:24:42 GMT

Hello @iti-attunity-sup ,

Please refer to the below user guide for the limitation

https://help.qlik.com/en-US/replicate/May2024/Content/Replicate/Main/Log%20Stream%20Staging/ls3.htm

Note : The Full Load and Store Changes replication options are not relevant.

Regards

Arun

Re: Is it possible to enable Store Changes when using Logstream ?

iti-attunity-sup — Fri, 28 Jun 2024 00:46:30 GMT

Hello Arun.
Thank you for your reply.

This is Log Stream Staging task limitation.
But replication task to transfer the staged data to the target also has same limitation ?

Best regards.

Re: Is it possible to enable Store Changes when using Logstream ?

sureshkumar — Fri, 28 Jun 2024 01:19:28 GMT

Hello @iti-attunity-sup

Replication (Child) Task is like normal task, there is no such limitation.

You can able "Store Changes/Apply Changes" once the Full Load completes.

Regards,
Suresh

Re: Is it possible to enable Store Changes when using Logstream ?

iti-attunity-sup — Fri, 28 Jun 2024 04:37:25 GMT

Hello Suresh.

What I want to do is following
-task configuration
Logstream task : Oracle → Logsteam
Repcation task : Oracle(logstream) → Snowflake ※Replication task is enable Store Changes

But I recognized first answer is "this task configuration is not appropriate usage".
After all this task configuration is possible? (But when task executed, task crashed)

Best Regards.

Re: Is it possible to enable Store Changes when using Logstream ?

SushilKumar — Fri, 28 Jun 2024 06:28:06 GMT

Hello @iti-attunity-sup

Logstream task and replicate task are there to serve Different purpose.

Logstream task mainly configure were you want to reduce no of process/query to get same kind of data and put less load on Connected Source Db. So only few processes engage Source db. to read the changes from Db or the logs and put them in a format where Replicate task extract changes and push them to intended targets.

So, in nutshell the Staging folder where logstream put the files to further read by replicate task provide kind of fault tolerant as well. And generated stored files only have the data related to participating tables while if you configure normal task which direct read from db. or redo logs.

when we choose LOGSTREAM task it will scan and push data work in this manner ( extract change and put into files for further read)

When we choose Normal REPLICATE task then it will scan and push data work in this manner ( extract change , Transform and push data to Target end)

Change table is created on the target side. hence, it's not possible for logstream to work with store changes

hope it helps.

if you like the explanation then request you to mark is as a solution.

Regards,

Sushil Kumar

Re: Is it possible to enable Store Changes when using Logstream ?

SachinB — Fri, 28 Jun 2024 07:29:29 GMT

Hello @iti-attunity-sup ,

Log Stream enables a dedicated Replicate task to save data changes from the transaction log of a single source database and apply them to multiple targets thereby eliminating the overhead of reading the logs for each target separately.

When you create task with Log stream you will store source data in stage and from stage you can consume the data as part of CDC it could be apply or store changes

If your task is crashing, I would request you create a case and will help you with screen sharing.

Hope this helps you.

Regards,

Sachin B

Re: Is it possible to enable Store Changes when using Logstream ?

iti-attunity-sup — Fri, 28 Jun 2024 09:40:50 GMT

Hello Sushil

Thank you for your reply.

To sum up,
there is limitation that Replication task with Logstream cannot use Store Changes ?

So following is impossible?
-task configuration
Logstream task : Oracle → Logsteam
Repcation task : Oracle(logstream) → Snowflake ※Replication task is enable Store Changes

Best Regards.

Re: Is it possible to enable Store Changes when using Logstream ?

kng — Fri, 28 Jun 2024 11:28:19 GMT

Hi @iti-attunity-sup

Good Day

Its possible to have a task configuration like this

logstream task: from source Oracle to Logstream

Replication task: Oracle(logstream) → Snowflake ※Replication task is enable Store Changes

In Replication task Yes we can enable store changes or apply apply changes also

Regards,

shivananda

Re: Is it possible to enable Store Changes when using Logstream ?

Heinvandenheuvel — Fri, 28 Jun 2024 17:01:01 GMT

@aarun_arasu wrote: Note : The Full Load and Store Changes replication options are not relevant.

As per documentation

@iti-attunity-sup replied: This is Log Stream Staging task limitation.

No it not a limitation. It is per design. Store-changes would be a nonsense-option as there is nothing to read as @sureshkumar indicates - storage changes is available as normal in the child = actual replication task.

>>> But I recognized first answer is "this task configuration is not appropriate usage".

That's just John being confused with your imprecise wording interpreting that as if you wanted the logstream 'parent' task to be configured for store-change - which is how I also read it at first.

>>> Repcation task : Oracle(logstream) → Snowflake ※Replication task is enable Store Changes >>> After all this task configuration is possible? (But when task executed, task crashed)

Yes. That should work. Tell us more about the crash - when? (startup? running? stopping?) where? (last successful acivity? What was expected next? Any -E- messages? Repeatable? Retry with verbose logging alll - just before an expected crash?)

Hein