topic Re: Logstream vs direct task in Qlik Replicate

Logstream vs direct task

suvbin — Thu, 22 Feb 2024 06:21:26 GMT

Hi Team,

Any link for this topic please for more information

Thanks.

Re: Logstream vs direct task

john_wang — Thu, 22 Feb 2024 06:24:52 GMT

Hello @suvbin ,

There is a chapter about Using the Log Stream in User Guide. Hope it helps.

John.

Re: Logstream vs direct task

SachinB — Thu, 22 Feb 2024 06:31:01 GMT

Hello @suvbin ,

Thank you for reaching out to the Qlik community!

Here are the link for Logstream and Direct Tasks

1. Using Log Stream
2. Defining and managing tasks

More information on below community Link :

Log Stream Staging - The Why and the How?

Regards,

Sachin B

Re: Logstream vs direct task

SushilKumar — Thu, 22 Feb 2024 11:06:33 GMT

Hello team,

As pervious communication talks more about the Usage and info around using. Logstream vs direct task.

but it something a part of Planning or Solution designing.

for example. logstream can be used to Brodcasting and provide you some control over if you have some issue with target endpoint.

Reagrds,

Sushil Kumar

Re: Logstream vs direct task

suvbin — Sat, 24 Feb 2024 01:44:30 GMT

any more points please (logstream vs direct task) with respect to hitting the source endpoint for cdc.

Re: Logstream vs direct task

SachinB — Sat, 24 Feb 2024 02:31:52 GMT

Hello @suvbin ,

Log Stream Staging provides for single reader of the source transaction log irrespective of how many variations of targets required.
Log Stream Staging is only storing and managing CDC events for the selected tables.
Log Stream Staging has its own Batching, Retention, Rollover parameters.

Regards,

Sachin B

Re: Logstream vs direct task

SushilKumar — Mon, 26 Feb 2024 06:09:05 GMT

Hello Team,

If our response has been helpful, please consider clicking "Accept as Solution". This will assist other users in easily finding the answer.

Regards,

Sushil Kumar

Re: Logstream vs direct task

suvbin — Wed, 28 Feb 2024 05:11:44 GMT

Hi @SachinB ,

Appreciate your response.

So in this case, for logstream task , only one session will be created (connection will hit to the source DB) for every change, and in the direct task, every time whenever there is a change in source , will it create different sessions for source DB . Have some confusion on this.

Thanks.

Re: Logstream vs direct task

SachinB — Wed, 28 Feb 2024 09:22:43 GMT

Hello @suvbin ,

To efficiently replicate data from a single source to multiple endpoints, streamline the process by creating a single connection to source using log stream and multiple child tasks to distribute the data. This approach minimizes the burden on the source connection while effectively managing data replication across multiple endpoints.

Regards,

Sachin B

Re: Logstream vs direct task

Heinvandenheuvel — Wed, 28 Feb 2024 13:43:51 GMT

@suvbin >>> So in this case, for logstream task , only one session will be created (connection will hit to the source DB) for every change, and in the direct task, every time whenever there is a change in source , will it create different sessions for source DB . Have some confusion on this.

There is some odd terminology used in your question which may explain difficulty understanding the concepts.

Any CDC task will open a connection to constantly read the change log whether this is from the DB itself, such as an Oracle Redo log, or from a logstream collect by a parent logstream task reading all changes for all tables for all child tasks.

This is NOT "for every change" but a single connection which gets polled over and over in a read loop.

Using logstream you move the polling overhead, reading and network overhead from the actual source reading to stage file reading local to the Replicate server..

To complicate matters a little, this is for the CDC action. When (child) tasks initiallize and/or full-load then that data comes from the actual source DB and a connection to the source will be established from all the (child) tasks to the source DB. That session will remain idle once the task is in CDC phase. The direct source connection will also be used to read LOB contents for certain sources.

Does that clarify a little?

Hein

Re: Logstream vs direct task

suvbin — Wed, 28 Feb 2024 15:41:45 GMT

@Heinvandenheuvel Thank you for the response