Logstream vs direct task

[suvbin]

Hi Team, 

Any link for this topic please for more information

Thanks.

[john_wang]

Hello @suvbin ,

There is a chapter about Using the Log Stream in User Guide. Hope it helps.

John.

[SachinB]

Hello @suvbin ,

Thank you for reaching out to the Qlik community!

Here are the link for Logstream and Direct Tasks

1. Using Log Stream 
2. Defining and managing tasks 

More information on below community Link :

Log Stream Staging - The Why and the How? 

Regards,

Sachin B

[SushilKumar]

Hello team,

As pervious communication talks more about the Usage and info around using. Logstream vs direct task.

but it something a part of Planning or Solution designing.

for example. logstream can be used to Brodcasting and provide you some control over if you have some issue with target endpoint. 

Reagrds,

Sushil Kumar

[suvbin]

any more points please (logstream vs direct task) with respect to hitting the source endpoint for cdc.

[SachinB]

Hello @suvbin ,

• Log Stream Staging provides for single reader of the source transaction log irrespective of how many variations of targets required.
• Log Stream Staging is only storing and managing CDC events for the selected tables. 
• Log Stream Staging has its own Batching, Retention, Rollover parameters. 

Regards,

Sachin B

[SushilKumar]

Hello Team,

If our response has been helpful, please consider clicking "Accept as Solution". This will assist other users in easily finding the answer.

Regards,

Sushil Kumar

[suvbin]

Hi @SachinB , 

Appreciate your response.

So in this case, for logstream task , only one session will be created (connection will hit to the source DB) for every change, and in the direct task, every time whenever there is a change in source , will it create different sessions for source DB . Have some confusion on this. 

Thanks.

[SachinB]

Hello @suvbin ,

To efficiently replicate data from a single source to multiple endpoints, streamline the process by creating a single connection to source using log stream and multiple child tasks to distribute the data. This approach minimizes the burden on the source connection while effectively managing data replication across multiple endpoints.

Regards,

Sachin B

[Heinvandenheuvel]

@suvbin >>> So in this case, for logstream task , only one session will be created (connection will hit to the source DB) for every change, and in the direct task, every time whenever there is a change in source , will it create different sessions for source DB . Have some confusion on this. 

There is some odd terminology used in your question which may explain difficulty understanding the concepts.

Any CDC task will open a connection to constantly read the change log whether this is from the DB itself, such as an Oracle Redo log, or from a logstream collect by a parent logstream task reading all changes for all tables for all child tasks. 

This is NOT "for every change" but a single connection which gets polled over and over in a read loop.

Using logstream you move the polling overhead, reading and network overhead from the actual source reading to stage file reading local to the Replicate server..

To complicate matters a little, this is for the CDC action. When (child) tasks initiallize and/or full-load then that data comes from the actual source DB and a connection to the source will be established from all the (child) tasks to the source DB. That session will remain idle once the task is in CDC phase. The direct source connection will also be used to read LOB contents for certain sources.

Does that clarify a little?

Hein