Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 
suvbin
Creator II
Creator II

Logstream vs direct task

Hi Team, 

Any link for this topic please for more information

Thanks.

 

Labels (1)
10 Replies
john_wang
Support
Support

Hello @suvbin ,

There is a chapter about Using the Log Stream in User Guide. Hope it helps.

John.

Help users find answers! Do not forget to mark a solution that worked for you! If already marked, give it a thumbs up!
SachinB
Support
Support

Hello @suvbin ,

 

Thank you for reaching out to the Qlik community!

Here are the link for Logstream and Direct Tasks

1. Using Log Stream 
2. Defining and managing tasks 

More information on below community Link :

Log Stream Staging - The Why and the How? 

Regards,

Sachin B

SushilKumar
Support
Support

Hello team,

As pervious communication talks more about the Usage and info around using. Logstream vs direct task.

but it something a part of Planning or Solution designing.

for example. logstream can be used to Brodcasting and provide you some control over if you have some issue with target endpoint. 

Reagrds,

Sushil Kumar

  

suvbin
Creator II
Creator II
Author

any more points please (logstream vs direct task) with respect to hitting the source endpoint for cdc.

SachinB
Support
Support

Hello @suvbin ,

 

  • Log Stream Staging provides for single reader of the source transaction log irrespective of how many variations of targets required.
  • Log Stream Staging is only storing and managing CDC events for the selected tables. 
  • Log Stream Staging has its own Batching, Retention, Rollover parameters. 

 

Regards,

Sachin B

SushilKumar
Support
Support

Hello Team,

 

If our response has been helpful, please consider clicking "Accept as Solution". This will assist other users in easily finding the answer.

 

Regards,

Sushil Kumar

suvbin
Creator II
Creator II
Author

Hi @SachinB , 

Appreciate your response.

So in this case, for logstream task , only one session will be created (connection will hit to the source DB) for every change, and in the direct task, every time whenever there is a change in source , will it create different sessions for source DB . Have some confusion on this. 

Thanks.

SachinB
Support
Support

Hello @suvbin ,

To efficiently replicate data from a single source to multiple endpoints, streamline the process by creating a single connection to source using log stream and multiple child tasks to distribute the data. This approach minimizes the burden on the source connection while effectively managing data replication across multiple endpoints.


Regards,

Sachin B

Heinvandenheuvel
Specialist II
Specialist II

@suvbin >>> So in this case, for logstream task , only one session will be created (connection will hit to the source DB) for every change, and in the direct task, every time whenever there is a change in source , will it create different sessions for source DB . Have some confusion on this. 

There is some odd terminology used in your question which may explain difficulty understanding the concepts.

Any CDC task will open a connection to constantly read the change log whether this is from the DB itself, such as an Oracle Redo log, or from a logstream collect by a parent logstream task reading all changes for all tables for all child tasks. 

This is NOT "for every change" but a single connection which gets polled over and over in a read loop.

Using logstream you move the polling overhead, reading and network overhead from the actual source reading to stage file reading local to the Replicate server..

To complicate matters a little, this is for the CDC action. When (child) tasks initiallize and/or full-load then that data comes from the actual source DB and a connection to the source will be established from all the (child) tasks to the source DB. That session will remain idle once the task is in CDC phase. The direct source connection will also be used to read LOB contents for certain sources.

Does that clarify a little?

Hein