QR: Log stream space estimates and tuning

[Jon_Donker]

Hi All,

We have an existing Qlik task that transfer changes from 30+ tables to S3.

The business wants to duplicate this pipeline and land the changes to another source.

To facilitate this duplication I am looking at creating a log stream for the source and running a pilot on a small database in Dev.

I have two questions:

1) Can we estimate the storage space requirements needed for log streaming from our current production task using the "Aggregate -> Change Volume" metric? 

2) Is there any guides to tuning log streaming performance?  Currently our pilot without tuning is running four time slower than direct transfer.

Thanks.

[DesmondWOO]

Hi @Jon_Donker ,

You can estimate the storage size by the change volume size. However,  log stream file includes additional information such as stream position, commit timestamp, and other information. Conducting multiple tests and evaluations would be necessary to obtain a more precise estimation of the required size. 

I would suggest setting up notification for the disk space also. This would help you stay aware of any potential issues with disk space becoming full. 

Direct transfer is faster due to Replicate processing changes in memory. In contrast, Log stream involves two-step process: writing changes to disk and then reading those changes from disk, which result in slower performance.

I haven't been able to locate any guide specifically focused on tuning log stream performance. Considering the numerous I/O operations involved in the log stream, faster disk can help.

Regards,
Desmond

[SachinB]

Hello @Jon_Donker 

You can refer the below article in order to create fresh logstream task.

https://community.qlik.com/t5/Official-Support-Articles/LogStream-Setup/ta-p/1743657

Regarding Space requirement for log stream, It's depend on how much source generating volume on daily basis and how long you would keep them in staging directory .

https://help.qlik.com/en-US/replicate/May2023/Content/Replicate/Main/Log%20Stream%20Staging/intro.ht...

Tuning of the task depends on the situation where your task is having source or target latency for log stream or child tasks, based on analysis we can fine tune the tasks.

We would request create a support ticket in order to investigate it further by attaching required logs and diagnostic package.

Regards,

Sachin B