Difference between Log streams and Apache Kafka

[MdFazil]

Hi, 


It's been quite some days I started using Qlik replicate. I recently came across "Log Streams" from Qlik replicate which is basically used to map from a single source to multiple targets. On the other hand I used to do replication using Apache kafka and similarly I have replicated a single source to multiple targets. 

Now my main question is how "Log Streams" in Qlik Replicate differs from Apache Kafka?

Regards
Fazil M

[aarun_arasu]

Hello @MdFazil ,

Thanks for reaching out to Qlik community

 Log Stream enables a dedicated Replicate task to save data changes from the transaction log of a single source database and apply them to multiple targets, thereby eliminating the overhead of reading the logs for each target separately.

Please refer to below guide which explains how logstream works in Qlik replicate with both pros and cons

https://help.qlik.com/en-US/replicate/November2023/Content/Replicate/Main/Log%20Stream%20Staging/ls1...

Regards

Arun

[aarun_arasu]

Hello @MdFazil ,

Additionally , to answer your question

Here's a simpler explanation:

Qlik Replicate logstream is  specially designed for capturing database changes using CDC , while Kafka is a general-purpose distributed streaming platform.

Regards

Arun

[MdFazil]

Hello @aarun_arasu ,

Thank your for the reply.


Yes, You're right That Qlik is specially designed for replication purpose. Apart from that is there any specific difference between them?


Regards
Fazil M

[SushilKumar · Accepted solution]

Hello @MdFazil 

Here are the differences you are looking for. 

Qlik Replicate Log Streams is a feature that allows you to replicate log data from source systems to target systems and enable Qlik replicate task to read transaction logs and fetch metadata from source put it into Staging area to be read again and push to multiple targets endpoint. thereby eliminating the overhead of reading the logs for each target separately.

Log streams refer to the continuous flow of log data generated by a system, application, or service. They provide real-time information about events, errors, and other activities that occur within the system. Log streams can be captured from various sources, including system logs, application logs, and network logs. The data in log streams can be analyzed to troubleshoot issues, monitor system performance, and gain insights into user behavior. Log streams can be stored in different formats, including CSV, JSON, and Kafka, and can be processed using various tools and technologies, such as Elasticsearch, Kibana, and Apache Spark.

Apache Kafka is a distributed streaming platform that is designed for handling live data streams in real-time. Apache Kafka is also designed for scalability, fault tolerance, and high throughput, making it a popular choice for handling large amounts of data from multiple sources.
Ultimately, the choice between log streams and Apache Kafka will depend on the specific needs of your use case.

Regards,

Sushil Kumar

If our response has been helpful, please consider clicking "Accept as Solution". This will assist other users in easily finding the answer.

[MdFazil]

Hello @SushilKumar ,

Thank you for replying. 

This reply seems to be legit and yes the fact Qlik is specially designed for replication task is alone enough. Which give us more advantage than Kafka. Thank you.

Regards 
Fazil M