Option to group records that form part of a single transaction when sending to Kafka
As suggested, I am breaking out my previous idea out into seperate Ideas.
Use case: We want to do CDC from a relational database to Kafka. We want to create a domain level event from the writes (multiple tables) that happen as part of a single database transaction. As a result we need to know the transaction boundaries and the writes that happened as part of that transaction.
In order to put in place an implementation for the above scenario, we need to capture all records that belong to a single transaction. In order to do so using the current functionality in 'Qlik replicate' requires complex aggregation, potentially repartitioning and the use of 'windows , E.g. Session Windows' in Kafka, in order to capture all records that belong to a single transaction. Writing to a buffer, as suggested by the docs is not seen as being efficient.
The implications are as follows:
The complexity of implementation.
The latency associated with having to use "Session Windows". Events are not real time.
Having to use a Single topic and partition, because of the limited partitioning options in the product.
What would make life easier, is if it was possible to have an option in 'Qlik Replicate' to send ALL (selected) records that form part of a single transaction in the same payload to Kafka. My understanding from the documentation is that the product already groups and sorts the records, if we could have the option to receive the entire transaction in a single payload, that would greatly simplify the implementation for the consumer.