Suggest an Idea

[joelm777]

Compose Live views very slow due to large number of sequence files in _CT Tables for real time latency use cases. Provide an ability to MERGE seq files in _CT Table or create large files

Why we need this: with databricks endpoint, we have to set “Change Processing” settings to default 30 seconds and 32 MB for a use case where we need near real time latency when querying live views. Doing so, replicate is creating like 3,600 files in each partition and we are partitioning every hour for the __CT Table. The impact of this is, over time, live views becomes very slow to query due to the large number of files for use cases specifically where we need the tables to have historical data or have high volumes in a short period of time.

Impact:  Slow live views and even though from replicate standpoint we have near real time latency, the live views are so slow that they defeat the purpose of having near real time latency.

Ask: replicate automatically merge the small sequence files once the current partition is closed or any other acceptable manner.

[Tzachi_Nissim]

Thank you for the suggestion. I understand exactly what you mean, but I do think that this should be opened as a support ticket for Compose. The performance of the live views should not deteriorate over time as long as you keep the same cadence of running the Compose task for actually merging changes into the target. This is likely either a tuning exercise, an issue in the Compose generated live views, or maybe some maintenance needed on the landing, but I don't think what you are suggesting. In any case - this should be in the Compose domain.