Suggested development methodology for new QCA Capacity licenses
Hello Community!
For a long time now we have been using the three stages methodology for Analytics applications development on premise (DATA, STG, APP).
However, the new capacity licensing for Qlik Cloud Analytics prevent us from using this approach as it is not efficient according to its Value Meters.
Acoording to Qlik Help, “The daily peak is calculated as the sum of all file sizes for formats like QVD, CSV, or text files plus the maximum bytes ingested from external sources for app reloads on that day”.
I can think of 2 main scenarios:
Build the analytics apps directly from the external database: The ingestion from external sources is reduced but I can think of several issues:
Data Base performance may be affected by continuous queries
Many reloads during the development and the ETL process
Different reload time for apps may result in different outcomes
You cannot work with incremental reloads
Use the DATA stage with incremental reloads and do the ETL process in the Analytic apps: The incremental reloads will reduce the ingestion from external sources but the DATA qvds will consume an important share of the daily quota. The challenges for this option are:
Qvds partition may result in unused data and should be deleted if not used (maybe through an automation?).
Can’t use Select * FROM as it will upload fields and data that may not be used.
ETL Process must be done in the Analytics apps and scripts will probable be more complex.
Can anyone think of any other option? Can you share your thoughts?
Yes, this is definitely an issue. I have started tracking consumption via the API, especially during development, and it quickly becomes clear that you cannot be as productive as you would like without spreading development over multiple days. The current capacity model effectively discourages established best-practice app development patterns in Qlik Cloud.
To mitigate this, one option is to shift transformations outside of Qlik Cloud using third-party ETL tools and only push curated datasets into the cloud, which helps reduce unnecessary field and data ingestion. Another practical approach within Qlik itself is to introduce a development mode in the load script, where you limit the number of rows during development so that generated QVDs remain small, for example:
AND ($(vDevMode) = 0 OR RowNo() <= $(vDevRowLimit));
This at least allows iterative development without immediately exhausting the daily capacity.
Third-party tools take us away from Qlik and I usually want to keep data transformation simpler and visible for the developer in one place.
What we had tested in the last months is to create DATA qvds with the personal space, at least in development mode. By doing so, you prevent the load statement from using the daily capacity!
@mspirandelli That's an interesting approach i need to test that approach with the personal space.However, I assume that only applies to load, and any qvd store still counts to capacity. That's the main thing that bothers me. It goes against the development best practices. How are you tracking the capacity. With the API? Since the dashboard has a 24h lag. Best!
