My company is moving to qlik cloud, now we are thinking to built a datwarehouse first, as we have none yet. As volumes and infrastructure is not the complicated, I am thinking just to use a data lake and do transformations in Qlik cloud itself, as it can be a fast delivery cycle en cost effective. Any experience/advise with this in the community?
You are going to hear multiple and different opinions about this topic. Qlik Platform has tools to build and manage proper Data Warehouses: Talend and Talend Cloud, however, in Qlik Cloud, you can mimic some features of Data Warehouses using Data files as outputs of your ETL/ELT processes. You can design Fact, Dimensions and Data Marts, even creating Bronze/Silver/Gold layers (in Qlik is mostly common to call these Layer 1, Layer 2 and Layer 3 files).
In general, without any specific requirements, you could create Layer 1 Builder or Extractor applications (use Qlik Script apps). Those Qlik Scripts are the only applications in your solution that will touch any data source. The purpose of those applications is just to connect to your data source and land the records to your Qlik Cloud - or any other storage platform - without any major transformations if possible (I recommend here only doing data reduction, for example, getting only the required fields and rows). You can use in this stage multiple techniques to increase the performance of your loads, such as data partitions and incremental loads. This is here you also should define how often you are going to refresh your data, settings up Schedulers, Qlik Application Automation or custom processes using Qlik APIs.
Usually, Qlik Developers choose by default to store their data in QVD format because it is fast and very compact, however, Parquet demonstrated to be much faster and smaller - especially with BROTLI compression - and there are new features coming to Qlik Cloud that may make Parquet files even better - ready about S3 Table Buckets, Iceberg and Upsolver.
Now, for transformations, you just need to create other Qlik Script applications to load, combine, transform the files generated during your lading/layer 1 process. Keep naming convention consistent (Data Space names, subfolders, file names, etc) and you should be good to go.
Don't be afraid about making mistakes, fixing and adjusting things later. Data Warehouse is a living thing and will change with time.
I think this is good for now. Please let us know if you have any problems during your Qlik journey.
