New job - with Qlik

[claus]

Hey guys,

I started a new job couple of months ago and I have questions for you.

Company : Retail company with millions and millions of rows of sales. Thousand of sales daily

Currently : Each morning Qlik automations retrieve data for the DB and store it in .qvd files (qlik format) and then the different dashboard read those files

Scenario :

I'm the only BI person of the company and my job consist to retrieve data from the so called PostgreSQL "data warehouse" (but for me it's more of a database because it's straight raw facts and dimensions tables that we need to join) that our ERP company made available for us, transform that data and create different dashboards with Qlik.

At the moment, the company decided to do everything in Qlik, so the data is extract from the DB with qlik, transform in qlik load script and then use in the different reports.

I'm looking to modernize the stack, implement a more robust ELT/ETL process and create a more scalable approach because the company plan to grow a lot in the future.

What would you suggest to use at each moment of the process ? (preferably open-source)

Just so you know, intermediate person here (great ELT/ETL, SQL and Python knowledge).

Thanks !

[marcus_sommer]

From my point of view doing everything within Qlik (View/Sense - without Compose/Replicate/Catalog) is a nearly a perfect scenario - especially by being a one-man-show. Reasons for it are the simplicity of the ETL jobs and the data-models and their powerful flexibility and performance. Millions of records could be classified as a "normal" data-set and is far away from being big or even huge. Nevertheless it will depend on the available hardware, the requirements and personal resources how well it would work.

Simplicity means of course to distribute the work into multiple layer and using a multi-tier data-architecture with at least 3 layers, like: generator --> data-model --> report with incremental load-approaches wherever it's needed. Important is the attempt to centralize the common stuff (paths + connections + variables + functions + filelist() loops + section access + ...) within external include-variables and also harmonizing all approaches (whereby being more practically as strict and rather skipping layers and logic as artificially creating ones if they aren't needed).

In this way is Qlik very  scalable and with a performance which you couldn't get with any other tools on a similar sized hardware. Especially beautiful is that everything could be done within Qlik and a normal storage - there is no need to handle with multiple tools and/or programming logic which usually creates more complexity as the single strengths of them are worth it. Having great ETL knowledge with SQL and Python could be quite valuable in Qlik, too but trying to implement their logic within Qlik is rather counterproductive as Qlik used an associative data-model and not a relational one like SQL tools for nearly everything exists simple and straightforward features in Qlik as that it would be sensible to do it programmatically in Python.