Sense ETL Methodology

[tseebach]

So in QlikView we used to have a QVW that creates library of QVDs in a folder. My question is what are we planning to do in Qlik Sense? What will become our new best practices?

1. Should we create a QVF called Extract, and then put this as the first job in our Task list?
2. Should we create a separate Stream for these files. And who should access these?
3. I would also create a separate data connection to allow Self Service users to access these files.
4. Is there a preferred location and structure to store QVDs?
   • I assume we would prefer to place them on a separate drive or SAN
   • I would suggest a structure that is somewhat similar to the QlikView Deployment Framework
   • But is there a Best Practice in place?
5. Is there a point in using a Content Library for QVDs? Because it can be synced across clusters, it could be convenient, but I'm unsure how it can handle large frequently updated files.

My answers currently are (And please challenge me and write your method):

1. Create a QVF called Extract and Transform, to match a 2 tier ETL process. Make these run as the first jobs in the task list.
2. Place these files in 2 separate streams, with admin rights
3. Create multiple data connections, to reflect roles that need access these files
4. I'd store QVD's on a separate drive or SAN. Like
   • D:\Qlik\Qlik Sense\Default\Data\Staging < First Tier
   • D:\Qlik\Qlik Sense\Default\Data\Model < Second Tier
     • Default refers to a role ie. a Department or Function
       
5. No (I might be wrong).
   

[Anonymous]

Hi Magnus,

Many thanks for your reply. It's very informative indeed.

I just want some clarification for containers. Could you please explain containers in detail? And if you have any examples for containers, I would really appreciate if I could have a look at it.

Many thanks

[Magnus_Berg]

Hi, please join the Qlik Deployment Framework community group, here we have content and videos regarding QDF and containers. Qlik Deployment Framework

Best regards

Magnus

[Anonymous]

The organization I work for uses a separate stream for ETL QVFs oddly enough called ETL.  We have a QVF for extract which builds QVDs and a QVF for transform which uses the extract QVDs to build the QVDs we use in our production QVFs.

[DataWrangler]

Hi Torben,

I recently posted two items of potential relevance to your question; I hope you find them useful:

1. Seek and Ye Shall Find: Introducing the Structured ETL Engine for Qlik
2. Thrones, Castles, and Catapults: The Value of Rule-Based ETL Engines in BI Projects

Cheers,

Jeff R. Robbins

[sandeepj13]

Thanks Magnus.. It was very helpful.

[vkish16161]

Michael,

You guys do a great job at Qlik. I have a question in this regard as well.

The documentation suggests that Synthetic Keys are bad and a Star Schema should be achieved all the time. At least, that's what the wording alludes to. The problem is that most of my users (I work for a Large INSURANCE company) don't have any Data Modeling skills.

They've been building great reports using QlikSense but unfortunately their apps are getting rejected by I.T. because of synthetic keys.

Their major gripe is that if a tool is advertised as a Self Service solution, it's should be more accommodating to minor "transgressions" in Data Modeling.

There is an aggressive debate going on between Business and IT. Is there any way Qlik can point to an official documentation that says synthetic keys are O.K but not circular references?

This'll be a major drag factor IMO for Qlik in the near future.

Please let me know if you can help.

P.S - Some of the sample apps in QS Desktop have synthetic keys.

mto‌ rwunderlich‌