I need to create a data model in Qlik Sense. The scenario is as follows: I have four tables imported from Databricks. (already imported with script) I have 3 fields that will be my key. one of the tables I need to do a left join with a QVD file. I also need to create a link/bridge table. how to create and organize sections? what order should I put my sections in? How do I know which Fields and which tables I need to DROP and when to do it? another doubt I have is when to use the sub and end sub, and at what time do I need to use the CALL function. If you can help me, I'd appreciate it.
There are many ways to build and organize your loader. You could use sub routines if you have repeated tasks throughout your code, loops for example. Just write it in a section and call it by its name and pass the arguments.
If you're extracting new data on a regular basis, let's say daily, it'd be smart to separate your ETL in different apps, a "Extractor" to save your data into qvds in a fully or incremental load, depending on your needs. Then you'd have another qvd to manipulate the data, where you'll build keys if needed and/or clean the data and then save the files ready for visualization.
As for dropping fields and tables, you will drop everything you don't need anymore, whether for memory optimization or to avoid unwanted relationships or auto concatenations. If you're extracting data, there's probably no need to keep any table after you have saved the data and be sure to drop temporary tables after you're done with them. If you're loading data for visualization you will need to relate the tables you've built and maybe that's where you'll create the bridge table.
