QVD generator - Run SQL in parallel instead of consecutively
Hey guys,
I have a QVD generator that is running 3 separate SQL queries from various databases. I'm using Qlik to generate a superset QVD, combining all three datasets. This is working, currently, to build my main QVD for it's corresponding front-end app.
The downside that I'm trying to tackle now is that, due to the SQLs running consecutively in section order, the QVD generator is taking 3x longer than expected. I'm hoping someone has a solution for running these SQL queries in parallel such that the QVD generator does not take much longer than the longest query, instead of a SUM of each individual query run-time.
Thanks.
**EDIT** Also trying to avoid 3 separate QVD generators. 😉
Qlik Script is wonderful as a basic ETL tool, but it does not support parallelism in a single script. For parallel processing as you've suggested I believe you will need to use a primary ETL tool like Talend.
Or...as you've already imagined, use four generators - three extractors and one combiner 🙂
