Qlik Community

Ask a Question

Qlik Compose for Data Lakes Discussions

Discussion board for collaboration on Qlik Compose for Data Lakes.

Announcements
QlikWorld Online 2021, May 10-12: Our Free, Virtual, Global Event REGISTER TODAY
cancel
Showing results for 
Search instead for 
Did you mean: 
Corne
Contributor
Contributor

Utilizing Provisioning Zone if HIVE projects were historically used

We have historically implemented Compose HIVE projects which does not have the provisioning zone functionality. We also have issues regarding large HDS tables and the current and full views that are not usable from our downstream tools. 

Some options we are exploring - creating daily snapshots of the large tables in order to bypass the current/full view.

Version 9 has been touted to solve some of the performance issues, hence wait for version 9

1. Can one create a decoupled SPARK compose job and only create a Provisioning Zone and keep the creation of the Storage Zones with the HIVE project. If technically this is possible is this a good idea?

2. IS there a similar "exit" in the compose HIVE step where one could potentially hook in a "snapshot" insert statement into new schema.

3. Wait for version 9

Any feedback would be appreciated

Labels (2)
2 Replies
TimGarrod
Employee
Employee

HI @Corne  - what cluster/hadoop platform are you on ? and do you have the ability to enable HIVE ACID transactions?   

#1 - You cannot take a Spark project and use a Hive based storage layer.  The projects are built to handle the end to end processing. 

2.  There is a Command Task which can run anything on the Compose server.  This could be a script or program / executable which could connect to Hive and run additional statements.   The Command task can be hooked to a Compose processing task via the Workflow feature. 

I'm not sure what "Version 9" is. 🙂

Corne
Contributor
Contributor
Author

Thanks - we are using Cloudera Data Platform. In short - no, we will not be able to use HIVE ACID transactions for the databases/tables in question. 

We will investigate your command task option, thanks.

Sorry, got my versions mixed up, I meant version 7. (Although I am sure version 9 will be even better 😉