Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Target Data System: Snowflake
GitHub : Project connected
Source Data System: QDI Project (Cross Project Pipeline)
- Source project is connected to a MS SQL server
Scenario: The Metadata project level settings affecting suffixes and prefixes for a Data Integration project are not being reflected into the push-down SQL. They in fact seem to be overwritten by Qlik Cloud and cannot be edited. We have only noticed this scenario when connecting projects to GitHub. This was not an issue prior to this week (3/31-4/4).
Details:
We are attempting to create a Data Integration project that we have versioned in GitHub by utilizing the steps mentioned in your help docs to "Share a project with another Space or Tenant" https://help.qlik.com/en-US/cloud-services/Subsystems/Hub/Content/Sense_Hub/DataIntegration/VersionC...
When following these steps and pulling down the changes from our main branch we then create a new branch to facilitate development for this project. Once the new branch is created, we then validate the "Metadata" project level settings to see that the suffix and prefix values are all set to a lowercase convention. We then prepare the first task in the pipeline and get the following error:
Prepare data task failed: Jobs execution failed. Job finished with error for process: "Create or replace transformation view 'HISTORY', for dataset 'AVT'". Failed to execute sql: Create or replace transformation view 'HISTORY', for dataset 'AVT'. SQL compilation error: Object '<Database>.<Schema>.AVT__HISTORY' does not exist or not authorized.
In the error above we can see that the suffix for the AVT history table is uppercase and not lowercase like specified at the project settings. However, when viewing the project settings again we can see that the lowercase convention has been overwritten to uppercase:
When checking the SQL Preview for this dataset we can also see that the query being executed has the same uppercase convention. Furthermore, in the Snowflake query history for the warehouse we can see the exact same SQL being executed:
When attempting to change the "Metadata" project level settings we can see that the "OK" confirmation button is grayed out and prevents the setting from being updated.
We also checked the source Storage Task dataset leading into this project (cross project pipeline) and the settings were also set to lowercase and when returning to the project they are defaulted to uppercase. The table was however, created with the lower-case suffix in Snowflake.
In summary, there seems to be an inconsistency on how Qlik Data Integration applies these settings when building the pushdown sql for executing data tasks. This is a complete show stopper for Snowflake based pipelines since Quoted Identifiers are used extensively in all Data Integration pushdown SQL for this target data system to which, the case of table names and other objects are incredibly important.
