Migrate Hive projects from Compose for Data Lakes 6.6 to Qlik Compose 2021.8

Troy_Raney — Tue, 10 May 2022 12:10:55 GMT

This is part 3 of 4 in a series of articles with information about migrating Hive HDS projects from Compose for Data Lakes 6.6 (C4DL 6.6) to Qlik Compose 2021.8 (Gen2)

Different Migration Modules for customers

Migration Pre-Requisites
- Pre-Requisites for Migrating Projects from Compose for Data Lakes to Qlik Compose
Databricks to Databricks
- Migrate Databricks projects from Compose for Data Lakes 6.6 to Qlik Compose 2021.8
Spark to Spark supported version
- Migrate Spark projects from Compose for Data Lakes 6.6 to Qlik Compose 2021.8

Module 3: Migrate Hive HDS projects from C4DL 6.6 to Gen2

There are two paths for Hive HDS project, you can choose one of the paths to finish migration.

Path 1: You can clean up storage database and replicate landing database (including underlaying files for attrep_cdc_partition). Reload replicate task and start the compose tasks as it is new project. It was covered in first demo.

Path 2: You can follow migration path, and we explained all the required steps in this document.

Here are the migration steps (Path 2) if you don’t want to reload the replicate task. You are going to completely migrate your project definition and data to Gen2.

Install Qlik Compose 2021.8 on a dedicated machine\the same machine where FDL 6.6 exist.

Select a project in C4DL 6.6 and run the CDC task last time with verbose.

Create the deployment package à Suresh_Hive_HDS_deployment_<datetime>.zip
Run the CDC task last time before migrate data and disable if there are any schedules on the tasks.

Create new project in 2021.8 with the same name and project type as it is in FDL 6.6. you have to migrate data also so create a new storage database with different name (we can’t use same database which is using in C4DL project) and provide it in Data Lake connection. Do the test connection to make sure you have correct drivers installed for your data lake. Now we have an empty project in gen2 with storage connection.

Deploy the projects on Qlik Compose 2021.8 and migrate the data

Migrate the project definitions

ComposeCli.exe adjust_cfdl_project --project Suresh_Hive_HDS --infile “C:\Program Files\Qlik\C4DL66\ Suresh_Hive_HDS_deployment_<datetime>.zip”

- C:\ProgramFiles\Qlik\Compose\data\projects\Suresh_Hive_HDS\deployment_package\ Suresh_Hive_HDS__<datetime>__QlikComposeDLMigration.zip

Deploy adjusted deployment package on gen2 project.
Validate model and create the storage tables.

Migrate the data

Generate tasks then execute a data migration SQL script from notebook.
Generate 'create data migration script’ using below cli (Qlik Compose 2021.8 cli) command and run it on storage database using External tool.

ComposeCli.exe create_cfdl_data_migration_script --project Suresh_Hive_HDS --infile “C:\Program Files\Qlik\C4DL66\ Suresh_Hive_HDS_deployment_<datetime>.zip”

verify the data is correct

NOTE:

We must use new storage database in path 2 because we are going to run select * from C4DL 6.6 storage database and insert into Qlik Compose 2021.8 storage database in the migrate data step (3 (b)).
If you want, you can run C4DL project and Gen2 project parallelly for few days.

After migration you can check\run

ODS and HDS live views before running CDC
CDC with updates\inserts\deletes
CDC with reload event
schema evolution - you should run schema evolution and skip all detected changes - to set the context

also set

workflow
scheduler
notification
work with multiple replicate servers

You can watch demo video How to Migrate Hive HDS projects.

Environment

From Compose for Data Lakes 6.6
To Qlik Compose 2021.8

The information in this article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support.

Re: Migrate Hive projects from Compose for Data Lakes 6.6 to Qlik Compose 2021.8

Abrie_M — Tue, 07 Feb 2023 06:04:35 GMT

Good morning

Running the data migration generation script, I get the following error:

[START: 2023-02-06 16:15:37 Qlik Compose Version: 2022.5.0.523]
1 2023-02-06 16:15:37 [Command ] [INFO ] Executing CreateDLDataMigrationScript command.
1 2023-02-06 16:15:42 [Command ] [ERROR] Command CreateDLDataMigrationScript failed.
1 2023-02-06 16:15:42 [Command ] [ERROR] SYS-E-HTTPFAIL, The given key was not present in the dictionary..
[END: 2023-02-06 16:15:42]

The documentation is not very clear of what could be missing. How do I troubleshoot this?

Thanx

Abrie Marais

article Migrate Hive projects from Compose for Data Lakes 6.6 to Qlik Compose 2021.8 in Official Support Articles

Migrate Hive projects from Compose for Data Lakes 6.6 to Qlik Compose 2021.8

Environment

Related Content

Re: Migrate Hive projects from Compose for Data Lakes 6.6 to Qlik Compose 2021.8