HDFS or Hadoop as target

[Pranita123]

Hello,

I'm currently using Oracle 19c as the source and HDFS as the target, with the file format set to Parquet. Additionally, I've configured a replication task with Full Load + Store Changes settings.

However, during my observations, I've noticed that Qlik Replicate is creating four tables:

- Full load
- Store Changes table
- Full load with suffix _att_temp
- Store Changes table with suffix _att_temp

I'm curious to understand the purpose of these _att_temp tables and why Qlik Replicate creates them.

Regards,

Pranita

[Pranita123]

Additionally, I've observed that in HDFS, Qlik Replicate creates two directories named "Delta" and "Base". I'm interested in understanding the significance of these directories and why Qlik Replicate generates them in HDFS.

[john_wang]

Hello @Pranita123 ,

Thanks for reaching out to Qlik Community!

Qlik Replicate need to use the interim files/tables in both Full Load stage and CDC stage especially if compression option is enabled. The data will be loaded into these _att_tmp tables first then merge to final target table by SQL likes (the table name is "KIT" in below sample):

INSERT INTO TABLE `default`.`KIT__ct` SELECT `header__change_seq` , `header__change_oper` , ...
FROM `default`.`KIT__ct_att_tmp`

Hope this helps.

John.

[john_wang]

Hello @Pranita123 ,

When it comes to the directories "Delta" and "Base", if I understood correctly, the "Base" folder is used to store the Full Load data, and the "Delta" for CDC store changes data. We can get double-confirmation from the data files name in the directories.

Regards,

John.

[SushilKumar]

Hello team,

If our response has been helpful, please consider clicking "Accept as Solution". This will assist other users in easily finding the answer.

Regards,

Sushil Kumar