Segments in Parallel Load

[Ginni]

What concept is used to create segments in Parallel Load?

Let say we have one table Emp having counts 50,00,000 to load with Parallel Load,

1. How will we come to know no of segments to be created and range of each segments?
2. How does each segment efficiently load large tables without taking more time?

[john_wang · Accepted solution]

Hello @Ginni ,

Thanks for reaching out to Qlik Community!

1. There are two data separating method in Qlik Replicate GUI:

• Use Data Ranges: Choose this option to segment the source data by user-defined columns and data ranges. You may define the no of segments to be created and range of each segments freely. 
• Use Partitions: Choose this option to segment the source data by existing partitions or sub-partitions of the source table(s). You may query the rows number of each partition in source database.

2. In Full Load replication mode, Qlik Replicate accelerates the replication of large tables by splitting the table into segments and loading the segments in parallel so far speed up the load process.

More detailed information can be found at here.

Hope this helps.

John.

[SushilKumar]

Hello @Ginni 

To add more to Expert comment here. By default, if you have table and no Range is defined by user. 

if you Have single table, it will use only single process to fetch/process the data irrespective of default parallel default value which is 5. 

if your table as partition then it will use partition by its own. for e.g. let's say your table have 7 partitions in total. and default value is 5 then it will spawn 5 parallel thread to read and push the data and once anyone of the channel process the data it will align those to capture or process the remaining partition. 

Seems it also depend upon the Source database. Database must support parallel as well. 

Regards,
Sushil Kumar

[sureshkumar]

Hello @Ginni 

Please refer the below article

Qlik Replicate: Identify Segment Boundaries for Da... - Qlik Community - 1966161

Regards,

Suresh

[Ginni]

Thanks @john_wang for the detailed explanation.

[Ginni]

Thank you @SushilKumar for the information.

[Ginni · Accepted solution]

Thank  you @sureshkumar for the information. It will be highly appreciated.

[john_wang]

Thank you for your great support @Ginni !