topic Segments in Parallel Load in Qlik Replicate

Segments in Parallel Load

Ginni — Wed, 24 Jul 2024 10:53:16 GMT

What concept is used to create segments in Parallel Load?

Let say we have one table Emp having counts 50,00,000 to load with Parallel Load,

How will we come to know no of segments to be created and range of each segments?
How does each segment efficiently load large tables without taking more time?

Re: Segments in Parallel Load

john_wang — Wed, 24 Jul 2024 13:17:25 GMT

Hello @Ginni ,

Thanks for reaching out to Qlik Community!

1. There are two data separating method in Qlik Replicate GUI:

Use Data Ranges: Choose this option to segment the source data by user-defined columns and data ranges. You may define the no of segments to be created and range of each segments freely.
Use Partitions: Choose this option to segment the source data by existing partitions or sub-partitions of the source table(s). You may query the rows number of each partition in source database.

2. In Full Load replication mode, Qlik Replicate accelerates the replication of large tables by splitting the table into segments and loading the segments in parallel so far speed up the load process.

More detailed information can be found at here.

Hope this helps.

John.

Re: Segments in Parallel Load

SushilKumar — Thu, 25 Jul 2024 01:13:26 GMT

Hello @Ginni

To add more to Expert comment here. By default, if you have table and no Range is defined by user.

if you Have single table, it will use only single process to fetch/process the data irrespective of default parallel default value which is 5.

if your table as partition then it will use partition by its own. for e.g. let's say your table have 7 partitions in total. and default value is 5 then it will spawn 5 parallel thread to read and push the data and once anyone of the channel process the data it will align those to capture or process the remaining partition.

Seems it also depend upon the Source database. Database must support parallel as well.

Regards,
Sushil Kumar

Re: Segments in Parallel Load

sureshkumar — Thu, 25 Jul 2024 10:21:57 GMT

Hello @Ginni

Please refer the below article

Qlik Replicate: Identify Segment Boundaries for Da... - Qlik Community - 1966161

Regards,

Suresh

Re: Segments in Parallel Load

Ginni — Fri, 26 Jul 2024 09:16:20 GMT

Thanks @john_wang for the detailed explanation.

Re: Segments in Parallel Load

Ginni — Fri, 26 Jul 2024 09:19:04 GMT

Thank you @SushilKumar for the information.

Re: Segments in Parallel Load

Ginni — Fri, 26 Jul 2024 09:20:44 GMT

Thank you @sureshkumar for the information. It will be highly appreciated.

Re: Segments in Parallel Load

john_wang — Fri, 26 Jul 2024 09:22:21 GMT

Thank you for your great support @Ginni !