topic Re: parallel load - table segmentation method in Qlik Replicate

parallel load - table segmentation method

al3x — Wed, 17 Jan 2024 10:35:16 GMT

Hello Qlik Community

I want to use prallel load for a table. The option "Use Partitions" is blurred.

Using the option "Use Data Ranges" causes the problem, that we don't have a equaly distributed column on the source database.

Here is a made up example. Using the year column isn't very efficient for a data range, because some ranges will get bigger than others.

It would be nice to have something like:

Segment 1: PRIMARY_KEY mod 4 = 0
Segment 2: PRIMARY_KEY mod 4 = 1
Segment 3: PRIMARY_KEY mod 4 = 2
Segment 4: PRIMARY_KEY mod 4 = 3

Can you propose some solution or workaround?

Regards
al3x

Re: parallel load - table segmentation method

DesmondWOO — Thu, 18 Jan 2024 04:11:26 GMT

Hi @al3x ,

The option "Use Partitions" is disabled, likely because your table is not configured as a partitioned table.

In regards to the data skew issue, it would be advisable to consult with your DBA. Resolving this issue requires a thorough understanding of your table structure and data distribution. For instance, if your table has a primary key, such as an invoice number with incrementally increasing values, you could use this column to distribute data evenly, as opposed to using the year.

Regards,
Desmond

Re: parallel load - table segmentation method

al3x — Thu, 18 Jan 2024 10:24:41 GMT

Hello Desmond,

Thank you for your informative reply. That would be an option for me if otherwise not possible.

I am looking for a solution that doesn't involve changing the condition from time to time. While using a "primary key, such as an invoice number with incrementally increasing values," I would need to analyze the table structure and add or change the pipeline definition from time to time.

I'm looking for a more elegant and future-proof solution, e.g., modulo operations.

Is there something like that, or is Qlik planning to have it in a future release?

Regards, Alex

Re: parallel load - table segmentation method

Dana_Baldwin — Thu, 18 Jan 2024 18:22:14 GMT

Hi @al3x

This is not a currently available option. I encourage you to request support for this directly from our Product Management team here: https://community.qlik.com/t5/Ideas/idb-p/qlik-ideas

For help implementing this with the current capabilities, you might consider contacting our Professional Services team. Please check with your Account Manager on this option.

Thanks,

Dana