Optimizing Data Transfer - Oracle to PostgreSQL with Qlik

[al3x]

We transfer data from Oracle to PostgreSQL via one Qlik task. Now, we have a use case where some tables should be available faster than others. One solution would be to split the Qlik task into two tasks and load sequentially. First, task A and then task B.

This introduces the danger of getting inconsistent data in the database tables and results in more code complexity in the following layer to counter that.

• Is it generally advisable to load data from one source to one target using multiple Qlik tasks?
• Are there other ideas to solve that problem, perhaps some best practices?"

[SachinB]

Hello @al3x ,

Thanks for reaching out to Qlik Community.

If you are looking for to load the data as part of full load and this can be achievable in only one task, we can set table load priorities for tables. Setting load order.

You can set the load order for each of the selected tables. This may be useful, for example, if your selected tables list contains tables of different sizes and you want the smaller tables to be loaded before the larger tables. When a group of tables are set with the same load order, Replicate will load the tables according to the table ID.

select one of the available priority levels (Lowest Priority, Low Priority, Normal Priority, High Priority, and Highest Priority).

You can find the more details in below table.

https://help.qlik.com/en-US/replicate/May2023/Content/Global_Common/Content/SharedEMReplicate/Design...

Regards,

Sachin B

[al3x]

Hello Sabine,

Thank you for the fast reply. Priority levels or parallel loads are helpful features. In our scenario, we need delta processing for the tables that load faster as well. In a sequential load, we would have to wait for the slower tables to finish before change processing starts.

I hope that's more precise. In general, I'm asking myself if it's wise/good practice to load data from one source to one target using multiple Qlik tasks, or if it's best practice to just use one Qlik task.

Regards, Al3x

[SushilKumar · Accepted solution]

Hello Team,

You Can try both option and see the result. However, it works in a similar way.

1. if you want to go with single tasks. if you increase the Parallel value from default (5) to 10 or 15. then it will allocate that many process to fetch the data from source.

2. if you go with multiple tasks let say 5 and keep the Parallel value from default (5) the total number of sessions created on more than 25 on the DB which may cause issue on source.

3. the ideal way to choose between the table segregation is data type. if tables have lobs and don't have primary keys then they should be kept in different task. and if table have partition then each partition treated as table in Qlik replicate.

hope this helps.

Regards,

Sushil Kumar