idea Re: Introduce a feature to execute post loading etls in parallel in Suggest an Idea

Introduce a feature to execute post loading etls in parallel

Aldy — Thu, 01 Apr 2021 16:24:05 GMT

In the existing scenario, Post Loading ETLs are running in sequential mode which is consuming more time, due to the number of scripts our application have. Can these be leveraged to run in parallel which will save lot of time.

Re: Introduce a feature to execute post loading etls in parallel - Status changed to: Open - Collecting Feedback

TimGarrod — Tue, 08 Feb 2022 18:14:16 GMT

Thank for you the ideation. I assume this would be relevant for DM and DWH Post-Load tasks?

Is it fair to say some level of parallel v serial requirement would still be involved? (ie. tasks with teh same sequence number would run in parallel then the next "sequence number" etc.)?

I'm opening this for additional community feedback .

Re: Introduce a feature to execute post loading etls in parallel

Aldy — Tue, 08 Feb 2022 18:18:44 GMT

Yes these are relevant for DM and DWH Post Loading.

Also, your other option too makes sense in term of few in parallel vs other in serial.

I would appreciate, if there is any work around for the same.

Re: Introduce a feature to execute post loading etls in parallel

TimGarrod — Tue, 08 Feb 2022 18:26:04 GMT

The only work-around I'm aware of right now would be to create multiple ETL Tasks (using the Pre-Load feature of the ETL tasks) and then configure the workload to execute those in parallel.

e.g.

ETL TASK 1 - Run 2 "pre-load" processes A & B (these would be serial)

ETL TASK 2 - Run 2 "pre-load" processes C &D (these would be serial)

In a workflow - execute ETLTask1 and ETLTask2 in parallel. eg. -

Re: Introduce a feature to execute post loading etls in parallel

Aldy — Tue, 08 Feb 2022 18:31:26 GMT

This will have a dependency if ETL Task 1 and ETL Task 2 are Query based ETL wherein, Task2 is dependent on task 1 data. How can this be leveraged, in a DM task where we have multiple Post Loading ETLs with 80% to run in parallel and the rest in serial.

Re: Introduce a feature to execute post loading etls in parallel

TimGarrod — Tue, 08 Feb 2022 18:37:42 GMT

This is providing a "work-around" for you for now, while we collect community feedback on the ideation.

ETL Task1 and ETL Task2 do not contain mappings in this instance. They do not have to touch the DW data, and could contain tasks you wish to run after the Data mart is completed.

They only contain "Pre-Loading ETL" steps which are free-form ETL processes (akin to the Post Load for data marts). - however via the workflow you can run them in parallel after the data mart load is complete.

I understand this requires additional tasks. As I said - this is a work-around at this point.

Feel free to PM me to discuss further.

From now on, please track this idea from the Ideation por...

Meghann_MacDonald — Wed, 20 Sep 2023 15:03:26 GMT

From now on, please track this idea from the Ideation portal.

Link to new idea

Meghann

NOTE: Upon clicking this link 2 tabs may open - please feel free to close the one with a login page. If you only see 1 tab with the login page, please try clicking this link first: Authenticate me! then try the link above again. Ensure pop-up blocker is off.

Re: Introduce a feature to execute post loading etls in parallel - Status changed to: Closed - Archived

Ideation — Wed, 20 Sep 2023 15:03:28 GMT