Skip to main content

Suggest an Idea

Vote for your favorite Qlik product ideas and add your own suggestions.

Announcements
Live today at 11 AM ET. Get your questions about Qlik Connect answered, or just listen in. SIGN UP NOW

Running data warehouse ETL sets in Parallel

sriniK
Contributor III
Contributor III

Running data warehouse ETL sets in Parallel

Compose Data warehouse workflow does not allow running multiple ETL set in parallel. It keeps throwing error - 

  • Task SRC_XREF and task SRC_DRM_FINANCE cannot be run in parallel as at least one of the source entities in their mappings is the same.
  • Task SRC_DRM_FINANCE and task SRC_XREF cannot be run in parallel as at least one of the source entities in their mappings is the same.
  • Task SAPS4HANA_SAPS4_ST_LR_Z3 and task SRC_DRM_FINANCE cannot be run in parallel as at least one of the source entities in their mappings is the same.

Currently we have 15 ETL sets and it takes 5.30 hours to complete the warehouse load. If its possible to run the ETL sets parallelly then  the run time can come down considerably.

Tags (1)
5 Comments
TimGarrod
Employee
Employee

Hi @sriniK , 

In Compose data warehouse projects, we currently don't allow for multiple parallel ETL  sets when the same target tables are in a task.   This is to prevent duplicate loading of data (if 2 tasks to load TABLEA are running in parallel, Compose could inadvertently load the same business / natural key to a target table).   This appears to be the case in your 1st instance. 

Typically, customers will put all mappings that load the same target structures into the same ETL set.  This ensures parallel loading of the mappings (even across sources) to specific target entities without concern for duplicate data being loaded to the vault. 

 

Can you confirm that the above scenario is what you are encountering with your scenario (different mappings loading to the same target tables ) ? 

Can you 

Status changed to: Open - New
TimGarrod
Employee
Employee
 
Status changed to: Open - Collecting Feedback
ubanerjee
Creator
Creator

Compose errors out an ETLSet if another is already running. Would it make sense to keep the scheduled ETLset as pending and run it once the currently running job is over? Workflows allow some of this, but jobs that need to start at fixed times fail if the previous job runs for an extended period.

Meghann_MacDonald

From now on, please track this idea from the Ideation portal. 

Link to new idea

Meghann

NOTE: Upon clicking this link 2 tabs may open - please feel free to close the one with a login page. If you only see 1 tab with the login page, please try clicking this link first: Authenticate me! then try the link above again. Ensure pop-up blocker is off.

Ideation
Explorer II
Explorer II
 
Status changed to: Closed - Archived