Iterate with Enable Parallel Execution in Large Jobs

[ml1662663516]

We have a large job with many subjobs within it.

Below I have included a screenshot of the job.

The subjob circled in green queries a DB to get back a list of products from our products table.

It then iterates (circled in green) over those products and performs various tasks associated with them on other tables within the DB.

It all works great. Except that depending on the number of products, it could take minutes to hours to complete the job.

If I were to activate Enable Parallel Execution on the iteration (circled in green), would that be "safe" considering all the subjobs that are flowed through below the iteration?

We do have various global variables that are set in the pre-job area. And, we have references to the product id's that are being iterated over (circled in green).

Any further or clearer understanding of how Enable Parallel Execution works (or doesn't work), would be greatly appreciated.

Thank you!

 

[quentin-vigne]

Hi @ml1662663516 

I've been using parallel execution for a while and there is some things that work great and sometimes doesn't work at all.

First of all, take care with the tDB components in parralel, from my experience you need each of them to use their separate connection (so don't use a shared connection)

If you're using file output or tDBOutputBulk component don't forget to add the value of the iteration to the name of the file or select "Append" in your tFileOutput.

Also if you have a lot of values from your first tDBInput it could actually slow down your process since the job will divide it in how many values you have. 

Your best option is to try it in a dev environment with 10-50 values at first and maybe try with more later.

Now about your tJava components, I don't know what you use them for, but if it's for displaying logs it might not work. When working in parallel global variable are not thread safe, meaning that you could actually get data mixed up depending on how you are working, and this might need some work to make it safe to use.

Good luck