topic Re: Task times for reports with multiple email destinations in Qlik NPrinting

Task times for reports with multiple email destinations

kdmarkee — Mon, 23 Nov 2020 22:29:16 GMT

I have an Excel report that uses a cycle, and has levels and paging, and the destinations include one network folder, 2 individual email addresses (one of which is myself), and an NP Group that has 3 users/emails. I have noticed that if I run this just for myself and the folder destinations, it runs in about 30 mins, however when all destinations are included (5 user emails and 1 folder) it runs well over 3 hours (I actually killed the task 3.5 hrs in; it was 83% done). I know cycles, levels and paging add a lot of performance overhead, but what else is happening to make this run so long? Does NP generate one report and then distribute it or is it going through the process of re-generating a report for each destination? Does it not like an NP Group as a destination? Looking for any insight. Thanks.

Re: Task times for reports with multiple email destinations

Lech_Miszkiewicz — Tue, 24 Nov 2020 00:43:03 GMT

Hi @kdmarkee

I am assuming this is the result of 5 user emails being part of the group which (i think) is resulting in NPrinting producing individual reports for each user (just if those users had any user filters in them).

@Ruggero_Piccoli, @Frank_S - do we have any official documentation regarding this? My understanding would be that if there is at least one person who is part of distribution and has the user filter assigned (filter from the NPrinting app report is created in) then NPrinting needs to create report for each user individually as it needs to assume that each user may have different filter hence requires different report.

Or is there any other behaviours in this regard. Would you be able to point us either to documentation or article where this is explained?

I would hope that if users dont have any user filters for NPrinting app the report is running in then report would be generated once (same for all) and distributed to user emails/folders... If this does not happen, then there is major risk in whole performance design.

Re: Task times for reports with multiple email destinations

Ruggero_Piccoli — Tue, 24 Nov 2020 09:06:45 GMT

Hi,

If all 5 users have not filters the report is generated only one time. If each user has a filter the report is generated for each one. So I think in this case the 5 users have filters.

You could upgrade to latest version but I think this will not increase the generation speed. You could evaluate to increase the hardware resource by following the suggestions in https://help.qlik.com/en-US/nprinting/November2020/Content/NPrinting/DeployingQVNprinting/Performance.htm.

Best Regards,

Ruggero

Re: Task times for reports with multiple email destinations

Lech_Miszkiewicz — Tue, 24 Nov 2020 09:17:13 GMT

Hi @Ruggero_Piccoli

can we precise this little bit more? is this global setting or App/connection based setting within NPrinting repository?

If all 5 users don't have any filters for one "1 NP" NPrinting app/connection but they have some filters for "2 NP" NPrinting app/connection then how many times report which is build in App "1 NP" gets generated? Is it only 1 time as none of the user filters is associated with that NPrinting app?

I know we can test it but I would like confirm this design rather than find out by assumption.

Appreciate your input on this - thanks!

Re: Task times for reports with multiple email destinations

Ruggero_Piccoli — Tue, 24 Nov 2020 09:40:03 GMT

Hi @Lech_Miszkiewicz ,

I'm collecting more details and opening a documentation improvement do document this topic. I think it is important to know to optimize the performances of Qlik NPrinting installation.

Best Regards,

Ruggero

Re: Task times for reports with multiple email destinations

Lech_Miszkiewicz — Tue, 24 Nov 2020 09:48:31 GMT

Thanks Ruggero - thats great!

Re: Task times for reports with multiple email destinations

kdmarkee — Tue, 24 Nov 2020 16:13:18 GMT

None of my users in this scenario have any user filters for the NPrinting app the report is running in.

Re: Task times for reports with multiple email destinations

Lech_Miszkiewicz — Tue, 24 Nov 2020 21:15:04 GMT

Hi @kdmarkee

Do your users have any other user filters for other NPrinting apps (I assume they do)?

I guess this may be clarification we need from @Ruggero_Piccoli once he gathers it. In my opinion if other apps filters can cause such performance decrease, then it would be something R&D should look at...

Lets wait for official documentation-cheers

Re: Task times for reports with multiple email destinations

Ruggero_Piccoli — Wed, 25 Nov 2020 13:17:23 GMT

Hi,

Are you using section access?

Re: Task times for reports with multiple email destinations

kdmarkee — Wed, 25 Nov 2020 21:56:16 GMT

Thanks to you both for your continued involvement. Here is some additional info...

Of the 5 users on this task, none of them have any user filters associated to any of my NP apps in my entire NP environment. The report associated to this task does have a report filter, however, if that is relevant. Additionally, if relevant, the users on this task do get other reports from other NP apps and tasks that do have their own report filters as well. Hopefully you followed that… short answer, no user filters at all anywhere for these 5 users.

There is no section access in place, to answer that question.

Additionally, I ran some more tests yesterday and today, and on average, it seems like NP needs 25 mins per user. With all users and folder included, the max testing time I got was 2.5 hrs (we got a faster vm this week, so my times are better this testing round than the original test time of 3.5+ hrs).

My gut tells me it has something to do with one or more of: cycles, paging, levels, or perhaps even just the report filter itself. I only say that because I have other excel reports, without any of these features, set up for the same set of users/destinations and they run very fast. I will try to do further testing by taking one feature off at a time to see if I can prove one of those features as being the culprit (which seems like a bug in that case, as if all the users are getting the same exact report, it shouldn’t add additional time for each user included in Destinations).

Re: Task times for reports with multiple email destinations

Lech_Miszkiewicz — Wed, 25 Nov 2020 22:26:40 GMT

@kdmarkee

One important thing to notice is your version (Sep 2018 - 2 years old and 10 major releases behind). I am not able to get 2018 environment setup to test it unfortunately.

In my current environments (and i have quite few clients who run similar setup on fresher versions) I don't have this kind of issue. If there are no user filter nor section access then report get generated only once...

regards

Lech

Re: Task times for reports with multiple email destinations

Ruggero_Piccoli — Thu, 26 Nov 2020 11:42:36 GMT

Hi @kdmarkee ,

I agree with @Lech_Miszkiewicz ; If you can please try also to upgrade to latest version (November 2020) and test the same configuration with it. You will benefit of all solved bugs, so we will be able to exclude that you are experiencing this for something already solved.

Compared with other Excel templates without cycles, levels and pages thos 3 feature are for sure the cause of the different generation time. In fact when the report is generated it need a number of filters that is the product of the number of distinct values of each so it can increase fast and you have to wait every single filter application.

You could keep the cycles, levels and pages in your template and create a new QVW with only the data you need for the report (and only the objects you need for the report). In this way you will not need to add a report filter so you can test the report generation time in this case.

Instead of adding 5 users in the publish task, you could also test to add a single user, use one of its alternate email and fill that alternate email with the list of recipients email addresses.

Let us know the results.

Best Regards,

Ruggero

Re: Task times for reports with multiple email destinations

Lech_Miszkiewicz — Tue, 01 Dec 2020 08:26:25 GMT

Hi @Ruggero_Piccoli

Have you got any updates how this should work by design?

If you have 5 users as recipients who dont have any filters and there is no section access - how many times NPrinting will produce report?

once for all users?
5 times - separately for each user?

just want to know how this should work by design

thanks

Lech

Re: Task times for reports with multiple email destinations

Ruggero_Piccoli — Tue, 01 Dec 2020 10:40:55 GMT

Hi @Lech_Miszkiewicz ,

If you have 5 users without filters in the same publish task the report should be created only one time and sent to all.

If the previous suggestions do not work I think the best at this point is to open a ticket with the environment so we can reproduce the issue and investigate it.

Best Regards,

Ruggero

Re: Task times for reports with multiple email destinations

kdmarkee — Tue, 01 Dec 2020 15:35:15 GMT

We are actually upgrading NP today so I'll do some additional testing and see how things go and report back.

Re: Task times for reports with multiple email destinations

kdmarkee — Wed, 02 Dec 2020 19:18:19 GMT

After upgrading and testing different scenarios, I'm able to confirm one thing right now... if I run the report for one user it took about 20 mins, but if I run it with all 5 users it ran for 1 hr 23 mins, so it still runs long as if the report is being generated more than once. I then tried having 1 user get the email and having the other 4 users added to the Email CC line and that test ran in 22 mins so this last scenario is the best option at this time to accomplish the delivery in the shortest time. I still hope to pinpoint which of the report features is causing the long run times, assuming one of them is the culprit (paging, cycle, level).

Re: Task times for reports with multiple email destinations

kdmarkee — Fri, 04 Dec 2020 17:40:39 GMT

Final observation: levels are the issue in combination with adding more than one individual user in the Destination tab

Below are the Excel report tests I ran and their results:

I also ran a pixel perfect report with levels and noticed the same behavior, that is, the time increased the more users I added on the Destinations tab but if I only had 1 user in the Destination tab with the other uses only in the email CC line, it ran in the same time as running a single Destination user.

So, with that being said, I have mixed feelings about using the email CC line to get these jobs distributed as soon as possible… I like that is speeds up the overall delivery time to end users, but what I don’t like is that if the job fails for whatever reason on the selected user in the Destinations tab, it is going to fail to get delivered to everyone since they are listed in the CC but NOT enabled as users in the Destinations.

I’m not sure if this is by design or if Qlik would research and change the behavior, but I can’t submit my qvw or report due to data privacy and the amount of work necessary to de-identify it, but I can provide this info below to generate one’s own qvw and report if looking to replicate as close to my scenario as possible.

Your qvw would have this amount of data values, seen below, or I suspect you could use the average count per team to achieve similar results. Not sure the number of expressions would matter or not; I had 9) Note the NP features per column.

The Excel report layout is like this below.

Re: Task times for reports with multiple email destinations

Lech_Miszkiewicz — Tue, 08 Dec 2020 20:53:54 GMT

Thanks for your insights Kris,

one last comment from me -it may be unrelated, but i also may be the main issue. Is your qlikview using qvp connection? and is it setup as per documentation: https://support.qlik.com/articles/000098313?

regards

Lech

Re: Task times for reports with multiple email destinations

kdmarkee — Tue, 08 Dec 2020 21:34:59 GMT

I am using qvp connections, but the times have been an issue in our older NP Sept 2018 version and in the NP Nov 2020 version.

Re: Task times for reports with multiple email destinations

Lech_Miszkiewicz — Wed, 09 Dec 2020 07:50:14 GMT

Hi @kdmarkee

As I mentioned - it may not be directly relevant to that topic, but regardless check the settings from the link I have given you in my previous reply. If you have different settings you will experience huge delays in all reports (no matter what their structure and builds are).