QLik and Python Data Conversion Speed

Anonymous — Sat, 16 Nov 2024 08:43:38 GMT

Hello,

I am writing a plugin for Qlik using Python 3.6.4. It takes a large amount of data passed to it from Qlik Sense Desktop through an open port. Once the data has been taken through the port, the Python program then converts the data handed to it from the Qlik Sense into a Python data format by essentially looping through the data. The problem is that this process takes about 2 minutes which is a rather long time compared to the rest of the program. Does anyone have any tips on how to speed up the data conversion process?

If it helps, I am already using maps to speed up part of the looping process and I have provided a snippet of the relevant part of my code below.

Here is a snippet of the relevant part of my code:

Note: request is the data sent to it by Qlik Sense, the data is 4 columns of data storing these items in this order: date, serial number, item name and item value. The date, serial, item_name and item_val variables are lists that correspond to the items passed to it by Qlik Sense.

for rows in request:

'''
This section still needs work to improve its efficiency.
'''
date += list(map(lambda c: dt.strptime(c.duals[0].strData,'%d/%m/%Y %H:%M:%S'),rows.rows))

serial += list(map(lambda c: c.duals[1].numData,rows.rows))

item_name += list(map(lambda c: c.duals[2].strData,rows.rows))

item_val += list(map(lambda c: c.duals[1].numData,rows.rows))

Thanks,

Matt

Re: QLik and Python Data Conversion Speed

SteveNewman — Fri, 09 Feb 2018 12:29:09 GMT

‌Can you clarify if you are doing this at Load Time (in Load Script) or at Run Time (on a Sheet, after user’s make selections)? This may be helpful to the answer.

Secondly, you state that you are sending a lot of data in, but are you also getting a lot of data back or just a small set of data? For example, IN: 10,000 rows * 4 columns (40,000 elements) vs OUT: 10 rows * 1 column (10 elements).

If you answered Run Time (#1 above) and a low result set ration (#2 above)...

Then I would suggest that you see what options exist to improve your python script. For example, you might be able to reduce it to only use 2 columns (date, serial), then have your Python script access the Name/Values (if needed) from a CSV file on its own. Doing this would reduce your IN: to 20,000 elements (above) which would significantly improve performance for your users.

Of course, the CSV file could be created by your Load Script during data updates so that it is always consistent with the Qlik App and up to date.

Hope that helps.

topic Re: QLik and Python Data Conversion Speed in Integration, Extension & APIs

QLik and Python Data Conversion Speed

Re: QLik and Python Data Conversion Speed