Qlik Request <-> Pandas Dataframe

Clever_Anjos — Sat, 16 Nov 2024 04:10:40 GMT

I've been using pandas as base for python SSE's

I've checked which part of the code takes longer times, so I might improve performance.

I came to case processing 510K rows x 4 Columns (https://www.kaggle.com/berkeleyearth/climate-change-earth-surface-temperature-data#GlobalLandTemperaturesByCountry.csv) and processed the files attributing the mean value for a country when a value as null (fillna() method from dataframe)

When I measured more than 70% of time is consumed converting from request values to Pandas dataframe back and forth

I'm using this to convert from request to dataframe

df = pd.DataFrame([(row.duals[0].strData, row.duals[1].strData,row.duals[2].strData,row.duals[3].strData) \

for request_rows in request \

for row in request_rows.rows], \

columns=['a_date','b_country','c_value','d_value'])

and this to convert back

df = df.values.tolist()

response_rows = [iter([SSE.Dual(strData=row[0]),SSE.Dual(strData=row[1]),SSE.Dual(strData=str(row[2])),SSE.Dual(strData=str(row[3]))]) for row in df]

response_rows = [SSE.Row(duals=duals) for duals in response_rows]

Any hints in how this could be improved?

Thanks in advance

Re: Qlik Request <-> Pandas Dataframe

jcbsorensen — Wed, 18 Dec 2019 18:31:41 GMT

Have you tried to play around with parallel computing libraries that are build with pandas in mind? - such as Modin , Dask or Swifter . If done right under specific circumstances, these libraries can be complete game-changers.

topic Qlik Request <-> Pandas Dataframe in Integration, Extension & APIs

Qlik Request <-> Pandas Dataframe

Re: Qlik Request <-> Pandas Dataframe