QLIK API Put Request 413 Error

[huan1996]

Hello,

I have an app and the ETL process work as follows:

1. Use python and its library to extract data from multiple on-prem databases (Microsoft SQL Server, IBM DB2 for LUW, and postgresql)

2. Doing Transform analysis in python and export results to csv files.

3. Load/upload these CSV files to the app's space via api_key and python subprocess library (request.put)

4. Load data into the apps as Load... From data.csv

My problem is in step 3, since the csv file is around 900MB, the request.put command result in 413 error (exceed transfer limit). Is there a work around?

My department is adamant about using this form of ETL instead of QLIK data integration.

[Leigh_Kennedy · Accepted solution]

This post may help:

https://community.qlik.com/t5/Qlik-Cloud-Data-Integration/Qlik-Cloud-data-files-upload-endpoint-not-...

API documentation is here: https://qlik.dev/apis/rest/temp-contents/#%23%2Fentries%2Fv1%2Ftemp-contents-post

Not a direct answer, but have you looked at using other formats that may compress the data such as Parquet?  Qlik apps can read parquet files. 

Regards.

[huan1996]

Ok so the first api call to upload into tempcontent end point works fine. But the second one is pushing an error: 

Temp content file with file ID: In-Return-Header-Prev-Request could not be retrieved from the temp content service: Received temp content file ID is in an invalid format: In-Return-Header-Prev-Request

I run curl from mac os terminal. Can you hellp me with this?