topic Re: Large data file upload using REST? in Integration, Extension & APIs

Large data file upload using REST?

yosuke-coupa — Fri, 06 Oct 2023 20:07:14 GMT

The qlik.dev documentation provides a few approaches for data file uploads.

Create data files in Qlik Cloud | Qlik Developer Portal

The issue we're having is uploading large data files, specifically QVD files. The documentation does state the following:

Note: If you are uploading data files larger than 500 MB, you will need to adjust the upload approach. This is because larger files must be uploaded to the temp-contents API, which supports chunked uploads for these larger files. When using qlik-cli, you add an additional flag, but for raw API approaches you will need to amend your code.

The documentation only shows how to upload large files via qlik-cli. Is there any information on how to use REST endpoints for large file uploads?

Re: Large data file upload using REST?

jprdonnelly — Mon, 09 Oct 2023 15:20:14 GMT

@yosuke-coupa - if you look at the temp-contents doc page on qlik.dev, it has a little drop-down on the right side that lets you choose between qlik-cli, node.js and curl examples:

curl "https://your-tenant.us.qlikcloud.com/api/v1/temp-contents" \ -X POST \ -H "Authorization: Bearer <API-key>" \ -H "Content-type: application/octet-stream" \ --data-binary "@/path/to/file"

Once you are able to upload to temp-contents, you can import the file into the destination Space. You can see an example of that in this qlik-oss GitHub action YAML.

Cheers!

Re: Large data file upload using REST?

Dave_Channon — Mon, 09 Oct 2023 15:59:57 GMT

You beat me to it, @jprdonnelly!

@yosuke-coupa I've been meaning to add some examples to that doc. What language are you trying to use?

In the example Justin posted, I used curl, but I've got some examples with Python which work in a similar way using stream, e.g.

headers = {'Authorization': auth, 'Content-type': 'application/octet-stream'} params = {'filename': 'import'} baseUrl = 'https://' + tenantUrl + '/api/v1/temp-contents' # Import to temp-contents try: with open(filePath, 'rb') as f: tempFilePath = requests.post(baseUrl, params=params, stream=True, data=f, headers=headers).headers['Location'] print("Temporary file path is " + tempFilePath) importFileId = tempFilePath.rsplit('/', 1)[-1] except: raise Exception("Failed while attempting to upload file to temp-contents.")

The only complex part is grabbing the ID of the uploaded file, since by default we pass a path. Both the action (see this line) and this python example separate out the ID, which you then use as input to create data file. So as an example (excuse the escaping, that's from a windows example):

curl "https://your-tenant.us.qlikcloud.com/api/v1/data-files" \ -X POST \ -H "Authorization: Bearer <API-key>" \ -H "Content-type: multipart/form-data" \ -F "Json={\"name\":\"MyFile.csv\",\"tempContentFileId\":\"theIdFromTempContsServiceYouGrabbedEarlier\"}"

I'll take a note to add examples to that page.

Re: Large data file upload using REST?

yosuke-coupa — Mon, 09 Oct 2023 22:29:28 GMT

Hi @Dave_Channon @jprdonnelly,

Thanks for the information. We will try it.

@yosuke-coupa I've been meaning to add some examples to that doc. What language are you trying to use?

We are using C#, but AFAIK, there is no C# SDK for Qlik. Thus, we were looking for examples with cURL so that we can build it in C#.

Re: Large data file upload using REST?

Øystein_Kolsrud — Tue, 10 Oct 2023 15:33:52 GMT

Hi! It's possible to use this library for doing this through .NET:

https://www.nuget.org/packages/QlikSenseRestClient/1.15.0

I've experimented with this today, and I had to expose some additional endpoints to make it possible to do those "multipart/form-data" calls, so make sure you pick the latest version if you use it. I've also added an example that illustrates how to do it:

https://github.com/kolsrud/qlik_rest_sdk/blob/master/Qlik.Sense.RestClient/Examples/QcsUploadDataFile/Program.cs