topic Re: Qix/datafiles parameters in Management & Governance

Qix/datafiles parameters

pedrobergo — Tue, 20 Apr 2021 14:48:59 GMT

Hi,

I'm asked how to delete 7,000 files automatically at this post: https://community.qlik.com/t5/Deployment-Management/How-delete-7-000-files-inside-SaaS-DataFiles-Folder/m-p/1760320/highlight/true#M18887. The solution works great, but it just operate with Personal Space.

How can I operate with other spaces? Where I can find qix/datafiles parameters ?

Tks

Pedro

Re: Qix/datafiles parameters

Levi_Turner — Wed, 21 Apr 2021 12:06:53 GMT

Hey there,

The general flow would be something like:

GET /api/v1/spaces

-> Get the id element for the desired space

GET /api/v1/data-connections?space=<spaceId>

--> Get the id element for the return with qName = DataFiles

Example response:

{ "id": "8ccd4eb8-7ba8-4add-8297-26c3e16f2b85", "links": { "self": { "href": "<removed>" } }, "privileges": [ "read" ], "qArchitecture": 0, "qConnectStatement": "CUSTOM CONNECT TO \"provider=qix-datafiles.exe;path=App Metadata QVDs:datafiles;\"", "qEngineObjectID": "8ccd4eb8-7ba8-4add-8297-26c3e16f2b85", "qID": "8ccd4eb8-7ba8-4add-8297-26c3e16f2b85", "qLogOn": 0, "qName": "DataFiles", "qType": "qix-datafiles.exe", "space": "5e3e045dcf6ae0000122ec7f" }

GET /api/v1/qix-datafiles?connectionId=<idFromPreviousStep>

Re: Qix/datafiles parameters

pedrobergo — Thu, 22 Apr 2021 22:40:02 GMT

Hi @Levi_Turner ,

It's almost that i'm looking for... I got a list with first 25 files in specified space, but is there possible to control the pagination response? Or is anyway to use a query parameter using wildcards or regex or anything else?

I miss the documentation for this api 😞

For example

Using Wildcards

"GET /api/v1/qix-datafiles?connectionId=195cb633-aed4-43ba-920e-bae6d76333f0&query=XXX*.qvd"

Using pagination:

"GET /api/v1/qix-datafiles?connectionId=195cb633-aed4-43ba-920e-bae6d76333f0&page=5 "

Can you show me how can do that?

Tks again,

Pedro

Re: Qix/datafiles parameters

Levi_Turner — Fri, 23 Apr 2021 00:46:38 GMT

Well, the API is private and therefore undocumented (and has the potential to change at any time).

But the param that you're looking for is top. So an example would be:

GET /api/v1/qix-datafiles?connectionId=8ccd4eb8-7ba8-4add-8297-26c3e16f2b85&top=10000

I would expect pagination to be a requirement for the API to go public and stable like the rest of the public and stable APIs on Qlik.dev: https://qlik.dev/apis/#rest

Re: Qix/datafiles parameters

wwengso314 — Tue, 27 Apr 2021 07:39:03 GMT

Hi, I followed the method above to delete the files in shared spaces, however, but the files are still in the shared spaces, I have tried to check the list using 'qlik raw get v1/qix-datafiles?connectionId={elementId}', the query listed down the files from personal space. What did I miss out? Thank you.

Re: Qix/datafiles parameters

pedrobergo — Wed, 28 Apr 2021 12:07:56 GMT

Hi @wwengso314

To use a qlik-clik withing line commands (Powershell, CMD or Bash), follow this to use it:

qlik raw get v1/qix-datafiles --query connectionId=dataconnectionid,top=100000

The parameter --query is same that '?' and you need to use comma (,) to separate the parameters.

And if you are using inside Powershell, add the '| ConvertFrom-Json' at end of line to better see it.

[],

Pedro

Re: Qix/datafiles parameters

gordon_ash — Tue, 11 Jul 2023 14:16:51 GMT

Hi Pedro,

I have followed this thread and used your scripts for the past year or so, thanks for that; I have accidentally uploaded over 90,000 files on top of an existing 40000 files and so need to delete them. When I use the top parameter it only seems to cope with about 35000 files, and they are in alphabetical order - the files I want to delete all begin with R so are probably from 50000 onwards so they never get returned. Is there any way of either ordering differently or having another pagination parameter, to get to the R*.qvd files to delete?

Thanks

Gordon

Re: Qix/datafiles parameters

Levi_Turner — Tue, 11 Jul 2023 14:40:47 GMT

The underlying API does not support filtering in this way so you will need to do it in your code. To get all data-files in a space using qlik-cli, you can use this approach:

$space_id = '5e42f2cc0ea40d0001214b3a' $data_files_connection_id = (qlik data-file connection ls --spaceId $space_id | ConvertFrom-Json).id $space_data_files_tmp = qlik data-file ls --connectionId $data_files_connection_id --raw | ConvertFrom-Json $space_data_files = $space_data_files_tmp.data do { $separator = "next=([^}]*)}" $nextURL = [regex]::match($space_data_files_tmp.links.next.href, 'next=(.*?)(&|$)').Groups.Value[1] $space_data_files_tmp = '' IF($nextURL -gt 0) { $space_data_files_tmp = qlik data-file ls --connectionId $data_files_connection_id --raw --page $nextURL | ConvertFrom-Json $space_data_files += $space_data_files_tmp.data } } while (($space_data_files_tmp.links.next).Length -gt 1)

Something like this will output the files with names which start with R or higher:

$space_data_files | Where-Object { $_.name -match '^[r-z]' }

Re: Qix/datafiles parameters

gordon_ash — Thu, 13 Jul 2023 09:36:42 GMT

Hi @Levi_Turner ,

Thanks for the quick response, I have implemented the code, but I never get anything in the @nextUrl when matching the regex, it always returns null. I have even just set @nextUrl to the links.next.href just to get the next full page of data but nothing gets returned. Presumably the regex is looking for something to define next=... The value of my

$space_data_files_tmp.links.next.href is :

"https://m3data.eu.qlikcloud.com:443/api/v1/data-files?connectionId=a34e64e7-d9b9-4f43-adc2-39516e61c02c&page=ZjNlODE1MDAtOWQ5Zi00ZjQ3LWEzMTgtNWIzYWI1YmRlYzcw"

I don't know what I am missing.

Re: Qix/datafiles parameters

Levi_Turner — Thu, 13 Jul 2023 12:27:37 GMT

Hmmm, it's possible that the platform changed the param for the next page of data. This should work:

$space_id = '5e42f2cc0ea40d0001214b3a' $data_files_connection_id = (qlik data-file connection ls --spaceId $space_id | ConvertFrom-Json).id $space_data_files_tmp = qlik data-file ls --connectionId $data_files_connection_id --raw | ConvertFrom-Json $space_data_files = $space_data_files_tmp.data $page_separator = 'page' do { $nextURL = [regex]::match($space_data_files_tmp.links.next.href, "$($page_separator)=(.*?)(&|$)").Groups.Value[1] $space_data_files_tmp = '' IF($nextURL -gt 0) { $space_data_files_tmp = qlik data-file ls --connectionId $data_files_connection_id --raw --page $nextURL | ConvertFrom-Json $space_data_files += $space_data_files_tmp.data } } while (($space_data_files_tmp.links.next).Length -gt 0)

Re: Qix/datafiles parameters

gordon_ash — Thu, 13 Jul 2023 13:53:59 GMT

Hi @Levi_Turner ,

Thats got it, should have spotted that myself. The final statement I had to add a .href to the .next part to get it to refill the tmp data set,

while (($space_data_files_tmp.links.next.href).Length -gt 1)

Now with a few extra mods to fit my use case this code pages the massive dataset as hoped.

Thanks very much for your speedy replies and getting this fixed for me.

Re: Qix/datafiles parameters

Jyoti007 — Mon, 27 May 2024 13:55:16 GMT

Hi Levi,

Is it possible to get all duplicate app names within a space in qlik cli. What is the command to extract and then display.

As a first step, I can get list of apps name within a space in qlik cli via >>> qlik app ls --spaceID <spaceID>
But in second step, I am not getting how to extract only the second column i.e. "NAME" (enclosed with blue color rectangle)
and the third step to check no. of duplicate app name and display.

Can you please help me in this with CLI script

Thanks and regards
Jyothi

Re: Qix/datafiles parameters

gordon_ash — Tue, 28 May 2024 07:20:33 GMT

Hi Jyothi,

If it is of any help to you this is how I handle file names in a list in powershell; convert to json and do a Where-object , like this:

$fileNames = 'rubbish*'

$space_data_files_tmp = qlik app ls --spaceId $spaceid --json --limit 100000 | ConvertFrom-Json

$space_data_files = $space_data_files_tmp | Where-Object {$_.NAME -like $fileNames}

foreach ($file in $space_data_files) {

if ($confirm -eq 'yes') {

Write-Log "Downloading file [$($file.NAME)] id [$($file.ID)]...";

....do things with the file....

Hope it helps,

Gordon

Re: Qix/datafiles parameters

Jyoti007 — Mon, 01 Jul 2024 06:59:17 GMT

Thanks @gordon_ash. It really helped me.

Regards
Jyothi