Qix/datafiles parameters

[pedrobergo]

Hi,

I'm asked how to delete 7,000 files automatically at this post: https://community.qlik.com/t5/Deployment-Management/How-delete-7-000-files-inside-SaaS-DataFiles-Fol.... The solution works great, but it just operate with Personal Space.

How can I operate with other spaces? Where I can find qix/datafiles parameters ?

Tks

Pedro 

[Levi_Turner]

Hey there,

The general flow would be something like:

GET /api/v1/spaces

-> Get the id element for the desired space

GET /api/v1/data-connections?space=<spaceId>

--> Get the id element for the return with qName = DataFiles

Example response:

        {
            "id": "8ccd4eb8-7ba8-4add-8297-26c3e16f2b85",
            "links": {
                "self": {
                    "href": "<removed>"
                }
            },
            "privileges": [
                "read"
            ],
            "qArchitecture": 0,
            "qConnectStatement": "CUSTOM CONNECT TO \"provider=qix-datafiles.exe;path=App Metadata QVDs:datafiles;\"",
            "qEngineObjectID": "8ccd4eb8-7ba8-4add-8297-26c3e16f2b85",
            "qID": "8ccd4eb8-7ba8-4add-8297-26c3e16f2b85",
            "qLogOn": 0,
            "qName": "DataFiles",
            "qType": "qix-datafiles.exe",
            "space": "5e3e045dcf6ae0000122ec7f"
    }

GET /api/v1/qix-datafiles?connectionId=<idFromPreviousStep>

[pedrobergo]

Hi @Levi_Turner ,

It's almost that i'm looking for... I got a list with first 25 files in specified space, but is there possible to control the pagination response? Or is anyway to use a query parameter using wildcards or regex or anything else?

I miss the documentation for this api 😞

For example

Using Wildcards

"GET /api/v1/qix-datafiles?connectionId=195cb633-aed4-43ba-920e-bae6d76333f0&query=XXX*.qvd"

Using pagination:

"GET /api/v1/qix-datafiles?connectionId=195cb633-aed4-43ba-920e-bae6d76333f0&page=5 "

Can you show me how can do that?

Tks again,

Pedro

[Levi_Turner · Accepted solution]

Well, the API is private and therefore undocumented (and has the potential to change at any time).

But the param that you're looking for is top. So an example would be:

GET /api/v1/qix-datafiles?connectionId=8ccd4eb8-7ba8-4add-8297-26c3e16f2b85&top=10000

I would expect pagination to be a requirement for the API to go public and stable like the rest of the public and stable APIs on Qlik.dev: https://qlik.dev/apis/#rest

[wwengso314]

Hi, I followed the method above to delete the files in shared spaces, however, but the files are still in the shared spaces, I have tried to check the list using 'qlik raw get v1/qix-datafiles?connectionId={elementId}', the query listed down the files from personal space. What did I miss out? Thank you.

[pedrobergo]

Hi @wwengso314 

To use a qlik-clik withing line commands (Powershell, CMD or Bash), follow this to use it:

qlik raw get v1/qix-datafiles --query connectionId=dataconnectionid,top=100000

The parameter --query is same that '?' and you need to use comma (,) to separate the parameters.

And if you are using inside Powershell, add the '| ConvertFrom-Json' at end of line to better see it.

[],

Pedro

[gordon_ash]

Hi Pedro,

I have followed this thread and used your scripts for the past year or so, thanks for that; I have accidentally uploaded over 90,000 files on top of an existing 40000 files and so need to delete them. When I use the top parameter it only seems to cope with about 35000 files, and they are in alphabetical order - the files I want to delete all begin with R so are probably from 50000 onwards so they never get returned. Is there any way of either ordering differently or having another pagination parameter, to get to the R*.qvd files to delete?

Thanks

Gordon

[Levi_Turner]

The underlying API does not support filtering in this way so you will need to do it in your code. To get all data-files in a space using qlik-cli, you can use this approach:

$space_id = '5e42f2cc0ea40d0001214b3a'
$data_files_connection_id = (qlik data-file connection ls --spaceId $space_id | ConvertFrom-Json).id
$space_data_files_tmp = qlik data-file ls --connectionId $data_files_connection_id --raw  | ConvertFrom-Json
$space_data_files = $space_data_files_tmp.data
do {
    $separator = "next=([^}]*)}"
    $nextURL = [regex]::match($space_data_files_tmp.links.next.href, 'next=(.*?)(&|$)').Groups.Value[1] 
    $space_data_files_tmp = ''
    IF($nextURL -gt 0) {
        $space_data_files_tmp = qlik data-file ls --connectionId $data_files_connection_id --raw --page $nextURL | ConvertFrom-Json
        $space_data_files += $space_data_files_tmp.data
    }
} while (($space_data_files_tmp.links.next).Length -gt 1)

Something like this will output the files with names which start with R or higher:

 $space_data_files |  Where-Object { $_.name -match '^[r-z]' }

[gordon_ash]

Hi @Levi_Turner ,

Thanks for the quick response, I have implemented the code,  but I never get anything in the @nextUrl when matching the regex, it always returns null. I have even just set @nextUrl to the links.next.href just to get the next full page of data but nothing gets returned. Presumably the regex is looking for something to define next=...  The value of my 

$space_data_files_tmp.links.next.href is :

"https://m3data.eu.qlikcloud.com:443/api/v1/data-files?connectionId=a34e64e7-d9b9-4f43-adc2-39516e61c..."

I don't know what I am missing.

[Levi_Turner]

Hmmm, it's possible that the platform changed the param for the next page of data. This should work:

$space_id = '5e42f2cc0ea40d0001214b3a'
$data_files_connection_id = (qlik data-file connection ls --spaceId $space_id | ConvertFrom-Json).id
$space_data_files_tmp = qlik data-file ls --connectionId $data_files_connection_id --raw  | ConvertFrom-Json
$space_data_files = $space_data_files_tmp.data
$page_separator = 'page'
do {
    $nextURL = [regex]::match($space_data_files_tmp.links.next.href, "$($page_separator)=(.*?)(&|$)").Groups.Value[1]
    $space_data_files_tmp = ''
    IF($nextURL -gt 0) {
        $space_data_files_tmp = qlik data-file ls --connectionId $data_files_connection_id --raw --page $nextURL | ConvertFrom-Json
        $space_data_files += $space_data_files_tmp.data
    }
} while (($space_data_files_tmp.links.next).Length -gt 0)