Suggest an Idea

[simonaubert]

Hello all,

Some actions are available when you select several app in the QMC : edit, etc...

However, we cannot export several application in once. Yesterday, I had 50+ app to export and doing that manually is a real pain. Moreover, this can lead to error.

Best regards,

Simon

[AndrewMcIlwrick]

Apart from the missing feature, the QMC inconsistency is also confusing / frustrating at times

[Thomas_Hopp]

Hi @simonaubert thanks for sharing this idea with us. Can you share a bit more about your use case and why you need to export so many Apps at once. We would like to understand the use case better to see if there are other ways to solve the problem.

Best regards,

Thomas

[simonaubert]

Hello @Thomas_Hopp 

Two main use cases with a customer with more than 1000 applications (front or "etl" ), My team of three developers owning 500 of it :
1/export/import from Development to Recept or Production Server. We have to deliver consistent packages (Etl layer 1, 2, 3... front) from sometimes 20 or more apps

2/a few weeks ago, we get a new environment who was initialized with app (let's say t0).  But then the IT Team stil has to do some work on it and we cannot use it. About 500 apps were modified before we could have the hand on it and we had to export each app by hand.

Best regards,

Simon

[howiekrauth]

@simonaubert , I have code written in c# that can export all the apps in a folder.  Currently working on code to re-import those same apps.  I've only tested this internally.  Let me know if you'd like to try it.

[_Adam_]

Hi @howiekrauth ,

Can you tell me if you managed to to the export via the code you wrote?

I also need to export around 400 apps from one environment to another and looking for more efficiant solution.

Thanks!

[mountaindude]

First. I assume this post is about client-managed Qlik Sense. It doesn't say, but reading between the lines that's the feeling I get.
The concept outlined below works for Qlik Cloud too, with some modifications.
-----------------------

I agree it would be nice to have this built into the QMC, but that model kind of breaks down if you are looking at more than 1-2 dozen apps.

If you are looking at exporting hundreds of apps I would argue that the QMC is the wrong tool, at least if want to avoid manual mistakes etc.  Especially if you expect a recurring need to export large number of apps.

I would instead turn to the excellent Qlik CLI tool.

While it mainly focuses on Qlik Cloud, the "qlik qrs" command is very powerful as it gives command line access to most parts of the QRS API. Including app export.

So.. you would have to 

1. Set up a virtual proxy that use JWT authentication. Qlik CLI relies on JWT auth when talking to client-managed Qlik Sense Enterprise on Windows.
2. Create a JWT to authenticate "qlik cli" with.
   The qs-jwt tool (created by yours truly) can be useful for doing this.
3. Run a command based on "qlik qrs app export create" and then the "qlik qrs download" command (if I recall correctly, it's been a while since I did this).
4. Once you have the exact command for exporting one app, you can script the whole thing in PowerShell:
   1. First use "qlik qrs app..." to get a list of all apps that should be exported.
   2. Loop over those apps, exporting them one by one
   3. Get a ☕, sit back and watch the export magic happen 🦄

[simonaubert]

Hello @mountaindude 

1/ yes, it was for client-managed in head (because as of today, it's mainly what I use), with several environments. But why not about cloud.

2/I'm working on qlik-cli in parallel. There are two hard parts : the first one, authentication, and the second one, I'm not that good at powershell and I spend hours even for a simple program. I also had to rewrite some code (if my memory works, the update task doesn't allow to change task name). So, that's a time-consuming process. There is also a more personal point : I don't like having too many different technologies, tools or languages in the same project, because that means managing skills in the team.

Best regards,

Simon

[stephenfr]

Hi Simon,

I made a Powershell script that makes automated backups of our apps using a Windows task, works perfectly. Organized by stream or user. With some modification it can export your apps in bulk. I do not have a script to import apps in bulk, so if someone has that or makes it based on this code, then please let me know!

I won't be able to explain or support this code, but the script has a lot of explanation in it, so you should be able to figure it out. Setting up Qlik-CLI is something you need to figure out yourself.

Good luck with it.

Stephen

^requires -version 4
<#
.SYNOPSIS
Export QVF files from Qlik Sense Enterprise server without data, grouped by user and stream.

.DESCRIPTION
Export QVF files from Qlik Sense Enterprise server without data, grouped by user and stream.
The number of backup versions to keep per app can be set using the variable versions_to_keep, with a
differentiation between workstream and published apps, as workstream apps are updated more frequently.
Files are stored in the format: [username]\[stream name]. If the QVF is not published,
the stream name 'Workstream' is used. This can be changed using the variable workstream_folder_name.

The script gets a list of all apps in the Qlik repository and then checks per app if the last
update date is later than the last time the app was saved (this date is available as a suffix
in the file name).
If the app is published, then the "date published" will used for the comparison. If the app is not
published, the "date modified" will be used. Note that unfortunately, this "date modified" is also
updated when the app is reloaded. Qlik doesn't keep track of when the script or objects were modified.
Also note that comparison is done per date and the time is ignored. If backups need to be made more
than once a day, you need to modify the script to include the time portion.
If the app was modified, it will be saved with the last modified date in the suffix. After the
app was saved, the scrips checks if there are more versions than the max number of versions
to keep. If so, it will delete the older versions. Note that apps that are deleted from Qlik,
will remain in the backup folders until manually deleted. Also if apps were transfered to another user,
or if the user was deleted from Qlik, the folder and apps of the original user will remain there.
There is a Qlik dashboard available that gives a good overview of all apps and their backups,
including the apps that were deleted from Qlik.

#>

Set-ExecutionPolicy Unrestricted -Scope CurrentUser

$start_time = (Get-Date) #Get todays date
$qlik_server = "qlik-dev.domain.com" #Qlik Server host name
$backup_root_folder = "\\servername\D$\Qlik\Backup\BackupUserApps" #The location to store the app backup files
$log_folder = "\\servername\D$\Qlik\Backup\BackupScriptLog" #The location to store the log files
$todays_date = (Get-Date).ToString("yyyy-MM-dd") #Todays date in text format
$stream_name = "" #The stream name of the current app
$workstream_versions_to_keep = "10" #The number of backup versions per workstream app to keep
$published_versions_to_keep = "5" #The number of backup versions per published app to keep (typically less than work stream apps)
$folder_naming_style = "user" #Values: "stream" or "user". If the folders will be named by stream (with subfolders by user) or by user name (with subfolders by stream).
$workstream_folder_name = "Workstream" #The folder name to use for unpublished apps that are not in a stream
$backup_versions_to_keep = "" #The number of backup versions to keep (will be set during runtime)
$app_change_date = "" #The date when the app was modified or published in case of published apps (will be set during runtime)
$app_counter = 0 #Count the apps that were found (will be set during runtime)
$app_backed_up_counter = 0 #Count the apps that were backed up (will be set during runtime)
$parameter_check_text = "" #This variable is used to set the error text for missing

#Store the log file with todays date
Start-Transcript -Path "$($log_folder)\backup_qvf_log_$($todays_date).txt" -Force

#Start new export/backup
Write-Output "Script start: $($start_time)"
Write-Output "Connecting to $qlik_server"
Connect-Qlik -TrustAllCerts -UseDefaultCredentials
#Connect-Qlik $qlik_server #not working without configuring the certificate first

#Check if all required parameters are available
If ( $workstream_versions_to_keep -lt 1 ) {
$parameter_check_text = $parameter_check_text + [char]10 + "The number of version to keep for the workstream has to be bigger than zero."
}

If ( $published_versions_to_keep -lt 1 ) {
$parameter_check_text = $parameter_check_text + [char]10 + "The number of version to keep for the published streams has to be bigger than zero."
}

If ( $folder_naming_style -ne "stream" -And $folder_naming_style -ne "user") {
$parameter_check_text = $parameter_check_text + [char]10 + "The folder naming style has not been set properly."
}

If ( $workstream_folder_name.Length -lt 1 ) {
$parameter_check_text = $parameter_check_text + [char]10 + "The workstream folder name has not been defined."
}

#If something was missing, then go to the end of the script to exit
If ($parameter_check_text.Length -gt 0 ) {

Write-Output "$($parameter_check_text)"
Write-Output "Exiting script"
Exit

}

#Examples of filtering apps
#foreach($qvf in $(Get-QlikApp -full -filter "stream.name eq 'Finance'")) {
#foreach($qvf in $(Get-QlikApp -full -filter "owner.name eq 'Doe, John'")) {
#foreach($qvf in $(Get-QlikApp -full -filter "owner.name so 'John Doe'")) {
#foreach($qvf in $(Get-QlikApp -full -filter "name so 'dummyTest'")) {

#Loop through all the QVFs on the server
foreach($qvf in $(Get-QlikApp -full)) {

$app_counter = $app_counter + 1

#Remove illegal characters from userId and QVF file name to avoid issues when exporting the file
#because the Qlik API can't handle some characters that are allowed to be used in the app name.
#Extra replace for [ and ], which for some unknown reason lead to an error
$user_id = ($qvf.owner.userId).Split([IO.Path]::GetInvalidFileNameChars()) -join ' '
$qvf_name = (($qvf.name).Split([IO.Path]::GetInvalidFileNameChars()) -join ' ').replace('[', '{').replace(']', '}')

#Get the app date modified for the file name suffix. Replace the / with -
$file_suffix = "_" + ($qvf.modifiedDate).substring(0, 10).replace('/', '-')

#Check if the app is published
if ($qvf.published) {

#If so, get the stream name. Check for and remove invalid characters if needed
$stream_name = ($qvf.stream.name).Split([IO.Path]::GetInvalidFileNameChars()) -join ' '

#Set the number of versions to keep
$backup_versions_to_keep = $published_versions_to_keep

#Store the published date
$app_change_date_txt = $($qvf.publishTime).substring(0, 10).replace('/', '-')

} else {

#If the app was not published, use the variable "workstream_folder_name" for the folder name
$stream_name = $workstream_folder_name

#Set the number of versions to keep
$backup_versions_to_keep = $workstream_versions_to_keep

#Store the modified date
$app_change_date_txt = $($qvf.modifiedDate).substring(0, 10).replace('/', '-')

}

#Check what the folder name style should be
If ( $folder_naming_style -eq "stream" ) {

#If it is a workstream, add the user name to the stream name
If (!$qvf.published) {

#Set the backup folder name
$backup_sub_folder = $stream_name + "\" + $qvf.owner.userId

} Else {

#Set the backup folder name
$backup_sub_folder = $stream_name

}

} ElseIf ( $folder_naming_style -eq "user" ) {

#Set the backup folder name
$backup_sub_folder = $qvf.owner.userId + "\" + $stream_name

}


# Check if the backup folder already exists
If(!(test-path "$backup_root_folder\$backup_sub_folder")) {

#If not, create it and log this action
New-Item -ItemType Directory -Force -Path "$backup_root_folder\$backup_sub_folder"
Write-Output "New folder created"

}

#Now only back up apps that were modified since the last time it was saved.
#Search the destination folder to retrieve the latest app file by ignoring the file suffix with the modified date
$file_dir = "$($backup_root_folder)\$($backup_sub_folder)\$($qvf_name)^$($qvf.id)*.qvf"
$last_backup_file = Get-ChildItem -Path $file_dir | Sort-Object LastAccessTime -Descending | Select-Object -First 1

#Check if a backup file was found or not
If (!$last_backup_file) {

#If not, set the "last_modified_date_file" far in the past so the app will be backed up.
$last_modified_date_file = [datetime]::ParseExact('1900-01-01', 'yyyy-MM-dd', $null)

} else {

#Else set last_modified_date_file to the app modified date, which is extracted from the file name suffix
$last_modified_date_file_txt = $($last_backup_file.Name).substring($last_backup_file.Name.length - 14, 10)
#Convert the date from text to date format, so it can be compared with todays date
$last_modified_date_file = [datetime]::ParseExact($last_modified_date_file_txt, 'yyyy-MM-dd', $null)

}

#Get the last changed date of the app
$app_change_date = [datetime]::ParseExact($app_change_date_txt , 'yyyy-MM-dd', $null)

Write-Output "App No. $($app_counter) ***************************************************" #separator line for logging app extract details

#Check if the current modified date is higher than the last modified date. If so, extract the app and save the file
If( $app_change_date -gt $last_modified_date_file ) {

#Set the name of the export file. Using the (cleansed) app name, the app id and the date modified
$export_filename = "$($qvf_name)^$($qvf.id)$($file_suffix).qvf"

#Extract the app from the repository, without data and save if as a qvf file in the designated folder
Export-QlikApp -id $qvf.id -skipdata -filename "$($backup_root_folder)\$($backup_sub_folder)\$($export_filename)"

#Keep a log of the app that was extracted
Write-Output "Stream: $($stream_name)"
Write-Output "App: $($qvf.name)"
Write-Output "Id: $($qvf.id)"
Write-Output "Owner: $($qvf.owner.name) ($($qvf.owner.userId))"
Write-Output "Exported to: $($backup_root_folder)\$($backup_sub_folder)\$($export_filename)"

#Count the number of files that were extracted
$app_backed_up_counter = $app_backed_up_counter + 1

#Count the total number of backup files for the current app
$backup_files_list = Get-ChildItem -Path $file_dir -Recurse | Where-Object {-not $_.PsIsContainer}

#Count how many backup files need to be deleted based upon how many versions should be kept
$number_of_files_to_delete = $backup_files_list.Count - $backup_versions_to_keep

#Check if there are more files than the maximum number of versions to keep
if ($number_of_files_to_delete -gt 0) {

#If so, sort the files by creation date and delete the oldest files
$backup_files_list | Sort-Object CreationTime | Select-Object -First $number_of_files_to_delete | Remove-Item -Force

#Keep a log of the number files that were deleted
Write-Output "Cleaning archive: $($number_of_files_to_delete) older backup file(s) deleted for app $($backup_sub_folder)\$($qvf.name)"
}

} else {

#If the app was not updated, no export is required. Keep a log of that
Write-Output "App not exported (no changes): $($backup_sub_folder)\$($qvf.name)"

}

Write-Output " "

}

$end_time = (Get-Date)
$elapsed_time = $end_time - $start_time
Write-Output "------------ Script Summary -----------"
Write-Output "Script start : $($start_time)"
Write-Output "Script end : $($end_time)"
Write-Output "Apps processed : $($app_counter)"
Write-Output "Apps unchanged : $($app_counter - $app_backed_up_counter)"
Write-Output "Apps new+updated : $($app_backed_up_counter)"
Write-Output $elapsed_time

Stop-Transcript

[howiekrauth]

I have created an executable for exporting hundreds of apps, but do not know where to put it here.