Solved: QDF containers in cloud - Qlik Community

bmenicucci · ‎2022-01-17

Hello everyone,

I have a question for you: is that possible to create containers or use QDF scripts stored in cloud drives like Google Cloud Storage, Dropbox etc.?

Thanks a lot!

Brunello

Magnus_Berg · ‎2022-03-21

Hi Brunello, not at the moment, but we see the need to enable this. Hope that this can be delivered, but I can not promise anything, as there could be unknowns here. Regards Magnus

View solution in original post

Magnus_Berg · ‎2023-05-30

Hi Brunello, as stated in the read me this is an alpha version that is not part of the deploy tool yet. You need to create your containers using deploy tool (or use a copy of containers you already have), you need after to replace the scripts with the new versions (git). You also need to modify container map to work with aOneDrive, as alternate path. Remember that its the path to the QDF root that you point, not the container itself (as the container name already exists in containermap.). Remember not to use excel when editing container map as this coverts it unreadable for QDF, instead use notepad.. Hope this helps..

View solution in original post

bmenicucci · ‎2024-06-18

Hello @Magnus_Berg ,

I have been able to successfully deploy an application in Qlik Saas using QDF and wanted to share with you my settings, that can be useful for others as well.

General Settings:

QDF folders are created in OneDrive, and all the scripts are there
As of now, I don't have created any DataGateway to collect data from real situation, just rely on some SQL data extracted previously and stored in 2.QVD/1.Extract/SQL
QDF container includes a DEV folder and a SHARED one
the Custom.Variables.csv file contains the following lines:
- VariableName,VariableValue,Comments,Tag
- vL.QVDPath,'lib://[space]:DataFiles',Path for QVD storing,
- vG.ExtractPath,'$(vG.QVDPath)/1.Extract',Path for QVD extracting,
- vG.LoadPath,'$(vL.QVDPath)',Path for QVD loading,
- Qvc.Global.v.QlikProduct,'SENSE',sets the product version,
- Qvc.Global.Extension.Directory,'$(vG.CustomPath)/4.Extensions/',sets the extensions folder,
the ContainerMap.csv contains the following lines:
- Prefix,ContainerName,Comments,AltPath
- Shared,Shared,Shared environment,lib://[spaces]:OneDrive/QDFSaaS,
- Admin,Administration,Mandatory administration container,lib://[spaces]:OneDrive/QDFSaaS,
- Dev,Dev,Dev environment,lib://[spaces]:OneDrive/QDFSaaS,
- Prod,Dev,Prod environment,lib://[spaces]:OneDrive/QDFSaaS,
- [spaces],[spaces],[spaces]Space in Saas,

My general loading script is the following:

//GLOBAL CONFIGS

SET vL.Space='[spaces]';

SET vG.RootContainer='lib://[spaces]:OneDrive/QDFSaas/$(vL.Space)';

LET vL.Environment=Subfield(DocumentTitle(),' ',1);

IF '$(vL.Environment)'='DEV' THEN

SET vG.HomeContainer='$(vG.RootContainer)/Dev';

ELSEIF '$(vL.Environment)'='PROD' THEN

SET vG.HomeContainer='$(vG.RootContainer)/Prod';

ENDIF

SET vL.Platform = 'SaaS';

// Basic search for 1.Init.qvs

$(include='$(vG.HomeContainer)\InitLink.qvs');

// Locale for Italian

$(Include=$(vG.LocalePath)/6.Ita.qvs);

$(Include=$(vG.SharedSubPath)/Qvc.qvs);

// Carica i dati

// $(Include=$(vG.CustomPath)\00.ExtractSQL.qvs);

$(Include=$(vG.CustomPath)\01.Extract.qvs);

$(Include=$(vG.CustomPath)\02.Transform.qvs);

$(Include=$(vG.CustomPath)\03.Load.qvs);

TRACE '##### END SCRIPT';

From there, you can see that "Shared" container is not used to run the init script, instead I worked to run it from the DEV folder, though Shared is loaded through the ContainerMap (I believe)

So, the scripts are read from OneDrive, while the QVDs, be them temporary or the final ones, will be stored in the [spaces]:DataFiles folder. I had to change some slashes (from '\' to '/') in my scripts and take care that all the STORE commands will point to the DataFiles folder. Then, I had to modify the Load script, to exclude the Temp QVDs (until I find a way to create/read folders in DataFiles, provided there's one).

Here's the QVDLoad script content:

LET QVD = '$(vL.QVDPath)'; //'lib://[spaces]:DataFiles';

FOR EACH vFileName IN filelist(QVD& '/*.qvd')

LET vFileBaseName=replace(subfield('$(vFileName)', '/', -1), '.qvd', '');

TmpQVDList:

LOAD

Autonumber(Name) AS ID,

Name,

FullName,

Size,

FileTime,

Path

WHERE NOT left(Name,4)='Temp'; // Do not consider Temp QVDs for loading

LOAD

'$(vFileName)' as FullName,

'$(vFileBaseName)' as Name,

FileSize('$(vFileName)') as Size,

FileTime('$(vFileName)') as FileTime,

FilePath('$(vFileName)') as Path

autogenerate 1;

next vFileName ;

// Generate the Loop for each value in the LIST_OF_TABLES table

for iQVDFile=0 to NoOfRows('TmpQVDList')-1;

let vQVDFile = peek('Name', iQVDFile, 'TmpQVDList'); TRACE '$(iQVDFile)';

let vQVDPath = peek('FullName', iQVDFile, 'TmpQVDList');

'$(vQVDFile)':

NOCONCATENATE

LOAD

*

FROM '$(vQVDPath)' (qvd);

//Variables CleanUp

LET QVD=;

LET vFileName=;

LET vFileBaseName=;

LET iQVDFile=;

LET vQVDFile=;

LET vQVDPath=;

Hope this will help!

Thanks for all your support in deploying QDF! Keep up the great job 😉

Brunello

View solution in original post

thomasmaure · ‎2022-01-17

Hello,

Very interesting question and I would like to get it answered as well.
I was able to use $(include= or $(must_include= with files on Azure blob storage and a directory tree similar to QDF one but I have no clue if the InitLink.qvs script works if the container is fully deployed on Azure Blob storage, nor if I can deploy it via the QlikDeploymentFramework tool .

For me the most important would be to check if the InitLink.qvs works correctly or if it requires some rework .

Regards

Thomas

bmenicucci · ‎2022-01-17

Hi Thomas,
exactly the same for me: I'm trying to use my QDF scripts from Google
Storage Cloud but without success as it seems not able to properly execute
InitLink.qvs
Brunello

thomasmaure · ‎2022-01-17

I think the InitLink.qvs is too sophisticated for cloud use with many subtilities like automated config file load or reverting to 99.SharedFolders when vG.HomeContainer does not point to valid directory .
It does not work probably because filesystem functions like filesize, or filetime are missing .

I am not working on this at the moment , but my idea was to rewrite a simplied InitLink.qvs with just
let vG.QVDPath= ...
let vG.ScriptPath = ...

and some code adpatation for the LCGV / DCGV functions .

albena_mitova · ‎2022-01-20

Hello there,

I am trying to do the same here but it seems the InitLink.qvs cannot be executed. Could anyone from Qlik provide a little bit more clearance on the topic? Any ideas how to replicate the idea of the QDF on the Qlik Claud? @Michael_Tarallo @rwunderlich

Thanks!

rwunderlich · ‎2022-01-20

I'll ping @Magnus_Berg here to see if he has any thoughts on addressing QDF SaaS compatibility.

-Rob

bmenicucci · ‎2022-01-20

That's great, Rob!

Thank you so much

-Brunello

Magnus_Berg · ‎2022-02-22

Hi Brunello, sorry for late reply. Ive been sick in Covid this winter. I have seen customers installing QDF including Qlik scripts using a s3 bucket. But you might need to change some code within QDF to make this work, as it's not supported out of the box today. Hope this helps Regards Magnus

Magnus_Berg · ‎2022-02-22

Thx @rwunderlich sorry for late reply, been sick in Covid. Good that you are testing this out, I have not started any work on this. Also think that functions like filesize will make it more difficult, as the initiation is validating what's available on disk and assigns global variables only to existing folders, will need to dig deeper into this. Regards Magnus

rwunderlich · ‎2022-02-22

Thanks for the response. I hope you are recovered.

-Rob