QDF containers in cloud

[bmenicucci]

Hello everyone,

I have a question for you: is that possible to create containers or use QDF scripts stored in cloud drives like Google Cloud Storage, Dropbox etc.?

Thanks a lot!

Brunello

[Magnus_Berg]

Great! Found and updated in git, please try without your change.
We need a generic solution that identifies between shared or "other home" container (as Qlik Cloud space can be home, thereby shared need to be used to read the containermap)

[Magnus_Berg]

Hi @bmenicucci  I found a bug related to exact your issue, I did not notice before due to an older saved variable. The bug is corrected and is now released in git. Please test and get back to me, if it all works I think we could release this version in QDF deploy package (after some testing). /Regards Magnus

[bmenicucci]

Hello Magnus,

I've tried using the QDF for cloud (Thank you!! very great job!), and was wondering why I'm not able to get the vG.QVDPath variable properly set. Looking at Init script, I've noted that actually it is setting that variable only for shared containers... is that correct? Please look at the following extract from the mentioned script, where you can see it only switches for shared containers and not the regular ones.

Furthermore, I have one last question: do you think it's possible to have just one QDF able to work for either cloud and on premises?

Thanks for your support!

//Set global variables for include scripts

//Determin if it's shared scripts used, by setting vG.SharedInclude or vG.Include in script initiaiton

LET vL.QDF.Home=lower(right('$(vG.HomeContainer)',6));

Switch '$(vL.QDF.Home)' // identifing shared container

   case 'shared'

      SET vL.QDF.SharedInclude='Shared'; //Shared does not contain slash

      //Set global variables for BasePath related to the space the app is executing from

      SET vG.BasePath= 'lib://:'; // same as current space lib: // [space]: Datafiles / *. qvd

      SET vG.QVDPath= '$(vG.BasePath)DataFiles/';

   case 'hared/' //Shared contains front-slash

      SET vL.QDF.SharedInclude='Shared';

      //Set global variables for BasePath related to the space the app is executing from

      SET vG.BasePath= 'lib://:'; // same as current space lib: // [space]: Datafiles / *. qvd

      SET vG.QVDPath= '$(vG.BasePath)DataFiles/';

   case 'hared\' //Shared contains back-slash

      SET vL.QDF.SharedInclude='Shared';

      //Set global variables for BasePath related to the space the app is executing from

      SET vG.BasePath= 'lib://:'; // same as current space lib: // [space]: Datafiles / *. qvd

      SET vG.QVDPath= '$(vG.BasePath)DataFiles/';

    default

    SET vL.QDF.SharedInclude=''; // No shared folder, use $(vG.HomeContainer) base path

end switch  

[bmenicucci]

Hello @Magnus_Berg ,

I have been able to successfully deploy an application in Qlik Saas using QDF and wanted to share with you my settings, that can be useful for others as well.

General Settings:

• QDF folders are created in OneDrive, and all the scripts are there
• As of now, I don't have created any DataGateway to collect data from real situation, just rely on some SQL data extracted previously and stored in 2.QVD/1.Extract/SQL
• QDF container includes a DEV folder and a SHARED one
• the Custom.Variables.csv file contains the following lines:
  • VariableName,VariableValue,Comments,Tag
  • vL.QVDPath,'lib://[space]:DataFiles',Path for QVD storing,
  • vG.ExtractPath,'$(vG.QVDPath)/1.Extract',Path for QVD extracting,
  • vG.LoadPath,'$(vL.QVDPath)',Path for QVD loading,
  • Qvc.Global.v.QlikProduct,'SENSE',sets the product version,
  • Qvc.Global.Extension.Directory,'$(vG.CustomPath)/4.Extensions/',sets the extensions folder,
• the ContainerMap.csv contains the following lines:
  • Prefix,ContainerName,Comments,AltPath
  • Shared,Shared,Shared environment,lib://[spaces]:OneDrive/QDFSaaS,
  • Admin,Administration,Mandatory administration container,lib://[spaces]:OneDrive/QDFSaaS,
  • Dev,Dev,Dev environment,lib://[spaces]:OneDrive/QDFSaaS,
  • Prod,Dev,Prod environment,lib://[spaces]:OneDrive/QDFSaaS,
  • [spaces],[spaces],[spaces]Space in Saas,

My general loading script is the following:

//GLOBAL CONFIGS

SET vL.Space='[spaces]';

SET vG.RootContainer='lib://[spaces]:OneDrive/QDFSaas/$(vL.Space)';

LET vL.Environment=Subfield(DocumentTitle(),' ',1);

IF '$(vL.Environment)'='DEV' THEN 

SET vG.HomeContainer='$(vG.RootContainer)/Dev';

ELSEIF '$(vL.Environment)'='PROD' THEN

SET vG.HomeContainer='$(vG.RootContainer)/Prod';

ENDIF

SET vL.Platform = 'SaaS';

// Basic search for 1.Init.qvs

$(include='$(vG.HomeContainer)\InitLink.qvs'); 

// Locale for Italian

$(Include=$(vG.LocalePath)/6.Ita.qvs);

$(Include=$(vG.SharedSubPath)/Qvc.qvs);

// Carica i dati

// $(Include=$(vG.CustomPath)\00.ExtractSQL.qvs);

$(Include=$(vG.CustomPath)\01.Extract.qvs);

$(Include=$(vG.CustomPath)\02.Transform.qvs);

$(Include=$(vG.CustomPath)\03.Load.qvs);

TRACE '##### END SCRIPT';

From there, you can see that "Shared" container is not used to run the init script, instead I worked to run it from the DEV folder, though Shared is loaded through the ContainerMap (I believe)

So, the scripts are read from OneDrive, while the QVDs, be them temporary or the final ones, will be stored in the [spaces]:DataFiles folder. I had to change some slashes (from '\' to '/') in my scripts and take care that all the STORE commands will point to the DataFiles folder. Then, I had to modify the Load script, to exclude the Temp QVDs (until I find a way to create/read folders in DataFiles, provided there's one).

Here's the QVDLoad script content:

LET QVD = '$(vL.QVDPath)'; //'lib://[spaces]:DataFiles';

FOR EACH vFileName IN filelist(QVD& '/*.qvd')

LET vFileBaseName=replace(subfield('$(vFileName)', '/', -1), '.qvd', '');

    TmpQVDList:        

    LOAD

      Autonumber(Name) AS ID,

        Name,

        FullName,

        Size,

        FileTime,

        Path

    WHERE NOT left(Name,4)='Temp'; // Do not consider Temp QVDs for loading

    LOAD

    '$(vFileName)' as FullName,

    '$(vFileBaseName)' as Name,

    FileSize('$(vFileName)') as Size,

    FileTime('$(vFileName)') as FileTime,

    FilePath('$(vFileName)') as Path    

    autogenerate 1;

next vFileName ;

// Generate the Loop for each value in the LIST_OF_TABLES table

for iQVDFile=0 to NoOfRows('TmpQVDList')-1;

let vQVDFile = peek('Name', iQVDFile, 'TmpQVDList'); TRACE '$(iQVDFile)';

let vQVDPath = peek('FullName', iQVDFile, 'TmpQVDList');

    '$(vQVDFile)':      

    NOCONCATENATE

    LOAD

      *

    FROM '$(vQVDPath)' (qvd);

next

DROP Table TmpQVDList;

//Variables CleanUp

LET QVD=;

LET vFileName=;

LET vFileBaseName=;

LET iQVDFile=;

LET vQVDFile=;

LET vQVDPath=;

Hope this will help!

Thanks for all your support in deploying QDF! Keep up the great job 😉

Brunello