Introduction of Artificial Intelligence

Sometimes, I want I might return to 1955 and forestall John McCarthy from occupation it “Artificial Intelligence.” It’s a term that, relying upon wherever you're employed, you can’t go five minutes while not hearing once or doubly — that is nice. It’s nice that folks are trying to the ‘future.’ It’s nice that society is pushing forward with growth and growth and everyone that heat and fuzzy stuff. sadly, AI doesn’t very do justice to what it’s describing. read this full from here https://www.edu-right.com/2019/08/what-is-application-of-artificial-intelligence.html

Qlik Deployment Framework for Cloud (alpha) 
These scripts are specific for Qlik Cloud, meaning that you can not run Qlik Sense desktop, QlikView or Qlik Sense server together with these scripts. Qlik Cloud spaces can be used as containers as well as classic containers stored under external storage that is mapped as a drive in Qlik Cloud.

This is an early alfa with several bugs and limitations in Qlik Cloud. Use standard QDF containers and replace the scripts with these. All QDF functions are loaded in during initiation but not all of functions are tested to work with Qlik Cloud.

• An external Shared container is a must containing these Qlik Cloud script (replacing older QDF container scripts) in an external storage. This container is need to initiate QDF using Qlik Sense reload script, using vG.HomeContainer as seen below:
  • SET vG.HomeContainer='lib://user:OneDrive - user/QDF_SaaS/Shared';
  • $(Include=$(vG.HomeContainer)\InitLink.qvs);
• In this release all container folders in external drives (folders you want QDF to identify as Global Variable path) need to include the Info.txt file, as this file identifies the folder as a Global path
• External drives are really picky on trailing slash, for OneDrive no trailing slash is possible
• To identify containers stored in external drives, the 'AltPath' to the external dirve and container root needs to be specified in vG.SharedBaseVariablePath/ContainerMap.csv
• Qlik Cloud spaces as containers works but with limited subfolders, only global variables vG.BasePath and vG.QVDPath are generated and used.
  • Mapp QDF container -> space by adding space name in Prefix and ContainerName under vG.SharedBaseVariablePath/ContainerMap.csv (no AltPath are specified in container map)
• Use $(include=$(vG.HomeContainer)/InitLinkSkip.qvs); to skip executing the initiation code that identifies related containers, this only works when initiation has executed successfully one time in the same location

Link to git Qlik Deployment Framework for Cloud (alpha) 

Hi,  Im so sorry for taking long time. My work and family has made this project to suffer, sorry for this. I have uploaded a QDF for cloud Alfa version here https://github.com/QlikDeploymentFramework/Qlik-Deployment-Framework-Cloud
There are several thinks to take in consideration:
This is an early alfa with several bugs and limitations in Qlik Cloud.
* Shared container is a must and need to be mapped within the reload script, as seen below:
SET vG.HomeContainer='lib://user:OneDrive - mail/QDF_SaaS/Shared';
$(Include=$(vG.HomeContainer)\InitLink.qvs);
* In this release all container folders in external drives (folders you want to QDF to identify as Global Variable path) need to include the Info.txt file, as this file identifies the folder as a Global path
* external drives are really picky on trailing slash, for OneDrive no trailing slash is possible
* To identify containers stored in external drives, the base URL need to be specified in vG.SharedBaseVariablePath/ContainerMap.csv
* To identify spaces as containers just add the space name under vG.SharedBaseVariablePath/ContainerMap.csv
* Qlik Cloud spaces has limited support for subfolders, so adding QDF containers to spaces (by same name) only vG.BasePath and vG.QVDPath will work
* To identify spaces as containers just add the space name under vG.SharedBaseVariablePath/ContainerMap.csv* Qlik Cloud spaces hav limited support subfolders, so adding QDF containers to spaces (by same name) only vG.BasePath and vG.QVDPath will work

The next generation BI tools have changed the IT landscape of enterprises with features such as self-service reporting, ease of development and deployment, cloud and big data support. With these rich features and initiatives to modernize IT, there has been surge in adoption and migration to tools such as Qlik and Tableau. The enterprise tools are now focused on automation and reducing IT development and maintenance overheads. And QA, being an integral and critical part of development process, must have clear automation approach supported by tools. The white paper describes a BI QA Automation framework.

Enterprises use BI tools such as SAP BusinessObjects, SAP Crystal Reports, OBIEE, Qlik and Cognos for catering their reporting needs. One of the key factors for successful BI project implementation is the level of trust on the data depicted in reports and dashboards. Lack of trust in data causes attrition in user adoption and often results in project failure.

BI testing is the process of validating the data, format and performance of the reports, subject areas and security aspects of the governing project. Emphasis on a thorough BI Testing is key for improving the quality of the reports and user adoption. However, testing of the BI projects is different from traditional web application testing since the contents of the report is automatically generated by the BI tool based on the tool metadata. The focus of this article is to propose an automated approach to test the report data and its utilization would be a report migration project.

Need for Report QA Automation

Report Data Testing is an important process of assessing the reliability of information visualized on the report. It is required to sustain the trust of the user on the reports because reports are often used as key basis for management decisions and the user assumes that the information on the report is representative of source data.

A typical approach to verify report quality is to verify calculations on the reports themselves to the data in the source report or data warehouse. Thus the BI testing process becomes a manual one, which increases the workload for testers and making it prone to mistakes. Most BI tools manipulate/transform the source data, which makes it even harder to manually test. Manual testing also is a slow and cumbersome approach which compromises the accuracy of the testing.

It needs to be ensured that report data exactly matches with the expected output, as the reports act as a basis to make critical decisions. Having a manual QA process to validate reports has multiple downsides:

• Reports with large data cannot be accurately validated by manual process, and leaves room for potentially uncovered mismatches.
• Huge efforts required for manually validating the report data.

QA Automation Solution Design

As described in the Solution Architecture below the QA automation is done in following steps:

1. 1. Qlik report extracts are generated by exporting the report contents
2. 2. Based on the dimensions and measures identified by the automation utility, code Generator generates the Qlik script file to validate Qlik report data. The Qlik script test engine (an automated process to run the generated script) compared the Qlik data against Gold dump / Legacy BI report extracts and logs the differences in the test log on a database.
3. 3. Gold dumps of expected report content are acquired from business user or by connecting to the legacy BI system which is to be migrated to Qlik.

Fig 1: QA Automation Solution Architecture

1. Legacy BI Report Data Extractor – Automated extract of data from  Legacy BI Report data based on gold dump is pulled in Excel or CSV format and stored at the Extract Repository, as shown in the above figure.
2. Qlik Data extractor – Automated extract of data from migrated Qlik report based on gold dump same as that of the Legacy BI Report is pulled in Excel or CSV format and stored at the Extract Repository.

3. Qlik Code Generator – It is the module developed as an independent tool where user identifies the dimensions or primary fields by which the rows are identified uniquely and the measure fields whose data is to be compared. The Dimension and Measure fields are placed in a Database source system. Once the Dimensions and measures are known to the system, the Qlik Code generator engine initiates and generates the QlikView script to populate the Qlik report with data discrepancies and save it in a Qlik script file.
4. Test Engine – Once the code is generated the user have option to run the Test Engine which will push the code to a Qlik interpretable QVS file on the Qlik Server and initiate the Qlik data comparison report load. On completion of the Qlik Load, the test result will be generated and displayed with option to export it into excel and preserve the result for future reference. The test result will provide the summary of the data discrepancies and the detailed view will highlight the missing data and data mismatches in red background. 

Generated Code Template

QA Automation Report

Fig 3: Report Output

The above figure is the snapshot of the test engine output where the “Comparison Summary” provides the count of Total mismatched data points and mismatch count by Measure. Also depicts count of data points which are present in either source or target.

The Detailed view highlights the mismatched data points in RED background. The column of interest here is “Table” from which one can identify to which source this data row belongs to i.e. in our case OLTP data means transaction data from SAP Crystal Reports and OLAP means Dimensional data from Qlik.

Benefit of ’s BI QA Automation Approach

As per the  Analyst Firm Gartner- “The average organization loses $14.4 million dollars annually through poor data quality.” 

• ’s automated approach saves a substantial amount of effort and time with enhanced accuracy
• To the organization which would adopt our approach will be benefited in terms of reduced project cost due to less number of testers and also reduced technical skills of the testers.
• A good automated BI report testing tool is also characterized by its user friendliness, which designed utility totally complements, so that even a non-technical user can also use it.
• Light weight and ease to configure are other features which makes the  BI QA automation solution a desired utility by any user.

Conclusion

Automated BI report Data Quality Analysis approach is a full-proof automated solution to the challenges faced by the Quality Analysts while performing data matches in the environments like report migration from one BI tool to another. It has multiple file format support feature and based on the pre requisite that the data is freeze at same point for both the reporting tools. The observation is that it reduces the testing effort by around 80 percent with reduced number of resources and lesser technically skilled. So it ensures that the trust on the reports is enhanced by reducing the efforts.

Current Version 3.20 (16-Oct-2015)
Now supports Partial Reload, runs faster with large data sets
and puts Num() function only on the outer KPI if KPIs are nested.

Download: Dropbox - KPI_Repository.zip

I recommend to name KPI variables starting with "_". Then you have a nice way to use type ahead formula completion in Sense and QlikView, as all your KPIs sort in a dropdown list next to the "_".

As Presales, you have probably done it a hundred times: Add lots of KPIs formulas and also try to keep a maintainable format with a "define once" principle. Formulas and Sets should be separated in their definitions and mixed and reused as needed.

This solution allows you to

• Define KPIs separate from the Apps and define in which app to use the KPI
• Make Set-Definitions separately from KPIs
• Use KPIs on basis of other KPIs (KPI3 = KPI1 / KPI2)
• Inject Set-Modifiers in the same KPI creating variants of the KPI without redefining.
• In large apps you will need just a Partial Reload to redefine formulas, not a entire data loading. The script runs also in Partial Reload mode.
  (for Sense 2.0 and 2.1 you need this external tool to execute a Partial Relaod https://community.qlik.com/docs/DOC-8866)

It is common knowledge that

• you can put your KPI formulas into a text file or  in a database tables
• load this table and loop through it while creating variables for each formula
• Anywhere in Qlik Sense or QlikView you may then call that formulas by the dollar-brackets syntax $(my_variable) ... this is a "code injection" interpreted ad-hoc.

But wait a minute! There are two traps, which are definately NOT common knowledge.

• Those dollar-bracket $(constructs) are immediately interpreted in the script during script execution. This makes a difference and often results in undesired variable content.
• The combination dollar-brackets-equal $(= ...) always leads to a hard to spot error. The script will not throw an error, but instead of where you had $(=something) you will find $(internal error) or simply anything removed between "$(" and ")".

So forget this manual approach and use my include script in one of the below ways: minimalistic flat file, maximum flat file, or database mode.

Using a KPI File (CSV, Excel)

Any textformat is supported. You have to specify the arguments for the text format.

Alternatively, use this in an Excel sheet. Same thing, easier to edit, but Excel is not always available during a SiB

If the KPI Definitions are in a Flat file and you use QlikView:

SET vKPI_Source = 'File';

SET vKPI_File = ;

SET vKPI_File_Params = [txt, codepage is 1252, embedded labels, delimiter is '\t', msq];

SET vKPI_KeepTable = 1; 

$(must_include=);

If the KPI Definitions are in a Flat file and you use Qlik Sense:

Create a folder connection to where the include-script.txt is placed and a folder connection (if different) where the KPI definition file is placed. All formats of the file are supported but you have to provide the file import parameters below, so it could be txt, biff, xlsx …

SET vKPI_Source = 'File';

SET vKPI_File = [lib://include/KPI_Def.txt]; // Qlik Sense

SET vKPI_File_Params = [txt, codepage is 1252, embedded labels, delimiter is '\t', msq];

SET vKPI_KeepTable = 1;

$(must_include=[Lib://include/Create_KPI_Repository-include.txt]);

Using a KPI Database

It doesn't matter which database you used. Attached is an MS Access example. Could be any other relational database.You will need 4 tables:

You know, sequence matters, so it is key to understand when you can already refer to a "previously" defined Formula.The sequence when using a database is like this

1. AggGroups are read
2. Set_Definitions are read in the sequence of SORTID
3. KPI_Definitions and KPI_Variants are cascaded together, the main order is driven by the SORTID of KPI_Defintions, but whenever a KPI_Variant refers to each given KPI_Definition it immediately defines the dependent KPI_Variants after its parent KPI_Definition.

Examples for each table

Table "AppGroups"

Table "Set_Definitions"

Table "KPI_Definitions"

Table "KPI_Variants"

Note that KPI_Variants has no FORMULA column, as the formula is inherited from the KPI defined under VARIANTOF.

Script for QlikView

//Put your database connector here

OLEDB CONNECT TO [Provider=Microsoft.ACE.OLEDB.12.0;User ID=Admin;Data Source=.............];

SET vKPI_Source = Database;

LET vKPI_SQL_AppGroups = 'AppGroups';

LET vKPI_SQL_Set_Definitions = 'Set_Definitions';

LET vKPI_SQL_KPI_Definitions = 'KPI_Definitions';

LET vKPI_SQL_KPI_Variants = 'KPI_Variants';

$(must_include=);

Script for Qlik Sense

Create a folder connection to where the include script is placed. In below example that connection is called “include” Create the database connection to where the definition table Is located. Connect to it, then provide the SQL command in the variable vKPI_Select before calling the include script.

LIB CONNECT TO 'kpi_definitions_db';

SET vKPI_Source = Database;

LET vKPI_SQL_AppGroups = 'AppGroups';

LET vKPI_SQL_Set_Definitions = 'Set_Definitions';

LET vKPI_SQL_KPI_Definitions = 'KPI_Definitions';

LET vKPI_SQL_KPI_Variants = 'KPI_Variants';

$(include=[Lib://include/Create_KPI_Repository-include.txt]);

The columns explained in detail

Provide columns in UPPER CASE. The sequence of the columns doesn't matter. You don't have to provide all columns, the 3 mandatory columns are enough. Any other combination is allowed.

USAGE (mandatory)work in progressNote: the script will not “find” apps with a matching title anywhere and inject the KPI definitions there. You need to add the load script yourself in any of the applicable apps.Be aware that a formerly created variable is not removed if later a wildcard expression is changed and is no longer applicable for a given app. The variable remains with its last loaded state unless removed in the frontend.

NAME (mandatory)Variable name under which the formula is stored. Brackets and single quotes are not allowed in the KPI_Name, the name is case-sensitive.Recommended characters: A-Z, a-z, 0-9, @, $, _ and space. Use a distict prefix for sets (like set_ or $) and another prefix for KPI formulas

FORMULA (mandatory)Put the correct formula or other definitions here.Altough you can assign all type of content into variables here, I see two typical ways of usage: Aggregation Formulas (aka "KPI") and Set Modifiers

• Important: The formula can hold a reference to a previously defined formula in the dollar-brackets-syntax: $(reference).
• Unless column “DONTRESOLVE” is set to 1, the reference is immediately resolved (replaced with the content) upon script execution. If this is not wanted and the reference should make it until later use in the GUI (frontend), put the column “DontResolve” to 1
• If you used $(= … fragment …) combination in the formula (watch for the equal sign(!) after dollar-open bracket) this fragment will not be interpreted by the script execution but passed into the variable definition as part of the formula.  E.g. Sum({< Year={$(=Year(Now())}>} Amount)

VARIANTOF (optional)work in progress

SEARCH (optional)work in progress

REPLACE (optional)work in progress

DONTRESOLVE (optional)A flag that defines whether a formula with a dollar-brackets $(reference) is resolved (=replaced by it’s referred content) immediately during Script execution or if the reference (the dollar-brackets fragment) is kept as part of the formula itself.

• put 1 to NOT resolve dollar-bracket $(fragments) ... if no such  fragment is used, 1 has no effect
• put 0 to resolve dollar-bracket $(fragments)
• If omitted, 0 is assumed.

NUMFORMAT (optional)Optionally, put a valid format-string here. That will cause the above formula to be wrapped into a NUM( … ) formula, where columns 6 and 7 are used as additional parameters for the NUM() formula.

• If omitted, no NUM( … ) formula will be used and formatting is up to the format of the data or chart default.
• Typical examples: 0.0%  #.##0  #,##0.00    #,##0.00 €

DECIMALSEP(optional)If column 5 is used, this may optionally define the decimal separator (one character only).If NumFormat is not used, this column is ignored

THOUSANDSEP(optional)If column 5 and 6 are used, this may optionally define the thousand separator (one character only).If DecimalSep is not used, this column is ignored. It only works in combination with columns 5 + 6

Traps avoided when using my script

If you would like to understand why defining KPIs in variables directly in the script will be problematic then try this in your Load Script

• LET set_LastYear = 'Year={$(=Year(Now())-1)}';
• See the content of the variable "set_LastYear"? It will read 'Year={}' and the lovely formula =Year(Now())-1 is gone. In earlier versions of QlikView you may read 'Year={(internal error)}' instead.
• Try the same with SET:
• SET set_LastYear = 'Year={$(=Year(Now())-1)}';
• No difference. As a general rule

So we conclude as a general ruleIconThe only way to define those formulas in a way to survive the script run and to make it into the frontend use, you have to either use my template or separate the dollar from the open brackets. Avoid the two characters unless you want the script immediately to inject something at this point. Try this:

• LET set_LastYear = 'Year={$' & '(=Year(Now())-1)}';  // the string is torn into two substrings and concatenated with &
• LET set_LastYear = Replace('Year={§(=Year(Now())-1)}', '§', '$');  // used § instead of $ in the formula and later replaced it

In both ways, the script doesn't "see" the dollar in combination with open-brackets. It will do nothing here but to save the result of the string operation (concatenation or Replace) into the variable. Then you will get what you wanted and in the GUI you can check, that the variable set_LastYear finally contains 'Year={$(=Year(Now())-1)}'You don't want to expose this to your customer in a SiB, do you? So use my simple template instead ...

Known limitations

• Defining the KPI formulas only prepares variables for later use. It does NOT create Measures in the Qlik Sense Master Items
• Measures which are removed in the definitions won't be deleted from the apps which were using it. Be careful with name-giving of those KPI variables, as they will remain in the apps. If you change the name the old ones will still be there together with the new.

Hi

We currently have Qlikview May 2021 (SR2) and QDF 1.7.1. Everything is working correctly.

We are now planning to upgrade to latest version of Qlikview (May 2022 SR1) and im wondering if we would also need to upgrade QDF?

Regards Mats

Hi,

We had received an input from CIS team in our organization to add cross site scripting vulnerability in Qliksense.

We also need to implement Content security policy?

Please find below snapshot of virtual proxies Additional response header:

Please let me know if i need to add any other parameters?

How do i implement content security policy?

Regards,

Rohit Gharat

Hi,

Anyone with experience of the migration of the Repository DB from self-hosting to GCP Postgres?

Any info or guidelines are appreciated,

Thanks in advance,

br

Paul

Hello
we've been using QDF for long now
but still couldn't resolve the issue of debugging a script in the QVF file
if I try to debug a script then the debugger would delve into all script files used by QDF and this takes ages
any idea of how be able to debug an app without having the debugger cursor go to all script files 

We have sporadically the effect, that the user can enter the hub and sees all the correct apps, but whenever he tries to open an app the Qlik loading bubbles appear and stay no matter how long he waits.
To make things worse, this bug can't be detected in the QMC, so we only notice it's occurrence when the users complain that Qlik isn't working.

This has happend on our Single-Server System as well as on on our 4-Server System with one central, two Reload and one Display Node.
The only way to remove the Bug ist to reboot the Display-Node-Server or on the Single-Server-System the whole system.

We are currently trying to find a html-query to detect when the error occurs to initiate an automatic Server reboot. While this would help us reduce the downtime for the Users, it still doesn't solve the actual Problem.

Any tips or ideas would be highly appreciated.

Hi All,

I have a general deployment question for the experts. We are currently looking into a multi-cloud setup for a customer how wants to use the Qlik SAAS environment for it's data viz app build capabilities as also for the data application to be used by viewers/consumers.

He would like to keep his data flow on his QSEoW environment (data loading from diff. systems and storing into QVD's) and store his datamarts to the cloud. Next step, he would setup a binary load from an app that he stores in a shared space on the Qlik SAAS environment and make the necessary front-end changes in Qlik SAAS. He would than publish the app to a managed space so that the end-users can consume the changes.

We investigated the usage of Qlik Data Transfer but this is kinda instable in my opinion and when it's stuck, you don't see it in any dataflow or task process on the QSEoW QMC so for administrator purposes, this is a no-go for a production deployment framework (in this case)

We went for the approach of using Qlik CLI for SAAS in combination with Qlik CLI for windows and implemented a trigger in the datamart scripts for triggering a powershell file to deploy an (export from QSEoW and import in Qlik SAAS) app to saas. BUT 🙂 the qlik app import functionality no longer support AUTOREPLACE mode. so everytime we now import an app, a new app-id is created. which break the binary load in the SAAS environment because this is setup to load from a specific app-id.

I'm kind off stuck in my setup on how to do this for a customer that has a multi-cloud license/setup and want the above approach.

How would you guys approach this request/deployment setup.

Thanks in advance,
Timmy

Hi,

We were using custom folder names in Qlikview for QVDs(Raw,Trandformed,Custom) and we want to move to QDF deploy tool for Qlik Sense, but it seems that whenever you deploy a new container, the tool copies the init.qvs to all container with a version most probably embedded in the exe. I tried changing the inline in the init.qvs from the template container, but the tool is copying another version.

Any ideas if this copy can be skipped or changed to copy from the template container?

Thanks,

Bogdan