Skip to main content
Announcements
Global Transformation Awards! Applications are now open. Submit Entry
hic
Former Employee
Former Employee

Today, the Qlik engine has some bugs in the area of a search and a subsequent select. These affect both interactive searches and searches in Set Analysis expressions. We are working on fixing them. However, one of these coming bug fixes may cause some backward incompatibility. This post explains what the bug fix will mean, and what you can do to avoid future problems.

When you use Set analysis, you can choose to have a condition in the form of a list of values in the Set expression, or you can choose to have a rule-based definition - a search for field values.

Examples of hard coded lists:

Sum( {$<Year = {2013,2014,2015}>} Sales)
Sum( {$<Country = {'Australia','Canada','France'}>} Sales)

Examples of searches:

Sum( {$<Year = {">=2013"}>} Sales)
Sum( {$<Country = {"Austr*"}>} Sales)
Sum( {$<Customer = {"=Sum(Sales)>100000"}>} Sales)

All the above constructions work today, and they will work correctly also after the bug fix.

Note the double quotes in the search expressions. These define the search strings, and between them you can write any search string – the same way as you would in a list box search.

However, you should not use single quotes to define a search string.

Single quotes are reserved for literals, i.e. exact matches. Hence they should be used when you want to make a list of valid field values, like the above list of explicit countries. But they should not be used for searches. Single quotes imply a case sensitive match with a single field value.

This difference between single and double quotes has been correctly described in the reference manual that states that searches are “always defined by the use of double quotes”. Also, all examples with literal field values in the documentation use single quotes.

Now to the bug: QlikView and Qlik Sense currently do not respect the above difference between single and double quotes. Instead, strings enclosed in single quotes are interpreted as search strings. As a consequence, it is not possible to make case sensitive matches with field values, something which you sometimes want to do.

This bug will be fixed in the coming November releases of Qlik Sense and QlikView. Then, wildcards in strings enclosed by single quotes will no longer be considered as wildcards. Instead, the strings will be interpreted as literals, i.e. the engine will try to match the string with a field value containing the ‘*’ character. The same logic will apply also to relational operators and equals signs.

Unfortunately, this bug has been incorrectly utilized by some developers: I have seen Set Analysis expressions with correct search strings, but erroneously enclosed in single quotes; for example

Sum( {$<Country = {'Austr*'}>} Sales)

This search should instead have been written

Sum( {$<Country = {"Austr*"}>} Sales)

Hence, there are documents with expressions that will not work in the corrected algorithm. However, the bug fix will be implemented in such a way that old documents will use the old logic, and new documents will use the new logic. In addition, it will be possible to force the new logic for all documents by using a parameter in Settings.ini.

You can of course also change the single quotes in existing search strings to double quotes, and the expression will continue to do what it always has done.

This post is mainly to give you a heads-up on a coming change, and give you a chance to make a quality control of your own documents. We will publish more information as we get closer to the release.

HIC

 

Further information related to Set Analysis and Searches:

The Search String

Introduction to Set Analysis (video) - Part 1

Set Analysis - Part 2 - Cool Stuff You Can Do (video)

61 Comments
hic
Former Employee
Former Employee

For those of you who want to test the above fix, I suggest you download the technical preview:

QlikView November 2017 Technical Preview

In this, you can tweak the behaviour by changing the Settings.ini file under the [Settings 7] tag. Introduce the following parameter:

   EnableSingleQuoteExactSearch=2

0 = all apps have old behavior

1 = new apps have new behavior, old apps have old behavior (default)

2 = all apps have new behavior

Further. if you have the default EnableSingleQuoteExactSearch=1, then you can force individual documents to behave as old/new documents by adding the below setting as one of the first lines in the script:

///$sqs new

///$sqs old

But I strongly suggest you don't force the old behaviour... 🙂

HIC

1,887 Views
blaise
Partner - Specialist
Partner - Specialist

well now in curious

Are there any other //$ commands/variables you can set ?

0 Likes
1,887 Views
evan_kurowski
Specialist
Specialist

Thank you Henric & Qlik, it is thrilling that this subtle wrinkle within set-analysis will be flattened, and expression syntax will truly operate under consistent quoting rules.  We understand it may have been hard to risk the ripples that might retroactively disrupt some proven apps, but going forward this strengthens the platform and elevates the 'rules of consistency'.

It's also exciting to have identification of the alternate quoting sets "", ``, [ ]  (doubles, graves, squares)

Without being sure if they were truly serious, for a while there it seemed like someone always had a request for "Hey can you now get me a set-analysis modifier that isolates 'Darryl O'Shaugnessey' for me?  We need his numbers.  Oh and don't confuse him with 'Darryl O'shaugnessey' from the Poughkeepsie office, who spells his 'shaugnessey' with a lowercase S"

or

"Hey listen, I've got this business named '* Asterisks Incorporated' and I need them plucked out, oh but please pair that with a wildcard search for anything having '*Aster*'.  That'd be great, thanks."

1,887 Views
Emmanuelle__Bustos
Partner - Specialist
Partner - Specialist

Thank you Henric

0 Likes
1,887 Views
YoussefBelloum
Champion
Champion

thank you for the explanations !

0 Likes
1,887 Views
evan_kurowski
Specialist
Specialist

 

This topic could be expanded upon a discussion about perception...

 

   

 

For example, we have this straightforward syntax for assigning the background color of an object.  The variable vColor is assigned to an RGB() value.

 

 

=IF(vBoolean,vColor)

 

 

 

And this simple expression works great, in the desktop and the desktop's web view... and even on the servers (for a while).

 

 

 

But inevitably someone will apply a patch or adjust a setting and viola, the expression can no longer interpret the vColor variable.  The expression returns null, and reworks are required.  If you're lucky, breakage waited until your application navigated a formalized & arduous path to reach a production environment.

 

 

 

The expression is then repaired with the following:

 

=IF(vBoolean,$(vColor))

 

 

But the point isn’t “how to repair”, it is the “integrity of the overall process”.  It doesn’t matter how significant the repair, so long as any correction is required.

 

 

As a long term effect of this 'wiggle' developers might adopt habits of "real world condition best practices" and begin always wrapping variables in dollar expansion (even in situations where the absence of wrapping seems to allow expressions to work fine).  When the syntax is then observed from outsiders, they may question, thinking "Why is this coder always wrapping these variables??  It's unnecessary."  Things always work great in the lab.  Make sure a corroboration of that perspective is sampled from the field.

 

This color syntax example is simple, but captures the nagging essence of a recurring issue, a kind of back-and-forth between one set of syntax and a variant, that has been a consistent observation through the entire Qlik experience.

           

 

This kind of functionality "weaving" between environmental layers has created inhibitions and reworks in almost every project I've been involved with, to the point I question whether they are entirely innocent (could they be a tactical form of technical dishonesty?). 

The developer has been lured into moving forward with
“compromised” syntax, because three layers of platform reported positive results.  But for whatever reason, a kind of "pocket card" allows the technical premise to be invalidated (and I’m sure there’s a population that would like to have a discussion with the dealer.  Either that, or they just got frustrated and found another table).

 

The contract between software and developer operates under the premise control is real, not “perception”.  Can’t imagine false-platform would ever be something touched by Qlik, it would represent an unsustainable violation toward our ecosystem.  But perhaps the possibility of interlopers is something to be taken into account when forming real-world solutions?

 

 

 

       

1,887 Views
hic
Former Employee
Former Employee

When developing software, we very often face choices between creating a "strict" syntax or a more forgiving one. We at Qlik usually choose to create a forgiving syntax, since many of our users are business users and not developers. "Forgiving" as in: The software should always make a best effort attempt to understand what the expression means.

Let's take your color expression as an example:

   =If(vBoolean,$(vColor))

Here a dollar expansion is used, which ALWAYS is expanded to a string. Ergo, a strict syntax would probably imply the use of single quotes

   =If(vBoolean,'$(vColor)')

but since this is non-intuitive for many people, we don't demand it.

A second example can be found in the above Set Analysis expressions. Both the following expressions are today valid:

   Sum( {$<Country = {Australia,Canada,France}>} Sales)

   Sum( {$<Country = {'Australia','Canada','France'}>} Sales)

But with a strict syntax, the first one wouldn't be a valid expression. And so, many users would fail to get this expression right.

I agree that there are cases where a stricter syntax would be beneficial, but it would also mean that fewer people could use the product.

HIC

1,901 Views
evan_kurowski
Specialist
Specialist

Hello Henric,

 

Thank you for taking another look at this. 

I understand and agree with the set-analysis quoting syntax adjustments described as an upcoming change in your original post. Thank you for making these changes and for shepherding the product.

 

Just to reiterate, topic has shifted somewhat and we are now describing something similar, which is dollar sign expansion and string interpretation.

 

For the color-expression, this is how I experienced the results of a variable assigned as follows:

vColor = RGB(100,255,100)

 

 

Expression used within user interface:

=IF(True(), vColor)

 

Worked consistently on

Desktop

Desktop Web View

Server

Yes

Yes

No*

 

*The expression would work at times on the server, but might break when updates were applied

 


=IF(True(), $(vColor))

 

Worked consistently on

Desktop

Desktop Web View

Server

Yes

Yes

Yes

 

In cases where the expression at first worked on the server, but then later experienced breakdown, these breakdowns seemed to coincide with patching on the server side.

When breakdown happened, our QlikView version and SR remained consistent, but without an administrator’s perspective I can’t confirm if the patches were specifically for QlikView, or applied to other areas of the OS.  My hypothesis is that server patching may have been effecting the QV implementation.

 

 

0 Likes
1,901 Views
hic
Former Employee
Former Employee

First, I am somewhat surprised that there is a difference between Desktop and Server. Secondly, I am surprised that the second expression works at all...

Further,

   =IF(True(), $(vColor))

should evaluate to

   =IF(True(), RGB(100,255,100))

which can be nicely evaluated to a color (a dual with integer value)

Whereas

   =IF(True(), '$(vColor)')

should evaluate to

   =IF(True(), 'RGB(100,255,100)')

which is a string that looks the same as above, but lacks numeric value.

HIC

0 Likes
1,901 Views
evan_kurowski
Specialist
Specialist

 

Yes Henric, this is happening. 

 

In a gaslighting effect, the rules of interpretation appear to be in motion (they may change across environments and modify through time).

I definitely
“repaired” a set of non-interpreting color expressions in the past week, by wrapping the color variables in dollar sign expansion, and doing so restored functionality. (and recall other occasions at other QlikView environments where this same technique produced similar effect).

  The question then reverts back to my original post, “perspective”.  Is my Qlik experience getting a “custom” set of interpretation rules?

0 Likes
1,901 Views