Qlikview 11.20 SR19 CRITICAL TROUBLE

[barakovskikh]

Hi! Beginning with QV 11.20 SR 16 and upper application confuses fields and data during calculations.

1. Open the application

2. Reload document

3. = Bug (See 1 screnshot in attachment) <- Field 1 = Field 2 & '|' & Field 3

4. Do not close document and reload again (See 2 screnshot in attachment) = all good!

Bug actual for desktop (QV) and server (QVB) load

Spoiler

tmp:
NoConcatenate
LOAD * Inline [
x,y,z
];

Concatenate(tmp)
LOAD
RowNo() as x
,RowNo()*-1 as y
,RowNo() as z
AutoGenerate(100000);

DATA:
NoConcatenate
LOAD
x & '|' & y as KEY,
x,
y,
fabs(z) as z
Resident tmp;

drop Table tmp;

First load after open document

Second load after open document

[rwunderlich]

Marcus,

When I reproduced the erroneous data was different each time. 

-Rob

[marcus_sommer]

Hi Rob,

I must admit that I have no real explanation for the seemingly rather randomly erroneous data - my thoughts behind this check was to find any pattern which leads nearer to the cause.

But nevertheless I have a guess which is related to the above mentioned bugfix in SR16 and that the cause is some kind of a timing-issue in the communication between the CPU + RAM + OS and QlikView in the direction how the OS the required RAM assigned to an application. By the application-start QlikView has just the RAM occupied which it needs as application and this opened document. Now the load is executed and QlikView required more RAM from the OS and gradually the RAM consumption increased - and somewhere here in the communication between the CPU + RAM + OS and QlikView is any conflict.

Before SR16 this conflict resulted in an error which hasn't an error-routine respectively they wasn't smart enough to handle it - maybe some kind of 'undefined' values - and QlikView froze, probably just waiting that anything would happens … The bugfix in SR16 changed this behavior maybe just by ignoring this error or implementing any kind of timeout and then skipping this part or something similar - and now there come more values as could be written in the RAM or some are overwritten from new ones and/or the ordering of the records becomes incorrect …

Here some further thoughts about the possible trigger of the issue in QlikView. I could imagine that's related to the fact that all fields in the raw-data have consecutive values like an auto-number-field which won't be stored like a normal field else the field-value and the pointer are the same - and if we look into the mem-file of it we see that the fields x and z have no own size else that they are just referenced to y.

I think it could be easily checked if this "special" constructed table is the real issue-trigger, for example with a where-clause like: where mod(rowno(), 10) > 0; or maybe by adding any string-values with a concatenate or any similar stuff to break the consecutive values.

Beside of the now - intentionally - created wrong values (which could be corrected in a later step) are there further erroneous data?

- Marcus

[barakovskikh]

Hi! I remind you, if you remove fabs error will not.

The error also occurs after loading from a file, not just autogenerate

Spoiler

SET ThousandSep=' ';
SET DecimalSep=',';
SET MoneyThousandSep=' ';
SET MoneyDecimalSep=',';
SET MoneyFormat='# ##0,00₽;-# ##0,00₽';
SET TimeFormat='h:mm:ss';
SET DateFormat='DD.MM.YYYY';
SET TimestampFormat='DD.MM.YYYY h:mm:ss[.fff]';
SET MonthNames='янв;фев;мар;апр;май;июн;июл;авг;сен;окт;ноя;дек';
SET DayNames='Пн;Вт;Ср;Чт;Пт;Сб;Вс';

DATA:
NoConcatenate
LOAD
x & '|' & y as KEY,
x,
y,
fabs(keepchar(z,'1234567890')) as z
FROM tmp.xlsx
(ooxml, embedded labels, table is tmp.xls);


but if we use this construction we have not error

DATA:
NoConcatenate
LOAD
x & '|' & y as KEY,
x,
y,
if(z<0,z * -1,z) as z
FROM tmp.xlsx
(ooxml, embedded labels, table is tmp.xls);

P.S

Spoiler

SIGN

[marcus_sommer]

Yes, the results are strange and I think my deducing of the possible trigger from the issue that the fieldvalue = pointer is wrong.

By comparing the input- and the output-data I noticed that all the erroneous data are within the first 10% of the records and many of the wrong values are just wrong because of an offset from one/two records. This means the error-rate of really wrong set values / missing values / overwritten values is not quite so high as the total amount of errors look like - nevertheless each single error is one too much and the result is not reliable.

The missing/overwritten values seems to fit within my above mentioned guess about a timing-issue in the process-communication but the really wrong values don't and indicate that there is another issue which results in real byte-errors. I assume that I'm now quite at the end what I could check and deduce with QlikView … but with all these hints someone by Qlik should be able to shed some more light into the matter.

- Marcus

[Brett_Bleess]

Alexandr, I noticed something this morning, can you uncheck the following settings in the Script Editor\Settings tab:

Open Databases in Read and Write mode

Can Execute External Programs

Those should be checked currently I believe, uncheck those and save things and rerun and see if you can still get the issue to occur or not at that point.  Let me keep messing with things as well, but that is one thing I noticed that is not normal and given the script, should not be needed...

Regards,
Brett

[Brett_Bleess]

Never mind, I was able to replicate it now even with those unchecked, the trick is whether I can get this to happen in a VM environment at this point such that the engineers can see it, will keep you posted.  I am not sure why Toni has been unable to replicate things either, but I know what I am looking for now, so should be easier I think.

Regards,
Brett

[barakovskikh]

I previously tested on VirtualBox + Windows 2012 r2 for validation. The problem exists

[Brett_Bleess]

What exact version of Window 10 are you running, I am on build 1903, but I noticed the VM Toni is running is on 1803, am just wondering if the rest of you are on 1903 as well, that may be part of the issue here...  I will see if I can update the VM Toni has to the newer release, but if you guys can confirm what you are running, that may help me feel more confident I am on the right track...

Cheers,
Brett

[barakovskikh]

Happy to answer questions for you, Windows 10 Pro (1903) Build 18362.207. Let me know if there's anything else I can do

[Brett_Bleess]

Thanks, I think that may be it, at least as far as Win 10 goes, let me see if I can get Toni's VM updated to 1903 as well and see if it will then result in the issue occurring, as it only took me about 3 tries on my laptop, and I am updated to 1903, so that is the only thing of which I can think at this point, but I will be stumped if I know why this is a problem! 🙂  Will let you know as soon as I can get things updated if I can replicate on Toni's machine or not, may take me a few hours.

Cheers,
Brett