topic Re: Calculation logic in text box in QlikView

Calculation logic in text box

sundarakumar — Tue, 18 Aug 2015 17:15:02 GMT

Hi Experts,

I tried writing the below in a text box

=101.02-101.01

The result 0.0099999999999909.

What could be the reason for this, how is this calculation being made in the back end.

Am using SR12.

Thanks in advance.

-Sundar

Re: Calculation logic in text box

Clever_Anjos — Tue, 18 Aug 2015 17:22:59 GMT

Use round

=round(101.02-101.01,0.01)

Re: Calculation logic in text box

maxgro — Tue, 18 Aug 2015 17:24:56 GMT

What could be the reason for this, how is this calculation being made in the back end.

Rounding Errors

Re: Calculation logic in text box

petter — Tue, 18 Aug 2015 17:26:56 GMT

QlikView is using 8 byte IEEE 754 double precison floating point number format. It does not represent every possible value accurately. So the number you get is correctly calculated according to the number format and precision you can expect. You should use rounding for display and final reporting purposes. Have a look at Wikipedia and other sources about IEEE 754. Microsoft Excel also stick to this standard as most commercial software also does. It is a very good trade-off between speed, space and necessary precision.

Re: Calculation logic in text box

petter — Tue, 18 Aug 2015 17:30:04 GMT

If you try the exact same calculation in Excel and increase the number of displayed decimals sufficiently you will get the same number in fact:

0,0099999999999909

Re: Calculation logic in text box

sundarakumar — Tue, 18 Aug 2015 17:30:18 GMT

Thanks All lot more to learn

Re: Calculation logic in text box

sundarakumar — Tue, 18 Aug 2015 17:31:22 GMT

Thanks for the reply, wanted to know what the logic was. Got it