Calculation logic in text box

[sundarakumar]

Hi Experts,

I tried writing the below in a text box

=101.02-101.01

The result 0.0099999999999909.

What could be the reason for this, how is this calculation being made in the back end.

Am using SR12.

Thanks in advance.

-Sundar

[Clever_Anjos]

Use round

=round(101.02-101.01,0.01)

[maxgro]

What could be the reason for this, how is this calculation being made in the back end.

Rounding Errors

[petter · Accepted solution]

QlikView is using 8 byte IEEE 754 double precison floating point number format. It does not represent every possible value accurately. So the number you get is correctly calculated according to the number format and precision you can expect. You should use rounding for display and final reporting purposes. Have a look at Wikipedia and other sources about IEEE 754. Microsoft Excel also stick to this standard as most commercial software also does. It is a very good trade-off between speed, space and necessary precision.

[petter]

If you try the exact same calculation in Excel and increase the number of displayed decimals sufficiently you will get the same number in fact:

0,0099999999999909

[sundarakumar]

Thanks All lot more to learn

[sundarakumar]

Thanks for the reply, wanted to know what the logic was. Got it