Issues with subtraction of fractions

[SureshUK]

Hi,

  

I have an issue with the floating point calculation as I suspect QV does something wired when a tried to subtract a 5 digit fraction value with another 5 digit fraction.

I tried and simplifed the problem to this:

When I create a text object with =70.1-70.0, I am getting 0.099999999999994  instead of .1. I tried this on QV 10 SR2.

I am not sure if any one faced this issue. Surely this can't be expected behaviour as a simple windows calculator gets me the correct value.

Did anyone ever faced this issue before?

Thanks,

Suresh

[swuehl]

Yes, I have.

in binary floating point representation, you can't represent all values exactely, so for some values or some results of a calculation, you will get a very small calculation error.

I believe it's just up to some "intelligent rounding", if you'll notice this error or not.

For example

=num(70.1-70.0,'#')

will show 0.1, which seems to be correct, but I am pretty sure the internal number representation will still show the small error. And I also believe Windows calculator suffers from the same problem, it might just be a little more clever showing you rounded values.

Hope this helps,

Stefan