Calculation error

[mikel_de]

Hello!

I am calculating the price per item by diving Value by Quantity. I decided to do this in the script and created a calculated column Price using the same expression. As you can see in the table below, the result is the same, but strangely enough, if I subtract  the new price from the old price, the result is not always 0.

This happens for only 2 out of 60 months in the analysis. What could be the reason?

[sunny_talwar · Accepted solution]

Seems like rounding error to me and nothing to be concerned about as the difference is almost negligible

Rounding Errors

[JustinDallas]

As Sunny said, this is a rounding error and is common in most computer languages that don't have a dedicated library to head off such things like Java's Big number classes.

I would just have an epsilon of some sort in my calculation

Script:

LET(?) vZeroEpsilon  = 0.000000001;

and then in my UI:

If( abs(Column(1) - Column(2) ) < '${vZeroEpsilon}', 0 , Column(1) - Column(2) )
;

This should go far in cutting down the nonsense numbers that are less than the circumference of an atom.

[mikel_de]

All clear! Thank you both!