Double vs FLOAT - unclear MathFloor error - page 7

New comment
 
Maxim Kuznetsov:
Depending on your goals, sometimes it's worth going to whole numbers altogether. It saves a lot of nerves :-) Once I had a task, where I had to precisely mark up many-many levels, at first I struggled with double, and then I converted everything to integer points from 0 and everything became easy, simple and error-free.
That's exactly what I did. I converted it to integers.
 
Vladimir:

You don't need to use rounding unless the task requires it.


That's the thing: tasks are different :-)

 

If you useMathFloor or MathCeil for rounding, you need to normalise the original result to the correct accuracy.

Because during calculation, instead of a whole number, you may get a fractional number, and because of this "tail" of the number is rounded incorrectly, the result is incorrect.
For example:

double point=0.01;
Print (DoubleToStr(1156.12/point, 15));
Print (NormalizeDouble(1156.12/point, _Digits));

now let's try rounding withMathFloor

double point=0.01;
Print (DoubleToStr(MathFloor(1156.12/point), 15));
Print (MathFloor(NormalizeDouble(1156.12/point, _Digits)));
 

If I normalise the number to 1.05 - it will normalise to 1.1, which in principle isnot what I need usingMathFloor :-)

For I would not useMathFloor at all.

But the problem is different.

 

Но задача то другая.  

Why is it different? You said you needed to round down - MathFloor does exactly that

 
Taras Slobodyanik:

Why is it different? You said you needed to round down - that's what MathFloor does.

yes!!! but if i do a normalisation it'll round down the maths andMathFloor won't make sense

 

the above example with the number 1156.12, this number is stored in memory as 1156.11999999999999891

i.e., if you try to round it down to a hundredth, you get 1156.11 instead of 1156.12, which is the "peculiarity" that caused you to rack your brains.

 
Taras Slobodyanik:

the above example with the number 1156.12, this number is stored in memory as 1156.11999999999999891

i.e., if you try to round it down to a hundredth, you will get 1156.11 instead of 1156.12 - this is the "peculiarity" of your problem.

Yes :-) Thanks.

1234567
New comment