| Re .0:
� I think I've just discovered a bug in WB 2.0's eval command.
I'd say you found out something about its implementation. See below.
� I was dividing 13823 by 1024 and of course got the answer - 13, but I
� was wanting a little more accuracy, so I started adding zeros to the
� numerator in order to use fixed point math so;
�
� 1> eval 13823/1024
� 13
� 1> eval 1382300/1024
� 1349 (13.49 - ok so far)
� 1> eval 138230000/1024
� 134990 (13.4990 - fine)
� 1> eval 1382300000/1024
� 1349902 (13.49902 - great)
� 1> eval 13823000000/1024
� 916111 (Huh???)
A 32-bit signed integer has a range from -2147483648 to +2147483647.
The numerator in the final formula is beyond that range, so all bets
are off as to how it might be interpreted. It DIDN'T simply wrap
around, as one might have expected. Make a note to yourself NOT to do
integer arithmetic with numbers (or intermediate results) exceeding 10
digits in length.
|
| >� 1> eval 1382300000/1024
>� 1349902 (13.49902 - great)
>� 1> eval 13823000000/1024
>� 916111 (Huh???)
>
> A 32-bit signed integer has a range from -2147483648 to +2147483647.
> The numerator in the final formula is beyond that range, so all bets
> are off as to how it might be interpreted. It DIDN'T simply wrap
> around, as one might have expected. Make a note to yourself NOT to do
> integer arithmetic with numbers (or intermediate results) exceeding 10
> digits in length.
Simple. Take 916111 * 1024 = 938097664.
938097664 is what you get when you multiply 1382300000 * 10 and 32 bit
hardware shifts bits off the end (and the program ignores the carry)
This is typical of many decimal input routines that simply read digits
from the input stream adding them to a sum and then multiplying the result
by 10.
john
|