Using windows calculator, 2957.45 / 9 = 328.60555555555555555555555555556. In my program I got 328.61 not the expected result of 328.605555...
What am I missing here? Using xHarbour+BCC55
TIA
p.s. Just tested in xbscript.exe by simple means of "? 2957.45/9". Got 328.61 here too
FiveTech Software tech support forums
www.FiveTechSoft.com
Some confusion in math result
3 posts • Page 1 of 1
Some confusion in math result
by hua » Thu Jan 07, 2010 3:25 am
Last edited by hua on Thu Jan 07, 2010 3:44 am, edited 1 time in total.
FWH 11.08/FWH 19.12
BCC5.82/BCC7.3
xHarbour/Harbour
BCC5.82/BCC7.3
xHarbour/Harbour
- hua
- Posts: 1072
- Joined: Fri Oct 28, 2005 2:27 am
Re: Some confusion in math result
by nageswaragunupudi » Thu Jan 07, 2010 3:41 am
Internal value has full accuracy. What is displayed may be rounded off to 2 decimal places.
msginfo( 2957.45 / 9 ) shows 328.605556
We can use the result for further processing without losing accuracy.
msginfo( 2957.45 / 9 ) shows 328.605556
We can use the result for further processing without losing accuracy.
Regards
G. N. Rao.
Hyderabad, India
G. N. Rao.
Hyderabad, India
-
nageswaragunupudi - Posts: 10626
- Joined: Sun Nov 19, 2006 5:22 am
- Location: India
Re: Some confusion in math result
by hua » Thu Jan 07, 2010 3:53 am
Thanks for the reply Rao.
I think age is catching up to me
Resolved the issue easily by just temporarily increasing SET DECIMALS value
I think age is catching up to me
Resolved the issue easily by just temporarily increasing SET DECIMALS value
FWH 11.08/FWH 19.12
BCC5.82/BCC7.3
xHarbour/Harbour
BCC5.82/BCC7.3
xHarbour/Harbour
- hua
- Posts: 1072
- Joined: Fri Oct 28, 2005 2:27 am
3 posts • Page 1 of 1
Return to FiveWin for Harbour/xHarbour
Who is online
Users browsing this forum: No registered users and 85 guests