I have 2 fields X and Y, both are type decimal(38,20). I need to have as many decimal places as possible in my calculations for these numbers. This is the first time I've had to deal with this type of "precise" data so if you see something completely wrong just say so...
Anyway, I need to divide 2 numbers to get an answer...
I'm dividing: -118.84 by 867561.993
selectcast(net_expenses/fund_shares_outstanding asdecimal(35,20)) from xyz where blah blah
My query returns: -0.00013600000000000000
This same calculation with the same numbers in Excel returns -0.00013698156553522500
What do I need to do to get the more precise answer?
Precision, scale, of output cannot exceede precision of input.
-0.00013698156553522500 is gibberish past the third decimal place unless you are working with
-118.84000000000000000000
and
867561.99300000000000000000
DECLARE @.float1float
DECLARE @.float2float
SET @.float1=-118.84
SET @.float2= 867561.993
SELECT @.float1/@.float2
No comments:
Post a Comment