Wednesday, March 7, 2012

Decimal Places

I have 2 fields X and Y, both are type decimal(38,20). I need to have as many decimal places as possible in my calculations for these numbers. This is the first time I've had to deal with this type of "precise" data so if you see something completely wrong just say so...

Anyway, I need to divide 2 numbers to get an answer...

I'm dividing: -118.84 by 867561.993

selectcast(net_expenses/fund_shares_outstanding asdecimal(35,20)) from xyz where blah blah

My query returns: -0.00013600000000000000

This same calculation with the same numbers in Excel returns -0.00013698156553522500

What do I need to do to get the more precise answer?

Precision, scale, of output cannot exceede precision of input.

-0.00013698156553522500 is gibberish past the third decimal place unless you are working with

-118.84000000000000000000

and

867561.99300000000000000000

DECLARE @.float1float

DECLARE @.float2float

SET @.float1=-118.84

SET @.float2= 867561.993

SELECT @.float1/@.float2

No comments:

Post a Comment