Thursday, May 15, 2014

BigDecimal and equals(); the hidden trap

On my last project we were using BigDecimal for calculations (you don't still use double, or god forbid float, do you?). Like a good programmer I created a helper function for doing the divide to do the checks for things like the numerator or denominator being null and for the denominator equal to 0.
You can imagine my surprise when we started getting divide by 0 errors in the app. I checked my unit tests (you do have unit tests for all your code, right?) and sure enough there was a test for divide by 0 so what gives?
    public static BigDecimal divide(BigDecimal numerator, BigDecimal denominator) {
        if (numerator == null || denominator == null || denominator.equals(BigDecimal.ZERO)) {
            return BigDecimal.ZERO;
        }
        return numerator.divide(denominator, RoundingMode.HALF_UP);
    }

Well, not only does BigDecimal keep more accurate calcs when you do things like divide two floating point numbers but it keeps track of something called precision. Now, for the non-technical out there, precision is used to determine how accurate a number is; let's look at pi to determine what we're talking about.
Since pi is infinite you need to cut it off somewhere for your calculation. For something that doesn't need to be accurate to a small degree (such as the area of a crop circle down to the square foot) you could just use 3 whereas the area of a machined screw may need to go all the way to 3.1415. In these cases the precision of pi for the crop area is 1 and the precision for the machine screw is 5. I won't get into the full definition of precision but as a rule the number of digits shown past the decimal point determine what the precision is for that number. So, if you have something like 5.25000 then the precision is 6 (5.25 makes 3 and the 3 0's make it 6).
Now, back to my issue above; I determined that I was checking the constant BigDecimal.ZERO (with a precision of 1) to an input value of 0.00 (a precision of 3) and was getting back a value of false for equals() (0 != 0.00). Problem.
Technically, these numbers aren't equal but logically I don't care about the extra precision stored in the second BigDecimal. So, whats a programmer to do? Well, if I were to compare these two numbers the extra precision isn't taken into account (0 is still 0 no matter how accurate you measure it) so if I were to change my equals to check to a compare to:

    public static BigDecimal divide(BigDecimal numerator, BigDecimal denominator) {
        if (numerator == null || denominator == null || denominator.compareTo(BigDecimal.ZERO) == 0) {
            return BigDecimal.ZERO;
        }
        return numerator.divide(denominator, RoundingMode.HALF_UP);
    }

you can now check two numbers for logical equality and my issue of divide by 0 goes away. Now, if you're doing highly scientific calcs this won't work and equals is the better choice but if you're just checking to see if two numbers are the same (especially if you're comparing them to some kind of constant) then precision is more than likely going to get into the way so you compareTo() == 0 instead.

No comments:

Post a Comment