NUM10-J. Beware of precision loss when converting primitive integers to floating-point

The following 19 specific conversions on primitive types are called the widening primitive conversions:

• byte to short, int, long, float, or double
• short to int, long, float, or double
• char to int, long, float, or double
• int to long, float, or double
• long to float or double
• float to double

Conversion from int or long to float, or long to double may lead to loss of precision (loss of least significant bits). In this case, the resulting floating-point value is a rounded version of the integer value, using IEEE 754 round-to-nearest mode. Despite this loss of precision, the JLS requires that the conversion and rounding occur silently; that is, without any runtime exception. See JLS, Section 5.1.2, "Widening Primitive Conversion" for more information.

Note that conversions from float to double can also lose information about the overall magnitude of the converted value. Specifically, on platforms whose native floating point hardware provides greater precision than double, the JIT is permitted to use floating point registers to hold values of type float (or type double), even though the registers support values with greater mantissa and/or exponent range. Consequently, conversion from float to double can cause an effective loss of precision, of magnitude, or of both. However, the lost precision or magnitude would also have been lost if the value were stored to memory, for example to a field of type float. See guideline FLP04-J. Use the strictfp modifier for floating point calculation consistency for additional information.

Noncompliant Code Example

In this noncompliant code example, two identical large integer literals are passed as arguments to the subFloatFromInt() method. The second argument is coerced to float, cast back to int, and subtracted from a value of type int. The result is returned as a value of type int.

This method may have unexpected results because of the loss of precision. Values of type float have 23 mantissa bits, a sign bit, and an 8 bit exponent. The exponent allows type float to represent a larger range than that of type int. However, the 23-bit mantissa means that float supports exact representation only of integers whose representation fits within 23 bits; float supports only approximate representation of integers outside that range.

class WideSample {
  public static int subFloatFromInt(int op1, float op2) {
    return op1 - (int)op2;
  }

  public static void main(String[] args) {
    int result = subFloatFromInt(1234567890, 1234567890);
    // This prints -46, and not 0 as may be expected
    System.out.println(result);  
  }

}

Note that conversions from long to either float or double can lead to similar loss of precision.

Compliant Solution (ArithmeticException)

This compliant solution range checks the argument of the integer argument (op1) to ensure it can be represented as a value of type float without a loss of precision.

class WideSample {
  public static int subFloatFromInt(int op1, float op2) throws ArithmeticException {

    // The significand can store at most 23 bits
    if ((op1 > 0x007fffff) || (op1 < -0x800000)) { 
      throw new ArithmeticException("Insufficient precision");	
    }

    return op1 - (int)op2;
  }

  public static void main(String[] args) {
    int result = subFloatFromInt(1234567890, 1234567890);
    System.out.println(result);  
  }
}

In this example, the subFloatFromInt() method throws java.lang.ArithmeticException. This approach should be used for conversions from long to either float or double.

Compliant Solution (wider type)

This compliant solution accepts an argument of type double instead of an argument of type float. Values of type double have 52 mantissa bits, a sign bit, and an 11 bit exponent. Consequently, integer values of type int and narrower can be converted to double without a loss of precision.

class WideSample {
  public static int subDoubleFromInt(int op1, double op2) {
    return op1 - (int)op2;
  }

  public static void main(String[] args) {
    int result = subDoubleFromInt(1234567890, 1234567890);
    // Works as expected
    System.out.println(result);  
  }

}

Note that this compliant solution is insufficient when the primitive integers are of type long, because Java lacks a primitive floating point type whose mantissa can represent the full range of a long.

Risk Assessment

Converting integer values to floating-point types whose mantissa has fewer bits than the original integer value will lose precision.

Automated Detection

Automatic detection of casts that can lose precision is straightforward. Sound determination of whether those casts correctly reflect the intent of the programmer is infeasible in the general case. Heuristic warnings could be useful.

Related Vulnerabilities

Search for vulnerabilities resulting from the violation of this guideline on the CERT website.

Related Guidelines

C Secure Coding Standard: FLP36-C. Beware of precision loss when converting integral types to floating point

C++ Secure Coding Standard: FLP36-CPP. Beware of precision loss when converting integral types to floating point

Bibliography

[[JLS 2005]] Section 5.1.2, "Widening Primitive Conversion"

INT02-J. Do not assume that the remainder operator always returns a non-negative result      03. Integers (INT)      INT04-J. Avoid using the char integral type to hold signed values