Serialization can prevent garbage collection and thus induce memory leaks. Every time an object is written out to a stream, a reference to the object is retained. If the same object (regardless of its contents) is written out to the same stream again, it is replaced with a reference to the originally cached object. The garbage collector cannot reclaim the memory associated with new objects as it cannot deal with live references.

Noncompliant Code Example

This noncompliant code example writes an array object arr to the underlying stream. The object is created afresh within the loop and filled uniformly with the value of the loop counter. Unfortunately, an OutOfMemoryError surfaces as the stream is kept open while new objects are being written to it.

class MemoryLeak {
  public static void main(String[] args) throws IOException {
    ObjectOutputStream out = new ObjectOutputStream(
        new BufferedOutputStream(new FileOutputStream("ser.dat")));
    for (int i = 0; i < 1024; i++) {
      byte[] arr = new byte[100 * 1024];
      Arrays.fill(arr, (byte) i);
      out.writeObject(arr);
    }
    out.close();
  }
}

Compliant Solution

Ideally, the stream should be closed as soon as the work is accomplished. This compliant solution adopts an alternative approach by resetting the stream after every write so that the internal cache no longer maintains live references, allowing the garbage collector to resume.

class NoMemoryLeak {
  public static void main(String[] args) throws IOException {
    ObjectOutputStream out = new ObjectOutputStream(
      new BufferedOutputStream(new FileOutputStream("ser.dat")));
    for (int i = 0; i < 1024; i++) {
      byte[] arr = new byte[100 * 1024];
      Arrays.fill(arr, (byte) i);
      out.writeObject(arr);
      out.reset(); // reset the stream
    }
    out.close();
  }
}

Risk Assessment

Memory and resource leaks during serialization can corrupt the state of the object or crash the JVM.

Automated Detection

TODO

Related Vulnerabilities

Search for vulnerabilities resulting from the violation of this rule on the CERT website.

References

\[[API 06|AA. Java References#API 06]\]
\[[Sun 06|AA. Java References#Sun 06]\] "Serialization specification"
\[[Harold 06|AA. Java References#Harold 06]\] 13.4. Performance 

SER00-J. Maintain serialization compatibility during class evolution      11. Serialization (SER)      11. Serialization (SER)