Page History

Legacy software frequently assumes that every character in a string occupies 8 bits (a Java byte). The Java language assumes that every character in a string occupies 16 bits (a Java char). Unfortunately, neither the Java byte nor Java char data types can represent all possible Unicode characters. Many strings are stored or communicated using an encoding such as UTF-8 that allows characters to have with varying sizes.

While Java strings are stored as an array of characters and can be represented as an array of bytes, a single character in the string might be represented by two or more consecutive elements of type byte or of type char. Splitting a char or byte array risks splitting a multibyte character.

Ignoring the possibility of supplementary characters, multibyte characters, or combining characters (characters that modify other characters) may allow an attacker to bypass input validation checks. Consequently, programs characters must not split apart a series of bytes that represent a single characterbe split between two data structures.

Multibyte Characters

Multibyte encodings such as UTF-8 are used for character sets that require more than one byte to uniquely identify each constituent character. For example, Shift-JIS (shown below), one of the Japanese encodings, supports multibyte encoding wherein the maximum character length is two bytes (one leading and one trailing byte).

...

The trailing byte ranges overlap the range of both the single byte and lead byte characters. When a multibyte character is separated across a buffer boundary, it can be interpreted differently than if it were not separated across the buffer boundary; this difference arises because of the ambiguity of its composing bytes \[[Phillips 2005|AA. Bibliography#Phillips 05]\]. 

Supplementary Characters

According to the Java API \[[API 2006|AA. Bibliography#API 06]\] class {{Character}} documentation (Unicode Character Representations):

The char data type (and consequently the value that a Character object encapsulates) are sic based on the original Unicode specification, which defined characters as fixed-width 16-bit entities. The Unicode standard has since been changed to allow for characters whose representation requires more than 16 bits. The range of legal code points is now \u0000 to \u10FFFF, known as Unicode scalar value.

The Java 2 platform uses the UTF-16 representation in char arrays and in the String and StringBuffer classes. In this representation, supplementary characters are represented as a pair of char values, the first from the high-surrogates range, (\uD800-\uDBFF), the second from the low-surrogates range (\uDC00-\uDFFF).

An int value represents all Unicode code points, including supplementary code points. The lower (least significant) 21 bits of int are used to represent Unicode code points, and the upper (most significant) 11 bits must be zero. Unless otherwise specified, the behavior with respect to supplementary characters and surrogate char values is as follows:

• The methods that only accept a char value cannot support supplementary characters. They treat char values from the surrogate ranges as undefined characters. For example, Character.isLetter('\uD840') returns false, even though this specific value if followed by any low-surrogate value in a string would represent a letter.
• The methods that accept an int value support all Unicode characters, including supplementary characters. For example, Character.isLetter(0x2F81A) returns true because the code point value represents a letter (a CJK ideograph).

Noncompliant Code Example (Read)

This noncompliant code example tries to read up to 1024 bytes from a socket and build a String from themthis data. It does this by reading the bytes in a while loop, as recommended by rule FIO10-J. Ensure the array is filled when using read() to fill an array. If it ever detects that the socket has more than 1024 bytes available, it throws an exception. This prevents untrusted input from potentially exhausting the program's memory.

public final int MAX_SIZE = 1024;

public String readBytes(Socket socket) throws IOException {
  InputStream in = socket.getInputStream();
  byte[] data = new byte[MAX_SIZE+1];
  int offset = 0;
  int bytesRead = 0;
  String str = new String();
  while ((bytesRead = in.read(data, offset, data.length - offset)) != -1) {
    offset += bytesRead;
    str += new String(data, offset, data.length - offset, "UTF-8");
    if (offset >= data.length) {
      throw new IOException("Too much input");
    }
  }
  in.close();
  return str;
}

This code fails to consider account for the interaction between characters represented with a multibyte encoding and the boundaries between the loop iterations. If the last byte read from the data stream in one read() operation is the leading byte of a multibyte character, the trailing bytes are not encountered until the next iteration of the while loop. However, multibyte encoding is resolved during construction of the new String within the loop. Consequently, the multibyte encoding is interpreted incorrectly.

Compliant Solution (Read)

This compliant solution does not actually create defers creation of the string until all the data is available.

...

This code avoids splitting multibyte multi-byte encoded characters across buffers by deferring construction of the result string until the data has been read in full.

Compliant Solution (Reader)

This compliant solution uses a Reader rather than an InputStream. The Reader class converts bytes into characters on the fly, so it avoids the hazard of splitting multibyte characters. This routine will abort aborts if the socket provides more than 1024 characters rather than 1024 bytes.

public final int MAX_SIZE = 1024;

public String readBytes(Socket socket) throws IOException {
  InputStream in = socket.getInputStream();
  Reader r = new InputStreamReader(in, "UTF-8");
  char[] data = new char[MAX_SIZE+1];
  int offset = 0;
  int charsRead = 0;
  String str = new String(data);
  while ((charsRead = r.read(data, offset, data.length - offset)) != -1) {
    offset += charsRead;
    str += new String(data, offset, data.length - offset);
    if (offset >= data.length) {
      throw new IOException("Too much input");
    }
  }
  in.close();
  return str;
}

Noncompliant Code Example (Substring)

This noncompliant code example attempts to trim leading letters from the {{string}}. It fails to accomplish this task because {{Character.isLetter()}} lacks support for supplementary and combining characters \[[Hornig 2007|AA. Bibliography#Hornig 07]\].

// Fails for supplementary or combining characters
public static String trim_bad1(String string) {
  char ch;
  int i;
  for (i = 0; i < string.length(); i += 1) {
    ch = string.charAt(i);
    if (!Character.isLetter(ch)) {
      break;
    }
  }
  return string.substring(i);
}

Noncompliant Code Example (Substring)

This noncompliant code example attempts to fixcorrect the problem by using the {{String.codePointAt()}} method, which accepts an {{int}} argument. This works for supplementary characters but fails for combining characters \[[Hornig 2007|AA. Bibliography#Hornig 07]\].

// Fails for combining characters
public static String trim_bad2(String string) {
  int ch;
  int i;
  for (i = 0; i < string.length(); i += Character.charCount(ch)) {
    ch = string.codePointAt(i);
    if (!Character.isLetter(ch)) {
      break;
    }
  }
  
  return string.substring(i);
}

Compliant Solution (Substring)

This compliant solution works both for supplementary and for combining characters \[[Hornig 2007|AA. Bibliography#Hornig 07]\]. According to the Java API \[[API 2006|AA. Bibliography#API 06]\] class {{java.text.BreakIterator}} documentation:

...

To perform locale-sensitive String comparisons for searching and sorting, use the java.text.Collator class.

Risk Assessment

Failure to correctly account for supplementary and combining characters can lead to unexpected behavior.

Bibliography

...