FIO32-C. Do not perform operations on devices that are only appropriate for files

File names on various operating systems, including Windows and UNIX, may be used to access "special" files, which are actually devices. Sample reserved MS-DOS device names include AUX, CON, PRN, COM1, and LPT1. Device files on UNIX systems are used to apply access rights and to direct operations on the files to the appropriate device drivers.

Performing operations on device files that are intended for ordinary character or binary files can result in crashes and denial-of-service attacks. For example, when Windows attempts to interpret the device name as a file resource, it performs an invalid resource access that usually results in a crash [[Howard 02]] .

Device files in UNIX can be a security risk when an attacker is able to access them in an unauthorized way. For instance, if attackers can read or write to the /dev/kmem device, they may be able to alter their priority, UID, or other attributes of their process or simply crash the system. Similarly, access to disk devices, tape devices, network devices, and terminals being used by others all can lead to problems [[Garfinkel 96]].

On Linux, it is possible to lock certain applications by attempting to open devices rather than files, for example:

/dev/mouse
/dev/console
/dev/tty0
/dev/zero
etc.

A web browser that failed to check for these devices would allow an attacker to create a website with image tags such as <IMG SRC=file:///dev/mouse> that would lock the user's mouse.

Non-Compliant Code Example

In this example, the user can specify a locked device or a FIFO filename, causing the program to hang on the call to open().

if (!fgets(filename, sizeof(filename), stdin)) {
    /* handle error */
}

if (open(filename, O_WRONLY) == -1) {
    /* handle error */
}

Compliant Solution (POSIX)

POSIX defines the O_NONBLOCK flag to open() which will ensure that delayed operations on the file do not hang the program.

When opening a FIFO with O_RDONLY or O_WRONLY set: If O_NONBLOCK is set:
  An open() for reading only will return without delay. An open() for writing only will return an error if no process currently has the file open for reading.
If O_NONBLOCK is clear:
  An open() for reading only will block the calling thread until a thread opens the file for writing. An open() for writing only will block the calling thread until a thread opens the file for reading.

When opening a block special or character special file that supports non-blocking opens:
If O_NONBLOCK is set:
  The open() function will return without blocking for the device to be ready or available. Subsequent behaviour of the device is device-specific.
If O_NONBLOCK is clear:
  The open() function will block the calling thread until the device is ready or available before returning.

Otherwise, the behaviour of O_NONBLOCK is unspecified.

Once the file is open, programmers can use the POSIX fstat() function to obtain information about a named file, and the S_ISREG() macro to determine if the file is a regular file.

Since the behavior of O_NONBLOCK on subsequent calls to read() or write() is unspecified, it is advisable to disable the flag once we are sure the file in question is not a special device.

When available (Linux 2.1.26+, FreeBSD, Solaris 10, POSIX.1-2008), the O_NOFOLLOW should also be used, see [[POS01-A. Check for the existence of links]]

#ifdef O_NOFOLLOW
  #define OPEN_FLAGS O_NOFOLLOW | O_NONBLOCK
#else
  #define OPEN_FLAGS O_NONBLOCK
#endif

/* ... */

struct stat orig_st;
struct stat open_st;
int fildes;
int flags;

if (!fgets(filename, sizeof(filename), stdin)) {
  /* handle error */
}

if ((stat(file_name, &orig_st) != 0) || (!S_ISREG(orig_st.st_mode))) {
  /* handle error */
}

/* A TOCTOU exists here, see note */

if ((fildes = open(filename, OPEN_FLAGS | O_WRONLY)) == -1) {
  /* handle error */
}

if (fstat(fildes, &open_st) != 0) {
  /* handle error */
}

if ((orig_st.st_mode != open_st.st_mode) ||
    (orig_st.st_ino  != open_st.st_ino) ||
    (orig_st.st_dev  != open_st.st_dev)) {
  /* file was tampered with */
}

/* Optional: drop the O_NONBLOCK now that we are sure this is a good file */
if ((flags = fcntl(fildes, F_GETFL)) == -1) {
  /* handle error */
}

if ((fcntl(fildes, F_SETFL, flags & ~O_NONBLOCK) != 0) {
  /* handle error */
}

/* operate on file */

Regarding the inevitable TOCTOU vulnerability between the stat() check and open() call, there are essentially four cases that could result from an attacker switching out the file for one of the following:

Type	Note on effect
another regular file	The `fstat()` verification will fail
FIFO	Either `open()` will error out and set `errno` to `ENXIO` or the `open()` will succeed and the `fstat()` verification will fail
link	`open()` will error out if `O_NOFOLLOW` is available, otherwise if the link does not point to a special device the `fstat()` verification will fail
special device	Most systems require superuser privileges to tamper with devices, at which point there is little that can be done

Keep in mind that this TOCTOU is not a problem if the program can guarantee a safe environment for itself by properly managing permissions and ensuring that an attacker cannot have write or modify access to files.

Compliant Solution (Windows)

The GetFileType() function can be used to determine if the file is a disk file.

HANDLE hFile = CreateFile(
  pFullPathName, 0, 0, NULL, OPEN_EXISTING, 0, NULL
);
if (hFile == INVALID_HANDLE_VALUE) {
  /* handle error */
}
else {
  if (GetFileType(hFile) != FILE_TYPE_DISK) {
    /* handle error */
  }
  /* operate on file */
}