Prerequisites

Table of Contents

1. File Descriptors
2. dup
3. dup2
4. Saving and Restoring File Descriptors
5. Executing Commands
6. fork
7. exec
8. Pipe
9. Using Pipes Across Processes

File Descriptors

Processes need to deal with files, but cannot do it directly. Writing to files requires access to an actual storage medium, such as a hard disk: you would not like any process to be able to wipe your disk, would you? Because of this reason and many others, operating systems manage files for processes.

When a process asks to open a file, the operating system returns a special identifier called a file descriptor.

On the operating system side, each process has an array of open files, called the file descriptor table. A file descriptor, often abbreviated to fd, is the index of a file in this table.

When a new file is opened, the first unused cell of the array is filled by a pointer to the open file. The operating system returns the file descriptor of this cell to the process, which then uses it to interact with the file.

The process can then read(2) and write(2) to the file using the file descriptor. Once the process stops working with the file, it can call close(2) to release it.

tip

stdin, stdout and stderr have fixed file descriptor numbers:

• stdin is 0
• stdout is 1
• stderr is 2

An Animated Example

A process was just started in a terminal. It reads input from stdin (file descriptor 0), writes to stdout (file descriptor 1) and in case of errors, to stderr (file descriptor 2).

As you can see, the operating system used the first available file number.

dup

The dup(2) function takes an open file descriptor as argument, and returns a new file descriptor referring to the same file (hence its name: it duplicates the given file descriptor).

This means that any operation performed on the newly created file descriptor will actually be done on the file descriptor referred to by the file descriptor which was given to dup(2) in the first place.

Example:

#include <unistd.h>
#include <string.h>
#include <stdio.h>

int main(void)
{
    const char *first_msg = "I will be printed on stdout!\n";
    size_t first_msg_size = strlen(first_msg);
    write(STDOUT_FILENO, first_msg, first_msg_size);

    int stdout_dup = dup(STDOUT_FILENO);
    printf("created a copy of stdout: %d\n", stdout_dup);

    const char *second_msg = "I also will be printed on stdout!\n";
    size_t second_msg_size = strlen(second_msg);
    write(stdout_dup, second_msg, second_msg_size);

    close(stdout_dup);

    return 0;
}

This program uses dup(2) to duplicate the standard output's file descriptor, writes a message to the standard output's file descriptor, and then writes another message to the duplicated file descriptor. Running this code writes both messages to the terminal's standard output.

dup(2) always uses the lowest available file descriptor number. For example, if all your file descriptors below 10 point to something and you close(2) two of them, the one with the lowest index will be reused.

danger

Do not forget to close(2) the file descriptor created with dup(2) after you are done using it.

dup2

Duplicating file descriptors is nice, but what we actually want to perform is a redirection. How do we make this happen?

The answer is located in the same manual page as dup(2): the dup2(2) function.

dup2(2) takes two arguments: an "old" file descriptor and a "new" file descriptor. It makes a copy of the "old" file descriptor, but instead of using the lowest available number, it uses the file descriptor number in "new".

You can think of it as making a copy of "old" and storing it into "new".

If "new" already points to something, the previous file descriptor is automatically closed.

For instance:

#include <unistd.h>
#include <stdio.h>
#include <fcntl.h>
#include <err.h>

int main(void)
{
    int file_fd = open("/tmp/dup2_test", O_CREAT | O_TRUNC | O_WRONLY, 0755);
    if (file_fd == -1)
        err(1, "open failed");

    if (dup2(file_fd, STDOUT_FILENO) == -1)
        err(1, "dup2 failed");

    puts("This will be written to /tmp/dup2_test");
    return 0;
}

note

puts(3) and printf(3) internally use a write(2) operation on STDOUT_FILENO. This is why the message printed by puts is redirected to the test file.

Saving and Restoring File Descriptors

Consider the example of the previous section. Can we print a message to stdout after performing the redirection to the file? No, we cannot: dup2(2) closed STDOUT_FILENO by copying file_fd into it.

Well, we can do something about it if we save stdout before performing the redirection. This way, we can reverse the redirection once it is not useful anymore.

We can save the file descriptor using dup(2), and restore it using dup2(2).

#include <unistd.h>
#include <stdio.h>
#include <fcntl.h>
#include <err.h>

int main(void)
{
    int stdout_dup = dup(STDOUT_FILENO);
    if (stdout_dup == -1)
        err(1, "dup failed");

    int file_fd = open("/tmp/dup2_test", O_CREAT | O_TRUNC | O_WRONLY, 0755);
    if (file_fd == -1)
        err(1, "open failed");

    if (dup2(file_fd, STDOUT_FILENO) == -1)
        err(1, "dup2 failed");

    puts("This will be written to /tmp/dup2_test");

    // Ensure the message is printed before dup2 is performed
    fflush(stdout);

    if (dup2(stdout_dup, STDOUT_FILENO) == -1)
        err(1, "dup2 failed");

    close(stdout_dup);

    puts("This will be printed on standard output!");

    return 0;
}

Executing Commands

To execute a command, you must:

• call fork(2) to create a child process. This creates a copy of the current process.
• from the child process, call a function from the exec(3) family to start the given command inside the process.

From the parent process, you always have to wait for the child process to terminate using waitpid(2).

When forking and using an exec function, file descriptors are not reset: the child process inherits its parent's file descriptors.

fork

The fork(2) syscall duplicates the current process to make an almost identical copy. Everything within the process is duplicated, including file descriptors.

Before fork(2)	After fork(2)

exec

The exec(3) family of functions all execute a program ¹, with a twist: instead of creating a new process, the process which calls exec(3) is re-used, and the current program is replaced.

File descriptors remain open across an exec(3) call ².

Consider the program myprogram.c:

#include <stdio.h>
#include <sys/wait.h>
#include <unistd.h>
#include <err.h>

int main(int argc, char *argv[])
{
    pid_t pid = fork();
    if (pid == 0)
    {
        printf("[PID %d] %s \tI am a new process\n", getpid(), argv[0]);
        char *argv[] = { "mychild", "Hello Child!\n", NULL };
        execve("mychild", argv, NULL);
        err(1, "failed to exec mychild");
    }
    else
    {
        printf("[PID %d] %s \t"
               "I am the parent process, waiting for its child to exit\n",
               getpid(), argv[0]);
        waitpid(pid, NULL, 0);
    }

    return 0;
}

As well as mychild.c:

#include <stdio.h>
#include <unistd.h>

int main(int argc, char **argv)
{
    printf("[PID %d] %s \tI am the new process, but with a different program\n",
           getpid(), argv[0]);
    return 0;
}

Before fork(2)	After fork(2)

After running:

execve("sh", [ "sh", "-c", "echo", "Hello World!", NULL ], NULL);

Let us compile both programs and execute myprogram:

42sh$ make myprogram mychild
42sh$ ./myprogram
[PID 673713] ./myprogram    I am the parent process, waiting for its child to exit
[PID 673714] ./myprogram    I am a new process
[PID 673714] mychild    I am the new process, but with a different program

Pipe

Pipes are a data queue provided by the operating system: you can write data on one end, and read the same data on the other end.

Pipes only work one way: one side can only be written to, and the other side can only be read from.

The pipe(2) syscall takes an array of two integers. Calling pipe(pipefds) fills pipefds with two file descriptors:

• pipefds[0] is the read end (the side processes read from).
• pipefds[1] is the write end (the side processes write to).

Pipes are mainly used to exchange data between processes: the read end of the pipe is in a process which consumes data, and the write end of the pipe is in a process which writes data.

danger

When two processes exchange data this way, the process which reads data has an easy way of telling whether the process which writes data has completed its task, and will no longer write to the pipe.

When all the file descriptors to the write side are closed, reading on the other side returns an EOF.

It only works if all the file descriptors to the write side are closed.

#include <unistd.h>
#include <stdio.h>
#include <fcntl.h>
#include <err.h>
#include <assert.h>

int main(void)
{
    // this program has no error checks
    // your code must check for errors
    int pipefds[2];
    pipe(pipefds);

    write(pipefds[1], "Hello", 5);
    write(pipefds[1], "World", 5);
    write(pipefds[1], "!", 1);

    char buffer[10];
    read(pipefds[0], buffer, 10);
    printf("%.10s\n", buffer);
    read(pipefds[0], buffer, 1);
    printf("%.1s\n", buffer);

    close(pipefds[0]);
    close(pipefds[1]);
    return 0;
}

Pipes have a limited size: if you try to write some data into a pipe and it happens to be full, your process will be blocked until something reads on the other side of the pipe.

If there is no process reading data on the other end, your program will get stuck.

This could happen in the above program: if the operating system's pipes were not large enough to hold a full "HelloWorld!", the process would get stuck trying to write it in the pipe.

danger

Never write data in a pipe unless you are sure another process will read it immediately.

Using Pipes Across Processes

When a process calls fork(2), the child process can still use the exact same file descriptors as the parent.

For that reason, if a pipe is created in a process which then calls fork(2) to create a new process, the new process will still be able to read from and write to the pipe.

danger

The read-end of the pipe will receive an EOF when the write-end is closed and if there is no data left to read. However, in order for a file descriptor to be closed, every process has to close it. Otherwise, the reader will never receive the EOF.

    Writer  : fds[1] -------+----> fds[0]
                        \ ↗
                         X
                        / ↘
    Receiver: fds[1] -------+----> fds[0]

In the example above, if the writer closes fds[1], the reader will never get an EOF since it is opened on its side. This is why unused pipe ends must be closed.

    Writer  : [opened fds[1]] ----      [closed fds[0]]
                                  \
    Receiver: [closed fds[1]]      ---->[opened fds[0]]

Now, when the writer is done writing on the pipe, it can close it and the receiver will be notified with an EOF since no other process has this file descriptor opened.

Below is a code example of what has been seen above:

#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <err.h>
#include <sys/wait.h>
#include <sys/types.h>

int main(int argc, char *argv[])
{
    if (argc != 2)
        errx(1, "Usage: %s MESSAGE", argv[0]);

    int fds[2];

    if (pipe(fds) == -1)
        errx(1, "Failed to create pipe file descriptors.");

    pid_t pid = fork();
    if (pid < 0)
        errx(1, "Failed to fork.");

    // if we are inside the child process,
    // read the data from end pipe and write it to stdout
    if (pid == 0)
    {
        // we NEED to close the write side: otherwise, we cannot
        // know when the parent is done writing
        close(fds[1]);

        char buf[10];
        size_t nb_read = 0;
        while ((nb_read = read(fds[0], buf, 10)))
            fwrite(buf, 1, nb_read, stdout);

        close(fds[0]);
        return 0;
    }
    // in the parent, write the data to the pipe
    else
    {
        close(fds[0]);

        size_t data_len = strlen(argv[1]) + 1;
        size_t nb_written = 0;
        while (nb_written < data_len)
            nb_written += write(fds[1],
                                argv[1] + nb_written,
                                data_len - nb_written);
        close(fds[1]);
        int status;
        waitpid(pid, &status, 0);
        return WEXITSTATUS(status);
    }
}

42sh$ make test
gcc test2.c -o test
42sh$ ./test "This is a test."
This is a test.

danger

In order to avoid leaks, always close the file descriptors in both processes.

For further information, refer to the pipe(2) man page.

---

1. Some variants of exec(3) look for the program in the folders listed with the PATH variable, and some take variadic arguments instead of an array. Check out man 3 exec.↩
2. You can explicitly tell the OS to close some file descriptors when exec is called using fcntl(fd, F_SETFD, FD_CLOEXEC).↩