Pipeline (Unix)

In Unix-like computer operating systems, a pipeline is a mechanism for inter-process communication using message passing.

The pipeline concept was invented by Douglas McIlroy[1] and first described in the man pages of Version 3 Unix.

[2][3] McIlroy noticed that much of the time command shells passed the output file from one program as input to another.

"The next day", McIlroy continued, "saw an unforgettable orgy of one-liners as everybody joined in the excitement of plumbing."

McIlroy also credits Thompson with the | notation, which greatly simplified the description of pipe syntax in Version 4.

In Tony Hoare's communicating sequential processes (CSP), McIlroy's pipes are further developed.

An open source third-party filter called bfr is available to provide larger buffers if required.

The shell starts the processes and arranges for the necessary connections between their standard streams (including some amount of buffer storage).

The standard shell syntax for anonymous pipes is to list multiple commands, separated by vertical bars ("pipes" in common Unix verbiage): For example, to list files in the current directory (ls), retain only the lines of ls output containing the string "key" (grep), and view the result in a scrolling page (less), a user types the following into the command line of a terminal: The command ls -l is executed as a process, the output (stdout) of which is piped to the input (stdin) of the process for grep key; and likewise for the process for less.

By default, the standard error streams ("stderr") of the processes in a pipeline are not passed on through the pipe; instead, they are merged and directed to the console.

The Bash shell can also merge standard error with |& since version 4.0[11] or using 2>&1, as well as redirect it to a different file.

To remedy this issue, the "pipemill" can instead be fed from a here document containing a command substitution, which waits for the pipeline to finish running before milling through the contents.

Named pipes may also be created using mkfifo() or mknod() and then presented as the input or output file to programs as they are invoked.