“[Errno 32] Broken pipe” in Python

At age 30, Python has been considered a mature language. The programming language is hugely popular among data scientist and AI engineers thanks to its simplicity and easy syntax. Despite that, its vague errors usually makes new users pull their hair out to debug.

In this article, we will discuss about [Errno 32] Broken pipe - a popular error message you often see when interacting with the file system. By the end of the article, you will understand why it happens and how to avoid it as well as how to fix your code. You might also be interested in finding out more about other common error messages of Python, such as locale.Error: unsupported locale setting or unindent does not match any outer indentation level.

What causes "[Errno 32] Broken pipe" in Python?

"Broken pipe" is essentially an IOError error (short for input/output error), which happened at the Linux system level. It usually occurs when reading and writing files, or in other words, doing file input/output or network input/output (via sockets).

The corresponding Linux system error is EPIPE, excerpted from GNU libc error codes:

Macro: int EPIPE

“Broken pipe.” There is no process reading from the other end of a pipe. Every library function that returns this error code also generates a SIGPIPE signal; this signal terminates the program if not handled or blocked. Thus, your program will never actually see EPIPE unless it has handled or blocked SIGPIPE.

From what we've just read, we know that [Errno 32] Broken pipe is caused by the system sending SIGPIPE signal, which is an inter-process communication mechanism of Linux.

For example, SIGINT is another signal used internally by Linux system. In Linux, Ctrl+C will send a SIGINT signal to end the process, or we can use the kill command to achieve the same effect.

Python does not ignore SIGPIPE by default. Instead, it translate the signal into an exception and raises IOError: [Errno 32] Broken pipe every time it receives a SIGPIPE.

[Errno 32] Broken pipe when pipe outputs in Linux terminal

If you encounter [Errno 32] Broken pipe when trying to pipe output of a Python script to another program such as the below example, read on.

python <filename>.py | head

This pipeline syntax will create a process that sends data upstream, and a process that reads data downstream. When the downstream does not need to read upstream data, it will send a SIGPIPE signal to the upstream process.

When downstream no longer needs to read upstream data? For example, the head command in the example only needs to read enough lines to tell the upstream that I no longer need to read it, and it will send the SIGPIPE signal to the upstream process.

When the upstream process is a Python program, an error such as IOError: [Errno 32] Broken pipe will occur.

Details about other cases : https://superuser.com/questions/554855/how-can-i-fix-a-broken-pipe-error

Avoid [Errno 32] Broken pipe by ignoring SIGPIPE

If you don't care too much about properly catching SIGPIPE and just need to get things running quickly, add the code snippet below to the top of your Python program.

from signal import signal, SIGPIPE, SIG_DFL 
#Ignore SIG_PIPE and don't throw exceptions on it... (http://docs.python.org/library/signal.html)
signal(SIGPIPE,SIG_DFL) 

What the code does is redirecting SIGPIPE signals to the default SIG_DFL, which the system usually ignore.

But beware, the Python manual on signal library warn against this type of handling SIGPIPE

Do not set SIGPIPE’s disposition to SIG_DFL in order to avoid BrokenPipeError. Doing that would cause your program to exit unexpectedly also whenever any socket connection is interrupted while your program is still writing to it.

Properly catch IOError to avoid [Errno 32] Broken pipe

Since [Errno 32] Broken pipe is actually a IOError, you can place a try/catch block to catch it like the code snippet below :

import sys, errno
try:
    ### IO operation ###
except IOError as e:
    if e.errno == errno.EPIPE:
        ### Handle the error ###

Possible solution for [Errno 32] Broken pipe in multi-process program.

In programs that uses worker processes to speed up processing and make use of multi-core CPUs, you can try reducing the number of the worker processes to see whether the error disappear or not.

A large number of worker processes may conflict with each other when they try to take control of system resources or the permission to write into disk.

Conclusion

Properly fixing [Errno 32] Broken pipe requires time and a close look at your code. Most of the time, you can safely ignore the problem if the Python program is relatively simple. Some other times, the solution involves in reducing worker processes in multi-process programs. But we hope that this article offer useful information and help you solve your problem.

Click to rate this post!
[Total: 1 Average: 5]

Leave a Comment