At age 30, Python has been considered a mature language. The programming language is hugely popular among data scientist and AI engineers thanks to its simplicity and easy syntax. Despite that, its vague errors usually makes new users pull their hair out to debug.
In this article, we will discuss about [Errno 32] Broken pipe - a popular error message you often see when interacting with the file system. By the end of the article, you will understand why it happens and how to avoid it as well as how to fix your code. You might also be interested in finding out more about other common error messages of Python, such as locale.Error: unsupported locale setting or unindent does not match any outer indentation level.
What causes "[Errno 32] Broken pipe" in Python?
"Broken pipe" is essentially an IOError error (short for input/output error), which happened at the Linux system level. It usually occurs when reading and writing files, or in other words, doing file input/output or network input/output (via sockets).
The corresponding Linux system error is EPIPE, excerpted from GNU libc error codes:
Macro: int EPIPE
“Broken pipe.” There is no process reading from the other end of a pipe. Every library function that returns this error code also generates a
SIGPIPEsignal; this signal terminates the program if not handled or blocked. Thus, your program will never actually see
EPIPEunless it has handled or blocked
From what we've just read, we know that [Errno 32] Broken pipe is caused by the system sending SIGPIPE signal, which is an inter-process communication mechanism of Linux.
For example, SIGINT is another signal used internally by Linux system. In Linux, Ctrl+C will send a SIGINT signal to end the process, or we can use the
kill command to achieve the same effect.
Python does not ignore SIGPIPE by default. Instead, it translate the signal into an exception and raises IOError: [Errno 32] Broken pipe every time it receives a SIGPIPE.
[Errno 32] Broken pipe when pipe outputs in Linux terminal
If you encounter [Errno 32] Broken pipe when trying to pipe output of a Python script to another program such as the below example, read on.
python <filename>.py | head
This pipeline syntax will create a process that sends data upstream, and a process that reads data downstream. When the downstream does not need to read upstream data, it will send a SIGPIPE signal to the upstream process.
When downstream no longer needs to read upstream data? For example, the
head command in the example only needs to read enough lines to tell the upstream that I no longer need to read it, and it will send the SIGPIPE signal to the upstream process.
When the upstream process is a Python program, an error such as IOError: [Errno 32] Broken pipe will occur.
Details about other cases : https://superuser.com/questions/554855/how-can-i-fix-a-broken-pipe-error
Avoid [Errno 32] Broken pipe by ignoring SIGPIPE
If you don't care too much about properly catching SIGPIPE and just need to get things running quickly, add the code snippet below to the top of your Python program.
from signal import signal, SIGPIPE, SIG_DFL #Ignore SIG_PIPE and don't throw exceptions on it... (http://docs.python.org/library/signal.html) signal(SIGPIPE,SIG_DFL)
What the code does is redirecting SIGPIPE signals to the default SIG_DFL, which the system usually ignore.
But beware, the Python manual on
signal library warn against this type of handling SIGPIPE
Do not set
SIGPIPE’s disposition to
SIG_DFLin order to avoid
BrokenPipeError. Doing that would cause your program to exit unexpectedly also whenever any socket connection is interrupted while your program is still writing to it.
Properly catch IOError to avoid [Errno 32] Broken pipe
Since [Errno 32] Broken pipe is actually a IOError, you can place a try/catch block to catch it like the code snippet below :
import sys, errno try: ### IO operation ### except IOError as e: if e.errno == errno.EPIPE: ### Handle the error ###
Possible solution for [Errno 32] Broken pipe in multi-process program.
In programs that uses worker processes to speed up processing and make use of multi-core CPUs, you can try reducing the number of the worker processes to see whether the error disappear or not.
A large number of worker processes may conflict with each other when they try to take control of system resources or the permission to write into disk.
Properly fixing [Errno 32] Broken pipe requires time and a close look at your code. Most of the time, you can safely ignore the problem if the Python program is relatively simple. Some other times, the solution involves in reducing worker processes in multi-process programs. But we hope that this article offer useful information and help you solve your problem.