Get shell output when calling shell from Python

Tags: ,

We have shown how to call shell in python using the subprocess communicate method. However, this method has some problems as the output is buffered into memory. We need to print out the shell output if the size is too large. Sometimes, when the shell scripts run for a long time, we may need to examine the output in real time to check the status of the problem.  In order to get the real time output from shell from python, we can using the following method:

Suppose we have a shell script, long_shell.sh,  like this:

Then we have the following python program to call the shell command and print the real output in screen.

We will immediately see the output when you run the python program. 

Note: this does not capture stderr. To capture stderr, we need to use p.communicate();  calling p.stdout.read() and then p.stderr.read() can deadlock if the buffer for stderr fills. To make them combined, we can use 2>&1 as part of the shell command. 

Reference: http://stackoverflow.com/questions/7353054/call-a-shell-command-containing-a-pipe-from-python-and-capture-stdout