Get shell output when calling shell from Python

July 15, 2016 Author: david

We have shown how to call shell in python using the subprocess communicate method. However, this method has some problems as the output is buffered into memory. We need to print out the shell output if the size is too large. Sometimes, when the shell scripts run for a long time, we may need to examine the output in real time to check the status of the problem. In order to get the real time output from shell from python, we can using the following method:

Suppose we have a shell script, long_shell.sh, like this:

#!/usr/bin/env bash
for i in $(seq 1 5)
do
 echo $i
 echo start sleep 1s
 sleep 1
done

#!/usr/bin/env bash

for i in $(seq 1 5)

echo $i

echo start sleep 1s

sleep 1

done

Then we have the following python program to call the shell command and print the real output in screen.

#!/usr/bin/python
import sys
import subprocess
cmd = ['sh', 'long_shell.sh']
p = subprocess.Popen(cmd, stdout=subprocess.PIPE)
def is_end(p, type):
    msg = type.readline()
    if msg == '' and p.poll() != None:
        return True
    if msg != '':
        sys.stdout.write("the msg is >>" + msg)
    return False
## start displaying output immediately ##
while True:
    if is_end(p, p.stdout):
        break

#!/usr/bin/python

import sys

import subprocess

cmd = ['sh', 'long_shell.sh']

p = subprocess.Popen(cmd, stdout=subprocess.PIPE)

def is_end(p, type):

msg = type.readline()

if msg == '' and p.poll() != None:

return True

if msg != '':

sys.stdout.write("the msg is >>" + msg)

return False

## start displaying output immediately ##

while True:

if is_end(p, p.stdout):

break

We will immediately see the output when you run the python program.

the msg is >> 1
the msg is >> start sleep 1s
the msg is >> 2
the msg is >> start sleep 1s
the msg is >> 3
the msg is >> start sleep 1s
the msg is >> 4
the msg is >> start sleep 1s
the msg is >> 5
the msg is >> start sleep 1s

the msg is >> 1

the msg is >> start sleep 1s

the msg is >> 2

the msg is >> start sleep 1s

the msg is >> 3

the msg is >> start sleep 1s

the msg is >> 4

the msg is >> start sleep 1s

the msg is >> 5

the msg is >> start sleep 1s

Note: this does not capture stderr. To capture stderr, we need to use p.communicate(); calling p.stdout.read() and then p.stderr.read() can deadlock if the buffer for stderr fills. To make them combined, we can use 2>&1 as part of the shell command.

Reference: http://stackoverflow.com/questions/7353054/call-a-shell-command-containing-a-pipe-from-python-and-capture-stdout