Python subprocess Popen Stdout and Stderr

Python's cool subprocess module allows you to launch new processes and make stuff happen for you. You can easily catch Stdout and Stderr from these processes and do post processing as you like.

You can redirect stdout and stderr into buffer by specifying stdout=subprocess.PIPE, stderr=subprocess.PIPE arguments into Popen. You can also ignore them be setting values to None. By default you are receiving Stdout and Stderr output normally.

Python documentation states that you should not use PIPE with Popen since it can fill up OS Buffer, if your Stdout generates enough output. However this can be prevented by communicating with your process regularly by using communicate().

In my script, I'm waiting for suprocess to complete in a while loop, I'm polling the status of the process and then communicating to get stdout and stderr from the OS pipe buffer. Its also wise to sleep during the while loop to save overhead on CPU :-)

Here is a small script to demonstrate this:

 import subprocess
 import time

 # Bash command that will be executed
 cmd = "sudo apt-get upgrade"  

 # Launch subprocess  
 print "Launching command: " + cmd  
 sp = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)  
 sp_data = []  
 # Wait for process to finish  
 while sp.returncode is None:  
  print "Polling.."  
  sp.poll()  
  sp_data = sp.communicate()  
  time.sleep(1)  
 # Print results  
 print "Finished, returncode: " + str(sp.returncode)  
 print "Stdout:"  
 print "------------------------"  
 print str(sp_data[0])  
 print "Stderr:"  
 print "------------------------"  
 print str(sp_data[1])  

Output from this command is following:

 Launching command: sudo apt-get upgrade  
 Polling..  
 Finished, returncode: 0  
 Stdout:  
 ------------------------  
 Reading package lists...  
 Building dependency tree...  
 Reading state information...  
 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.  
 Stderr:  
 ------------------------  

To test that Stderr is also working, lets try to execute same command without sudo:

 Launching command: apt-get upgrade  
 Polling..  
 Finished, returncode: 100  
 Stdout:  
 ------------------------  
 Stderr:  
 ------------------------  
 E: Could not open lock file /var/lib/dpkg/lock - open (13: Permission denied)  
 E: Unable to lock the administration directory (/var/lib/dpkg/), are you root?  

You might also want to look at my previous post about Popen, how to spawn parallel processes:
Experimenting with Python subprocess Popen