Python subprocess Popen Stdout and Stderr

Python's cool subprocess module allows you to launch new processes and make stuff happen for you. You can easily catch Stdout and Stderr from these processes and do post processing as you like.

You can redirect stdout and stderr into buffer by specifying stdout=subprocess.PIPE, stderr=subprocess.PIPE arguments into Popen. You can also ignore them be setting values to None. By default you are receiving Stdout and Stderr output normally.

Python documentation states that you should not use PIPE with Popen since it can fill up OS Buffer, if your Stdout generates enough output. However this can be prevented by communicating with your process regularly by using communicate().

In my script, I'm waiting for suprocess to complete in a while loop, I'm polling the status of the process and then communicating to get stdout and stderr from the OS pipe buffer. Its also wise to sleep during the while loop to save overhead on CPU :-)

Here is a small script to demonstrate this:

 import subprocess
 import time

 # Bash command that will be executed
 cmd = "sudo apt-get upgrade"  

 # Launch subprocess  
 print "Launching command: " + cmd  
 sp = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)  
 sp_data = []  
 # Wait for process to finish  
 while sp.returncode is None:  
  print "Polling.."  
  sp.poll()  
  sp_data = sp.communicate()  
  time.sleep(1)  
 # Print results  
 print "Finished, returncode: " + str(sp.returncode)  
 print "Stdout:"  
 print "------------------------"  
 print str(sp_data[0])  
 print "Stderr:"  
 print "------------------------"  
 print str(sp_data[1])  

Output from this command is following:

 Launching command: sudo apt-get upgrade  
 Polling..  
 Finished, returncode: 0  
 Stdout:  
 ------------------------  
 Reading package lists...  
 Building dependency tree...  
 Reading state information...  
 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.  
 Stderr:  
 ------------------------  

To test that Stderr is also working, lets try to execute same command without sudo:

 Launching command: apt-get upgrade  
 Polling..  
 Finished, returncode: 100  
 Stdout:  
 ------------------------  
 Stderr:  
 ------------------------  
 E: Could not open lock file /var/lib/dpkg/lock - open (13: Permission denied)  
 E: Unable to lock the administration directory (/var/lib/dpkg/), are you root?  

You might also want to look at my previous post about Popen, how to spawn parallel processes:
Experimenting with Python subprocess Popen

Experimenting with Python subprocess Popen

Process control and management with Python is pure greatness. Parallel tasks can be easily launched and monitored, output can be captured and used later on if needed.

You can communicate with your processes by sending data to stdin. You can also send in signals and all processes will give you return code after their execution.

Official subprocess documentation can be found here

I made small script to demonstrate subprocess.Popen

 import random  
 import subprocess  
 import time
  
 # Amount of spawned processes  
 amount = 10  
 # Process time  
 min = 5  
 max = 20  
 print " "  
 print "Subprocess example starting"  
 print "....................................."  
 time.sleep(1)  
 print " "  
 print "Spawning " + str(amount) + " processes"  
 # Store subprocesses in this array  
 process_list=[]  
 # Spawn all processes  
 for i in range (1, amount + 1):  
      timer = random.randint(min, max)       
      sp = subprocess.Popen("sleep " + str(timer), shell=True)  
      print "Spawning process nro " + str(i) + " | PID: " + str(sp.pid) + " | Timer set to: " + str(timer)  
      process_list.append(sp)  
 print "All processes spawned"  
 print " "  
 print "Waiting.."  
 print " "  
 time.sleep(5)  
 # Loop for processes, wait for completion  
 stop = 0  
 while stop != 1:  
      for sp in process_list:  
           sp.poll()  
           if str(sp.returncode) != "None":  
                print "Process (PID: " + str(sp.pid) + ") finished working! returncode: " + str(sp.returncode)  
                process_list.remove(sp)  
           time.sleep(0.1)  
      if len(process_list) <= 0:  
           stop = 1  
 print " "  
 print "....................................."  
 print "All subprocesses completed!"  

In this example script, first loop spawns 10 shells that will execute sleep timer randomly between 5 and 20 seconds. Second loop will just wait until every process has finished. poll() will set return code for process so it must be done once in a loop, so my application will notice finished processes.

Its handy to store your processes in a list!

Like in this case, if shell argument is set to true in Popen() .pid() (process ID) will belong to spawned shell. Alternatively you could do this:

 sp = subprocess.Popen(["./sleep_subprocess.sh", str(timer)])  

Here is the output:

 Subprocess engine starting  
 .....................................  
 Spawning 10 processes  
 Spawning process nro 1 | PID: 14644 | Timer set to: 19  
 Spawning process nro 2 | PID: 14645 | Timer set to: 5  
 Spawning process nro 3 | PID: 14647 | Timer set to: 10  
 Spawning process nro 4 | PID: 14649 | Timer set to: 12  
 Spawning process nro 5 | PID: 14651 | Timer set to: 6  
 Spawning process nro 6 | PID: 14653 | Timer set to: 17  
 Spawning process nro 7 | PID: 14655 | Timer set to: 7  
 Spawning process nro 8 | PID: 14657 | Timer set to: 17  
 Spawning process nro 9 | PID: 14659 | Timer set to: 19  
 Spawning process nro 10 | PID: 14661 | Timer set to: 17  
 All processes spawned  
 Waiting..  
 Process (PID: 14645) finished working! returncode: 0  
 Process (PID: 14651) finished working! returncode: 0  
 Process (PID: 14655) finished working! returncode: 0  
 Process (PID: 14647) finished working! returncode: 0  
 Process (PID: 14649) finished working! returncode: 0  
 Process (PID: 14653) finished working! returncode: 0  
 Process (PID: 14661) finished working! returncode: 0  
 Process (PID: 14657) finished working! returncode: 0  
 Process (PID: 14644) finished working! returncode: 0  
 Process (PID: 14659) finished working! returncode: 0  
 .....................................  
 All subprocesses completed!  

If your process stays too long you can terminate it with .terminate() or if its stuck you can .kill() it!