Experimenting with Python subprocess Popen

Process control and management with Python is pure greatness. Parallel tasks can be easily launched and monitored, output can be captured and used later on if needed.

You can communicate with your processes by sending data to stdin. You can also send in signals and all processes will give you return code after their execution.

Official subprocess documentation can be found here

I made small script to demonstrate subprocess.Popen

 import random  
 import subprocess  
 import time
  
 # Amount of spawned processes  
 amount = 10  
 # Process time  
 min = 5  
 max = 20  
 print " "  
 print "Subprocess example starting"  
 print "....................................."  
 time.sleep(1)  
 print " "  
 print "Spawning " + str(amount) + " processes"  
 # Store subprocesses in this array  
 process_list=[]  
 # Spawn all processes  
 for i in range (1, amount + 1):  
      timer = random.randint(min, max)       
      sp = subprocess.Popen("sleep " + str(timer), shell=True)  
      print "Spawning process nro " + str(i) + " | PID: " + str(sp.pid) + " | Timer set to: " + str(timer)  
      process_list.append(sp)  
 print "All processes spawned"  
 print " "  
 print "Waiting.."  
 print " "  
 time.sleep(5)  
 # Loop for processes, wait for completion  
 stop = 0  
 while stop != 1:  
      for sp in process_list:  
           sp.poll()  
           if str(sp.returncode) != "None":  
                print "Process (PID: " + str(sp.pid) + ") finished working! returncode: " + str(sp.returncode)  
                process_list.remove(sp)  
           time.sleep(0.1)  
      if len(process_list) <= 0:  
           stop = 1  
 print " "  
 print "....................................."  
 print "All subprocesses completed!"  

In this example script, first loop spawns 10 shells that will execute sleep timer randomly between 5 and 20 seconds. Second loop will just wait until every process has finished. poll() will set return code for process so it must be done once in a loop, so my application will notice finished processes.

Its handy to store your processes in a list!

Like in this case, if shell argument is set to true in Popen() .pid() (process ID) will belong to spawned shell. Alternatively you could do this:

 sp = subprocess.Popen(["./sleep_subprocess.sh", str(timer)])  

Here is the output:

 Subprocess engine starting  
 .....................................  
 Spawning 10 processes  
 Spawning process nro 1 | PID: 14644 | Timer set to: 19  
 Spawning process nro 2 | PID: 14645 | Timer set to: 5  
 Spawning process nro 3 | PID: 14647 | Timer set to: 10  
 Spawning process nro 4 | PID: 14649 | Timer set to: 12  
 Spawning process nro 5 | PID: 14651 | Timer set to: 6  
 Spawning process nro 6 | PID: 14653 | Timer set to: 17  
 Spawning process nro 7 | PID: 14655 | Timer set to: 7  
 Spawning process nro 8 | PID: 14657 | Timer set to: 17  
 Spawning process nro 9 | PID: 14659 | Timer set to: 19  
 Spawning process nro 10 | PID: 14661 | Timer set to: 17  
 All processes spawned  
 Waiting..  
 Process (PID: 14645) finished working! returncode: 0  
 Process (PID: 14651) finished working! returncode: 0  
 Process (PID: 14655) finished working! returncode: 0  
 Process (PID: 14647) finished working! returncode: 0  
 Process (PID: 14649) finished working! returncode: 0  
 Process (PID: 14653) finished working! returncode: 0  
 Process (PID: 14661) finished working! returncode: 0  
 Process (PID: 14657) finished working! returncode: 0  
 Process (PID: 14644) finished working! returncode: 0  
 Process (PID: 14659) finished working! returncode: 0  
 .....................................  
 All subprocesses completed!  

If your process stays too long you can terminate it with .terminate() or if its stuck you can .kill() it!