2013/05/11

Python subprocess Popen Stdout and Stderr

Python's cool subprocess module allows you to launch new processes and make stuff happen for you. You can easily catch Stdout and Stderr from these processes and do post processing as you like.

You can redirect stdout and stderr into buffer by specifying stdout=subprocess.PIPE, stderr=subprocess.PIPE arguments into Popen. You can also ignore them be setting values to None. By default you are receiving Stdout and Stderr output normally.

Python documentation states that you should not use PIPE with Popen since it can fill up OS Buffer, if your Stdout generates enough output. However this can be prevented by communicating with your process regularly by using communicate().

In my script, I'm waiting for suprocess to complete in a while loop, I'm polling the status of the process and then communicating to get stdout and stderr from the OS pipe buffer. Its also wise to sleep during the while loop to save overhead on CPU :-)

Here is a small script to demonstrate this:

 import subprocess
 import time

 # Bash command that will be executed
 cmd = "sudo apt-get upgrade"  

 # Launch subprocess  
 print "Launching command: " + cmd  
 sp = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)  
 sp_data = []  
 # Wait for process to finish  
 while sp.returncode is None:  
  print "Polling.."  
  sp.poll()  
  sp_data = sp.communicate()  
  time.sleep(1)  
 # Print results  
 print "Finished, returncode: " + str(sp.returncode)  
 print "Stdout:"  
 print "------------------------"  
 print str(sp_data[0])  
 print "Stderr:"  
 print "------------------------"  
 print str(sp_data[1])  

Output from this command is following:
 Launching command: sudo apt-get upgrade  
 Polling..  
 Finished, returncode: 0  
 Stdout:  
 ------------------------  
 Reading package lists...  
 Building dependency tree...  
 Reading state information...  
 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.  
 Stderr:  
 ------------------------  

To test that Stderr is also working, lets try to execute same command without sudo:
 Launching command: apt-get upgrade  
 Polling..  
 Finished, returncode: 100  
 Stdout:  
 ------------------------  
 Stderr:  
 ------------------------  
 E: Could not open lock file /var/lib/dpkg/lock - open (13: Permission denied)  
 E: Unable to lock the administration directory (/var/lib/dpkg/), are you root?  

You might also want to look at my previous post about Popen, how to spawn parallel processes:
Experimenting with Python subprocess Popen

Install webmin on Ubuntu server (12.04 LTS)

Webmin is powerful web-based application for administering your Unix based or Windows system. With webmin, you can modify basic configuration of your server, manage cron jobs, execute commands etc.

Webmin is modular, which makes it very flexible. You can see list of standard modules of latest version from this link:
http://www.webmin.com/standard.html

First up, you should look up most recent version of webmin, you can check latest version from here:
http://freefr.dl.sourceforge.net/project/webadmin/webmin/

Download package with wget:
 wget http://freefr.dl.sourceforge.net/project/webadmin/webmin/1.620/webmin_1.620_all.deb  

Webmin is going to need some dependencies in order to install, handy way to install all required packages is to use tool called gdebi. Run gdebi as sudo against your deb packet to install all dependencies.

First you need to install packet gdebi-core:
 sudo apt-get install gdebi-core  

Then you can run it against your .deb pakcage:
 sudo gdebi webmin_1.620_all.deb  
 Requires the installation of the following packages:  
 apt-show-versions libapt-pkg-perl libauthen-pam-perl libio-pty-perl libnet-ssleay-perl  
 web-based administration interface for Unix systems  
  Using Webmin you can configure DNS, Samba, NFS, local/remote filesystems  
  and more using your web browser. After installation, enter the URL  
  https://localhost:10000/ into your browser and login as root with your root  
  password.  
 Do you want to install the software package? [y/N]:y  

Now everything should be set up and we can proceed with webmin install:

 sudo dpkg -i webmin_1.620_all.deb  
 ...  
 Webmin install complete. You can now login to https://ip-192-168-1-1:10000/  
 as root with your root password, or as any user who can use sudo  
 to run commands as root.  

Now everything should be set! You can start configuring your server from https://<your server>:10000/

2013/05/09

Grep running process

If you need script to see if specified process is running, its easy to grep it from ps aux.

 m@box:~$ ps aux | grep mysqld  
 mysql   1090 0.0 3.1 322660 48892 ?    Ssl 19:53  0:00 /usr/sbin/mysqld  
 m       1903 0.0 0.0  4384  808 pts/0  S+  20:29  0:00 grep --color=auto mysqld  

Problem here is that you can see your grep process which has your process as parameter. Script that relays on exit code ends up always being successful since grep process is on the list. You can easily fix this problem by adding additional pipe trough grep with -v grep parameter.

-v option is also --invert-match

 ps aux | grep mysqld | grep -v grep 1>/dev/null&& echo Process is runing || echo Process is not running  

Its a lot easier to do this by using pgrep.

pgrep scans running processes and returns process pid into your stdout.

 m@box:~$ pgrep mysqld 1>/dev/null&& echo Process is runing || echo Process is not running  
 Process is runing  

2013/01/20

CIDR and VLSM

For a while it was very hard for me to understand what exactly is the difference between CIDR and VLSM. And since it took time to figure it out, i decided to post about it in a way how i would understand it myself.

Classless Inter-Domain Routing is a method to allocate IP addresses into multiple logical networks. You may know CIDR for its notation, for example 192.168.0.0 with mask 255.255.255.0 would be notated as 192.168.0.0/24.

CIDR has a 33 blocks of subnets, ranging from 0 to 32. Which makes subnetting a lot more efficient than classful subnetting. CIDR doesn't cover all possible subnet masks. Check the CIDR block table from wikipedia page

What if i want to use subnet mask of 255.255.255.253, which doesn't have CIDR notation ?

This is still indeed a valid subnet mask, but using it is discouraged. There might be incompatibility with some routing hardware depending how they are parsing their bits. In CIDR bits are expected to be 1's from left to right.

Where does the /24 come from ?

24 presents the amount of "turned on" bits in subnet mask's binary format. IPv4 address is a 32bit, so every octet in the IP address has 8 bit's. Every bit in octet has a value which you can either turn on or off.

1286432168421
11111111

If every bit is turned on, result would be 255

Consider this IPv4 addreess:

11000000 10100000 00001010 01100101

First octet: 11000000: 128 + 64 = 192
Second octet: 10100000: 128 + 32 = 168
Third octet: 00001010: 8 + 2 = 10
Fourth octet: 01100101: 64 + 32 + 4 + 1 = 100

Subnet mask 255.255.255.0 in binary would look like this:

11111111 11111111 11111111 00000000

It has 24 "turned on" bits and 8 "turned off" bits, so thats /24

255.255.255.255.192 in the other hand would look like this:

11111111 11111111 11111111 1100000

It has 26 one's and 6 zeroes, so this would be notated as /26

Then what is VLSM ?

VLSM stand for Variable Lenght Subnetmask. Name kind of gives it away, so for example: instead of splitting your 192.168.0.0/24 network into 4 same size pieces, you can split it into multiple variable sized networks.

So if i would want to hack 192.168.0.0/24 into 4 pieces it would look like this:

Network A: 192.168.0.0/26 (64 hosts)
Network B: 192.168.0.64/26 (64 hosts)
Network C: 192.168.0.128/26 (64 hosts)
Network D: 192.168.0.192/26 (64 hosts)

So now all networks can have total of 64 hosts (minus network and broadcast).

But what if my network A would have demand for more hosts, and 192.168.0.0/24 would be the only IP block that we can spare to allocate ?

Let's pretend that B and C networks would need only half of the currently allocated IP's. So instead of 64 hosts they would have 32 hosts. We could mask these networks with 255.255.255.224 which is equivalent to /27, and has 32 hosts. And now we have 64 IP's unallocated which we can lend to network A!

Using VLSM, we split the network more logically to serve all network better for their needs, our new networks would look like this:

Network A: 192.168.0.0/25 (128 hosts)
Network B: 192.168.0.128/27 (32 hosts)
Network C: 192.168.0.160/27 (32 hosts)
Network D: 192.168.0.192/26 (64 hosts)