Setting up LAMP in Ubuntu Server

This guide will take you trough process of installing fully functional LAMP server in Ubuntu. You can setup this easily in few minutes.

 sudo apt-get update  
 sudo tasksel  

Move to LAMP server and check it by pressing space, then click TAB to move to Ok and press Enter.

tasksell will download and install all necessary packets for you. Next you must enter MySQL root password and confirm it.








Once install is completed, verify it by connecting to http://<your public dns or ip>/

You should see this:

Yup! it works, now we will set up phpmyadmin for administrating MySQL

 sudo apt-get install phpmyadmin  

Setup will automatically configure phpmyadmin for you.

Select "apache2" from the list by pressing space

Configure database for phpmyadmin with dbconfig-common? -> Yes

Here you must enter your MySQL root password. The one you entered earlier!

Enter password for application password for phpmyadmin

This is password for user "phpmyadmin" that will be used for communication between MySQL Server and phpmyadmin.

And we are set again, verify installation by visiting http://<your public dns or ip>/phpmyadmin

You can log in using "root" and your MySQL root password or with "phpmyadmin" and application specific password.

Generate .ppk out of .pem using PuTTYgen

First you must download PuTTYgen, you can download it from here.

Open PuTTYgen and select Load

Note that you must select All Files (*.*) from filter or you cannot locate the file. Click Open 

Next up "Save private key" should no longer be greyed out, click it and name your .ppk file.

And you are done!

Linux: Using FTP in bash

Default FTP client in Linux is simply called "ftp", it covers mostly all the basic needs and its easy to use.

Using FTP in command line

Lets pretend that i have file called data.dat that i need to upload to ftpserver.net. First we start FTP by simply typing in "ftp".

 m@srv:~$ ftp  
 ftp> o ftpserver.net  
 Connected to ftpserver.net.  
 220 ProFTPD 1.3.4b Server ready.  
 Name (ftpserver.net:m): myusername  
 331 Password required for myusername  
 Password:  
 230 User myusername logged in  
 Remote system type is UNIX.  
 Using binary mode to transfer files.  
 ftp>  

So first of all we connect with command "o", followed by the server address, if connection is successful   server will prompt you for username and password.

After server lets you in, you can list directories by doing "dir" or "ls"

 ftp> ls  
 200 PORT command successful  
 150 Opening ASCII mode data connection for file list  
 drwx--x---  9 ftp   ftp     4096 Sep 14 11:45 .  
 drwx--x---  9 ftp   ftp     4096 Sep 14 11:45 ..  
 drwx--x--x  4 ftp   ftp     4096 Sep 3 20:00 .appdata  
 -rw-r--r--  1 ftp   ftp      220 May 12 2008 .bash_logout  
 -rw-r--r--  1 ftp   ftp     3116 May 12 2008 .bashrc  
 -rw-------  1 ftp   ftp      89 Sep 13 22:12 .clipboard.txt  
 -rw-r--r--  1 ftp   ftp      675 May 12 2008 .profile  
 -rw-r-----  1 ftp   ftp      34 Jun 5 2009 .shadow  
 drwxrwx--x  2 ftp   ftp     4096 Sep 14 07:21 .spamassassin  
 drwx------  2 ftp   ftp     4096 Sep 3 20:01 application_backups  
 drwxr-xr-x  2 ftp   ftp     4096 Apr 21 2011 backups  
 drwx--x--x  3 ftp   ftp     4096 Jun 5 2009 domains  
 drwxrwx---  3 ftp   ftp     4096 Jun 5 2009 imap  
 drwxr-xr-x  2 ftp   ftp     4096 Sep 14 11:45 upload  
 226 Transfer complete  
 ftp>  

Now we can proceed and upload file, there is a folder called upload so I'm going to use that one. You can create directories with "mkdir".

 ftp> cd upload  
 250 CWD command successful  
 ftp> put /home/m/data.dat data.dat  
 local: /home/m/data.dat remote: data.dat  
 200 PORT command successful  
 150 Opening BINARY mode data connection for data.dat  
 226 Transfer complete  
 ftp> ls  
 200 PORT command successful  
 150 Opening ASCII mode data connection for file list  
 drwxr-xr-x  2 ftp   ftp     4096 Sep 14 11:50 .  
 drwx--x---  9 ftp   ftp     4096 Sep 14 11:45 ..  
 -rw-r--r--  1 ftp   ftp       0 Sep 14 11:50 data.dat  
 226 Transfer complete  
 ftp>  

With "put" command we first specify local location of the file,  following that only the file name, since we are already in a correct folder. We could also use absolute paths here, like this: "/upload/data.dat". We verified that our file is indeed in upload folder with "ls".

Note: You can delete files with command "del" and remove directories with "rm".

Now we can exit

 ftp> exit  
 221 Goodbye.  

Polite FTP server will tell you Goodbye!

Upload file using NcFTP

I personally wanted to have a ftp program that can upload a file in a single command, i couldn't do this with ftp so i needed to look for alternatives.

Client called NcFTP can do this. This doesn't come as a default so you need to install it.

 m@srv:~$ sudo apt-get install ncftp  

With ncftp, you get a program called ncftpput. If i would want to upload same file as above with single line command it would look something like this:

 m@srv:~$ ncftpput -u myusername -p mypassword ftpserver.net /upload /home/m/data.dat  
 /home/m/data.dat:                 183.41 kB 208.00 kB/s  
 m@srv:~$  

Here you first specify remote folder and after that local file location, exact opposite to ftp's put.

Very handy for scripting!

Send commands to Linux screen

GNU Screen is very powerful piece of software that lets you multiplex virtual consoles. I personally use it for running jobs & servers on background. I also use irssi trough screen.

Name your screens!

When you run stuff at screen its recommended to set name for that session using -S switch.

For example with irssi i would specify irssi also as session name.

 m@srv:~$ screen -S irssi irssi  
 m@srv:~$ screen -list  
 There are screens on:  
     16425.irssi   (09/13/2012 01:28:36 PM)    (Detached)  
     16323.mc    (09/13/2012 01:24:43 PM)    (Detached)  
 2 Sockets in /var/run/screen/S-m.  

ID in front of your session name is PID of virtual screen console.

 m@srv:~$ ps -A | grep 16425  
 16425 ?    00:00:00 screen  

As you can see there is now two screens running, irssi and mc (mc is my minecraft server :-)). Why I'm telling this is simply because this way its bit easier to track what is running in your screen. Its not mandatory.

You can resume this sessions now with name, for example

 screen -r irssi  

If you are new to screen here is a good tip for you, you can detach from screen by doing ctrl+a+d

How to send commands to screen?

Especially with Minecraft server its handy to communicate with your server without attaching your screen session. For automation this is good, you can do daily backups, restart your server and before all that, you can alert your users!

This doesn't work if your screen is password protected.

 screen -S mc -X stuff 'say Server going down for maintenance in 20 seconds'$'\012'  

First you specify session with -S, then you need to do -X stuff '<message here>'

"stuff" needs to be there because.. you are sending stuff to it. Sorry i couldn't find any details about this.

\012 is a newline character, if you don't pass this, screen will input your message but wont "press enter" for you, so your message will remain in the console input screen.

OMG! Server going down

Replicate linux data using rsync

I¨ve always had multiple Linux boxes at my disposal, usually they are just old laptops or desktop that i have planned to use for something but never had time to do anything useful with them, so i have them just lying around.

They never had any kind of raid or redundancy and my data has been "kept save" just by holy spirit :-)

Easy way to sleep without worrying is to replicate your data between servers. Easy way to achieve this is by using rsync and cron. In this example we don't use rsync protocol but we are using SSH connection for the transfer.

First you should create SSH keys to your server to make them access each other without need to enter credentials.

In this example i will create rsync from my home server to external server, in this case i have only 1 account for both servers that i will be using.

1. Create SSH Keys

Im using ubuntu servers so first i need to edit file /etc/ssh/sshd_config and uncomment one line

#AuthorizedKeysFile     %h/.ssh/authorized_keys
(remove # from the start)

Then i will login to my home server and create SSH key with command ssh-keygen -t rsa

I didn't use any passphrase

 m@homeserver:~$ ssh-keygen -t rsa  
 Generating public/private rsa key pair.  
 Enter file in which to save the key (/home/m/.ssh/id_rsa):   
 Enter passphrase (empty for no passphrase):   
 Enter same passphrase again:   
 Your identification has been saved in /home/m/.ssh/id_rsa.  
 Your public key has been saved in /home/m/.ssh/id_rsa.pub.  

Now we need to open this public key in /home/profile/.ssh/ that we just created, and copy paste all of its contents to external server.

login to external server
Authorized keys will be stored at file "authorized_keys" in /home/profile/.ssh

File didn't exist so i had to create it

 m@external:~/.ssh$ touch authorized_keys  

Open file with your favourite editor and paste contents of your public key into it + save.

We should be all done, time to test

 m@homeserver:~/.ssh$ ssh m@extserver  

Works! (at least for me), no password prompted, yippee.

2. rsync

I want to keep contents between my servers always identical, so i will use following rsync command:

 rsync -avz --delete /home/m/samba m@external:/home/m/backup/  

-a is for archive mode, its very good for backups it will preserve when file was modified and who is the owner and so on

-v is for verbose, so you will get some output about transfer itself

-z is for compress, will save some bandwidth

--delete will remove files on extserver that don't exist on sender, in my case homeserver

If you need to specify alternate port for your connection, add --rsh='ssh -p1234' modify 1234 to your port

3. Cronjob

Now we create cron in order to make this rsync run automatically. I want to replicate all changes once per day, there might be some huge changes in my samba directory so i want to give some time for this process. 24 hours should be enough..

Access crontab with:

 crontab -e  

In my case this will be easy, there is some predefined scheduling definitions that will work for me.
@daily will run job everyday at midnight. Please see more at Cron wiki page

So my crontab looks like this:

 @daily rsync -avz --delete /home/m/samba m@external:/home/m/backup/  

Some final words

Actually this is kind of bad backup, since one human error may compromise all of my data. So if i delete a very important file.txt from my samba share, rsync will destroy it from external server also.

There are many ways to overcome this, for example you could zip backup folder everyday for last x days, in my case this is not possible since i don't have enough disk space :-(

Anyway, be careful !

Experimenting with Python subprocess Popen

Process control and management with Python is pure greatness. Parallel tasks can be easily launched and monitored, output can be captured and used later on if needed.

You can communicate with your processes by sending data to stdin. You can also send in signals and all processes will give you return code after their execution.

Official subprocess documentation can be found here

I made small script to demonstrate subprocess.Popen

 import random  
 import subprocess  
 import time
  
 # Amount of spawned processes  
 amount = 10  
 # Process time  
 min = 5  
 max = 20  
 print " "  
 print "Subprocess example starting"  
 print "....................................."  
 time.sleep(1)  
 print " "  
 print "Spawning " + str(amount) + " processes"  
 # Store subprocesses in this array  
 process_list=[]  
 # Spawn all processes  
 for i in range (1, amount + 1):  
      timer = random.randint(min, max)       
      sp = subprocess.Popen("sleep " + str(timer), shell=True)  
      print "Spawning process nro " + str(i) + " | PID: " + str(sp.pid) + " | Timer set to: " + str(timer)  
      process_list.append(sp)  
 print "All processes spawned"  
 print " "  
 print "Waiting.."  
 print " "  
 time.sleep(5)  
 # Loop for processes, wait for completion  
 stop = 0  
 while stop != 1:  
      for sp in process_list:  
           sp.poll()  
           if str(sp.returncode) != "None":  
                print "Process (PID: " + str(sp.pid) + ") finished working! returncode: " + str(sp.returncode)  
                process_list.remove(sp)  
           time.sleep(0.1)  
      if len(process_list) <= 0:  
           stop = 1  
 print " "  
 print "....................................."  
 print "All subprocesses completed!"  

In this example script, first loop spawns 10 shells that will execute sleep timer randomly between 5 and 20 seconds. Second loop will just wait until every process has finished. poll() will set return code for process so it must be done once in a loop, so my application will notice finished processes.

Its handy to store your processes in a list!

Like in this case, if shell argument is set to true in Popen() .pid() (process ID) will belong to spawned shell. Alternatively you could do this:

 sp = subprocess.Popen(["./sleep_subprocess.sh", str(timer)])  

Here is the output:

 Subprocess engine starting  
 .....................................  
 Spawning 10 processes  
 Spawning process nro 1 | PID: 14644 | Timer set to: 19  
 Spawning process nro 2 | PID: 14645 | Timer set to: 5  
 Spawning process nro 3 | PID: 14647 | Timer set to: 10  
 Spawning process nro 4 | PID: 14649 | Timer set to: 12  
 Spawning process nro 5 | PID: 14651 | Timer set to: 6  
 Spawning process nro 6 | PID: 14653 | Timer set to: 17  
 Spawning process nro 7 | PID: 14655 | Timer set to: 7  
 Spawning process nro 8 | PID: 14657 | Timer set to: 17  
 Spawning process nro 9 | PID: 14659 | Timer set to: 19  
 Spawning process nro 10 | PID: 14661 | Timer set to: 17  
 All processes spawned  
 Waiting..  
 Process (PID: 14645) finished working! returncode: 0  
 Process (PID: 14651) finished working! returncode: 0  
 Process (PID: 14655) finished working! returncode: 0  
 Process (PID: 14647) finished working! returncode: 0  
 Process (PID: 14649) finished working! returncode: 0  
 Process (PID: 14653) finished working! returncode: 0  
 Process (PID: 14661) finished working! returncode: 0  
 Process (PID: 14657) finished working! returncode: 0  
 Process (PID: 14644) finished working! returncode: 0  
 Process (PID: 14659) finished working! returncode: 0  
 .....................................  
 All subprocesses completed!  

If your process stays too long you can terminate it with .terminate() or if its stuck you can .kill() it!

Google Drive lets you stream your videos directly from Drive

I recently uploaded some of my videos into Drive for safe keeping, later i noticed that i can actually stream these videos straight from Google Drive.. Cool!

I tried some other formats as well and they all seem to work, it will just take some time for google to convert your videos.

More information about this feature can be found from Google Help

Here is an example, I'm uploading a random_video.mp4 to Drive

After the upload is completed, and you try to open the file, drive will let you now download your file but not stream it. And there is actually a notice about video processing. hmm i didn't notice this earlier, maybe it was added recently?

It took 34 seconds (i took time) for my video to process and now its ready to roll.

Good example where cloud storage is heading in the future, battle cannot be won with only prices and bulk storage.

Users want to do much as possible in the cloud for example edit their photos and videos, in  the future maybe everything we need is just a browser!

Python and MySQL (MySQLdb)

If you need to access MySQL with Python MySQLdb is your friend.

Very good User guide can be found here. I lost few hairs when i was first playing with this so i decided to create small tutorial of my own. Hopefully will save some hair from someone

This is very basic stuff

Installing MySQLdb

In ubuntu system you need to install packet called python-mysqldb. Sources itself can be found from here

apt-get install python-mysqldb

Connecting database

 import MySQLdb   
  try:    
    conn = MySQLdb.connect(host=address,user=username,passwd=password,db=database)   
  except:    
    # Connection failed  

Executing commands

Here we pass two variables into INSERT command, first you need to create a cursor, then execute against this cursor.

After execute, commit must be applied or changes wont happen in your database.

 cursor = conn.cursor()  
 cursor.execute("""INSERT INTO data (col1,col2) VALUES (%s, %s)""", (var1,var2,))  
 conn.commit()  

TIP: After execute, you can get primary ID from previous insert with:

 print str(cursor.insert_id())  

Select from database

  sql = "SELECT * FROM stuff"  
  # Query  
  conn.query(sql)  

At this point, your query is executed, now you need to store your results

  # Store results  
  result_store = conn.store_result()  

Note: By default mysql db will return you only 1 row from result set, so if you specify maxrows=0 in your fetch_row, it will return all rows from the results set.

  # Fetch row from store  
  results = result_store.fetch_row(maxrows=0)  

Oh Yes.. this was my biggest issue, i didn't realize that result tuple had one tuple inside of it, stupid me!

Basically at this point if you print results output is:

 >>> print results  
 (('row1col1', 'row1col2'), ('row2col1', 'row2col2'))  

You can simply loop trough results with for each or get individual row by giving and index, row1 is naturally index number of 0 and row2 is index number of 1

 >>> row1 = results[1]  
 >>> print row1  
 ('row2col1', 'row2col2')  

And same applies to column..

 >>> col1 = row1[1]  
 >>> print col1  
 row2col2  

Closing connection

 conn.close()