Showing posts with label Linux. Show all posts
Showing posts with label Linux. Show all posts

2014/02/26

Linux: Bash single line for loop examples

Knowing for loop in bash is definitely one of the most powerful tricks. For instance, it makes processing and reading files quick and painless in most cases. Since there can be countless use cases for this, i will write down some basic use scenarios.

Here is few basic examples that could be useful.

Creating files:

 for i in 1 2 3 4;do touch file"$i";done  
 ls -l  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:31 file1  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:31 file2  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:31 file3  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:31 file4  

Renaming all files within directory:

 ls -l  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic01.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic02.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic03.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic04.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic05.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic06.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic07.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic08.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic09.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic10.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic11.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic12.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic13.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic14.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic15.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic16.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic17.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic18.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic19.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic20.JPG  

I want to change rename these pictures as picture_01, picture_02 and so on. I could do this:

 for i in *;do mv $i $(echo $i | sed "s/pic/picture_/");done  
 ls -l  
 total 0  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_01.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_02.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_03.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_04.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_05.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_06.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_07.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_08.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_09.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_10.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_11.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_12.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_13.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_14.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_15.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_16.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_17.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_18.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_19.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_20.JPG  

* wildcard in this bash loop (for i in *) refers to all files within current working directory. That's why this for loop will catch all files.

2013/05/11

Install webmin on Ubuntu server (12.04 LTS)

Webmin is powerful web-based application for administering your Unix based or Windows system. With webmin, you can modify basic configuration of your server, manage cron jobs, execute commands etc.

Webmin is modular, which makes it very flexible. You can see list of standard modules of latest version from this link:
http://www.webmin.com/standard.html

First up, you should look up most recent version of webmin, you can check latest version from here:
http://freefr.dl.sourceforge.net/project/webadmin/webmin/

Download package with wget:
 wget http://freefr.dl.sourceforge.net/project/webadmin/webmin/1.620/webmin_1.620_all.deb  

Webmin is going to need some dependencies in order to install, handy way to install all required packages is to use tool called gdebi. Run gdebi as sudo against your deb packet to install all dependencies.

First you need to install packet gdebi-core:
 sudo apt-get install gdebi-core  

Then you can run it against your .deb pakcage:
 sudo gdebi webmin_1.620_all.deb  
 Requires the installation of the following packages:  
 apt-show-versions libapt-pkg-perl libauthen-pam-perl libio-pty-perl libnet-ssleay-perl  
 web-based administration interface for Unix systems  
  Using Webmin you can configure DNS, Samba, NFS, local/remote filesystems  
  and more using your web browser. After installation, enter the URL  
  https://localhost:10000/ into your browser and login as root with your root  
  password.  
 Do you want to install the software package? [y/N]:y  

Now everything should be set up and we can proceed with webmin install:

 sudo dpkg -i webmin_1.620_all.deb  
 ...  
 Webmin install complete. You can now login to https://ip-192-168-1-1:10000/  
 as root with your root password, or as any user who can use sudo  
 to run commands as root.  

Now everything should be set! You can start configuring your server from https://<your server>:10000/

2013/05/09

Grep running process

If you need script to see if specified process is running, its easy to grep it from ps aux.

 m@box:~$ ps aux | grep mysqld  
 mysql   1090 0.0 3.1 322660 48892 ?    Ssl 19:53  0:00 /usr/sbin/mysqld  
 m       1903 0.0 0.0  4384  808 pts/0  S+  20:29  0:00 grep --color=auto mysqld  

Problem here is that you can see your grep process which has your process as parameter. Script that relays on exit code ends up always being successful since grep process is on the list. You can easily fix this problem by adding additional pipe trough grep with -v grep parameter.

-v option is also --invert-match

 ps aux | grep mysqld | grep -v grep 1>/dev/null&& echo Process is runing || echo Process is not running  

Its a lot easier to do this by using pgrep.

pgrep scans running processes and returns process pid into your stdout.

 m@box:~$ pgrep mysqld 1>/dev/null&& echo Process is runing || echo Process is not running  
 Process is runing  

2012/12/30

Inodes in Linux

What is a inode, Why we need them ?

Have you ever wondered where is your access permissions located? How does your system know that you are the owner of your home folder? Is it written on the file itself ?

Answer to your questions is the Inodes also known as Index nodes.

Inode is a data structure, inodes stores all data about the file objects, except data content and file name. Unix doesn’t use the file name to refer to the file, it uses the inodes and inodes are pointing to actual block addresses where the data is located. inodes reside in Inode table (see Ext4 Disk Layout)

Windows (NTFS) equivalent is Master file table
Max OSX (HFS) equivalent is Catalog File

POSIX standard description for inode is:


  • File size in bytes
  • Owner and Group of the file
  • Device ID (device where file is contained)
  • file mode (access permissions)
  • Timestamps (Content last modified (mtime), Inode last modified(ctime) and Last accessed (atime))
  • Link count (how many hard links point to the inode)
  • Pointers to actual disk blocks that are containing the data
  • Additional system and user flags (limit its use and modification)


Unix or Linux systems doesn't usually store creation date, but ext4 has an attribute for creation date.

crtime (Create time) and also a dtime (Delete time)

inodes allows us to do linking and gives us significant performance increase, because inodes points us to the right data block when we are querying for files.

You can run out of inodes, what happens then ?

inode limit is decided at file system creation time. Its not a fixed value. You can check your current inodes usage with df -i


 Filesystem      Inodes  IUsed  IFree IUse% Mounted on  
 /dev/sda1      14983168  95016 14888152  1% /  

If you happen to run out of Inodes before you run out of disk space, you simply cannot create any more files or directories that consume Inodes, your things start to get very messy and unstable :)

stat command

stat displays file or file system status.

You can query file or file system with stat, for example for file:

 m@srv:~/symlink_test$ stat file1  
  File: `file1'  
  Size: 65         Blocks: 8     IO Block: 4096  regular file  
 Device: 801h/2049d     Inode: 5506851   Links: 1  
 Access: (0644/-rw-r--r--) Uid: ( 1000/  m)  Gid: ( 1000/  m)  
 Access: 2012-12-30 14:38:23.983590932 +0200  
 Modify: 2012-12-30 14:38:49.112092037 +0200  
 Change: 2012-12-30 14:38:49.112092037 +0200  

and for file system (use -f flag)

 m@srv:~/symlink_test$ stat -f /dev/sda1  
  File: "/dev/sda1"  
   ID: 0    Namelen: 255   Type: tmpfs  
 Block size: 4096    Fundamental block size: 4096  
 Blocks: Total: 191863   Free: 191821   Available: 191821  
 Inodes: Total: 191863   Free: 191262  


Inode data pointers, direct blocks and indirect blocks

One inode can point only to 12 blocks of data.

So if your block size is 4096, that would mean 12x4096 = 49152 bytes (48kb). This limitation would suck so that's why inode also has 3 indirect block pointers.

Indirect block pointer points to a block of block pointers, which.. again can point to indirect blocks that will point to new set of block pointers! This allows us to have very large files in our file system.

I tried to illustrate this with a picture but i ended up with a big messy pile of lines. Please check the wikipedia page for more info.


2012/12/29

Linux Symbolic links: Soft and Hard links

What is a symbolic link? Why its important ?

Symlink (Symbolic link) is a reference to another file or directory. It allows you to point multiple file names to single inode that points to a physical block address on the disk.

Symlink's  are mostly used as a shortcut's, they can make your life a lot easier by making your directories of data available for you where you want it.

How do i make a symlink ?

Use the following option in ln command:

 -s, --symbolic
              make symbolic links instead of hard links

For example:

 ln -s file1 file1_symlink  
 ls -l
 -rw-r--r-- 1 m m 0 2012-12-29 14:57 file1  
 lrwxrwxrwx 1 m m 5 2012-12-29 14:58 file1_symlink -> file1  


How do i make a hard link ?


 ln file1 file1_hardlink  
 ls -l  
 -rw-r--r-- 2 m m 0 2012-12-29 15:03 file1  
 -rw-r--r-- 2 m m 0 2012-12-29 15:03 file1_hardlink  

What is the difference between soft and hard link ?

Most important difference between hard and soft is that soft link depends on the original file.

For example, if i create a soft link and delete the original file that i linked, i cannot access my file trough symlink anymore and it becomes broken link.

 m@srv:~/symlink_test$ cat file1_symlink   
 Hello World!  
 m@srv:~/symlink_test$ rm file1  
 m@srv:~/symlink_test$ cat file1_symlink   
 cat: file1_symlink: No such file or directory  

But in the case of hard link, my file is always accessible if there is one hardlink existing of the file

 m@srv:~/symlink_test$ cat original  
 Hello World!  
 m@srv:~/symlink_test$ ln original hard_linked  
 m@srv:~/symlink_test$ cat original hard_linked   
 Hello World!  
 Hello World!  
 m@srv:~/symlink_test$ rm original   
 m@srv:~/symlink_test$ cat hard_linked   
 Hello World!  

Also hard links do not link paths on different volumes or file systems, soft links do. Soft links also consume inodes but hard links always shares the same inode.

Where is the symbolic link stored ?

Symlinks are just references to a physical location of a file, if you make symlinks, its doesn't meant that you are losing disk space by creating a copies. Symbolic links are stored in the same place as inodes, inodes are the place where disks resides its block addresses. 

2012/10/01

Generate .ppk out of .pem with Linux (Ubuntu)

Here is a example how to convert .pem to .ppk using Ubuntu.

First you need to install package putty-tools
 sudo apt-get install putty-tools  

After install, all you really need to do is this:
 puttygen key.pem -o key.ppk  

But.. with -P switch you can set passphrase for extra security, this is recommended and easy to do:
 puttygen key.pem -o key.ppk -P -C "My server key"  

It is also recommended to set comment for your key using -C switch, because this string will be prompted to you when you are entering your passphrase.





Note that you can also change passphrase afterwards by using -P switch

 m@box:~/Downloads$ puttygen -P key.ppk   
 Enter passphrase to load key:   
 Enter passphrase to save key:   
 Re-enter passphrase to verify:   

And you are done!


2012/09/14

Linux: Using FTP in bash

Default FTP client in Linux is simply called "ftp", it covers mostly all the basic needs and its easy to use.

Using FTP in command line

Lets pretend that i have file called data.dat that i need to upload to ftpserver.net. First we start FTP by simply typing in "ftp".

 m@srv:~$ ftp  
 ftp> o ftpserver.net  
 Connected to ftpserver.net.  
 220 ProFTPD 1.3.4b Server ready.  
 Name (ftpserver.net:m): myusername  
 331 Password required for myusername  
 Password:  
 230 User myusername logged in  
 Remote system type is UNIX.  
 Using binary mode to transfer files.  
 ftp>  

So first of all we connect with command "o", followed by the server address, if connection is successful   server will prompt you for username and password.

After server lets you in, you can list directories by doing "dir" or "ls"

 ftp> ls  
 200 PORT command successful  
 150 Opening ASCII mode data connection for file list  
 drwx--x---  9 ftp   ftp     4096 Sep 14 11:45 .  
 drwx--x---  9 ftp   ftp     4096 Sep 14 11:45 ..  
 drwx--x--x  4 ftp   ftp     4096 Sep 3 20:00 .appdata  
 -rw-r--r--  1 ftp   ftp      220 May 12 2008 .bash_logout  
 -rw-r--r--  1 ftp   ftp     3116 May 12 2008 .bashrc  
 -rw-------  1 ftp   ftp      89 Sep 13 22:12 .clipboard.txt  
 -rw-r--r--  1 ftp   ftp      675 May 12 2008 .profile  
 -rw-r-----  1 ftp   ftp      34 Jun 5 2009 .shadow  
 drwxrwx--x  2 ftp   ftp     4096 Sep 14 07:21 .spamassassin  
 drwx------  2 ftp   ftp     4096 Sep 3 20:01 application_backups  
 drwxr-xr-x  2 ftp   ftp     4096 Apr 21 2011 backups  
 drwx--x--x  3 ftp   ftp     4096 Jun 5 2009 domains  
 drwxrwx---  3 ftp   ftp     4096 Jun 5 2009 imap  
 drwxr-xr-x  2 ftp   ftp     4096 Sep 14 11:45 upload  
 226 Transfer complete  
 ftp>  

Now we can proceed and upload file, there is a folder called upload so I'm going to use that one. You can create directories with "mkdir".

 ftp> cd upload  
 250 CWD command successful  
 ftp> put /home/m/data.dat data.dat  
 local: /home/m/data.dat remote: data.dat  
 200 PORT command successful  
 150 Opening BINARY mode data connection for data.dat  
 226 Transfer complete  
 ftp> ls  
 200 PORT command successful  
 150 Opening ASCII mode data connection for file list  
 drwxr-xr-x  2 ftp   ftp     4096 Sep 14 11:50 .  
 drwx--x---  9 ftp   ftp     4096 Sep 14 11:45 ..  
 -rw-r--r--  1 ftp   ftp       0 Sep 14 11:50 data.dat  
 226 Transfer complete  
 ftp>  

With "put" command we first specify local location of the file,  following that only the file name, since we are already in a correct folder. We could also use absolute paths here, like this: "/upload/data.dat". We verified that our file is indeed in upload folder with "ls".

Note: You can delete files with command "del" and remove directories with "rm".

Now we can exit

 ftp> exit  
 221 Goodbye.  

Polite FTP server will tell you Goodbye!

Upload file using NcFTP

I personally wanted to have a ftp program that can upload a file in a single command, i couldn't do this with ftp so i needed to look for alternatives.

Client called NcFTP can do this. This doesn't come as a default so you need to install it.

 m@srv:~$ sudo apt-get install ncftp  

With ncftp, you get a program called ncftpput. If i would want to upload same file as above with single line command it would look something like this:

 m@srv:~$ ncftpput -u myusername -p mypassword ftpserver.net /upload /home/m/data.dat  
 /home/m/data.dat:                 183.41 kB 208.00 kB/s  
 m@srv:~$  

Here you first specify remote folder and after that local file location, exact opposite to ftp's put.

Very handy for scripting!






2012/09/13

Send commands to Linux screen

GNU Screen is very powerful piece of software that lets you multiplex virtual consoles. I personally use it for running jobs & servers on background. I also use irssi trough screen.

Name your screens!

When you run stuff at screen its recommended to set name for that session using -S switch.

For example with irssi i would specify irssi also as session name.

 m@srv:~$ screen -S irssi irssi  
 m@srv:~$ screen -list  
 There are screens on:  
     16425.irssi   (09/13/2012 01:28:36 PM)    (Detached)  
     16323.mc    (09/13/2012 01:24:43 PM)    (Detached)  
 2 Sockets in /var/run/screen/S-m.  

ID in front of your session name is PID of virtual screen console.

 m@srv:~$ ps -A | grep 16425  
 16425 ?    00:00:00 screen  

As you can see there is now two screens running, irssi and mc (mc is my minecraft server :-)). Why I'm telling this is simply because this way its bit easier to track what is running in your screen. Its not mandatory.

You can resume this sessions now with name, for example
 screen -r irssi  

If you are new to screen here is a good tip for you, you can detach from screen by doing ctrl+a+d

How to send commands to screen?

Especially with Minecraft server its handy to communicate with your server without attaching your screen session. For automation this is good, you can do daily backups, restart your server and before all that, you can alert your users!

This doesn't work if your screen is password protected.

 screen -S mc -X stuff 'say Server going down for maintenance in 20 seconds'$'\012'  

First you specify session with -S, then you need to do -X stuff '<message here>'

"stuff" needs to be there because.. you are sending stuff to it. Sorry i couldn't find any details about this.

\012 is a newline character, if you don't pass this, screen will input your message but wont "press enter" for you, so your message will remain in the console input screen.



OMG! Server going down



Replicate linux data using rsync

I¨ve always had multiple Linux boxes at my disposal, usually they are just old laptops or desktop that i have planned to use for something but never had time to do anything useful with them, so i have them just lying around.

They never had any kind of raid or redundancy and my data has been "kept save" just by holy spirit :-)

Easy way to sleep without worrying is to replicate your data between servers. Easy way to achieve this is by using rsync and cron. In this example we don't use rsync protocol but we are using SSH connection for the transfer.

First you should create SSH keys to your server to make them access each other without need to enter credentials.

In this example i will create rsync from my home server to external server, in this case i have only 1 account for both servers that i will be using.

1. Create SSH Keys

Im using ubuntu servers so first i need to edit file /etc/ssh/sshd_config and uncomment one line

#AuthorizedKeysFile     %h/.ssh/authorized_keys
(remove # from the start)

Then i will login to my home server and create SSH key with command ssh-keygen -t rsa

I didn't use any passphrase


 m@homeserver:~$ ssh-keygen -t rsa  
 Generating public/private rsa key pair.  
 Enter file in which to save the key (/home/m/.ssh/id_rsa):   
 Enter passphrase (empty for no passphrase):   
 Enter same passphrase again:   
 Your identification has been saved in /home/m/.ssh/id_rsa.  
 Your public key has been saved in /home/m/.ssh/id_rsa.pub.  


Now we need to open this public key in /home/profile/.ssh/ that we just created, and copy paste all of its contents to external server.

login to external server
Authorized keys will be stored at file "authorized_keys" in /home/profile/.ssh

File didn't exist so i had to create it

 m@external:~/.ssh$ touch authorized_keys  

Open file with your favourite editor and paste contents of your public key into it + save.

We should be all done, time to test

 m@homeserver:~/.ssh$ ssh m@extserver  

Works! (at least for me), no password prompted, yippee.

2. rsync

I want to keep contents between my servers always identical, so i will use following rsync command:

 rsync -avz --delete /home/m/samba m@external:/home/m/backup/  

-a is for archive mode, its very good for backups it will preserve when file was modified and who is the owner and so on

-v is for verbose, so you will get some output about transfer itself

-z is for compress, will save some bandwidth

--delete will remove files on extserver that don't exist on sender, in my case homeserver

If you need to specify alternate port for your connection, add --rsh='ssh -p1234' modify 1234 to your port

3. Cronjob

Now we create cron in order to make this rsync run automatically. I want to replicate all changes once per day, there might be some huge changes in my samba directory so i want to give some time for this process. 24 hours should be enough..

Access crontab with:
 crontab -e  

In my case this will be easy, there is some predefined scheduling definitions that will work for me.
@daily will run job everyday at midnight. Please see more at Cron wiki page

So my crontab looks like this:
 @daily rsync -avz --delete /home/m/samba m@external:/home/m/backup/  

Some final words

Actually this is kind of bad backup, since one human error may compromise all of my data. So if i delete a very important file.txt from my samba share, rsync will destroy it from external server also.

There are many ways to overcome this, for example you could zip backup folder everyday for last x days, in my case this is not possible since i don't have enough disk space :-(

Anyway, be careful !


2012/09/12

Experimenting with Python subprocess Popen

Process control and management with Python is pure greatness. Parallel tasks can be easily launched and monitored, output can be captured and used later on if needed.

You can communicate with your processes by sending data to stdin. You can also send in signals and all processes will give you return code after their execution.

Official subprocess documentation can be found here

I made small script to demonstrate subprocess.Popen

 import random  
 import subprocess  
 import time
  
 # Amount of spawned processes  
 amount = 10  
 # Process time  
 min = 5  
 max = 20  
 print " "  
 print "Subprocess example starting"  
 print "....................................."  
 time.sleep(1)  
 print " "  
 print "Spawning " + str(amount) + " processes"  
 # Store subprocesses in this array  
 process_list=[]  
 # Spawn all processes  
 for i in range (1, amount + 1):  
      timer = random.randint(min, max)       
      sp = subprocess.Popen("sleep " + str(timer), shell=True)  
      print "Spawning process nro " + str(i) + " | PID: " + str(sp.pid) + " | Timer set to: " + str(timer)  
      process_list.append(sp)  
 print "All processes spawned"  
 print " "  
 print "Waiting.."  
 print " "  
 time.sleep(5)  
 # Loop for processes, wait for completion  
 stop = 0  
 while stop != 1:  
      for sp in process_list:  
           sp.poll()  
           if str(sp.returncode) != "None":  
                print "Process (PID: " + str(sp.pid) + ") finished working! returncode: " + str(sp.returncode)  
                process_list.remove(sp)  
           time.sleep(0.1)  
      if len(process_list) <= 0:  
           stop = 1  
 print " "  
 print "....................................."  
 print "All subprocesses completed!"  

In this example script, first loop spawns 10 shells that will execute sleep timer randomly between 5 and 20 seconds. Second loop will just wait until every process has finished. poll() will set return code for process so it must be done once in a loop, so my application will notice finished processes.

Its handy to store your processes in a list!

Like in this case, if shell argument is set to true in Popen() .pid() (process ID) will belong to spawned shell. Alternatively you could do this:

 sp = subprocess.Popen(["./sleep_subprocess.sh", str(timer)])  

Here is the output:

 Subprocess engine starting  
 .....................................  
 Spawning 10 processes  
 Spawning process nro 1 | PID: 14644 | Timer set to: 19  
 Spawning process nro 2 | PID: 14645 | Timer set to: 5  
 Spawning process nro 3 | PID: 14647 | Timer set to: 10  
 Spawning process nro 4 | PID: 14649 | Timer set to: 12  
 Spawning process nro 5 | PID: 14651 | Timer set to: 6  
 Spawning process nro 6 | PID: 14653 | Timer set to: 17  
 Spawning process nro 7 | PID: 14655 | Timer set to: 7  
 Spawning process nro 8 | PID: 14657 | Timer set to: 17  
 Spawning process nro 9 | PID: 14659 | Timer set to: 19  
 Spawning process nro 10 | PID: 14661 | Timer set to: 17  
 All processes spawned  
 Waiting..  
 Process (PID: 14645) finished working! returncode: 0  
 Process (PID: 14651) finished working! returncode: 0  
 Process (PID: 14655) finished working! returncode: 0  
 Process (PID: 14647) finished working! returncode: 0  
 Process (PID: 14649) finished working! returncode: 0  
 Process (PID: 14653) finished working! returncode: 0  
 Process (PID: 14661) finished working! returncode: 0  
 Process (PID: 14657) finished working! returncode: 0  
 Process (PID: 14644) finished working! returncode: 0  
 Process (PID: 14659) finished working! returncode: 0  
 .....................................  
 All subprocesses completed!  

If your process stays too long you can terminate it with .terminate() or if its stuck you can .kill() it!