Linux: Bash single line for loop examples

Knowing for loop in bash is definitely one of the most powerful tricks. For instance, it makes processing and reading files quick and painless in most cases. Since there can be countless use cases for this, i will write down some basic use scenarios.

Here is few basic examples that could be useful.

Creating files:

 for i in 1 2 3 4;do touch file"$i";done  
 ls -l  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:31 file1  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:31 file2  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:31 file3  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:31 file4  

Renaming all files within directory:

 ls -l  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic01.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic02.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic03.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic04.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic05.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic06.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic07.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic08.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic09.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic10.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic11.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic12.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic13.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic14.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic15.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic16.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic17.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic18.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic19.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 pic20.JPG  

I want to change rename these pictures as picture_01, picture_02 and so on. I could do this:

 for i in *;do mv $i $(echo $i | sed "s/pic/picture_/");done  
 ls -l  
 total 0  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_01.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_02.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_03.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_04.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_05.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_06.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_07.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_08.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_09.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_10.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_11.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_12.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_13.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_14.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_15.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_16.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_17.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_18.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_19.JPG  
 -rw-r--r-- 1 m 5000 0 Feb 26 21:18 picture_20.JPG  

* wildcard in this bash loop (for i in *) refers to all files within current working directory. That's why this for loop will catch all files.

Jinja2 template engine, simple basics.

Template engines will allow you to keep your HTML code outside of your code base. Managing and generating HTML in code is a bad practise. Templates will also give nice boost for performance and give you more secure system. this naturally depends how you are using your templates.

This is what Jinja2 is hyping in their features list:

Sandboxed execution

Powerful automatic HTML escaping system for XSS prevention

Template inheritance

Compiles down to the optimal python code just in time

Optional ahead of time template compilation

Easy to debug. Line numbers of exceptions directly point to the correct line in the template.

Configurable syntax

I was surprised to see how flexible Jinja2 for instance is. Variables can be modifier by filters you can see the full list of built in filters from here: http://jinja.pocoo.org/docs/templates/#builtin-filters

Here is some basics, which are usually enough to get going.

I'm passing this test data to Jinja2 (I'm using Python):

 template_data = {"page_title": "Testing templating",  
                  "page_text": "This is my Jinja2 rendered web page!",  
                  "loop_data": ["item1", "item2", "item3"]}  

Data is in dict format. It consists title, text and some data for demonstrating looping. Loop data is placed inside a list.

This is my template file:

 <html>  
 <head>  
 <title>{{ page_title }}</title>  
 </head>  
 <body>  
 <h1>{{ page_text }}</h1>  
 Loop data:  
 {% for row in loop_data %}  
  {% if row == "item2" %}  
   <font color="red">{{ row }}</font>  
  {% else %}  
   {{ row }}  
  {% endif %}}  
 {% endfor %}  
 </body>  
 </html>  

Variables are referred with {{ var_name }}.

For loop works like in any other programming language. Just loop list named loop_data and represent its data with variable row in the loop.

There is also a example how simple IF works. Basically when row value hits "item2" in the loop, font color should be in red. You can also specify {% elif %} blocks.

Python subprocess Popen Stdout and Stderr

Python's cool subprocess module allows you to launch new processes and make stuff happen for you. You can easily catch Stdout and Stderr from these processes and do post processing as you like.

You can redirect stdout and stderr into buffer by specifying stdout=subprocess.PIPE, stderr=subprocess.PIPE arguments into Popen. You can also ignore them be setting values to None. By default you are receiving Stdout and Stderr output normally.

Python documentation states that you should not use PIPE with Popen since it can fill up OS Buffer, if your Stdout generates enough output. However this can be prevented by communicating with your process regularly by using communicate().

In my script, I'm waiting for suprocess to complete in a while loop, I'm polling the status of the process and then communicating to get stdout and stderr from the OS pipe buffer. Its also wise to sleep during the while loop to save overhead on CPU :-)

Here is a small script to demonstrate this:

 import subprocess
 import time

 # Bash command that will be executed
 cmd = "sudo apt-get upgrade"  

 # Launch subprocess  
 print "Launching command: " + cmd  
 sp = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)  
 sp_data = []  
 # Wait for process to finish  
 while sp.returncode is None:  
  print "Polling.."  
  sp.poll()  
  sp_data = sp.communicate()  
  time.sleep(1)  
 # Print results  
 print "Finished, returncode: " + str(sp.returncode)  
 print "Stdout:"  
 print "------------------------"  
 print str(sp_data[0])  
 print "Stderr:"  
 print "------------------------"  
 print str(sp_data[1])  

Output from this command is following:

 Launching command: sudo apt-get upgrade  
 Polling..  
 Finished, returncode: 0  
 Stdout:  
 ------------------------  
 Reading package lists...  
 Building dependency tree...  
 Reading state information...  
 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.  
 Stderr:  
 ------------------------  

To test that Stderr is also working, lets try to execute same command without sudo:

 Launching command: apt-get upgrade  
 Polling..  
 Finished, returncode: 100  
 Stdout:  
 ------------------------  
 Stderr:  
 ------------------------  
 E: Could not open lock file /var/lib/dpkg/lock - open (13: Permission denied)  
 E: Unable to lock the administration directory (/var/lib/dpkg/), are you root?  

You might also want to look at my previous post about Popen, how to spawn parallel processes:
Experimenting with Python subprocess Popen