Showing posts with label Python. Show all posts
Showing posts with label Python. Show all posts

2014/02/26

Jinja2 template engine, simple basics.

Template engines will allow you to keep your HTML code outside of your code base. Managing and generating HTML in code is a bad practise. Templates will also give nice boost for performance and give you more secure system. this naturally depends how you are using your templates.

This is what Jinja2 is hyping in their features list:


  • Sandboxed execution
  • Powerful automatic HTML escaping system for XSS prevention
  • Template inheritance
  • Compiles down to the optimal python code just in time
  • Optional ahead of time template compilation
  • Easy to debug. Line numbers of exceptions directly point to the correct line in the template.
  • Configurable syntax

  • I was surprised to see how flexible Jinja2 for instance is. Variables can be modifier by filters you can see the full list of built in filters from here: http://jinja.pocoo.org/docs/templates/#builtin-filters

    Here is some basics, which are usually enough to get going.

    I'm passing this test data to Jinja2 (I'm using Python):

     template_data = {"page_title": "Testing templating",  
                      "page_text": "This is my Jinja2 rendered web page!",  
                      "loop_data": ["item1", "item2", "item3"]}  
    

    Data is in dict format. It consists title, text and some data for demonstrating looping. Loop data is placed inside a list.

    This is my template file:

     <html>  
     <head>  
     <title>{{ page_title }}</title>  
     </head>  
     <body>  
     <h1>{{ page_text }}</h1>  
     Loop data:  
     {% for row in loop_data %}  
      {% if row == "item2" %}  
       <font color="red">{{ row }}</font>  
      {% else %}  
       {{ row }}  
      {% endif %}}  
     {% endfor %}  
     </body>  
     </html>  
    

    Variables are referred with {{ var_name }}.

    For loop works like in any other programming language. Just loop list named loop_data and represent its data with variable row in the loop.

    There is also a example how simple IF works. Basically when row value hits "item2" in the loop, font color should be in red. You can also specify {% elif %} blocks.


    2013/05/11

    Python subprocess Popen Stdout and Stderr

    Python's cool subprocess module allows you to launch new processes and make stuff happen for you. You can easily catch Stdout and Stderr from these processes and do post processing as you like.

    You can redirect stdout and stderr into buffer by specifying stdout=subprocess.PIPE, stderr=subprocess.PIPE arguments into Popen. You can also ignore them be setting values to None. By default you are receiving Stdout and Stderr output normally.

    Python documentation states that you should not use PIPE with Popen since it can fill up OS Buffer, if your Stdout generates enough output. However this can be prevented by communicating with your process regularly by using communicate().

    In my script, I'm waiting for suprocess to complete in a while loop, I'm polling the status of the process and then communicating to get stdout and stderr from the OS pipe buffer. Its also wise to sleep during the while loop to save overhead on CPU :-)

    Here is a small script to demonstrate this:

     import subprocess
     import time
    
     # Bash command that will be executed
     cmd = "sudo apt-get upgrade"  
    
     # Launch subprocess  
     print "Launching command: " + cmd  
     sp = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)  
     sp_data = []  
     # Wait for process to finish  
     while sp.returncode is None:  
      print "Polling.."  
      sp.poll()  
      sp_data = sp.communicate()  
      time.sleep(1)  
     # Print results  
     print "Finished, returncode: " + str(sp.returncode)  
     print "Stdout:"  
     print "------------------------"  
     print str(sp_data[0])  
     print "Stderr:"  
     print "------------------------"  
     print str(sp_data[1])  
    

    Output from this command is following:
     Launching command: sudo apt-get upgrade  
     Polling..  
     Finished, returncode: 0  
     Stdout:  
     ------------------------  
     Reading package lists...  
     Building dependency tree...  
     Reading state information...  
     0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.  
     Stderr:  
     ------------------------  
    

    To test that Stderr is also working, lets try to execute same command without sudo:
     Launching command: apt-get upgrade  
     Polling..  
     Finished, returncode: 100  
     Stdout:  
     ------------------------  
     Stderr:  
     ------------------------  
     E: Could not open lock file /var/lib/dpkg/lock - open (13: Permission denied)  
     E: Unable to lock the administration directory (/var/lib/dpkg/), are you root?  
    

    You might also want to look at my previous post about Popen, how to spawn parallel processes:
    Experimenting with Python subprocess Popen

    2012/10/30

    App Engine Templates and CSS with Python

    HTML is much easier to maintain in App Engine if you use templates. Templates are way of storing you HTML code, it also has syntax to show your application data where you want it to be.

    Django's templating engine is included in webapp2 framework.

    Here is a sample code how to render html template:

     import webapp2  
     from google.appengine.ext.webapp import template  
     class index(webapp2.RequestHandler):  
      def get(self):  
       self.response.out.write(template.render('index.html', {}))  
     app = webapp2.WSGIApplication([('/', index)])  
    

    Just remember to store your html file in the same folder.

    Then you probably want to add CSS to your project? Create folder called css (or whatever you want to call it) and add it as a static directory to your app.yaml:

    This will map your "physical" css directory into <project url>/css url.

     - url: /css  
      static_dir: css  
    

    Create your style.css file and refer it in index.html and you are done!

     <link href="css/styles.css" rel="stylesheet" type="text/css">  
    


    2012/09/12

    Experimenting with Python subprocess Popen

    Process control and management with Python is pure greatness. Parallel tasks can be easily launched and monitored, output can be captured and used later on if needed.

    You can communicate with your processes by sending data to stdin. You can also send in signals and all processes will give you return code after their execution.

    Official subprocess documentation can be found here

    I made small script to demonstrate subprocess.Popen

     import random  
     import subprocess  
     import time
      
     # Amount of spawned processes  
     amount = 10  
     # Process time  
     min = 5  
     max = 20  
     print " "  
     print "Subprocess example starting"  
     print "....................................."  
     time.sleep(1)  
     print " "  
     print "Spawning " + str(amount) + " processes"  
     # Store subprocesses in this array  
     process_list=[]  
     # Spawn all processes  
     for i in range (1, amount + 1):  
          timer = random.randint(min, max)       
          sp = subprocess.Popen("sleep " + str(timer), shell=True)  
          print "Spawning process nro " + str(i) + " | PID: " + str(sp.pid) + " | Timer set to: " + str(timer)  
          process_list.append(sp)  
     print "All processes spawned"  
     print " "  
     print "Waiting.."  
     print " "  
     time.sleep(5)  
     # Loop for processes, wait for completion  
     stop = 0  
     while stop != 1:  
          for sp in process_list:  
               sp.poll()  
               if str(sp.returncode) != "None":  
                    print "Process (PID: " + str(sp.pid) + ") finished working! returncode: " + str(sp.returncode)  
                    process_list.remove(sp)  
               time.sleep(0.1)  
          if len(process_list) <= 0:  
               stop = 1  
     print " "  
     print "....................................."  
     print "All subprocesses completed!"  
    

    In this example script, first loop spawns 10 shells that will execute sleep timer randomly between 5 and 20 seconds. Second loop will just wait until every process has finished. poll() will set return code for process so it must be done once in a loop, so my application will notice finished processes.

    Its handy to store your processes in a list!

    Like in this case, if shell argument is set to true in Popen() .pid() (process ID) will belong to spawned shell. Alternatively you could do this:

     sp = subprocess.Popen(["./sleep_subprocess.sh", str(timer)])  
    

    Here is the output:

     Subprocess engine starting  
     .....................................  
     Spawning 10 processes  
     Spawning process nro 1 | PID: 14644 | Timer set to: 19  
     Spawning process nro 2 | PID: 14645 | Timer set to: 5  
     Spawning process nro 3 | PID: 14647 | Timer set to: 10  
     Spawning process nro 4 | PID: 14649 | Timer set to: 12  
     Spawning process nro 5 | PID: 14651 | Timer set to: 6  
     Spawning process nro 6 | PID: 14653 | Timer set to: 17  
     Spawning process nro 7 | PID: 14655 | Timer set to: 7  
     Spawning process nro 8 | PID: 14657 | Timer set to: 17  
     Spawning process nro 9 | PID: 14659 | Timer set to: 19  
     Spawning process nro 10 | PID: 14661 | Timer set to: 17  
     All processes spawned  
     Waiting..  
     Process (PID: 14645) finished working! returncode: 0  
     Process (PID: 14651) finished working! returncode: 0  
     Process (PID: 14655) finished working! returncode: 0  
     Process (PID: 14647) finished working! returncode: 0  
     Process (PID: 14649) finished working! returncode: 0  
     Process (PID: 14653) finished working! returncode: 0  
     Process (PID: 14661) finished working! returncode: 0  
     Process (PID: 14657) finished working! returncode: 0  
     Process (PID: 14644) finished working! returncode: 0  
     Process (PID: 14659) finished working! returncode: 0  
     .....................................  
     All subprocesses completed!  
    

    If your process stays too long you can terminate it with .terminate() or if its stuck you can .kill() it!

    Python and MySQL (MySQLdb)

    If you need to access MySQL with Python MySQLdb is your friend.

    Very good User guide can be found here. I lost few hairs when i was first playing with this so i decided to create small tutorial of my own. Hopefully will save some hair from someone

    This is very basic stuff

    Installing MySQLdb

    In ubuntu system you need to install packet called python-mysqldb. Sources itself can be found from here

    apt-get install python-mysqldb

    Connecting database

     import MySQLdb   
      try:    
        conn = MySQLdb.connect(host=address,user=username,passwd=password,db=database)   
      except:    
        # Connection failed  
    

    Executing commands

    Here we pass two variables into INSERT command, first you need to create a cursor, then execute against this cursor.

    After execute, commit must be applied or changes wont happen in your database.

     cursor = conn.cursor()  
     cursor.execute("""INSERT INTO data (col1,col2) VALUES (%s, %s)""", (var1,var2,))  
     conn.commit()  

    TIP: After execute, you can get primary ID from previous insert with:

     print str(cursor.insert_id())  

    Select from database

      sql = "SELECT * FROM stuff"  
      # Query  
      conn.query(sql)  
    

    At this point, your query is executed, now you need to store your results

      # Store results  
      result_store = conn.store_result()  
    

    Note: By default mysql db will return you only 1 row from result set, so if you specify maxrows=0 in your fetch_row, it will return all rows from the results set.

      # Fetch row from store  
      results = result_store.fetch_row(maxrows=0)  
    

    Oh Yes.. this was my biggest issue, i didn't realize that result tuple had one tuple inside of it, stupid me!

    Basically at this point if you print results output is:

     >>> print results  
     (('row1col1', 'row1col2'), ('row2col1', 'row2col2'))  
    

    You can simply loop trough results with for each or get individual row by giving and index, row1 is naturally index number of 0 and row2 is index number of 1

     >>> row1 = results[1]  
     >>> print row1  
     ('row2col1', 'row2col2')  
    

    And same applies to column..

     >>> col1 = row1[1]  
     >>> print col1  
     row2col2  
    

    Closing connection
     conn.close()