From strings to a centipede!

Long title: "From strings to a centipede - a web developers dive into frameworks, and the realization that tools should not abstract away from technology".

So, a while back I was fed up with not knowing what was going on inside all these web application frameworks and application servers I was struggling with all day long, and why they sometimes was so hard to debug. So I set out to build my own in hope of some better understanding of the situation.

This resulted in a project I called strings ( not because of string theory or "strings" or anything - rather cause I like picking my guitar ). It was a fun project and I've learned a great deal doing it.

One especially important lesson was the way these type of frameworks tend to abstract away important parts of the technology in use, stuff that a web developer should know and think about. Stuff like http requests and responses, http headers, character encoding, data storage. Frameworks tend to create their own metaphors for these things, in order to "simplify" the development process. But what often happens is that this locks developers in some framework specific world where they don't really have control over what is going on. This in turn is a big cause for confusion, debug trouble, unforeseeable time consumption, etc.

Tools should not abstract away from technology!

One can even be tempted to say that these frameworks hides the technology in order to lock developers to specific tools, though I don't think that is the case. But it sure feels that way some times.

There is of course the inverted argument, that developers should not use these tools or frameworks if they are not familiar with their inner workings and the technologies they use. And thats a valid argument ... in some theoretical world without any deadlines, raving project managers, broke-down coffee machines & cars, crippled backs, facebooks to be update, babies to be fed and dogs to be walked. Come on, who ever has the time ( or the ability ) to read and understand 20.000 lines of someone else's code. Not me anyway...

One can always discuss if the technology in use should be more approachable or modeled differently, but thats another discussion. One we would gladly participate in, as soon as we know what the frack we're dealing with today!

Still, there is no need to dangle from the velvet curtains in desperate tears just yet. In the era of free software, ideas flow freely, solutions are swiftly shifting and the distant drums of frustrated developers are becoming ever louder. There has been a lot of very cool things surfacing these last couple of years aiming to solve these exact issues.

Hidden away in a mozilla project called bespin I found a brilliant solution that fits my way of thinking perfectly. It makes use of the WSGI and a brilliant little decorator based router called urlrelay. Adding a startup script and a tiny bit of glue - centipede was born.

Centipede also makes use of other peoples code through the use WSGI middleware ( but these are often small modules aimed at doing one thing - that I can handle ), and might be even be called a framework ( though I don't think it is in the previously discussed sense of the word ) but imho it models the http protocol in a way that puts me in contact with the underlaying principles of web technology. Have a look at the code and you'll see what I'm talking about.

It feels like real knowledge and control to dabble around close to the protocol and still have a great range of tools at my fingertips, and no framework ball and chain to drag around. It's like I now know both the hammer and the bronze, and why it acts the way it does when I hit it.

I'd like to wrap up with a quote I stumbled across on my journey.

"Making a complicated design work is not a sign you're clever, it is a sign you failed" - Gilad Bracha

enjoy.

Google picasa integration

There seems to be developing a series of integration blogs as I build my new website - http://asbjornenge.com - and the next step in the series will be a picasa integration.

We will be using the gdata python client library to get albums and photos from picasa. There is an egg available on pypi, so the first thing we need to do is to grab that.

Obtaining the required packages - the gdata client library

The simplest way to get the gdata python client is to install it using easy_install.

$ easy_install gdata

That being said, using the easy_install binary directly like this is not the greatest idea. It will install the library globally, and cause clutter with different versions of the library, different versions in collaboration with other library's, different versions for different python versions. A mess...

One easy way to control your environment is to use a buildout based system, like strings, or make sure you isolate the packages required some other way.


Using the gdata client library to get albums and photos

Lets have a look at how we can use the gdata client library to get albums and photos from picasa.

import gdata.photos.service

# Set up the connection
client = gdata.photos.service.PhotosService(email='username@picasa.com',
password='password',
source='username-someapplication.com-1.0')

# Get the albums
albums = client.GetUserFeed(user='username')
photos = {}
for album in albums.entry:
photos[album.title.text] = []
# Get the photos for each album
aphotos = client.GetFeed(
'/data/feed/api/user/%s/albumid/%s?kind=photo' % ('username', album.gphoto_id.text))

# Get information about the photos
for photo in aphotos.entry:
p = {}
p['title'] = photo.title.text
p['url'] = photo.content.src
p['thumb'] = photo.media.thumbnail[2].url
photos[album.title.text].append(p)

# Print the result
print photos


Its pretty self explanatory, so I'll let the code speak for itself. Remember to instantiate the PhotosService only once ( !!not for every request if your building a web app!! ).

For a more detailed example using strings, check out this page.

enjoy.

Google blogger integration

So, a first blog with some relative meaning to it..

Alright, theres a million ways to integrate a blog. But you want a fast and clean code integration right? Preferably using python? Well, here it is...

Google offers a python client library for the Google data API's, witch has a blogger module. There is an egg available over at pypi, and some outdated documentation on the googlecode site.

Obtaining the API wrapper - the gdata python client library

The simplest way to get the gdata python client is to install it using easy_install.

$ easy_install gdata

Using the gdata python client library

First, lets go through the interesting bit - how to use the gdata client to get your blogs.

import gdata.blogger.client

# Get a client and log in
client = gdata.blogger.client.BloggerClient()
client.client_login('googleaccount_username',
'googleaccount_password',
source='someuserorcomp-someapp-1.0',
service='blogger')

# Build the query.
query = gdata.blogger.client.Query(order_by='updated')

# Request the feed.
feed = client.get_posts(self.blog_id, query)

# Get the blogs.
blogs = []
for entry in feed.entry:
blog = {}
blog['title'] = entry.title.text.encode('utf-8')
blog['link'] = entry.GetHtmlLink().href
blog['content'] = entry.content.text.encode('utf-8')
blogs.append(blog)

# Print the results.
print blogs


Its pretty straight forward.

Remember to only do the client_login bit once ( !!not for every request if your building a web app!! ) and to put restraints in the query to limit load and the amount of data transferred.

For a strings example, check out this page.

enjoy.