Digitalistic

In this blog I have earlier declared my love for both CodeIgniter and Wordpress, but I am very unfaithful when it comes to frameworks and programming tools, so now I am testing out the Python framework Django instead (but I am still a newbie). It’s my experiences with Google App Engine that lead me into the Python/Django world, and when I am now considering what framework to use for an upcoming big project Django is definitly on the shortlist. It will not be a Google App Engine project due to that some of the limitations of that platform makes it a bad fit for the project in question, so it would be my first standalone Django project. So far I have only installed all the stuff I need and got it to play nicely together, but that was a real learning experience in itself, and by sharing it in this post I hope to save some other fellow geek some time and peace of mind.

Before getting started I want to thank Alexis Bellido at ventanazul.com for his patients with my Django questions, check out his post Django questions and answers with a Swedish guy for some more info on Django setup (if you didnt figure it out I am that Swedish guy).

Installation guides
When I set up Django locally I wanted to make my local development environment as similar to a production environment as possible, it will hopefully make production deployment easier down the line. To do this I am running Python 2.5, Django 1.0, PostgreSQL 8.3 (and thus the neccessary python driver psycopg2) and mod_python on Apache. There are some good installation guides out there, so setting up this stuff was mostly a walk in the park, check out the install guide in the Django Book, the quick install guide at Django Project or the install guide at WebMonkey. Of course there are some gotchas that took me quite some time to figure out…

Uninstall old versions of Django
Before getting started I already had Django 0.96 installed, and I  just installed Django 1.0 over it assuming that it was going to replace the old version. I was wrong and when I tried to run my new Django installation I got errors like “NameError: name ‘url’ is not defined” and “ImportError: cannot import name WEEKDAYS_ABBR” - neither which made much sense to me. It turns out that if you do not uninstall an old Django installation first you get a mix of new and old Django files, and that just do not work very well. So take care to acctually read and follow the instructions at Remove any old versions of Django.

Use the right version of the db driver (duh!)
To get the PostgreSQL working correctly with your Python install you have to use a version of the driver Psycopg2 that matches both your Python version and your Apache version, otherwise things just do not work and you do not really get a helpful error message. You can get the window versions of Psycopg2 here.

Configure Apache for mod_python
Once Django and Apache are installed you can create your new Django project (via django-admin.py startproject myNewProject) where Apache can find it (in htdocs for example). To be able to use Apache as your webserver you also need to install and configure mod_python. Installing is straight forward (see the install gudies above for more info), but configuring Apache took me some time. You need to edit httpd.conf and add the following to the end of the file:

<Location "/myNewProject">
SetHandler python-program
PythonHandler django.core.handlers.modpython
SetEnv DJANGO_SETTINGS_MODULE settings
PythonOption django.root /myNewProject
PythonDebug On
PythonPath "['C:\Program Files\Apache Group\Apache2\htdocs\myNewProject'] + sys.path"
</Location>

myNewProject is of course the name of your own Django project.

Hopefully this is of use to someone other than me, if not, then at least I have my notes organised for the next time. If I have missed anything or gotten anything wrong I would very much appreciate your feedback!

Using Google App Engine has great advantages, but there are also serious limitations in the platform that it’s important to be aware of. Not all applications can be implemented and executed with Google App Engine. Google is working hard on removing a lot of these limitations, and they will probably do so eventually, but in the meantime that isn’t really any help if you are writing a web app today.

This post was originally published in swedish on Mashup.se, my blog about swedish mashups and APIs.

Only Python
If you don’t know Python you don’t have choice if you want to use Google App Engine, you just have to learn. It’s not a difficult programming language to learn if you already know how to program. Django, the Python framework that really speeds up writing applications for Google App Engine, is also quite easy to pick up (tip: Use the Google App Engine helper for Django). Before I started with Google App Engine I hadn’t written a single line pf Python and after a few intense days of concentration and coffee consumption I knew Python, Django and Google App Engine quite well.

Most 3rd party python libraries work perfectly on Google App Engine (Django is just one example), but there are limitations. Only libraries that are 100% Python can be used, so if the library has any code in C it can’t be used on Google App Engine. If a librarry has any code that makes a HTTP request or similar it can’t be used on Google App Engine either. All HTTP requests have to be done via Google App Engines URL Fetch API.

Differences between local dev environment and the production environment
One of the advantages with Google App Engine is that there is a local development server that simulates how Google App Engine works when an application is in production. This makes it easy to develop an application locally and then deploy it to the live servers on Google App Engine. As a developer it is important to pay attention to the differences between executing an application on the local server compared to the production environment. It’s no fun to spend time writing code that just don’t work once deployed.

The biggest difference betweent the two environments are that the requests made to 3rd parties work differently. If you have an application that uses the Delicous API it will work fine locally, but once deployed it wont work at all. The reason is that Delicous is blocking all requests from Google App Engine IP-addresses. Same thing is true for the Twitter API due to some HTTP headers that Google App Engine sets (Twitter claims to have fixed this now, haven’t had a chance to test yet). To avoid these problems you need to test your application often in the live environment, especially when you use APIs to call other services.

Datastore
The Google App Engine datastore has some limitations due to being a distributed database. The most obvious limitation is that there isn’t an “OR” operator in GQL (the Datastore version of SQL), but that is easily handled when coding. A more annoying limitations is that it is not possible to create a new entity (data object) in the Datastore via the Google App Engine dashboard unless there already exists an entity of this type, and that there is a real noticable delay between what one can see in the Datastorer in the Dashboard compared to what really is stored in the Datastore. This makes it very hard to really check what data that is stored in the Datastore in any given moment, which makes debugging more difficult.

No scheduled processes
One of the most restrictive limitations with Google App Engine is that there is no way to start a process, other than via a HTTP request. There is no type of recurring scheduled process (like a cron job) and no triggers or hooks to use to start a process when a special event occures. Almost all web applications need som kind of scheduled process to function correctly - to clean up old data, send emails, consolidate statistics or fetch data from an RSS feed once an hour.

The easiest way to get around this limitation is to create a cron job on another server that calls an URL in the Google App Engine application. If you have a lot of visitors you can also perform the background process as part of a user request. For example you could import data from an RSS feed when a user logs in to your application. This will of course make the user experience slow and there is no guarantee that users will perform the action to trigger the background process att the right times.

No matter which solution is implemented one will quickly bump into the next limitation of Google App Engine, that only short processes are permitted.

Only short processes
Something I have learned the hard way is that Google App Engine only is built to handle applications where a users makes a request and quickly gets an answer back. There is no support for a process that  executes for a longer time, and for the time being this limitations seems to be approximatly 9 secondes/process. After that an exception is thrown and the process killed. It does not end there, even if you are nowhere close to use the assigned CPU resources you can quickly use too many resources with long running processes since they have their own unique resource pool. If you use too many resources you application is shut down for 24 hours, and right now there is no way to buy extra resources.

This is a really serious limitation in Google App Engine that it is really difficult to get around. If you need heavy processes it is recommended that you use Amazon EC2 or something similar. To handle (not get around) the limitation you have to handle exceptions in a nice way and use transactions. More about this in William Vambenepe’s very informative post Emulating a long-running process (and a scheduler) in Google App Engine. He has some tips on how to get around this limitation, even if it is not recommended since you risk that your application is shut down.

More limitations
There are more limitations, read  Google App Engine: The good, the bad, and the ugly? for a longer list. Just keep in mind that some of the limitations mentioned in that post already have been addressed by Google.

To summerize you need to really know what limitations there are in Google App Engine before you spend time and energy on developing an application. If the limitations are not a problem then there is a lot to gain by using Google App Engine.

I’ve used dozens of Content Manage Systems, both open source versions like Drupal and Joomla and Enterprise level ones as Microsoft CMS and others. They all have one thing in common - they suck (that is a technical term for “providing less than optimal results”). Either they are too complicated or too simplistic, or in some cases, both at the same time. After some experimentation I have found a CMS that works perfectly for most of my needs, and if it do not fit a project I write my own (as I did for WebHostNinja.com). This perfect(ish) CMS is the blogging platform Wordpress.

At the moment I am using Wordpress as a CMS for several websites. Of course this blog is running on Wordpress, not that suprising really since it is a blog. The main site of blendapps.com is all Wordpress, while the acctual chat application that integrates Meebo chat with Ning social networks, run on the subdomain chat.blendapps.com and is implemented on Google App Engine. For Blendapps.com there is both a seperate blog section and a content section, so it is by no means a pure blog. Mashup.se is also running Wordpress, and there I have tried to have more of a newspaper feel than a pure blog. The purest use of Wordpress as a CMS is probably on restillmexico.se (swedish travel guide to Mexico), that site doesn’t even have a blog.

Why Wordpress?
My requirements for a CMS are simple - it should be quick to implement a new site, it should be easy to maintain a site and it should be easy to expand and customize the site when it grows. A plus is if it runs on LAMP, so I can hack it if neccessary and so I can host it anywhere. There is not really any need to support multiple concurrent users since it is mostly going to be yours truley doing the work. Not rocket science really, but evidently damn hard to implement since most CMSes miss the mark by miles.

Let’s take a look why Wordpress live up to these requirements:

• Quick to implement a new site - setting up Wordpress is a matter of minutes. Get the Wordpress files, set up a database and go. For some extra bonus you can add the plugins you need and choose one of 100’s of free well done themes to get the look you want. You can of course also do as I do and get a professional design and just add that on top of an exising theme.
• Easy to maintain - the Wordpress control panel is easy to use, and if you dont like it you can change it around with plugins (see below for some such plugins). Wordpress and most plugins are continously updated and to install updates is a breeze. All you need - such as static pages, blog posts, tagging, categories, multiple users, comments, spam protection (Akismet) etc is all part of the basic Wordpress install.
• Easy to expand and customize - again, the plugins are really great for customization. Also, Wordpress have really powerful and usefull template tags that allows you to customize the theme well beyond recognition. See tips and tricks below for more details.
• LAMP - check. All written in PHP and uses a MySQL database perfectly. Even if I at times have hacked the Wordpress core files it is not really necessary, but the key thing is that it is possible.

Pages and Custom Fields are your friends
Content in Wordpress can be either shown in “posts” or “pages”, where posts are blog posts and pages are static content. Under the surface it is all the same, both posts and pages are stored in the same tables, but they are still treated a bit differently. If you are using Wordpress as a CMS then pages are definitly your friends. I have found that pages are a little easier to organise if you are not writing a blog. You can set a page as the front page of your site in the Wordpress control panel. Go into settings -> reading and choose what page you want as front page.

Custom Fields are data fields that are not part of the core Wordpress installation but that you define yourself. A custom field is attached to a post or a page and can be used to store any extra info that Wordpress or plugins dont handle. For restillmexico.se I use it to store the URL or the images for each page, as well as the attribution of the images etc (a trick from WordPress Custom Fields: Adding Images To Posts).

Plugins to turn Wordpress into a CMS
Even if a basic Wordpress installation works fine as a CMS there are a few plugins that makes your life easier:

• CMS-like admin menu -  all it does is make posts second to pages in the menus, ie puts pages before posts. If all you have on your site is pages then this plugin is a nice touch.
• PageMash - drag and drop interface to order pages. Makes it very easy to change order as well as set parent/child relationship among pages. I especially use this one heavily to organise the hierarchy at restillmexico.se.
• Page Excerpt - adds the excerpt functionality of posts to pages
• Query Child Of - makes it possible to list the children of a page in your theme, great in combination with PageMash mentioned above
• Admin drop down menu - it gives the control panel a drop down menu, which is quicker and easier to use than the standard control panel menu. Not really unique for using Wordpress as a CMS, but I really like this plugin.

Tips and tricks
To hack a wordpress theme to work as well as a CMS there are a few tips and tricks. The first tip is to not hack at all, just use the great CMS2 theme. It already has a page focus and most of the organisation needed for a static web site. For the sites I have on Wordpress I have either customized this theme or one of the themes that comes with the basic Wordpress install.

When customizing a theme there are some Wordpress template tags that will save you a lot of time:

• wp_list_pages - allows you to list the pages you have in your site, pefect to create a menu with. You can include or exclude any pages you want.
• query_posts - get the posts you want by using the parameters to this tag. Since pages and posts really are the same thing this tag can be used to list pages when wp_list_pages doesn’t do exactly what you need.
• get_post_meta - get the value of a specific custom field for a page or a post, necessary to be able to get the most out of custom fields.

Summary
Wordpress is a very good option as a CMS, not just for blogs but for static sites as well. With the great choice in plugins and the open source code of Wordpress you can get it to do almost anything you want without serious hacking. The huge installbase ensures that you can find help and resources if you need it.

Last week I finally relaunched mashup.se. I have had that domain for a couple of years now and started my blogging there in 2006. But I was never happy with the content and the direction it took, in the end it became nothing more than general tech news and I lost interest. Now, at the relaunch, it is much more focused. The idea is to write about mashups and APIs from a swedish angle (yes, the site is all in Swedish, but thanks to Google Translate that is no problem, click here for the English version). There are so many mashups and APIs, but most of them are from the US and have very US specific content. An API that only produces content about the US in English is of very little help to a developer writing apps for the Swedish market.

Hopefully I can create a good collection of swedish mashups and APIs and at the same time get to know more what is going on in Sweden. Since I am not only going to write about specifically Swedish stuff, but also US APIs that are of interest to Swedes, I think that it will also help me write more posts for digitalistic.com. After all a mashup built with Yahoo’s BOSS mashup framwork is of interests to geeks everywhere (at least if you are the kind of geek I am).

The 3G iPhone was released a few weeks ago, so it’s time to write a page or application for the iPhone and try to ride the hype wave for all it is worth. So I started working on an iPhone specific page for enlatele.com.mx - after all why shouldn’t Mexicans be able to find their TV-schedule on their iPhone? You can see the finished result on iphone.enlatele.com.mx. Since all I want to do is show web based content in the iPhone there is no need to start using the iPhone API and build an actual application, so instead I built a web page aimed specifically at the iPhone, and it was suprisingly easy!

Some general lessons I learned before we jump into the actual code:

• iPhone pages are all about lists, the more lists the better apparently. On the smaller screen you can not really do much more than lists, and if you want to do more then just let the iPhone user browse your normal web page.
• Write a seperate page for the iPhone, do not detect if a site visitor is using the iPhone and automatically server up the iPhone page. One of the cool things about the iPhone is that it is possible to interact with normal web pages quite well on a small screen, dont take this away from the iPhone user. Instead create a subdomain specifically for this, for example iphone.enlatele.com.mx, or m.enlatele.com.mx.
• The iPhone connection is not always the best, so do not serve lots of images etc if there is not a real need for it. This might not be true with the new 3G iPhone in the middle of New York or Stockholm, but writing an iPhone page for Mexico I’d rather be safe than sorry and keep the number of images to a minimum.

I used just HTML, CSS and JavaScript, packaged up very nicely in the IUI framework. Basically all you do is download a CSS file and a JavaScript file, import them into your HTML page and write lists with certain CSS classes and IUI takes care of all the rest. IUI even has minimized versions of the CSS and JavaScript file you can use to make the page even smaller (and thus a quicker download to the iPhone). This is what I have in the header of my page:

<meta name="viewport" content="width=320; initial-scale=1.0; maximum-scale=1.0; user-scalable=0;"/>
<style type="text/css" media="screen">@import "iui/iuix.css";</style>
<style type="text/css" media="screen">@import "styles.css";</style>

The meta tag “viewport” is needed to let the Safari browser on the iPhone know how to deal with the content, without that tag you risk that the page acctually looks worse in the iPhone than in a regular browser.

In the body of my HTML document I specify the menus of my iPhone, ie lists lists and more lists. The code for the start menu is:

<ul id="home" title="Enlatele.com.mx" selected="true">
<li class="group">Canales</li>
<li><a href="channelgroup.html?=canales_abc">A-B-C</a></li>
<li><a href="channelgroup.html?=canales_def">D-E-F</a></li>
<li><a href="channelgroup.html?=canales_ghi">G-H-I</a></li>
<li><a href="channelgroup.html?=canales_jkl">J-K-L</a></li>
<li><a href="channelgroup.html?=canales_pqr">P-Q-R</a></li>
<li><a href="channelgroup.html?=canales_stu">S-T-U</a></li>
<li><a href="channelgroup.html?=canales_vwxyz">V-W-Y-X-Y-Z</a></li>
<li class="group">Proveedores</li>
<li><a href="#sky">Sky</a></li>
<li><a href="#cablevision">Cablevision</a></li>
<li><a href="#mastv">MASTV</a></li>
<li><a href="#tvabierta">TV Abierta</a></li>
</ul>

And this results in this page (as can be seen on iphone.enlatele.com.mx):

IUI styles it all nicely. The key parts of this list is:

• the id “home” on the ul tag. Ids like this are used throught IUI to identify submenus etc. For example the link to Sky is <a href="#sky">Sky</a> this links to another ul list on the same page with the id “sky” - <ul id="sky" title="Sky">. This means that I can have most of my menu structure, submenus and all, on one page. So once this page is downloaded to the iPhone the phone do not need to access the internet just because you click on a link to a submenu.
• title="Enlatele.com.mx" is the header of the page
• selected="true" on the ul means that this is the first menu to load.
• <li class="group">Canales</li> creates a grouping in the menu, a nice way to make things look fancy without acctually doing any work
• now to a really nice thing about IUI, it takes care of loading content dynamically into your current page seamlessly. For the enlatele iPhone page there are listings of TV channels that are not loaded to the iPhone unless the user clicks on a link in the menu to do this. So if a user wants a list of all channels that starts with A, B or C he would click on the following link in the menu - <li><a href="channelgroup.html?=canales_def">D-E-F</a></li>. IUI would then insert the content of channelgroup.html into the page basically using AJAX (don’t worry about the ?=canales_def - that is just so that the enlatele backend knows what to load). The channelgroup.html page should _only_ contain a list of channels to show once this link is clicked, nothing else:

<ul id="canales_abc" title="Canales A-B-C">
<li><a href="channel.html?canal=a_e_mx_west">A&E MX West</a></li>
<li><a href="channel.html?canal=american_network">American Network</a></li>
<li><a href="channel.html?canal=animal_planet_latinam">Animal Planet LatinAm</a></li>
<li><a href="channel.html?canal=animax">Animax</a></li>
</ul>
So no HTML start tags or BODY tags or anything else, just the list you see above, that is the whole of channelgroup.html.

IUI also deals with other things you really want on an iPhone page, such as automatically inserting a back button (that only looks good in Safari since it is using Safari specific functions, so if it looks crappy when you test in Firefox do not worry), showing a nice loading graphic etc. There are also a lot of more fancy stuff that can be done with IUI, but just the basic can get you a long way. This is pretty cool stuff I think, and it makes is really quick to implement something that works great and looks great on the iPhone. Of course there is a big part missing here, and that is the backend acctually filling these pages with content (in the enlatele case dynamically creating the channel list based on what is in the enlatele database), but that is standard programming stuff so I wont cover it here.

If you are interested in doing more mobile content, especially design, check out Yahoo’s Design Stencil Kit for mobile. It contains all the building blocks you need to do mockups and designs for the iPhone and other mobile platforms.