Welcome to diagrammes modernes, a web log all about Python, Javascript, HTML, Blogger hacks and other programming escapades. This is not at all the place you want to be if your desire is to learn about flying sheep. Nor is it in any way affiliated with the French language.

Rather I have decided, in time-honoured Python fashion, to take one of my favourite comedy skits by the Monty Python troupe and riff on that. Be thankful you did not get Spiny Norman.

This content used to be hosted under the Theatre of Noise, but I decided that all the programming material should be in one place. If you wish to be entertained with strange web links, humour, poetry, sound art, photography, design and other artistic escapades, you should go to that site.

"Dinsdale?"

"Dinsdale?"

Excuse me, I must fly.

more

Here's my sneak peak at the upcoming Python 2.5. There are no earth-shattering changes, rather a gentle increase in functionality, extending syntax to reduce code verbosity.

First, you may be pleased to know that the standard library will be enhanced to include "setuptools" and Fredrik Lundh's ElementTree. The new "hashlib" module will provide a superset of "md5" and "sha".

The new absolute import syntax will make it clear whether the module being imported is within the current package.

The "with" statement allows for constructs that have guaranteed exit behaviour. It took me a while to understand this, but basically it's a way of getting around writing lots of try/finally blocks.

A conditional expression is being added in the form "X if C else Y". Python will finally have a ternary! But rather than operator form, this looks like list comprehensions.

Generators will be enhanced to make them usable as coroutines.

There will be a new formulation to allow try-except-finally, saving some further code verbosity.

The exception hierarchy will change so that KeyboardInterrupt, SystemExit and Exception inherit from BaseException. This means that Exception includes only exceptions that signify errors.

Currying is to be provided by a function in the new "functional" module.

Arbitrary objects can be used for slicing, where currently integers are required.

There is a new standard mechanism for adding metadata to Python packages. The Distutils "sdist" command will write metadata fields to a file named PKG-INFO.

For excrutiating details, have a look at PEP 356. There's also a tentative What's New page available.

Expect Python 2.5 to be available in release form by October 2006.

more

Guido has been interviewed by eWeek, and discusses his work at Google and the future of Python. It's not a terribly in-depth article, but contains one big revelation. The mythic Python 3000, which will reformulate some basic concepts and hence break backwards compatibility, may actually be available within two to three years. Guido's position at Google leaves half his time free to work directly on Python. Sounds like a brilliant job!

It's nice to see further mainstream attention being given to Python. This article was picked up by ACM TechNews and no doubt other aggregators.

more

This is it! Part 4! We've made it! Exclamation! After parts one, two, and three, I can finally introduce you to today's top players in the web application arena. We've got nine titles to cover, so let's hop right in and see how they stack up.

This is not going to be a detailed review, because that is impossible. By the time I installed every package and gave them a good workout some would be well on to newer versions, and so the comparison would be stale. And the more details one gets bogged down in the less one sees the important overview. So rather I'll summarise other reviews and some of the voluminous discussions that have occurred on the web of late. All this will be filtered through my particular biases and instincts. Which is why you are reading this blog and not some other¹.

TurboGears is designed to select the best components and bring them together in a well organised manner. As such it packages CherryPy, SQLObject, and Kid, with some bonuses like Mochikit for JavaScript. The upside to this approach is that as TurboGears gains popularity, it helps all of its components along. The downside is that if you don't want to use this combination of tools you'll go somewhere else. And I don't, so I will. But here's someone who does. (Sorry, that was difficult to follow. I need more caffeine.)

Proponents of TurboGears will say that if you don't like CherryPy (for example) you don't have to use it. TurboGears also works with mod_python and lighthttp. CherryPy itself is an example of this approach: it has a default way of working that you don't need to use. For example, sessions works over cookies but you can choose a different method. Classes compile to standalone servers but you can use other (more standard) web connectors. And so on. What some see as flexibility I see as confusion. What is the point of choosing a product designed to integrate "best-of-breed" solutions if you are going to swap them out?

Put another way: I would rather build up from independent components than have to tear down an integrated structure.

Subway also packages CherryPy and SQLObject, this time with Cheetah. This is also not appealing to me. In any case, the project has been terminated.

Django is the biggest "competitor" to TurboGears. It has its own templating system and a large list of functions. The API appears to be designed with little structure, in what has been called the PHP "bag of functions" style. Yuck. Also there's a lot of "clever-clever" stuff going on behind the code, so much so that they have a "magic removal" version due for release soon. Even if they remove some of the fairy dust I am not confident on this product from a design perspective. The developers have gone on record as having no interest in being Pythonic, preferring to attract an audience of non-programmers.

Still, here's someone who likes it.

Pylons operates at a decidedly granular level, as the developer explains. He lists 17 components of a WAF, even more than I came up with in Part 2 of this article. This toolkit approach saves time in tedious integration without constraining the developer to one particular tool for each of the slots. The trick is in making it easy to swap these in and out. Pylons supports FastCGI, SCGI, AJP, or a standalone server. It does not dictate an ORM, though SQLObjects again seems to be the favoured choice.

A good number² of the components are taken from Myghty which has templating based on HTML::Mason, a component-based architecture, caching, threading support, sessions and is WSGI-compliant. This looks sharp to me, and gives further incentive to check out Pylons.

At a similar library level of (non-)integration is Paste which contains a server, authentication, testing, URL mapper, logging, exception handling and once again conforms to WSGI. The docs put me off, I must say. But once again, Pylons takes what is good from this toolkit and integrates it.

Jon's Python modules are a disparate collection including connectors for CGI, mod_python, and FastCGI. Additionally you get web templating, session management and a database connection pool.

James Gardner's webmodules provide an ORM, forms, sessions, error handling, WSGI interface, authentication and template parsing (supporting Cheetah).

Finally, mention must be made of Mod_python itself. This is the well known Apache module that embeds a Python interpreter within the web server. As such, it is supported by almost all the other solutions here. But less obvious is the fact that it includes a library of functions for handling requests, sessions, cookies, authentication... many of the elements of a WAF in fact. The major limitation would be the obvious one that it is specific to Apache, not supporting other web server connections. For this reason I would not choose it.

Let's take a big breath and look at what we have. It's quite a list, and even so doesn't come close to the vast number of lesser-known offerings out there floating in the Python universe.

At the top of the list are slick packages solutions that have a greater degree of integration and hence deserve the label "framework". As we get closer to the bottom the glue becomes less binding and the proper description of the product becomes "library". I think the best place to be is somewhere in the middle, where there is a depth of components but a loose coupling that does not lock you in. Building a website is never a simple process repeated for each client; it's different depending on platform, server, and other architectural constraints. And different also based on the product, market, goals, etc. One needs to be flexible, but re-use as much as possible from one gig to the next.

In comparing these products all I can say for sure is that the market is very much more mature than it was even two years ago. No matter which toolkit you choose you will be saved much development time and hassle. You will be able to focus your time not on repetitive infrastructure but on the particular distinct characteristics of the job.

Python is not like Ruby. It does not have a single good solution. It's got eight good solutions (or more!) all worthy of your attention. No matter which choice you make it's the right one³. Nonetheless, I hope this article has saved you much reading and helped point you in a fruitful direction.

In a follow-on article I'll give you some further references.

Me, I'll be looking at Pylons. I will try to give pointers as I go for those following the same path.


¹ What's that? You say you are reading some other blog? How metaphysical of you!

² If you think that five is a good number. Personally, I think it is just fine.

³ And no, I don't think this is a cop-out. Any developer worthy of the name could put one of these products to work and get up and running in a short time.

more

In Part 1 of this series I gave you a perspective on the evolution of Web Application Frameworks (WAFs) and in Part 2 delineated the types of components they might contain. In this article I will let you know about some of the specific component products that have made it to the top of the foodchain. I will focus on those that are a) popular or b) my choices. No use discussing the rest -- this series is supposed to save you time!

In the following the bold terms match the components discussed in my last article.

Routes is a Python implementation of the Ruby On Rails URL-to-object mapper. It is in use in several of the top products. The blog from the developer, Groovie, is a great source of information and informed opinion. In particular this comparison article is relevent here.

WebStack is a web server connection that I mention here since it supports no less than BaseHTTPRequestHandler, CGI, Jython/Java Servlet API, mod_python, Twisted, Webware and Zope 2.

SQLObject is the most popular object-relational mapper, used by Subway, TurboGears, and Pylons. Despite this, it is not preferred by those who have worked on large-scale systems. This seems to be one of those cases where an easy-to-use product at the entry-level provides barriers to use later one. You know, like PHP.

PyDO and especially SQLAlchemy get the nod instead. This last product supports SQLite, Postgres, MySQL, and Oracle, a standard list. It provides for thread-safe and pooled connections. Notable is the design decision to keep the actual database mapping and the class design separate. In particular, objects can map to joins, subqueries, and unions, not just simple tables. I think I need to shrug off my aversion to ORMs and try it.

Cheetah is likely the most popular Python template language. It provides a host of markup, an approach which I have already explained I do not like. The argument can be made that just because the markup is there doesn't mean you have to use it, but I prefer a tool that encourages best practice. That is one reason I am fond of Python itself.

The new, ahem, Kid on the block is similar, perhaps, though simpler. That's a good thing. It does not allow you to generate output within a code block, a restriction which discourages arbitrarily mixing code and markup. Already I like the look of this fresh-faced youngster!

Kid is not only based on XML, it guarantees well-formed XML output given a valid template. Output is compiled to Python byte-code so it can be used like any other Python module. Used in TurboGears, Kid would also be my choice, if I didn't have an enormous hunger to implement the Wasp templating system one more time. (That is a joke. I think.)

I've got my own session and cookie modules but I'm sure the counterparts that come with most WAFs are at least as good. The same goes for other components I don't specifically mention here.

FormEncode has rapidly become the default way of handling forms. It's in use by Subway, TurboGears, and Pylons. Who am I to argue? Oh, yeah, well, I might argue, except I hate my own way of managing this mess of HTML.

For installation tasks we have a brand new approach, courtesy of Paste Deploy. This module allows the configuration of WSGI applications using the soon-to-be ubiquitous Egg installation format. All it takes is a simple configuration file and your WAF can go from installation package to functioning web server in two commands. It's difficult to comprehend from the developer's terse site, but an explanation by Groovie helps, in case my own attempt was not good enough for you.

Did you notice I threw a new acronym at you? WSGI is a developing Python standard that unpacks to Web Server Gateway Interface. The proposal intends to add a level of conformity to all the WAFs wafting about. It works at about the same level as the DBAPI, which could be considered not strict enough. But it's a decent step in the right direction and will hopefully become "law". There's a bit of a technical intro at... wait for it... yes, you're right, it's at Groovie's site. Sure hope he appreciates all the new traffic I'm sending his way.

Now that we have chosen best-of-breed components we can take a stab at the major frameworks. That's the task ahead of us in Part 4, the conclusion of this little series, which will be coming your way in no time at all.


^† What are you doing reading a footnote that has no reason for existence?

more

The recent debate on application frameworks was galvanised when Python creator and Benevolent Dictator For Life Guido van Rossum (currently employed by Google) asked the community to "Please Teach Me Web Frameworks for Python". It turned out that WAFs are different things to different people, so it's likely best if I start with a bit of a definition.

But before you read that you may want to start with Part 1 of this article, so you can follow my perspective. As incentive I will say that the first part was highly entertaining. This part will not be, although I do promise one pun.

Next, put on your well-worn copy of the Arcade Fire album and sing along while reading. Ready? Then we'll begin.

What is a Web Application Framework (WAF)?

In Python terms it's a bunch of modules for building web apps, modules which may contain one or more components. In order to establish what these are, let's look from the perspective of a typical victim user. Said person is browsing a particular page served up by your application. Let's examine what components will be needed behind the scenes.

First, the user types in the URL. Many WAFs contain an URL to object mapper, which interprets a given address as a method call on a particular class. This means that the web site does not contain a pile of HTML files in a doc folder, but rather a pile of classes in a custom module. Whether you like this idea or not will determine a lot about which WAF you choose.

The request handler wraps around the URL component and provides for the proper interpretation of the HTTP spec: GET, PUT, and so on.

This needs to talk to the server, and for that you need some sort of web server connection. The classic old-school slow-but-sure way is CGI. But there's all sorts of improvements which embed the Python interpreter into the web server. The main advantage of these methods is performance, since the interpreter does not need to be loaded each time a request is made. (Examples include mod_python and FastCGI.)

Since memory space is shared between requests, these methods allow database connection pooling and other enhancements. The downside to embedding the interpreter is that you have to be careful how you code, and need special mechanisms to dynamically load code changes without restarting the server.

On the topic of databases, a WAF may have its own database interface. A special type is proving popular these days, namely an Object-Relational Mapper (ORM). This allows a more Pythonic¹ mode of database access, hiding SQL, cursors and the like behind a familiar sheen of classes, methods, and iterators.

Unless the HTML is totally programme generated (generally not a good way to make friends with your designer) there will be a templating language, designed to extend HTML to allow for dynamic content. Some templates allow for embedding arbitrary Python code directly in the page (me hates). Others supply a limited but powerful menu of custom tags (me likes).

For example, Wasp had tags for including another file, expanding a simple macro, running a custom procedure to embed HTML, and also a simple conditional. I still do not see any advantage to providing more than that. As soon as you get into loops and variables and stuff you are better off in a code module².

Some WAFs do not have their own template markup, but rather a plugin architecture to install someone else's. In this day and age that makes total sense, because this particular wheel does not need reinventing. (I will try not to use the word "interoperability" in this article but I trust you will forgive me if it slips out.)

In order to provide some state to the stateless world of web pages, WAFs might have a session mechanism in combination with, maybe, cookies. At least that's how I did it. On top of this an authentication mechanism fits well. For increased performance (especially if the framework is slow) you may need page caching. I'm betting you don't need it, but your client thinks it makes them sound like one of the big boys. Cache on the line, that's how they like it!³

It goes without saying that forms are a particular pain. You may find your WAF has a special library to make them easier. Other features may include logging, error handling, testing,... the list goes on. (Funny how people always say "the list goes on" in precisely those situations where the list does not go on.)

In fact, there are so many components that this has spawned the need for one more: an installation component to get all the others set up nicely.

To summarise, here are common WAF components:
1. URL mapper
2. request handler
3. web server connection
4. database interface
5. object-relational mapper
6. templating
7. sessions
8. cookies
9. authentication
10. caching
11. forms
12. logging
13. error handling
14. testing
15. installation

Furthermore, there are some features you may see bandied about. There's the aforementioned "connection pooling" and "multi-threading" and "object publishing" and "iron-fortified"⁴.

There's also lots of talk of "middleware" which, near as I can figure, just means one method calling another.

I believe we are now equipped with the domain knowledge and terminology necessary to evaluate specific WAF products. That's what I'll do in the next article. But before then you may want to look over that tall list of features and assign priorities to the items based on what you want to see and how important it is to you. That will help tailor this discussion to your needs.

See you in Part 3.


¹ Pythonic means "behaving like Python", which is to say in a pragmatic object-oriented style free of fluff, fuss, and Freon. Beyond that it is in the eye of the beholder. Though you may want to type "import this" into the Python command interpreter window -- surprise!

² How are you enjoying Arcade Fire?

³ That's my pun for this article. You can now breathe easy. The worst is over.

⁴ OK, I admit it. I made that last one up.

more

If you are trying out Pylons on a Windows computer there are a few "gotchas". And if you are running Python 2.3 there are a few more. After posting requests for help to the friendly Pylons Group I have come up with the following quick instructions. I hope it helps those following my series of articles on Python Web Application Frameworks.

1. Create paster.bat somewhere on your path, containing this line:
python C:\Python24\Scripts\paster %1 %2 %3 %4 %5 %6 %7 %8 %9

2. Update setuptools:
python ez_setup.py -U -D setuptools

3. If you are running Python 2.3, get the subprocess module here.

4. Ensure you have the version of Routes that works with Python 2.3:
easy_install -U -D Routes==dev

5. Install Pylons:
python ez_setup.py Pylons==dev

Installation is done. To setup a test project, do this:
paster create --template=pylons testproject

If this works you are all set and can continue following the Beginners Tutorial

more