Recently in Programming Category

Not to long ago I began work on a very cool new theme, or Template Set, for Movable Type. I had built out all the templates and nearly completed all of my work without having to write a single line of Perl code, which was one of those unstated design goals for the project because let's face it, no one should have to write a single line of code, be it Perl, PHP, Python or Ruby, to create a theme for any blog and web site. That is outside the skill set of a typical designer.

But then I got to a point in the project where I wanted to surface configuration options to the user for the theme.

Byrne, meet Wall. Wall, meet Byrne. Smack.

That is when the "unstated design goal" permuted into a new proposal and prototype for how Movable Type can be extended in the future to support 100% perl-less plugins. No kidding. The experimental plugin is called "Movable Type Configuration Assistant."

How it Works

Say you are a designer. You want to build a theme for a web site that someone can just drop in to their blog and have it work. You also want to make the theme configurable while keeping your users from having to know anything about HTML, CSS, PHP or Perl themselves. Just fill out a form and be done with it. Here's the challenge: you don't know how to do that because while you know HTML and CSS like the back of your hand, your knowledge of Perl and/or PHP is some what lacking.

Enter Configuration Assistant.

Configuration Assistant for Movable Type allows you to easily design an entire settings page for your theme or plugin complete with field sets, labels, default values, pull down menus, radio buttons, check boxes, text areas, and text input fields for all the options you wish to expose. You can do this all through a simple config file. Did I say that no perl is required?

The settings a user then saves using this form will automatically be made available to you, the designer, through a set of template tags you choose and that you can use within your templates. Here is a simple example:

Here is an excerpt from a Movable Type config.yaml file that will define two settings for your plugin:

id: MyPluginID
name: My Plugin
blog_config_template: '<mt:PluginConfigForm id="MyPluginID">'
plugin_config:
    MyPluginID:
        fieldset_1:
            feedburner_id:
                type: text
                label: "Feedburner ID"
                hint: "This is the name of your Feedburner feed."
                tag: 'MyPluginFeedburnerID'
            use_feedburner:
                type: checkbox
                label: "Use Feedburner?"
                tag: 'IfFeedburner?'

This is the settings page it creates:

And here is the template code that utilizes the tags it auto-creates for you:

<mt:IfFeedburner>
  My feedburner id is <mt:FeedburnerID>.
<mt:Else>
  Feedburner is disabled!
</mt:IfFeedburner>

Documentation

• Read the complete README.txt file found inside the plugin.
• MTOS Proposal

Status of the Plugin

This plugin is a prototype for something I believe strongly should be in the core of Movable Type. I have built this plugin to serve as a working example for a proposal made to the Movable Type Open Source community.

The plugin needs additional refinement before being fully integrated. Namely:

• this plugin should extend the existing settings registry key as opposed to defining its own plugin_config registry key.
• this plugin should support additional form elements like radio, multiple checkboxes, and more.
• simplify the semantics of the configuration options.
• this plugin needs to be used by users, have its documentation completed, and be more thoroughly tested.
• this plugin needs endorsements and approval from the community.

Download

I need help testing this concept. Please consider downloading it and playing around with iit:

• Download 0.9

Recently I received a comment on a post about modperlite alleging that modphp, an Apache module mod_perlite is modeled after, is falling out of favor with hosting providers. That is quite an allegation - one that I initially shrugged off because it seemed preposterous to me. I mean just consider for a moment all of the incredibly popular applications including WordPress, Drupal, PostNuke, phpBB, Mediawiki and many more too numerous to list here let alone count that all rely on PHP exclusively.

Not wanting to completely discount this comment I decided to consult a close acquaintance of mine that works with one of the most popular hosting providers on the Internet. In working with him on a number of projects I have come to respect him on a number of levels, but especially for his knowledge of operations and system architecture. He spends a great deal of his time every day trying to maintain systems running software he didn't write or install on machines run by users he has little or no control over. He is an expert in cleaning up after others. So I asked him if he had an opinion about this statement.

What I learned through our email dialog was enlightening, and I wanted to share it with a larger audience. He permitted me to republish parts of our correspondence, but asked to remain anonymous. (Editorial note: I have taken some of his text and elaborated as best that I could in order to provide a more complete overview of this issue.)

This is a rather multifaceted issue, but to summarize I'd venture that mass virtual hosting on the whole is trending away from mod_php/mod_perl.

He proceeded to back this claim up:

• Real file I/O is impossible or insecure - it is difficult to harden a mod_perl/mod_php application because these Apache modules only run in the permissions context of the web server's user. Software that helps resolve this issue, like suPHP, which allows a script to run in the permissions context of the script's owner, negate the performance benefits of the underlying mod_perl/mod_php module.

• One of the more viable solutions to this permissions dilemma, an Apache module called mpm_perchild, died on the vine years ago. "This was the Apache 2+ intended answer for the security problem and was supposed to (correctly, IMO, given Apache's architecture) have each request thread out based on file ownership, sort of a rolled in suexec."

• "There is growing adoption of code frameworks (rails, django, catalyst, etc etc) that just don't play with mod_php/perl. Building an effective cgi/fastcgi architecture covers bases better."

• "There is a growing adoption of very powerful web servers like lighttpd and nginx whose only interface is cgi/fastcgi. To many developers today Apache really is an anachronism."

• "Another side effect of cgi/fastcgi that I neglected to mention is that it is much more apparent which users' code/processes are monopolizing resources" thereby making it easier for hosting providers to monitor, enforce resource utilization policies and recover from systems under load.

Then y friend also responded to this comment:

@saj - really? You think mod_php is falling out of favor? That is quite a statement... one that I can't imagine is true. PHP is one of the most ubiquitous web scripting languages, second only to Perl. A cannot imagine hosts degrading their support for this language.

With this response:

I think this is the most important misconception to dispel. In an Apache vhosted setup, modphp *is* degraded support. Settings like safemode are generally implemented and users don't have access to edit their own php.ini, relying on .htaccess hackery to eke out the limited php settings customization available.

He concluded, "there is a ton more to the issue and these points really don't do it full justice."

Now lets be real, I don't think my friend is saying that hosting providers will drop support for PHP, not in the foreseeable future, but if hosting providers tend not to prefer mod_perl and mod_php based applications what will this mean for all us developing on top of these languages and deployment platforms?

And ask yourself, when you write a web based application, are you thinking about the company that is going to be hosting your application for your customers/users? And what are you doing for them to make their lives easier?

I have had a theory for a long time that perl based CGI scripts could be sped up considerably by simply by keeping the interpreter resident in memory. Not only that, but the module that makes that possible wouldn't be a pain in the ass to install because of a key design goal: simplicity, borrowing heavily from a module that I think does a lot right, mod_php.

Well, if you followed the thread you would have noted that Aaron just recently got a version of mod_perlite to work against basic CGI functions. Totally awesome.

Next step: benchmarking. There are some skeptics out there, and those skeptics are a helluva lot smarter then me, so I have to see for myself: does this approach have actual merit?

And if the theory holds water, then I will setup a homepage for the project, get some builds spun up, and start working on documentation. For those of you who expressed interest in helping, by all means check out the latest from subversion and give mod_perlite a whirl.

Over the many years of their coexistence, the terms CGI and Perl have become virtually synonymous. This perception that CGI and Perl are one and the same has contributed to some small degree to the perception that Perl is outdated and an inappropriate language for web programming - unlike its more modern counterparts like Ruby or PHP.

I of course know this to be a complete fallacy. First of all, the company I work for is a devout Perl shop and our products, all written in Perl, are collectively some of the largest and most scalable on the Internet. Few other companies in fact present more widely attended seminars and tutorials about scaling for the Web then ours do at conferences like eTech and OSCON.

But perhaps the more significant fallacy is that CGI and Perl are synonymous. In actuality, they technically have very little to do with one another, and there is technically no reason one should be hampered by the other, despite all evidence to the contrary.

A little history and background

CGI stands for "Common Gateway Interface" and was invented to allow any script, written in any scripting language, to also act as a web based application. In the early days of the Internet this was incredibly helpful to the first Web programmers (a.k.a. system administrators) proficient in sh, bash, csh, tcsh and perl because it allowed them to quickly deploy simple web based automation tools based on scripts and libraries they had already written. Cool.

But the inherent flexibility of language agnosticism is also CGI's greatest liability, and also by association, Perl's as well. You see, CGI is based upon the principle that when the web server receives a request, it does not know what scripting language will interpret that request. Therefore, it defers processing directly to the operative system, and so must do something geeks call "forking and exec'ing" - or in other words, the web server must start up an entirely new process on your server to handle the request. This may not sound like a big deal, but it sometimes can be as each forked process holds the entire Perl interpreter in memory. And that is a big deal. First it can consume a lot of your server's memory, and depending upon the size of your application, it can be slow to initialize. It works - and works well by virtue of working, but it by no means ideal.

More modern languages have been designed to avoid this. Let's take PHP for instance. PHP is a language that was designed for web programming exclusively. Therefore its architects made a critical (perhaps even obvious) decision early on: if the web server is going to handle a lot of PHP scripts, why bother forking a process to determine what scripting language will handle the request (this is expensive) - why not just load the PHP interpreter into memory once and interpret the request within the web server (which is much cheaper)?

So when it comes to CGI vs PHP, it is not really about Perl vs. PHP at all. It is really about understanding two solutions to two different problems - one operating under the assumption that every request will be processed by the same interpreter and the other designed to execute any script via the web.

The solution as it stands today

In the Perl world, there are actually two Apache modules that attempt to do what PHP does inherently: load the Perl interpreter into memory so that you no longer have to spawn a new process each and every time your web server receives a CGI request. Those two modules are mod_perl and mod_fastcgi.

However, these two modules have a critical flaw: they are incredibly complex because they attempt to solve a huge problem set having to do creating a persistent and stateful execution context. The result are two modules that are not only too heavy weight for the average user but also incredibly difficult to install - even for myself.

How to actually fix the problem

The more I have thought of this problem, the more I have come to believe that there is little standing in Perl's way to have all of the benefits that PHP has gained from being a language designed exclusively for the Web.

In theory, one should be able to take the source code of mod_php (the Apache module that dispatches web requests to a PHP interpreter) and swap out the component responsible for dispatching a request to PHP for one that dispatches the request to Perl. In theory it should just work (more or less). The result would be an Apache module that would be easy to install, and be much more efficient in handling and processing requests online.

Granted, this solution would not be stateful and persistent the way mod_perl and mod_fastcgi are, but that is not a problem this solution is engineered to solve.

Introducing mod_perlite

All of this is a really long-winded setup for what is a very quick conclusion.

I shared this hypothesis with an engineer at work, Aaron Stone, who shares a passion for Perl with me, but who also has a passion for operations. He took on this challenge and devoted part of his 20% time to testing this hypothesis.

The output of his work is called mod_perlite. It is largely derivative of PHP and is capable of processing Perl scripts quickly and efficiently. The next step of the project is to make it compatible with the CGI protocol, which can be done by gutting parts mod_cgi and dropping them into mod_perlite.

So far our results are promising, and it is possible that with a little hacking we may have just made Perl faster on the web and easier to deploy for everyone.

If you are interested in helping or participating in this project please let me know -- we could certainly use the help.

Last week I was in New York participating in a Movable Type Hackathon and Blogging Summit I helped to organize. While I wasn't planning on presenting at the summit, fate had another opinion on the matter. In the end I had just a couple of days to come up with a fifty minute presentation about Movable Type Enterprise's Extensible Authentication Framework - a framework on which Movable Type's LDAP, Oracle Single Sign-On and Shibboleth drivers are built.

I guess I work pretty well under pressure because despite a total and complete lack of sleep, I managed to hack a pretty good presentation together. If you don't mind the 156MB download, you can watch it. But for the rest of you - a simple PDF download is also available.