Implementing Ping Back for Movable Type

Posted by Byrne, 1 Feb 2008

Years ago Ben Trott introduced one of the first seminal technologies and protocols that would shape the very nature of blogging and our industry today. Subsequently, WordPress released an alternate Link Back protocol, the umbrella term to refer to all of these notification schemes.

The release of a second notification protocol spurred some mild debate among protocol junkies about which one is better. Let's review the general state of affairs:

TrackBack is pretty broken as it stands today. The discovery mechanism is god-awful, and the protocol is prone to exploitation and spam. Generally speaking, given the percentage of TrackBacks that are spam, it makes me wonder why bother at all. The spec is so bad for example that WordPress claims it supports TrackBack, but blatantly doesn't even implement it as written.
PingBack does not have the breadth of adoption that TrackBack has (at least when TrackBack was first released), but it is moot given the sheer market force of WordPress. It is in some respects more secure, but not fundamentally, just consequentially. That too doesn't matter because the net result is the same: there is less PingBack spam then TrackBack spam.

So in this week's hackathon I thought I would take a stab at implementing PingBack for Movable Type. I am not done yet, but there is code available and it does technically work.

Up until now, all of my opinions on PingBack were merely philosophical, but now that I have an implementation under my belt I have a deeper more informed position on the pros and cons of the protocol.

XML-RPC - the fact that PingBack utilizes XML-RPC is most likely the reason why the protocol is marginally less spammy than TrackBack, because it raises the bar for spammers and eliminates random HTTP POST spam from the equation. But here is the rub - because PingBack has no authentication layer associated with it, it is still fundamentally as insecure as TrackBack.
Weak Requirements - The idea behind PingBack is more reliable stems from the requirement that a server ("A") should confirm that the client ("B") indeed has a valid link from B to A. However, the requirement, as stated, is actually:

The server MAY attempt to fetch the source URI to verify that the source does indeed link to the target.

Something I learned from Brad Fitzpatrick when helping to shape the YADIS protocol is that any specification that includes a MAY requirement, might as well not stipulate the requirement at all because you are signaling to the developer that the requirement is not important. Even though authors of the PingBack specification cannot control the implementations, they should at least put a stake in the ground and say - if you want to be compliant with this spec, then you MUST take steps to ensure the validity of the ping. As it stands now, that step is optional.
Insufficient Data Exchange - strictly from a developer perspective the fact that the protocol only calls for two pieces of data to be sent from a client to a server, the source URL and target URL, makes the process of implementation a little frustrating. With so little data being exchanged, it is incumbent upon the developer to extract any meaningful data at all from the source URL. There is no transmission of an excerpt, an author, a source title, or anything. I must determine all that for myself.

Now in theory this is not so bothersome, because in theory that is all the data you should need. But in practice the fact that there is no deterministic, specified or even recommended way for me to do so, makes the process of implementation bothersome. What PingBack needs is a binding to Atom, or a recommendation that states a source URI should publish an XML representation of the originating resource to make it easier for a server to extract an excerpt, author name, post title, etc. Then again perhaps rewriting a spec to work around my laziness isn't always the best approach. :)

Does it matter?

Actually no. My opinions actually have no bearing on whether or not I will implement or how I will implement PingBack. Because the reality is that with so many WordPress blogs out there, it would be a disservice to Movable Type users not to.

Plus, let's face it: noone is going to change the PingBack protocol simply because I have some issues with it. C'est la vie.

Which leaves us with...

Where to go from here?

What this process has left me with is a renewed appreciation for how important notifications are in the world of online communications. Ben Trott invented something fundamental to modern communications and the fabric of interconnected online content, and just because the current set of specifications all kind of suck in their own way, does not lessen that importance.

Therefore, the industry really should reinvest in these protocols in to improve security, reduce spam, and broaden adoption to more communications tools.

I don't know what the solution is, but my growing belief is that neither TrackBack or PingBack is adequate anymore. I am confident however that there is no need to waste time in the politics of inventing a new standard when we already have one that is more then sufficient: Atom).

So I wonder, does Atom have a role to play in enabling publishing systems to help communicate and establish links between one another?

Filed in Blogging, Movable Type, Open Source, Web Services and tagged apis, movable type, mtos, notifications, pingback, plugins, protocols, trackback, wordpress

11 Comments

Byrne

A very interesting and educational post. You've managed to explain a couple of things that I never fully understood until now!

It's also refreshing to see how open and transparent your attitude is towards supporting "bits and pieces" that one of your competitors "offers" even if it is a bit "broken"

I'll try to test the new plugin later this evening if I can

Thanks

Michele

blacknight at 2008-02-07T11:36:28-08:00

What is lost (or has to be worked around for other blogging software) in the way WordPress implements TrackBack?

Lloyd Budd at 2008-02-10T12:32:06-08:00

A conversation is evolving on LLoyd Budd's website about this article...

Link to article

Byrne at 2008-02-11T12:18:15-08:00

Lloyd, thank you for dropping by. I have posted a follow up comment on your post, but thought I would repost it here, just to keep my readers in the loop as well:

Originally posted at A Fool's Wisdom

@Lloyd - Wordpress’ support of TrackBack is tricky from what I can tell. I am certain the server infrastructure supports it but no all themes follow the spec in regards to autodiscovery, resulting in clients wishing to send ping to be SOL. So “support” seems to me, but I could be wrong, theme-dependent.

This is what I have observed - some themes in WordPress make no mention of TrackBack at all. No big deal IMHO. Other themes though utilize a discovery mechanism that uses a rel=”trackback” attribute on an anchor tag encapsulating the TrackBack endpoint, like this:

Send TrackBacks to <a rel="trackback"
  href="http://…/tb.php">http://…/tb.php</a>

I am sure there are other permutations, but I have not really been looking for them.

The proper mechanism is ugly and involves embedding an RDF document inside your HTML like so:

<p>My post content here.</p>
<!--
<rdf>….</rdf>
-->

Ugly - I know. But technically the only valid discovery mechanism.

Regardless of TB’s viability, we really should push to update the protocol as outlined on the TrackBack wiki.

Byrne at 2008-02-11T13:44:25-08:00

WP has both trackback and pingback built in. At least in the 2.3.3. version that I use. I've never even had to think about all this jazz. It just works.

Val at 2008-02-11T21:29:59-08:00

As it should. A very interesting revelation though about WordPress is that TrackBack doesn't always work. At least it is not all automagical and whatnot. I would be curious to know Val, what is the theme you are using?

Byrne at 2008-02-12T11:08:00-08:00

I've been having a conversation spread across a few blog posts where we discussed whether pingbacks / trackbacks were even useful.

We all agreed that by and large the information that ends up on the target site often just gets in the way and that really we needed something more, like you suggest, to pass meaningful information about the source post so that it didn't interrupt the conversation.

Andrew at 2008-02-22T09:05:40-08:00

I couldn't agree more. There is too much of a burden on the developer to discern the data relevant in between two parties pinging one another. The end result of these exchanges is often relatively meaningless beyond the link itself. I believe it has evolved that way because TrackBacks and Pingbacks are mostly for the ego of the author/publisher, "look at me, someone linked to me!"

What we really should do is approach this from a reader perspective and ask, what kind of data would be most valuable to the reader of a blog? How should a post quoting or linking to another be presented remotely?

Byrne at 2008-02-27T21:00:23-08:00

http://www.commentback.org/ may be an opportunity to improve the technical approach for this style of problem solution.

Lloyd Budd at 2008-05-13T20:17:06-08:00

Hi!

I had some problems implementing pingback at my movable type installation (MT 4.21). Here it is the changes I did:

/path/to/mt/plugins/PingBack/mt-pingback.cgi add the line that start with BEGIN
```
#!/usr/bin/perl -w
BEGIN { $ENV{'MT_HOME'} = '/path/to/mt/'; }
```

If work with Askinet spam detection, some how the thinks messes up. To fix this, add at line 213 in /var/www/mtos/rentalios/plugins/PingBack/lib/MT/PingbackServer.pm

$ping->junk_status(-1);

It should looks like

[...]
   if ( !$ping ) {
        $ping ||= MT::TBPing->new;
        $ping->blog_id( $tb->blog_id );
        $ping->tb_id($tb_id);
        $ping->source_url($sourceURI);
        $ping->ip( $ip || '' );
        $ping->junk_status(0);
        $ping->visible(1);
    }
    $ping->junk_status(-1);
    my ($blog_name, $entry_title, $excerpt) = _extract_excerpt($sourceURI,$targetURI,$source_html);

    $ping->title($entry_title);
    $ping->blog_name($blog_name);
    $ping->excerpt($excerpt);
[...]

pingbacks should be appear as spam trackbacks. You just publish it (I ll work to fix this)

You need to install a perl module. I did it using cpan shell, you can do it with yum, apt ... If you didn't configure cpan shell yet, just trust all default questions (they will work on most installations)
```
perl -MCPAN -e 'shell'
cpan[1]> install HTML::RelExtor
```
You can test the pingback stuff using the file stored on /path/to/mt/plugins/PingBack/tools/pingtest.pl but you need to fix it: Search the line:
```
my $call = $jira->call("pingback.pingback",
```
and replace by:
```
my $call = $jira->call("pingback.ping",
```

I think that's all!!! I recommend fix this and write some installation and testing instructions, as far as wordpress is often used, pingback must be working at any blog.

Ruben at 2008-11-17T04:37:53-08:00

Thanks for your contributions. I have folded most of them into the core and released version 1.01. Check out my projects page to find a link to the new distro.

Byrne at 2008-11-21T08:02:53-08:00

majordojo