saturday, 5 february 2011

posted at 09:13

Hmm, I don't write here much anymore. As is the case for lots of people, my blogging has suffered in favour of spewing random crap into Twitter, Facebook and elsewhere. I'm actually doing a lot more stuff via "social" sites in general, so I think a redesign of this site might be necessary soon to bring all that stuff into one place so I don't look dormant to anyone that just follows here.

Anyway. Christmas and holidays have come and gone. I spent a good amount of time on the big idea that I mentioned last time, and got it close to finished, but then in performance tests found that that the naive old-school Ajax implementation I'd done wouldn't scale much past 30 users. Thats unacceptable, so I started to read up WebSockets, Comet and other things to reduce network and processing on a web application. I settled on using Tatsumaki to implement a long-polling version, but that meant a rewrite of much of the server and the client. At this point I was well and truly on holiday and my brain had shut off, so I threw the project on the backburner.

This can be a dangerous thing for me, because I inevitably change my mind and do something else. I started watching Babylon 5, a show I'd somehow missed back in the day. Anyone that's read for a while knows my long infatuation with space games, so of course I started looking around for something spacey to do or play. And last week, I found Pioneer.

Pioneer is a clone of Frontier, the sequel to Elite. I always much preferred it to Elite. I think it was mostly because of the scale - I could fly forever, land on planets, and just bum around in my spaceship. So I grabbed the code, built it and had a play. And I got sold on it quickly because its awesome, but has a giant stack of things that need work still. In the spirit of the new "just do it" style I'm trying to live up to, I decided first that I wanted to hack on it and then started playing and figuring out what I wanted to hack on. After a couple of hours play I found a heap of little bugs and tweaks that needed fixing, and because the main coder is an awesome guy, lots of my stuff has already been merged.

Not much else to write. This is mostly a "here's what I'm up to" kind of post, so now you know. If you like space games do grab Pioneer (and pull my integration branch if you just want to see what I'm doing). Cheers :)

saturday, 6 november 2010

posted at 16:48

I've recently picked up maintainer duties for Net::OpenID::Consumer and Net::OpenID::Server which have needed some love for a while. I'm starting to get them into shape and have today released developer versions to CPAN. If you're using these modules to implement an OpenID provider or relying party, then I would reatly appreciate you taking the new versions out for a spin to make sure nothing breaks.

All the details are in this post to the openid-perl list: Net-OpenID-* 1.030099_001 now on CPAN

friday, 6 august 2010

posted at 19:23

Disclaimer: I work for Monash University, but I don't speak for them. Anything I write here is my own opinion and shouldn't be relied upon for anything.

So Google Wave has been put out to pasture. That makes it a good time for me to write a bit about what I've been working on in the last few months and what I think about the whole thing.

For those that don't know, I work for Monash University as a web developer and Google Apps specialist. We've spent the last ten months rolling out the Google Apps suite to our staff and students. We completed our rollout to some ~150K students last month, and so far have about 10% of our staff across. A big part of the reason we've been able to move that fast is that for the most part, people are extremely excited about Google technologies and how they might use them for education. That excitement goes to the highest levels of the University (one of our Deputy Vice-Chancellors was the first production user to go) and has seen Google Apps being included in our recently-announced strategy for online and technology-assisted learning and teaching, the Virtual Learning Environment.

The interest from our users in Google extends beyond the Apps suite of products to pretty much every product that Google offers, and perhaps none more so than Wave. Through the eEducation centre Monash has already been doing a lot of research into how teachers and students can teach and learn from each other (instead of the traditional top-down lecture style) and how technology can assist with that. Groups sharing and building information together is really what Wave excels at, so it wasn't long before we started seriously considering whether or not Wave was something we could deploy to all of our users.

There were three main issues that needed to be addressed before this could happen:

  • Wave doesn't have enough features to allow a lecturer or tutor to control access and guide the conversation flow.
  • The sharing model opens some potential legal issues surrounding exposure of confidential information, particularly to third-party robots.
  • The Wave UI does not meet the stringent accessibility requirements that University services must meet

Over the last few months we've been working with the Google Wave team to address these issues.

The first is simply a case of the Wave team writing more code. Its well known that they have been thinking and working on access control stuff. Plans exist for limiting access to a wave to a set group of users, allowing robots to better manage the participants in the wave, locking the conversation root down so that users can only reply, and so on. In many ways its the easiest thing to fix, and given the commitment from the Wave team to talk to us and do something to help with what we needed we were never particularly concerned about this stuff.

I won't comment much on the legal side of things, mostly because I don't understand most of it. I do know that its a serious issue (eg Victorian privacy law is perhaps the strictest in the world) but its something that our solicitors have been working on and it probably would have come out ok in the end, if for no other reason than if it didn't people would just use the public Wave service with no protection at all. Users are notoriously bad at looking after themselves :)

The accessibility issues are where my interest in Wave came from so I'll spend a little time there.

I'll be the first to admit that I don't really get accessibility. I am in the happy position of having my whole body working as designed and to my knowledge all my close friends and family are the same, so I really have very little exposure to the needs of those who are perhaps not so fortunate. What I do understand though is that its critically important that information be available to everyone equally and achieving that is far more complicated than the old tired lines of "add alt attributes to your images" and "don't use Javascript". So I'm very happy to follow the lead of those who do know what they're talking about.

Not far away from me in my building we have a wonderfully competent team of usability and accessibility experts. They were asked to do an accessibility review of the Wave client and perhaps not surprisingly, it failed hard. Most of it comes from the difficulty of expressing to assistive technologies (eg screen readers) that something in a page has changed, particularly with proper context. The Wave client builds a complex UI on the fly (eg as the wave is incrementally loaded) and of course has realtime updates. At a more basic level though the static parts of the interface are constructed without using semantically-correct markup. A user agent (eg a screen reader) that scans the page looking for interesting things like links pretty much comes up with nothing.

The accessibility team presented their findings to some people from the Wave team and the response from where I sat appeared to be equal parts of surprise and dismay. They were receptive to the issues raised though. I travelled to Sydney shortly afterwards for DevFest and had the opportunity to chat to some of the team and they all had seen or heard of the report, so it would appear that it was taken seriously.

For me though, I could see that this had the potential to be a real showstopper to our deployment and I didn't want that as I could see the potential for Wave to be a game-changer. Since at the time I knew very little about accessibility, I started work on answering more technical but somewhat related question: "can Wave work without Javascript?". The Wave team had just released a data access API so I set to work trying to build a client using it. That work grew into the (still unfinished) ripple which more or less answers the question in the affirmative. This type of client doesn't solve the accessibility issues but its definitely a step in the right direction.

The part of ripple that I'm most proud of is the renderer. Rendering a wave well is actually quite a complicated prospect. Styles and links are represented as ranges over the static textual content. Its possible for these ranges to overlap it complex ways that make it difficult to produce semantically-correct HTML. It took three rewrites to get it there, and there's still a couple of little nits that I would have addressed sometime if this code had a future, but I mostly got there and I was happy with it :)

Anyway, these problems were being addressed, a few areas around the university started doing research and small pilots usng Wave, and it all seemed to be only a matter of time. I started work on a robot/data API client library for Perl for two reasons, one being that ripple really needed its server comms stuff abstracted properly and two being that we're a Perl shop and we would soon want to host our own robots and integrate them properly into our environment.

This was a great opportunity for me to learn Moose and my suspicions have been confirmed - Moose is awesome and I'll use it for pretty much everything I do with Perl moving forward. A few of weeks later and we get to Wednesday night and I've got things to the point where you could have a nice conversation with a Perl robot. And then I got up Thursday morning and heard that Wave was going away and all my code just got obsoleted.

I was shocked initially, but I was surprised that I didn't feel angry or sad or anything. I can hardly call the time I spent on it a waste as I learned so much (Moose, tricky HTML, accessibility, operational transform) and met some incredibly smart an awesome people, both at Monash, at Google, and elsewhere. I think for the most part though I was ok with it because its probably the right decision.

We (as in the Wave users and developers everywhere) have been playing with Wave for over a year, and we still don't know what it is and what its for. Unless you have the ability to build your own extensions it doesn't really do much for you. The interface is painful, the concepts within don't match anything else we're used to and despite various mechanisms for exposing internal data, you're still pretty much confined to the Wave client if you want to get anything useful done.

The technical issues would have been addressed with time. We would have gotten enough functionality to write a full-blown replacement client. It would have gotten much easier to expose not only data but the structure of data in other applications. But if you take that to its conclusion, Wave becomes a backing store for whatever frontend applications you build on top of it.

But what of the interface? By having lots of different ways to structure and manipulate data Wave tries to let you focus on the task at hand rather than the structure of the data. Traditional applications (web-based or desktop) are tailored to their own specific data models, so we have seperate apps for email, calendars, spreadsheets, etc. Wave wanted to pull all that information together so you could work on all the pieces of your puzzle in the same space. You start a document then realise you need some help, so you bring in friends and talk about the doc as you build it together. You need to process some data, so you drag in a spreadsheet gadget. You embed images or videos or whatever else you need to add to the discussion. Robots can help out by mining external databases and manipulating in-wave data to present even more rich information and even allow feedback via forms. Its all a nice idea, but how do you represent the different kinds of data and structure effectively? Wave tried, and we tried, but I'm not convinced anyone really had a clear idea of how to build an interface that makes sense.

It might not have been an interface issue. It might be that people want to have seperate loosely-integrated applications, one for each of the different types of data they want to manipulate. I don't think thats the case, but I think that a clearer migration path from those other applications would have helped a lot. People first came to Wave wanting to do their email in it. What if from the outset they could have easily pulled mail into Wave and if there was a "mail mode" that allowed some manipulation of Wave data in a way that they were familiar with? What about doing similar things for other data types? I'm don't know how much difference that sort of thing would have made, but something, anything to answer the "what do I do with this" question that everyone had that the start couldn't have hurt.

Wave's failure may also just be a problem of timing and circumstance. The Wave team have regularly acknowledged that they were surprised by the response. The message was supposed to be "we made something different, what do you think?". Unfortunately it was painted in the tech media as an "email killer", which of course it wasn't, but of course that's going to get everyone interested. Being such an early preview Wave was naturally buggy and slow and couldn't accomodate the load caused by the droves of users that wanted to play. So you got swarms of people banging down the door to see what all the fuss is about, and the few that got in found that it wasn't what they'd been led to believe it was and none of their friends could get in so they couldn't try it for what it was. So naturally they disappeared, disappointed, and even later when the bugs were fixed and the system was stable the first impression stuck and those users couldn't be lured back. And although there was a bit of a second wind a couple of months ago after I/O 2010, the same "what to do I now?" question came up.

From what I've seen of Google in the past, they're willing to take a risks if they see a likely or even possible positive outcome. But looking at Wave, how much future did it really have? We loved it, and we saw that it could do things better than existing services (though with some effort), but was it really going to displace them for the casual user? Was it going to make any serious money for Google? Was it ever even going to break even (remember that it takes plenty of infrastructure and manpower to develop and maintain things like this).

Based on all of this, you can totally understand an executive saying "guys, I see what you're trying to do, and thanks for trying, but the numbers just don't add up". Its not like its been a complete waste - there's some awesome technology thats already finding its way into other Google applications (eg Docs now has live typing just like Wave).

So is Wave dead? The product is, but as a concept it lives on. We're fortunate that Google and others have given us plenty of docs and code and their pledge to open-source everything remains. Then there's the third-party protocol implementations that already exist, both open-source (eg PyGoWave, Ruby on Sails) and commercial (eg Novell Pulse, SAP StreamWork). It will take some work, but any one of us could build and deploy another Wave. The question is, would you want to? I think its more likely that we'll see people incorporating bits of the technologies and concepts into new products. And maybe, just maybe, in a few years time some of the work that Wave pioneered will be commonplace and people will be amazed and we'll be those old curmudgeons saying "eh, Wave did that years ago".

So for Monash, we'll continue working on our existing plans. We've mostly been looking at Wave as a delivery platform for what we wanted to do. Not having it available means we'll have to look elsewhere for the technology we need (whether thats buying or building), but our direction won't change.

And for me? I won't continue work on ripple and Google::Wave::Robot code but they'll live on in GitHub should anyone want to rip them off for anything. My next project is building an OpenSocial container in Perl with a view to integrating it into the Monash portal (my.monash, which is where my "Web Developer" duties lie); hopefully I'll write something about it! I will however be hanging around Wave until the bitter end and I would like to do something with operational transforms in the future as they look really cool and interesting. See, its not dead, really!

And to any Wave team reading this, thanks guys. You've kept my interest and my enthusiasm alive, you've put up with my incessant questioning and harassment and you've contributed more good ideas and happiness to me and my colleagues than you're probably aware of. For the few of you that I've met and worked with already, I really hope that this isn't the end and that we get to work together in the future. I'll probably stalk you for a while to see where you end up because frankly, people are far more interesting than technology and you've all proven yourselves. Cheers :)

saturday, 12 june 2010

posted at 21:32

Today I declared myself officially on the Google Wave bandwagon when I released a tiny Wave client called ripple. I wrote it to see if it would be possible to make Wave work in only HTML, something we may soon want for work if we're to provide an accessible alternative interface to Wave for our users.

From what I'm hearing from the Wave crew, this is also the first example of doing something significant with Wave in Perl. That's exciting.

There's more detail on the splash page, but here's some quicky links to get you started:

friday, 14 may 2010

posted at 09:28

Another little program from my toolbox. This one I'm quite proud of. Its a tiny little file transfer tool I call otfile (that is, one time file).

The idea is this. Quite often I need to send a file to someone on the work network. These can vary from small data files or images to multiple gigabytes of raw data or confidential documents. Our network is fast and the network and servers themselves are considered secure so I don't have to worry about eavesdropping, but there's a real problem with the transport mechanisms - they all suck.

I can put the file in an email, but there are transmission and storage size restrictions. Its also fiddly - create message, attach file, send.

I can put the file on a web server, but the only ones I have ready access to are publically-accessible, so I have to set up an access control. If its a large file then I have to think about disk space on the server (usually an issue) and then I have to wait while the file copies before sending the recipient a link. Oh, and I have to test that link myself because invariably I've screwed up file permissions or something else.

Probably the closest to what I want is file transfer via IM, but for various reasons that's currently blocked at the network level. I could probably get that block changed but it'd mean a bunch of negotiations for something that isn't actually related to my job. Its not worth my time.

So I wrote otfile. You run it with a single file as an argument, and it creates a web server on your machine with a randomised url for the file. You paste the url into an instant messaging session (I'm chatting with my team all day long). They click it, file downloads directly from the source, and then crucially, the script exits and the web server goes away. That url, while open for anyone to connect to, is near impossible to guess and only works once. That's secure enough for me.

The major thing I think this is missing right now is the ability to do multiple files at once. Its not that big of an issue because its pretty easy to run multiple instances - just a shell loop. If I went for multiple files I'd have to decide if I want to make it produce multiple urls (a pain to paste and to then require someone to click on them all), produce a directory listing (what are the semantics? when do the files disappear? when does the server shut down?) or build some kind of archive on the fly (cute, but is that painful for the receiver?). I'll probably just dodge it until I use it like that enough to be able to ask the receiver what they would have expected.

#!/usr/bin/env perl

use 5.010;

use warnings;
use strict;

use autodie;

use File::MMagic;
use File::stat;
use UUID::Tiny;
use Sys::HostIP;
use URI::Escape;
use Term::ProgressBar;

use base qw(HTTP::Server::Simple);

my @preferered_interfaces = qw(eth0 wlan0);

say "usage: otfile <file>" and exit 1 if @ARGV != 1;

my ($file) = @ARGV;

open my $fh, "<", $file; close $fh;

my $mm = File::MMagic->new;
my $type = $mm->checktype_filename($file);

my $size = (stat $file)->size;

my ($fileonly) = $file =~ m{/?([^/]+)$};

my $uuid = create_UUID_as_string(UUID_V4);

print "I: serving '$file' as '$fileonly', size $size, type $type\n";

my $server = __PACKAGE__->new;

my $interfaces = Sys::HostIP->interfaces;
my ($ip) = grep { defined } (@{$interfaces}{@preferered_interfaces}, Sys::HostIP->ip);

my $port = $server->port;
my $path = "/$uuid/".uri_escape($fileonly);
my $url = "http://$ip:$port$path";

print "I: url is: $url\n";

$server->run;

my $error;

sub setup {
    my ($self, %args) = @_;

    print STDERR "I: request from $args{peername}\n";

    if ($args{path} ne $path) {
        $error = "403 Forbidden";
        print STDERR "E: invalid request for $args{path}\n";
    }
}

sub handler {
    my ($self) = @_;

    if ($error) {
        print "HTTP/1.0 $error\n";
        print "Pragma: no-cache\n";
        print "\n";
        return;
    }

    open my $fh, "<", $file;

    print "HTTP/1.0 200 OK\n";
    print "Pragma: no-cache\n";
    print "Content-type: $type\n";
    print "Content-length: $size\n";
    print "Content-disposition: inline; filename=\"$fileonly\"\n";
    print "\n";

    my $p = Term::ProgressBar->new({
        name => $fileonly,
        count => $size,
        ETA => "linear",
    });
    $p->minor(0);

    my $total = 0;
    while (my $len = sysread $fh, my $buf, 4096) {
        print $buf;
        $total += $len;
        $p->update($total);
    }

    $p->update($size);

    close $fh;

    exit;
}

sub print_banner {}

I really need to set up a repository for things like this. Not hard to do of course, I'm just not sure if I should have one repository per tool, even if its just a single file, or all these unrelated things in one repository. I'll probably just do the latter; its way easier to manage.

wednesday, 28 april 2010

posted at 22:36

Its kind of hilarious that out of everything I've done in the last couple of months this is the thing I decide to come up for air with, but its been that kind of a day. This is the result of three hours of study and hacking. Its using the new IMAP OAUTH mechanism implemented by Gmail to let me login as one of my users via IMAP.

#!/usr/bin/env perl

use warnings;
use strict;

use Net::OAuth;
use URI::Escape;
use MIME::Base64;
use Mail::IMAPClient;

# user to connect as
my $username = q{some.user};
# apps domain
my $domain   = q{some.domain.com};
# oauth consumer secret. dig it out of the "advanced settings" area of the apps dashboard
my $secret   = q{abcdefghijklmnopqrstuvwx};

my $url = 'https://mail.google.com/mail/b/'.$username.'@'.$domain.'/imap/';

my $oauth = Net::OAuth->request('consumer')->new(
    consumer_key     => $domain,
    consumer_secret  => $secret,
    request_url      => $url,
    request_method   => 'GET',
    signature_method => 'HMAC-SHA1',
    timestamp        => time,
    nonce            => int(rand(99999999)),
    extra_params => {
        'xoauth_requestor_id' => $username.'@'.$domain,
    },
);
$oauth->sign;

my $sig = $oauth->to_authorization_header;
$sig =~ s/^OAuth/'GET '.$oauth->request_url.'?xoauth_requestor_id='.uri_escape($username.'@'.$domain)/e;
$sig = encode_base64($sig, '');

my $imap = Mail::IMAPClient->new(
    Server        => 'imap.gmail.com',
    Port          => 993,
    Ssl           => 1,
    Uid           => 1,
);
$imap->authenticate('XOAUTH', sub { $sig }) or die "auth failed: ".$imap->LastError;

print "$_\n" for $imap->folders;

I guess three-legged OAuth would be pretty similar to get going, but I don't have a particular need for it right now.

tuesday, 9 february 2010

posted at 08:49

My laptop has had an interesting couple of days. The main filesystem went read-only a couple of nights ago after a couple of random journal errors. After being fsck'd and cleaned up it did it again, so I reinstalled it and restored it from backup yesterday. Then last night it overheated, leading me to open the case and clean the wall of dust out of the fan. Its back together now, but a couple of lost of tiny parts means I have no indicator lights and no trackpoint. Fortunately the trackpad still works, but its taking a little getting used to. On the other hand, its not burning my lap or my hands anymore, so its probably an overall victory though its not quite feeling that way yet.

One of the things I did lose in the rebuild, due to it not living in one of my backup locations (which is /etc/ and /home/rob) is my cute little mobile roaming script. I rewrote it on the bus on the way home yesterday and thought that perhaps its interesting enough to post here.

The basic idea is that every day I switch between at least two networks. My home network has a PC in the hall cupboard which among other things runs a web proxy and a mail server. Its also the firewall, so web and mail traffic can't go out directly. Work on the other hand, implements transparent proxying (with some network authentication) and has a SMTP server, but naturally it has a different address. I also occassionally use other networks (friend's places, coffee shops, etc) which usually have no facilities at all, requiring me to fend for myself.

My laptop runs a SMTP server (Postfix) of course, because that's just what you do on Unix. I also run a Squid proxy which I point all my local HTTP clients at. This way, when I move networks, I only have to reconfigure the local proxy rather than tweak every web client I have.

I spent a long time looking for a decent roaming reconfiguration package, but I never managed to find one. Some would try to do network detection and too often get it wrong. Some would have overly complicated and/or feature deficient configuration languages. I vaguely recall that I really liked one of them but it was tightly integrated with NetworkManager, which I don't use because it could never seem to keep the network alive for more than a few minutes (and it appears to be pretty much tied to the GUI, which is painful when I need network on the console).

So, in the finest open source tradition, I rolled my own. The script itself is trivial; its just a tiny template expander. I'll list the script in a moment, but first I'll talk about its operation.

The script, which I call location, takes a location name on the command line (like home or work), runs over a (hardcoded) list of config files, reads them in, modifies them, and spits them out to the same file. It makes modifications according to templates that may exist in the file. If the file has no template, then location ends up emitting the unchanged file.

In any file you want it to modify, you add an appropriate template. This is the template I have in my /etc/postfix/main.cf:

### START-LOCATION-TEMPLATE
##@ home relayhost = lookout.home
##@ work relayhost = smtp.monash.edu.au
##! /etc/init.d/postfix restart
##! sleep 1
##! /usr/bin/mailq -q
### END-LOCATION-TEMPLATE

When it finds itself inside a template, location stops its normal operation of outputting the lines of the file as-is and instead starts parsing. Interesting lines begin with ##, anything else is ignored. Its the third character that determines how the line is interpreted. So far I have the following functions:

  • #: do nothing, just output the line
  • @: emit if at location. If the location specified on the command line matches the first argument to @, then the rest of the line is added to the file as-is.
  • !: run command. Calls a shell to run the specified command after the file has been generated.
  • >: interpolate line. Include the rest of the line in the file, but expand any %variable%-type markers. So far only %location% is defined, and is replaced with the location specified on the command line.

(I'll provide an example of that last one in a moment).

So in the case of main.cf, lets say we ran location with home as the location. This would result in the template section being written to the output file as:

### START-LOCATION-TEMPLATE
##@ home relayhost = lookout.home
relayhost = lookout.home
##@ work relayhost = smtp.monash.edu.au
##! /etc/init.d/postfix restart
##! sleep 1
##! /usr/bin/mailq -q
### END-LOCATION-TEMPLATE

The listed commands are then run, which cause Postfix to be restarted and the mail queue to be flushed:

/etc/init.d/postfix restart
sleep 1
/usr/bin/mailq -q

Naturally Postfix interprets the template parts of the file as comments, so nothing to worry about. The next time location is run, the "bare" relayhost line is ignored, so it doesn't get in the way.

The config for Squid is similar. Because Squid's config file is huge, I don't quite trust my script to handle the whole thing sanely, so at the bottom of squid.conf I've added:

include /etc/squid/location.conf

And in location.conf I have:

### START-LOCATION-TEMPLATE
##@ home cache_peer lookout.home parent 8080 0 default
##@ home never_direct allow all
##! /etc/init.d/squid restart
### END-LOCATION-TEMPLATE

By default Squid will try and hit the internet directly, which is fine for work and unknown locations. For home, i need to force it to always go to an upstream proxy, which is what those the cache_peer and never_direct directives will achieve.

The proxy at work used to be an authenticating proxy, so I had to specify both a peer and a username/password combination. This made the required amount of variable config a little unwieldy to be include in a template, which is where the > function came from. location.conf used to have this:

##> include /etc/squid/upstream.%location%.conf

Which would arrange for upstream.home.conf, upstream.work.conf, etc to be included depending on the location. There's every chance this will come in useful again one day, so I've left the code in there for now.

Here's the script in its entirety:

#!/usr/bin/env perl

use 5.010;

use warnings;
use strict;

my @files = qw(
    /etc/squid/location.conf
    /etc/postfix/main.cf
);

use autodie qw(:default exec);

use FindBin;

if ($< != 0) {
    exec "/usr/bin/sudo", "$FindBin::Bin/$FindBin::Script", @ARGV;
}

say "usage: location <where>" and exit 1 if @ARGV != 1;

my ($location) = @ARGV;

for my $file (@files) {
    say "building: $file";

    my @out;
    my @cmd;

    open my $in, "<", $file;

    my $in_template = 0;
    while (my $line = <$in>) {
        chomp $line;

        if ($line =~ m/^### START-LOCATION-TEMPLATE/) {
            $in_template = 1;
            push @out, $line;
            next;
        }

        if ($line =~ m/^### END-LOCATION-TEMPLATE/) {
            $in_template = 0;
            push @out, $line;
            next;
        }

        if (!$in_template) {
            push @out, $line;
            next;
        }

        my ($tag) = $line =~ m/^##([#@!>])/;
        if (!$tag) {
            next;
        }

        given ($tag) {
            when ('#') {
                push @out, $line;
                next;
            }

            when ('@') {
                push @out, $line;

                my ($want, $rest) = $line =~ m/^##@ (\w+) (.*)/;
                if ($want eq $location) {
                    push @out, $rest;
                }

                next;
            }

            when ('!') {
                push @out, $line;

                my ($cmd) = $line =~ m/^##! (.*)/;
                push @cmd, $cmd;

                next;
            }

            when ('>') {
                push @out, $line;

                my ($rest) = $line =~ m/^##> (.*)/;

                $rest =~ s/%location%/$location/g;

                push @out, $rest;
            }
        }
    }

    die "$file: unclosed location template" if $in_template;

    close $in;

    open my $out, ">", $file;
    say $out $_ for @out;
    close $out;

    for my $cmd (@cmd) {
        say "running: $cmd";
        system $cmd;
    }
}

Because its so trivial and I only run it a couple of times a day, I just run it when I get to work (location work) or when I get home (location home). If I felt inclined I could probably hook it up to my network stuff but I think that would be more trouble than its worth.

On occassion I have to use Windows on the same machine. I have no idea how to achieve something similar there, so I just reconfigure my browser. Fortunately I don't go there often, and almost never from work. This is why I like open source. I can make my system work in exactly the way I want and usually with a minimum of fuss.

thursday, 10 december 2009

posted at 21:20
tags:

Day two of training today. I had to run off early, but not before getting a crash course in Moose. I've been watching Moose for a couple of years, and have a project in mind for it, but haven't got around to doing anything with it yet. Doing a few basic exercises with it was awesome just to see what it can do, but I did manage to get frustrated by three things within the first ten minutes.

  1. The first thing I noticed is that every attribute accessor created is publically available. There isn't really a way to make an attribute that is read/write from inside the class but read-only from outside. The fine manual suggests this:

    has 'attr' => (
        is     => 'ro',
        writer => '_set_attr',
    );
    

    This works ok, but its still possible for code outside the class to call _set_attr directly. Until we get lexical subs its impossible to make the writer method invisible to the outside world, but until then I'd still like it to be possible for Moose to produce an accessor that can check its caller.

    In a similar vein, its not possible to create proper protected or private attributes. Private attributes can sort of be done by assigning directly to the object hash:

    $self->{attr} = 1;
    

    I don't like this because it because it makes assumptions about the internal implementation of the objects (and with Moose I'd like to remain as ignorant as possible on this point), but also because it provides no type or constraint checking.

    Protected attributes (that is, attributes private to a class and its subclass) seem to be completely impossible.

  2. By default, a Moose class quietly accept any and all parameters passed to its constructor, regardless of whether or not they correspond to an attribute in the class or its parents. This confused me for a moment as I've come from Params::Validate which allows you to declare parameter types and constraints much like Moose attribute declarations, but dies if you provide a parameter that is not declared. The fine inhabitants of #moose on irc.perl.org pointed me at MooseX::StrictConstructor, which does what I want - dies if undefined parameters are provided.

    It gets better though. I was declaring an attribute that I wanted to be impossible to initialise via the constructor as I planned to set its initial value in BUILD, and to allow the user to provide a value only to ignore it is confusing. The manual explains that specifying init_arg => undef in the attribute definition will arrange for that parameter passed to the constructor to be ignored, but again, it does it quietly.

    It turns out (again via #moose) that combining MooseX::StrictConstructor with init_arg => undef yields the desired results. I can live with that, but I would never have anticipated that result from the documentation. Hmph.

  3. Moose doesn't provide any syntactic sugar for class attributes/methods. A quick search just now turns up MooseX::ClassAttribute which will probably be as much as I'll need, at least initially, but I was surprised that core Moose didn't have anything for this. Are class attributes so uncommon?

At the end of the day though, these are all pretty minor nits. Moose is awesome. Its very actively maintained and developed by a number of incredibly smart people, so its not going away any time soon. I'm looking forward to having the time to do something serious with it.

thursday, 10 december 2009

posted at 11:26
tags:

Work is sending me on a Perl Training Australia course this week, so I'm getting to hang out with Paul and Jacinta and get a good refresher on Perl OO. I wouldn't say I needed it, but I've been enjoying the discussion and it never hurts to make sure that your accumulated understanding matches the current reality.

One of the exercises involved a class representing a coin with methods to flip the coin. One of the things we were asked to do at one point was to create an array of coins and do various things to them. My first instinct to create the array was to do this:

my @coins = (Coin->new) x 10;

I was saddened but not surprised to find that this doesn't work. As the following test demonstrates the left hand side is only evaluated once and then just copied, so I ended up with an array containing ten references to the same object:

$ perl -E '$c = 0; @x = ($c++) x 10; say @x'
0000000000

The best I could come up with is this, which I don't think reads anywhere near as well:

my @coins = map { Coin->new } (1..10);

We briefly discussed whether it would be worth developing a core patch to do something like it, but realistically the only option that preserves a reasonable amount of backward compatibility is to only reevaluate the left side for a very specific set of types, namely code references, giving something like this:

my @coins = (sub { Coin->new }) x 10;

Given that that really doesn't read particularly better than the version using map, and not knowing if anything smarter is possble (and how to do it if it is), and knowing that the core developers aren't particularly keen on new features to existing constructs at the best of times, I've opted to leave it for now but keep my eyes open for things like this.

One part of another exercise had me dealing with decks of cards. Internally I represented suits as integers, with the following list to assist with the as_string method:

my @suits = qw(hearts spades clubs diamonds);

When I got to adding the initialiser for the class, naturally I wanted to be able to specify a string. The usual thing I'd do here is create a hash from @suits with the values as the integer array indexes. This time I came up with this one-liner to determine the index of a value in an array:

my $index = do { my $found; grep { $found = 1 if $_ eq $needle; $found ? 0 : 1 } @haystack };

It plays on the fact the grep in scalar context returns the number of matches; that is, the number of times the code block evaluates true. All this does is arranges it such that the block is true for every array index before the wanted value but false for every index after (and including) the wanted index. If $index == @haystack, then it wasn't found.

Its certainly not optimal - a binary search is always going to be quicker, and you'd nearly always want to use the hash method if you were doing it many times, but it was certainly fun to write a cute oneliner to do it.

monday, 2 november 2009

posted at 08:17
tags:

A couple of months ago I started work in a new job, doing various programmery things for the Monash student intranet thing The job description says "web programmer", but there hasn't been much web yet. At this point I'm mostly concentrating on the glue code needed to hook up various Google services to our environment. 99% of the code I'm writing and working on is Perl.

I've been using Perl for ages though. For the last nine-and-a-bit years I've been a mail sysadmin at Monash, and while the "core" of our systems has always been full-on professional mail software packages (both proprietary and open-source), all the bits in between have always been Perl. We've written all sorts of stuff, from full web environments and workflow packages to all the traditional sysadmin tools like log parsers, report generators, config builders and everything in between. There's never been any question for us - Perl is just so obvious for this kind of work.

Previously though, Perl was merely a tool that I used to get the job done. In many ways though its now become the job itself. For any given day I can be reasonably confident that most of it will be reading or writing Perl code, whereas before I'd only bust it out when I needed it. Additionally, I haven't really done this type of work before so I'm getting lots of ideas for stuff I want to play with on my own time and also finding gaps in my knowledge that I want to fill out. So now I'm finding just about every moment I'm at the computer I'm doing something with Perl, far more than ever before.

So, I've decided that it would be really good to finally join the mob rather than just hang around the edges looking in. I'm signing up for the Iron Man challenge to keep me honest, and I'm moving my code to Github for a bit more visibility. This is going be an interesting change for me, as in both blogging and coding I'm used to producing something large and fully-formed before showing the world, but obviously that doesn't work if you need to post once a week. I've started making a list of little Perl things I can write about in a couple of paragraphs, so hopefully I'll be able to keep it fresh.

Additionally, I've committed myself to writing everything I possibly can in Perl. I have a long history with C as well, and for the longest time always reached for it for anything closer to the hardware/OS (a fuzzy line, but typically that means server-type things). No more. From now on unless there's a very specific reason why Perl is unsuitable, I'll be choosing Perl for my code.

So that's it. Hi :)

thursday, 22 october 2009

posted at 13:33

As is often the case with me, what started as a small hack to blosxom to make it do tags and per-tag feeds turned into me rewriting it from the bottom up. I quite like what its become, though I doubt its of much use to anybody but myself. Give me a yell if you want the code.

Anyway, now the whole site has proper tags, and you can use them to subscribe to just bits of my ramblings rather than the whole lot, which should make a huge difference considering how much I don't actually write. Oh, and there Atom feeds too if you like that sort of thing.

To celebrate this momentous occasion, I've moved the whole mess to a new and somewhat relevant domain, eatenbyagrue.org. I think I got all the redirects right, so existing subscriptions should work ok.

That's all. Back to work shortly.

tuesday, 29 september 2009

posted at 21:16

I love blosxom, so I persist with it, but its a nightmare to refactor. It could have done what it does just as well without being quite as clever. I wonder if that's why development on it is mostly dead. In any case, the refactoring goes well, and I hope soon to have something that does exactly what I want (which I'll talk about more soon; better to spend my time on code right now).

thursday, 24 september 2009

posted at 22:59

Tonight I wrote a simple tag plugin for blosxom (the blog engine I use), imported all the categories from the old Wordpress blog as tags, and wrote some CSS to make it work properly. There's still a bit more to do, notably making the tags into links that take you to ther posts tagged the same thing as well as getting per-tag feeds going (though the RSS plugin needs a lot of work anyway).

The plugin, for the curious:

package tags;

use vars qw($tags);

sub start { 1 }

sub story {
    my ($pkg, $currentdir, $head_ref) = @_;

    $tags = '';
    if ($meta::tags) {
        my @tags = split /\s+/, $meta::tags;
        $tags = "<div class='tags'>tags: <ul>";
        $tags .= "<li>$_</li>" for @tags;
        $tags .= "</ul></div>";
    }
}

1;

Obviously, its really the meta plugin that does most of the heavy lifting.

So now at the top of a post I write something like:

meta-tags: site perl

and tags pop out. Lovely!

wednesday, 28 january 2009

posted at 22:26

After my lack of motivation I've had a couple more interesting ideas and so I've started very slowly poking at them, being very careful to not overdo it in an attempt to avoid the burnout. So far it seems to be working!

The short is that my mate Sam and I have been pondering for a year or more the idea of building an arcade cabinet that runs emulators for various old systems (MAME and such). He's a high school woodworking teacher and cabinetmaker by trade so he's perfectly qualified to build the box. I know a thing or two about computers, so I can do that bit. The problem so far is that we've never really had any good place to build it - I have a garage but no tools, Sam had no space at all.

He's just recently moved house and is finishing getting his own garage all kitted out with workbenches and drop saws and drill presses and other things that scare me and he's now ready to build something, so we're taking another look at it. I've managed to scrounge enough parts to build a reasonably good rig, though the ridiculous weather is making me reluctant to go out to the shed to work on it. Arcade Gaming Australia have all the buttons, joysticks and other bits that we'll need. There's only one thing left - an awesome UI for choosing games and things. Yay software!

So tonight I've been sitting under the air conditioner fiddling with Clutter. Its a library for making fancy interfaces by using lots of 3D stuff under the hood. As far as I can tell the most well known example of the type of thing its for is Apple's Cover Flow. Just from playing with some of the samples I already have some idea of how I'd like a game selector to look, so I've started experimenting using the Perl bindings.

The basic idea is that you set up a bunch of actors, which are basic visual elements - some text or an image for example. You can specify various transformations for an actor, eg scaling, rotating, etc. After that, you place your actors somewhere on the stage, which is roughly analogous to a window.

Next is where I get a little confused, but not so much that I can't get something done. You setup a timeline, which has two paramaters - number of frames, and frames per second. You hook up an "alpha" to the timeline, which is a function that gets called every frame, and returns a number that I don't fully understand the purpose of yet. The number is used to drive "behaviours" attached to each actor, which makes them do something depending on the current distance through the timeline. A behaviour might be to move the actor around the stage, rotate it, or something more clever.

There's also an input layer, but I haven't really started looking at that yet.

So here's the fruits of my evening. It takes a random image and rolls it around a window.

#!/usr/bin/env perl

use 5.10.0;

use strict;
use warnings;

use Glib qw( :constants );
use Clutter qw( :init );

say "usage: roll image" and exit -1 if !@ARGV;

my $stage = Clutter::Stage->get_default;
$stage->set_color(Clutter::Color->parse("DarkSlateGray"));
$stage->signal_connect('key-press-event' => sub { Clutter->main_quit });
$stage->set_size(800, 600);

my $actor = Clutter::Texture->new($ARGV[0]);
$actor->set_anchor_point($actor->get_width / 2, $actor->get_height / 2);
$actor->set_position($stage->get_width / 2, $stage->get_height / 2);
$stage->add($actor);

my $timeline = Clutter::Timeline->new(100, 26);
$timeline->set(loop => TRUE);

my $alpha = Clutter::Alpha->new($timeline, sub {
    my ($alpha) = @_;
    return int($alpha->get_timeline->get_progress * Clutter::Alpha->MAX_ALPHA);
});

my $rotate = Clutter::Behaviour::Rotate->new($alpha, "z-axis", "cw", 0.0, 359.0);
$rotate->apply($actor);

my $path = Clutter::Behaviour::Path->new($alpha, [ $actor->get_width,                     $actor->get_height                      ],
                                                 [ $actor->get_width,                     $stage->get_height - $actor->get_height ],
                                                 [ $stage->get_width - $actor->get_width, $stage->get_height - $actor->get_height ],
                                                 [ $stage->get_width - $actor->get_width, $actor->get_height                      ],
                                                 [ $actor->get_width,                     $actor->get_height                      ]);
$path->apply($actor);

$timeline->start;

$stage->show;

Clutter->main;

Hard to show it here, but here you go:

Of course I have no idea if this is the "right" way to do it, but it seems to perform well enough so it will do for now. Next is to make a little photo thumbnail viewer, using the arrow keys to scroll through the photos and a little zooming magic.

thursday, 19 june 2008

posted at 00:49
tags:
  • mood: modular

A few random things I've been working on lately:

  • Started migrating this blog into ikiwiki. Stuck on a problem with file create/modify times not being preserved, which makes complete sense but is annoying. I think a plugin is required, much like one I wrote for blosxom once upon a time.

  • Made progress with Test::MockTerm. The terminal itself works, with open() and POSIX::isatty() being overidden correctly. I'm currently planning the interface for sending and handling control characters (and from there emulating Term::ReadKey). Its hairy.

  • Made a new release of XML::Quick after a couple of years. The test suite now runs (the were bugs in the suite itself) and an ancient bug was taken care of to boot.

  • Dug out XML::Spice, something I started a couple of years ago as an answer to Python's stan XML generation library. It already has some funky magic in place that lets you call the generator function to create some piece of XML which totally complete and usable, but then call the generator again with that returned chunk as one of the arguments to have it embed that chunk into another chunk of XML. At this point both are valid, but it can do some funky stuff like move and reprefix namespace declarations to make the result more concise without changing the semantics. It does this without having to reparse the original XML. You have to read the tests to find out more. In any case, its not finished yet and perhaps never will be since I'm long over XML.

  • Also dug out HTML::Calendar::Render (no link yet), which you give a bunch of events and it creates a calendar using HTML tables. This was done for work, at a time when we temporarily needed an alternative view to our corporate calendar system. Producing calendars in HTML is nothing new, but this is prettier than any of others I've seen on CPAN. It works really hard to produce stuff that looks like your calendar in something like Outlook, where overlapping events appear next to each other at half-width. This one I'm kinda interested in getting into some kind of release quality. The plan is to split out the tree generator from the renderer itself, so you can add modules that might render it with or without HTML tables, in PDF, or via cairo. Coming soon I guess.

wednesday, 4 june 2008

posted at 23:33
tags:

I've been getting into Perl again in a big way. I'd been getting in the mood for code again, and started looking through my old stuff to see if anything tickled my fancy. I found some Perl stuff, which I've been chucking in git, and was poking about it, when a question about IO::Prompt came up. Since I'm theoretically responsible for that module, it seemed like a good place to kick off.

I started work again on Test::MockTerm, this time making a pure-Perl version rather than relying on IO::Pty, which had a few issues that made it unsuitable for what I needed. As mentioned in that old post of mine though, without real terminal handles I have no way to test the code paths that expect them.

During my break, Perl 5.10.0 was released, but the hoped for patch that would allow me to override the -t file test operator never made the cut, so I'm still stuck without a solution. The short-term workaround will therefore be to modify IO::Prompt to use POSIX::isatty, which does essentially the same thing. Hopefully it'll work on Windows when that time comes.

Longer term, there needs to be a way to make -t overridable in some way, so I started poking at the code a bit, and thought about extending tied handles to do the work.

For the uninitiated, Perl has a facility called "tying" where you can essentially tuck an object reference into a variable. When operations are performed on that variable (like storing a value in it), Perl arranges to call methods on the object to do the work. perltie explains the details.

You can tie a filehandle to an object in the same way, so its the ideal way to do what I want. Except that in the BUGS section, we see:

Tied filehandles are still incomplete. sysopen(), truncate(), flock(), fcntl(), stat() and -X can't currently be trapped.

-X is the general way to refer to the file test operators. Damn.

Tonight I sent off a patch that implements stat() and -X for tied filehandles.

It was quite a challenge to write, but very fulfilling. One of the really interesting things about hacking on Perl is that the culture of writing comprehensive tests and documentation is deeply entrenched, so no matter what you do, you end up with high-quality work at the end of it because its provably correct.

Unfortunately even if it is accepted it seems unlikely to me that the patch will appear anytime soon. I don't know what the policy is on new features on the current maintenance branch, but I wouldn't be surprised if it was slated for inclusion in 5.12. It took over three years for 5.10 to appear after 5.8, so I won't have this code available in a stable Perl for quite a while yet, but at least I have something to point to.

Next is to get Test::MockTerm finished, which I'll get back into tomorrow.

friday, 15 september 2006

posted at 07:23
tags:

So a couple of weeks ago I got the chance to chat to Damian about IO::Prompt and my completion patch. While he rejected the patch because the interface was sucky (and I agree), he accepted my offer to take on maintenance duties for the module. Thats not it for completion though; we're currently designing a much better completion and history interface, which I'll write more about that some other time. My first trick will be to get a test suite up and running.

IO::Prompt doesn't currently have a test suite, and I'm not confident that I'll be able to make any significant changes without breaking whats there, so the current functionality has be recorded. The difficult thing about it is that we're testing something terminal based, so we have to pretend to type something, and then watch not only what the API returns, but also watch what appears on the screen.

This turns out to be quite complicated. The module opens /dev/tty directly, both for reading and writing, so we need to intercept the calls to open (via CORE::GLOBAL::open) and returns some filehandles we can manipulate directly. My first cut used basic scalar handles, but then I ran into further trouble when I found that the module uses -t to see if its talking to a terminal. Obviously my scalar handles are not terminals, so I needed a way to convince -t otherwise.

After a deep tour into the guts of Perl itself (a fascinating and scary place) I determined that there's really no pleasant way of overriding -t, though there is a patch under consideration for 5.10, and I did figure out a really evil way that might do it by twisting the optree in ways that I wouldn't dare give to the world. So the only other option is to somehow produce filehandles that are in fact terminals.

IO::Pty provides the answer, by allowing me to get pseudo-terminals from the operating system. I kinda didn't want to go there, because it ties the implementation to systems that have terminals, which doesn't include Windows, but I've since decided that it'll be fine for now since the current code hits /dev/tty directly, and that doesn't exist on Windows either.

Time passes. I play with this module, figure out the difference between master and slave and make a note of it because its stupid and I can never remember, and finally produce Test::MockTerm. Its not CPAN-ready yet, its currently a build helper for IO::Prompt, but I think it may have a life of its own someday. Using it, I write some basic tests for IO::Prompt, and run it .. and it hangs, waiting to read from standard input.

After further perusal of the code, it seems that IO::Prompt only reads directly from /dev/tty when the -tty or -argv options are specified. Otherwise, it arranges to read from standard input. However, it does this not be simply using the STDIN handle, but by using the first file in ARGV, and if that doesn't work, trying to open - (using the single-arg scalar-and-filehandle form of open). I think (more testing required) Damian did it this way because STDIN may have been redirected away from wherever ARGV and - pointed initially.

This presents an interesting problem. I now need to arrange for opening - to actually cause my pseudo-terminal input handle to be used instead. But, I've already overridden open, and you can't have multple overrides, so I need some kind of multiplexing/dispatch thing to figure out which open replacement to use.

Except I don't. I've just now had a good idea. What if you specified /dev/tty explicitly on the command line as the input source? Wouldn't we want that intercepted also? And isn't that in the scope of what Test::MockTerm should do? The answer is yes. I'm going to modify my code to look for /dev/tty in the one-arg form of open, as well as to look for - and use the same handles. That should take care of it. Epiphany!

So thats where I'm at for now. This has been an incredibly challenging project so far, and I haven't actually written any real tests yet! I intend for this code to be released in IO::Prompt 0.99.5 or 0.99.6, depending on how long it takes.

friday, 1 september 2006

posted at 15:53
tags:

Damian dropped in on Monday to word us up on the finer points of VIM insanity. So now I have an editor that can do anything, one way or another. And if it can't, Perl can, and the two of them work well enough to get things done. As a test, I implemented an insert-mode LDAP lookup which uses Perl to do the LDAP stuff, leaving the editor bits to VIMs internal scripting language.

The Perl interface for talking back into VIM itself (eg reading and setting variables) is pretty horrendous, but I've already written a proof-of-concept for a module that will make life far more Perly, eg instead of this:

VIM::Msg(VIM::Eval("a:arg"));

we can do this:

print $a{arg};

I know which I prefer :)

When and if I get this module into some kind of usable state, it'll go up to CPAN as VIM::Sane.

I also deleted my entire .vimrc and .vim/ and started again. Its now clean and nicely commented, and makes me happy. I finally know what I'm doing :)

I also got a chance on Monday to talk Perl with Damian at lunch, and got an answer to my Class::Std problems as well as lots of IO::Prompt related stuff. But I'm out of time now, will talk about that more later.

sunday, 27 august 2006

posted at 15:21
tags:

The Midnight clone I keep talking about (and I am getting around to writing about, really) is actually a port to Perl of a Java application. My initial goal was to get the thing working, then make it more Perl like.

One interesting pattern that is used throughout the original is special classes that act only as enumerated types - they simply create a bunch of instances of themselves with specific names that act as constants. These classes are known in Java as "typesafe enumerations".

So I needed something to replace them with. My first attempt was to simply create empty packages with a pile of use constant declarations in them. This worked well enough, but the original enumeration classes had additional features - they needed to stringify properly (via a toString method), they needed to compare equal only to themselves (something use constant can't do, since its constants resolve to just plain scalars) and the constants can be objects with methods declared on them.

Now I don't know if all these extras stop the classes are "correct" by some theoretical definition of what a typesafe enumeration should be (if such a definition exists), but there's no denying that this stuff is useful, and Perl has always been about practicality over usefulness. Plus, I've started to think of places where I could use something similar in other code of written - pretty much anywhere that a variable could have a limited number of possible values, like state variables.

So I implemented something generic and useful, that I'm now quite proud of: Class::Constant. It makes it dead simple to declare a basic C-style enumeration, but has enough wackiness to do some really crazy things. I'll talk a bit more about some real examples once I start writing about my project, real soon now. Until then, this is just highly ineffective advertising :)

sunday, 20 august 2006

posted at 14:27
tags:

I gave Daniel a demo of the Midnight clone the other day, and one of the first things he did was to try and do a tab complete in the little command-line interface. It was immediately obvious that any modern command-based interface needs completion and history, so I set out to find a way to provide it.

A quick CPAN search didn't really uncover anything. A couple of the Term::Readline variants claim to have support, but the interface seemed rather clunky (reasonable, since it comes from C). I use and love IO::Prompt, largely because of its trivial interface. The answer became clear - IO::Prompt requires tab completion.

The whole thing took three evenings to implement. It was pretty straightforward. I started by adding support for a -complete option, which took a list of possible values. After that it was just a case of hooking up the tab key, comparing the current input to the values in the list, and replacing part or all of it with the matched value. This worked wonderfully well, and did great things for my confidence - I'm always a little unsure if I'm doing something the correct way when I go to work on someone elses code - particularly when its written by a known hero like Damian Conway :P

Adding the characteristic "beep" when you only get a partial match was next. Trivial, of course - just emit ASCII code 0x7 at the proper time; the terminal takes care of the rest. A -bell option went in alongside - I'm a firm believer in being as flexible as possible.

The next hallmark of tab completion is displaying possible values when there is more than match for the current input. Since I already knew about the available matches, its no effort to print them out, but it wants to look nice too. A little column calculator went in to make things pretty. I also added a "show all <N> matches?" prompt when its likely that showing them all will scroll your terminal. Obviously, getting a prompt is no problem (this is a prompting module, after all :P ), but I also found that prompt() is totally reentrant - it doesn't restore the previous terminal settings when it exits, opting instead to return it to "cooked" mode. I haven't looked in any depth, so I don't know if IO::Prompt or Term::ReadKey is at fault here. Either way, it caused tab and other keys to not be detected correctly after the "show all" prompt. The workaround was to simply chuck the terminal back into raw mode, and it coped nicely.

So that just about finished it, but then while writing examples I started to realise that the whole thing was actually pretty useless. The reason: the vast majority of command line inputs are actually a set of individual and sometimes unrelated fragments. Think about completion for files. If I had to provide an array of every possible file that could be chosen in a given situation, I'd have to provide a list of every file on the system, each with full path. Obviously, this is ridiculous. What's wanted is a way to complete only portions of the input line, with the possible values selected by looking at the surrounding context. More than a humble array can achieve. What was needed was a callback.

I made it so that -complete could handle a coderef as well as an array - the simple array code might still be usable in places, and its certainly easier to understand. I figured it would be enough to simply pass the current input to the callback, and have it look at the contents and return a list of possible values based on it.

This worked, but had problems. The callback code was pretty complex, and when there were multiple possible values, displaying them was awful, because my code only knew how to complete entire input lines, not fragments of lines. So the callback would have to return the full input line with each possible outcome. Perhaps an example: we're writing a program that has to load files, using a "load /path/to/file" command. We want to do shell-style completion for the file/path portion of the input. To work correctly, the callback has to look for the "load " at the start, then split up the file path, look inside the directory, and return any files there. But, it has to return the full input line, so it returns something like:

  • load /path/to/file/foo
  • load /path/to/file/bar
  • load /path/to/file/baz

As well has having piles of redundant information, if my code were to display them, it would show the entire lines, when it should have just shown foo, bar and baz (just like your shell). Obviously, IO::Prompt needed to be smarter - it had to understand fragments and do the splitting itself.

This actually took me about two days of thought to figure out - the bus to work is a great place for pondering. The solution was to have prompt() split the input and pass all of those to the callback, and only do completion on the final item in the split. So in the above example, the callback would return qw(foo bar baz), and thats that. A -split option was added that takes a regex to be passed to Perl's split builtin.

Implementing this took quite a bit of internal gymnastics to achieve because I was having to essentially write the same code for two cases (full line and split fragments). Generalising aspects of the code (mostly the list matching and the inpup replacement code) was proving quite finnicky, until I made the leap of logic that told me I'd made the correct choice with the interface. If we're trying to complete for a full line, that line becomes a split with no delimiting character, and one fragment.

Five minutes later, it was done. And the callback is trivial. The most complex callback you'll ever likely write for this sort of thing is one to do file paths, because of the special cases - there's files and subdirs. I included that in the examples. It weighs in at just eight lines.

All in all, a roaring success. Damian now has the patch. Haven't heard back yet, but I'll see him next week so I'll hassle him then if necessary.