eaten by a grue

I've been learning a lot about fonts in the last week as I work on the foundations needed to turn this:

into this:

I'm not going into detail right now, because I could write reams and still not end up saying much - font rendering is a real dark art. I mostly just wanted to share a little hack that I put together that might be useful to anyone experimenting in the same space.

The short of it is that Pioneer uses FreeType. Since I was getting into working on the font code I took the opportunity to see if there's any lighter alternatives so that we could remove a dependency. The answer is that there really isn't for the kind of things we need, but there is one worthy contender: stb_truetype.h

Its a very simple TrueType renderer in ~1800 lines of C (of which ~500 of that is comments and documentation). Its missing support for a lot of things, but its a single file to include in your project and does a good job of the common fonts that most people have.

The only trouble I had with it is that the very few examples I could find assume you're using it directly with OpenGL. Its probably not an unreasonable assumption, but it made things a little difficult for me because I'm still not great with GL but I'm much better with SDL. What I really wanted was a simply example for SDL so I could get a feel for the API and check its output without having having to wrestle with GL and then wonder if I was getting odd results because I'd done something wrong.

Alas, no such example existed, so I wrote one. Here it is, for internet's sake:

/*
 * sdl_stbtt - stb_truetype demo using SDL
 * Robert Norris, May 2011
 * Public Domain
 *
 * Compile:
 *   gcc --std=c99 -o sdl_stbtt sdl_stbtt.c `sdl-config --cflags --libs` -lm
 *
 * Run:
 *   ./sdl_stbtt <path-to-ttf-file> <text-to-render>
 */

#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/mman.h>
#include <SDL.h>

#define FONT_HEIGHT 32

#define STB_TRUETYPE_IMPLEMENTATION
#include "stb_truetype.h"

int main(int argc, char **argv) {
    if (argc != 3) {
        printf("usage: sdl_stbtt <path-to-ttf-file> <text-to-render>\n");
        exit(-1);
    }

    /* getting the font into memory. this uses mmap, but a fread variant would
     * work fine */
    int fontfd = open(argv[1], O_RDONLY);
    if (fontfd < 0) {
        perror("couldn't open font file");
        exit(1);
    }

    struct stat st;
    if (fstat(fontfd, &st) < 0) {
        perror("couldn't stat font file");
        close(fontfd);
        exit(1);
    }

    void *fontdata = mmap(NULL, st.st_size, PROT_READ, MAP_SHARED, fontfd, 0);
    if (!fontdata) {
        perror("couldn't map font file");
        close(fontfd);
        exit(1);
    }

    if (SDL_Init(SDL_INIT_VIDEO) < 0) {
        fprintf(stderr, "sdl init failed: %s\n", SDL_GetError());
        munmap(fontdata, st.st_size);
        close(fontfd);
        exit(1);
    }

    /* creating an off-screen surface to render the glyphs into. stbtt outputs
     * the glyphs in 8-bit greyscale, so we want a 8-bit surface to match */
    SDL_Surface *glyphdata = SDL_CreateRGBSurface(SDL_SWSURFACE, 512, 512, 8, 0, 0, 0, 0);
    if (!glyphdata) {
        fprintf(stderr, "couldn't create sdl buffer: %s\n", SDL_GetError());
        munmap(fontdata, st.st_size);
        close(fontfd);
        SDL_Quit();
        exit(1);
    }

    /* 8-bit sdl surfaces are indexed (palletised), so setup a pallete with
     * 256 shades of grey. this is needed so the sdl blitter has something to
     * convert from when blitting to a direct colour surface */
    SDL_Color colors[256];
    for(int i = 0; i < 256; i++){
        colors[i].r = i;
        colors[i].g = i;
        colors[i].b = i;
    }
    SDL_SetPalette(glyphdata, SDL_LOGPAL|SDL_PHYSPAL, colors, 0, 256);

    /* "bake" (render) lots of interesting glyphs into the bitmap. the cdata
     * array ends up with metrics for each glyph */
    stbtt_bakedchar cdata[96];
    stbtt_BakeFontBitmap(fontdata, stbtt_GetFontOffsetForIndex(fontdata, 0), FONT_HEIGHT, glyphdata->pixels, 512, 512, 32, 96, cdata);

    /* done with the raw font data now */
    munmap(fontdata, st.st_size);
    close(fontfd);

    /* create a direct colour on-screen surface */
    SDL_Surface *s = SDL_SetVideoMode(640, 480, 32, 0);
    if (!s) {
        fprintf(stderr, "sdl video mode init failed: %s\n", SDL_GetError());
        SDL_FreeSurface(glyphdata);
        SDL_Quit();
        exit(1);
    }

    /* the actual text draw. we loop over the characters, find the
     * corresponding glyph and blit it to the correct place in the on-screen
     * surface */

    /* x and y are the position in the dest surface to blit the next glyph to */
    float x = 0, y = 0;
    for (char *c = argv[2]; *c; c++) {
        /* stbtt_aligned_quad effectively holds a source and destination
         * rectangle for the glyph. we get one for the current char */
        stbtt_aligned_quad q;
        stbtt_GetBakedQuad(cdata, 512, 512, *c-32, &x, &y, &q, 1);

        /* now convert from stbtt_aligned_quad to source/dest SDL_Rects */

        /* width and height are simple */
        int w = q.x1-q.x0;
        int h = q.y1-q.y0;

        /* t0,s0 and t1,s1 are texture-space coordinates, that is floats from
         * 0.0-1.0. we have to scale them back to the pixel space used in the
         * glyph data bitmap. its a simple as multiplying by the glyph bitmap
         * dimensions */
        SDL_Rect src  = { .x = q.s0*512, .y = q.t0*512, .w = w, .h = h };

        /* in gl/d3d the y value is inverted compared to what sdl expects. y0
         * is negative here. we add (subtract) it to the baseline to get the
         * correct "top" position to blit to */
        SDL_Rect dest = { .x = q.x0, .y = FONT_HEIGHT+q.y0, .w = w, .h = h };

        /* draw it */
        SDL_BlitSurface(glyphdata, &src, s, &dest);
    }

    /* done with the glyphdata now */
    SDL_FreeSurface(glyphdata);

    /* wait for escape */
    SDL_Event e;
    while(SDL_WaitEvent(&e) && e.type != SDL_KEYDOWN && e.key.keysym.sym != SDLK_ESCAPE);

    SDL_FreeSurface(s);
    SDL_Quit();

    exit(0);
}

This only uses the "simple" API, so its results aren't as good as stb_truetype is capable of, but it was enough to play and see what the output is like (very good).

As it is, I've settled on sticking with FreeType for a number of reason, but that doesn't take anything away from stb_truetype. If you're looking for basic font rendering without much overhead, do give it a try!

Like many games, Pioneer uses Lua for its content generation and dynamic world stuff. It doesn't expose enough of its lucky charms to the world this way though (in my opinion), which is why I'm currently doing a major overhaul of everything Lua related.

From looking at the code there's a couple of ways that Lua has been integrated over time. All the models in the game (ships, space stations, buildings) consist of static data (textures, object definitions) and a Lua script to hook it all together. The script can define a function that gets called as the model is rendered that can actually modify the model. This is how flashing running lights, rotating radar dishes and other things are done. Its quite a clever system. The Lua style used is pretty much plain calls to the Lua C API, with some hand-coded Lua classes for matrices, vectors, and so on.

The other place Lua is used is in the modules. These are essentially plugins that add stuff to the game based on triggers. Its how missions and smugglers appear on the bulletin board and pirates appear in the system when you enter it, to name but two functions. This interface uses a combination of normal Lua and OOLua.

OOLua is a C++ binding for Lua, which provides lots of macro and template magic to make it easy to expose your C++ classes as Lua classes. It automatically handles unpacking the Lua stack and argument typing and whatnot so that when in Lua do something like o = SomeClass:new("abc", 1) and o:thing(23) it'll arrange calls to SomeClass::SomeClass(const char* a, int b) followed by void SomeClass::thing(int a). I'll leave it to you to go and read the theory and code linkes from the OOLua homepage. Its quite interesting, though it took me quite a bit of use and misuse before I really started to get my head around it.

My plan, which of course you read because I linked it above, is to expose pretty much everything that might be useful to people wanting to add to the Pioneer universe as Lua, since that's theoretically easier for non-programmers to get to grips with. A good start has already been made to getting OOLua hooked up, so I decided after a few experiments that my first steps should be to convert all remaining non-OOLua stuff to use it. The big one here is all the code in LmrModel.cpp, which is where all the model magic mentioned above happens (indeed, LMR stands for "Lua Model Renderer", which should give some idea of just how central Lua is to all of this).

The way a Lua model definition works is pretty straightforward. At boot Pioneer loads all the model Lua files, which typically contain calls to define_model(). The arguments to define_model() contain lots of call to functions that define the model. Examples are cylinder(), texture() and so on. As you'd expect, Lua calls all of these functions to assemble the arguments before calling define_model() with the whole lot. LmrModel turns this inside out. When a call is made to eg cylinder(), it actually pushes a bunch of commands like "draw triangle", "draw billboard", "use light" and so on onto a stack. When the final call to define_model() is made, LmrModel attaches that stack to a global model definition. Its a bit unusual and can be prone to errors (eg currently if you call one of the element functions outside of define_model(), you'll usually get a segfault), but it also simplifies the code a great deal because it greatly reduces the amount of data that needs to be passed back and forth between C++ and Lua.

The difficult thing about converting all of this to OOLua is that its all the element functions are static functions, not object methods. OOLua is really only built for proper classes, and has only the most minimal support for calling an unadorned function from Lua. That minimal support does do the stack and type signature handling that I described above, so I've built an entire layer on top of it to sanely handle calling static functions from Lua. Its still under heavy development (you can follow my lua-overhaul branch if you're interested) but its already very functional. Here I want to describe a bit about how it works, because I'm quite proud of what I've been able to do. The details are at the bottom of LmrModel.cpp and in OOLuaStatic.h if you want to follow along.

OOLua requires that all static functions be registered against a Lua class (causing them to appear under its namespace). What that means is that we have to define a class, even its empty. Sucks, but lets do it.

Once there's a class in place, its trivial to register functions against it. A typical call is:

OOLUA::register_class_static<pi_model>(l, "call_model", &static_model::call_model);

pi_model is the already-registered class we're hooking the function to. call_model is the name of the function that appears on the Lua side (so in this case we've just registered pi_model.call_model). The final arg is a pointer to the C function that will be called when the function is invoked.

The function is a standard Lua callback with the signature int func(lua_State*). If you like you can use this as-is, but OOLua provides some extra magic here to use this function as a thunk that does stack unpacking and type checking before passing the call on to a real handling function. A typical callback function for OOLua looks likes:

int static_model::call_model(lua_State *l) {
    OOLUA_C_FUNCTION_5(void, call_model, const std::string&, const pi_vector&, const pi_vector&, const pi_vector&, float)
}

The arguments are straightforward - the return type, the name of the function to call, and the types of its arguments. This will result in a call to:

static void call_model(const std::string& obj_name, const pi_vector& pos, const pi_vector& _xaxis, const pi_vector& _yaxis, float scale)

If the types or number of arguments are wrong, then instead a Lua error will be generated.

So this is all very nice, but has some shortcomings. The simplest is the amount of boilerplate that needs to be written to set up a function. Some simple start/end macros to define the thunk function are all thats necessary.

The next thing I stumbled on is the need for a form of multiple dispatch. OOLua already does this for constructors, but not for method calls, which I find a little odd. What it meant is that I had to impelement it myself. Since Lua is typeless there's really no way short of some educated guessing to make a choice based on types, but choosing the function is possible based on the number of arguments. This is expected in a few places in the existing model interface. For example, texture() in its simplest form requires the name of a texture only, but its also possible to call it with extra args specifying position and transformation of the texture. So we now have two possible functions that we could call. OOLua can't support this directly, so I wrote some macros that when used, expand to (slightly simplified):

int static_model::texture(lua_State *l) {
    const int n = lua_gettop(l)-1;
    if (n == 1) {
        OOLUA_C_FUNC_1(void, texture, const std::string&)
    }
    if (n == 4) {
        OOLUA_C_FUNC_4(void, texture, const std::string&, const pi_vector&, const pi_vector&, const pi_vector&)
    }
    _static_dispatch_fail();
    return 0;

Using the macros, this gets written as:

STATIC_DISPATCH_START(pi_model,texture)
    STATIC_FUNC_1(void, texture, const std::string&)        
    STATIC_FUNC_4(void, texture, const std::string&, const pi_vector&, const pi_vector&, const pi_vector&)
STATIC_DISPATCH_END

That works well. Later I found another problem, which was a bit trickier to solve. The function extrusion() looks like this in Lua:

function extrusion (start, end, updir, radius, ...)

The first three args are vectors, the fourth is a number (float). That's not important though. What's important here is that following the required args comes an arbitrary number of vectors to define points for the shape to extrude. This posed a problem - its easy to create a macro that says "expect 4 or more arguments), but OOLua's function call mechanism fails outright if the number of args on the Lua stack don't match the number of arguments called for.

The solution I settled on was to define a seperate STATIC_FUNC_4_VA macro. When this appears in the thunk definition, it looks for extra arguments on the stack and puts them into a Lua table. It then pushes the table and the number of items in it onto the stack and calls the function with two extra arguments. All this gives the following:

STATIC_DISPATCH_START(pi_model,extrusion)
    STATIC_FUNC_4_VA(void, extrusion, const pi_vector&, const pi_vector&, const pi_vector&, float)
STATIC_DISPATCH_END

static void extrusion(const pi_vector& start, const pi_vector& end, const pi_vector& updir, float radius, OOLUA::Lua_table t, int nt)

STATIC_FUNC_4_VA expands like so:

if (n >= 4) {
    _static_varargs_table(l,n);
    OOLUA_C_FUNC_6(void, texture, const std::string&, const pi_vector&, const pi_vector&, const pi_vector&, OOLUA::Lua_table, int)
}

You can go read _static_varargs_table() too. Its interesting but not really relevant to this discussion.

So right now this is all working wonderfully well. I'm not quite finished refactoring all the functions, but its only a short hop away. But there is one fatal flaw in all this which I'm really struggling with right now. The problem is that all calls made by OOLua via its method/function macros don't have ready access to the lua_State representing the interpreter, which means if at any point OOLua can't do something for you (which is often) and you need to drop back down to the standard Lua API, you're stuck.

In LmrModel this is not a problem as the state is held in a global. In LuaUtilFuncs however there's no such global and indeed, you wouldn't want one, as these functions as used by several different interpreter contexts throughout the codebase.

Its actually tricky to solve this one. Obviously when the registered function is called its called with the Lua context in the args, so we do know it. But we lose it as soon as we ask OOLua to call our function with its fancy typechecks and stuff. We can store it in a global just for the duration of that call, but then we aren't re-entrant which could be a real problem down the track. I don't want to do that.

The only idea I have at this point is to push a pointer to the context onto the Lua stack so that OOLua can unpack it and pass it, but these seems rather heavy. Its not just a pointer either; due to the way OOLua does its type handling I have to push a full object instance. Thats a slight lie; a few primitive types like int and float don't need an object, but I don't want to do crazy stuff like casting pointers to integers to make this work.

I will try that option, but I'm keen to find something else. A thought that occured to me is that perhaps this is all wrong; perhaps it should always be the case that these functions are actually called as object methods. It makes a certain amount of sense. The model definitions could be built up in an object rather than in globals, which paves the way for object loading to be done in parallel in the future. The difficulty with this however is that the pseudo multiple-dispatch that I've implement for functions is not available for method calls, so I've undone a good amount of work.

I think at some point I'm going to need to take all this over to the OOLua author and discuss getting all this implemented properly. Its a fantastic system and I don't want to have to move away from it, but its starting to run out of steam. The author has been very responsive and helpful so I expect there's lots that can be done, which is hopeful.

That'll do for now.

Hmm, I don't write here much anymore. As is the case for lots of people, my blogging has suffered in favour of spewing random crap into Twitter, Facebook and elsewhere. I'm actually doing a lot more stuff via "social" sites in general, so I think a redesign of this site might be necessary soon to bring all that stuff into one place so I don't look dormant to anyone that just follows here.

Anyway. Christmas and holidays have come and gone. I spent a good amount of time on the big idea that I mentioned last time, and got it close to finished, but then in performance tests found that that the naive old-school Ajax implementation I'd done wouldn't scale much past 30 users. Thats unacceptable, so I started to read up WebSockets, Comet and other things to reduce network and processing on a web application. I settled on using Tatsumaki to implement a long-polling version, but that meant a rewrite of much of the server and the client. At this point I was well and truly on holiday and my brain had shut off, so I threw the project on the backburner.

This can be a dangerous thing for me, because I inevitably change my mind and do something else. I started watching Babylon 5, a show I'd somehow missed back in the day. Anyone that's read for a while knows my long infatuation with space games, so of course I started looking around for something spacey to do or play. And last week, I found Pioneer.

Pioneer is a clone of Frontier, the sequel to Elite. I always much preferred it to Elite. I think it was mostly because of the scale - I could fly forever, land on planets, and just bum around in my spaceship. So I grabbed the code, built it and had a play. And I got sold on it quickly because its awesome, but has a giant stack of things that need work still. In the spirit of the new "just do it" style I'm trying to live up to, I decided first that I wanted to hack on it and then started playing and figuring out what I wanted to hack on. After a couple of hours play I found a heap of little bugs and tweaks that needed fixing, and because the main coder is an awesome guy, lots of my stuff has already been merged.

Not much else to write. This is mostly a "here's what I'm up to" kind of post, so now you know. If you like space games do grab Pioneer (and pull my integration branch if you just want to see what I'm doing). Cheers :)

Anyone who has spent any time around me knows that I'm something of a dreamer. I think big, and if I get an idea I tend to overthink it to the point where even the thought of starting is completely demotivating. As a result, something I've been thinking about and working on for over ten years has perhaps a thousand lines of code over several rewrites to show for it.

Two articles I've read this week have both challenged me and inspired me to produce something. The first is a blog post by Tom Critchlow describing how he built booklist sharing site 7bks.com in four weeks with no prior programming experience. This is a hugely impressive feat and shows just how much the web has changed in the last few years. It used to be that sites like Geocities existed to make it easy for newcomers to build a website. Now with web frameworks Django and Ruby on Rails and easy-to-use infrastructure like App Engine its easy for newcomers to build a webb application. I've been following Pamela Fox's efforts to bring the Girl Develop It program to Australia. Although its goals are different (bring more women into IT) its recent success reinforces the point: its now easy for people with no prior experience to get into web application development.

The second article is from is from the ever-insightful Rands in Repose on how the recently-released Instagram is so awesome because of everything it doesn't do. I'm not entirely sure of the timespan involved, but the story goes that they've built an amazing mobile photo sharing application and it only took them eight weeks. They did this by remaining completely focused on the core feature set and not allowing any distractions in. Conceptually what they're doing has been done several times before, but by remaining focused they've collected huge amounts of users and standa good chance of beating out the competition.

I found both these stories rather inspiring. I've have my great idea, and I have more experience than many of the people doing these amazing things. It should be easy for me to do all manner of amazing and interesting things, but get mired in details. However I'm now resolved. I've spent much of the day distilling all my plans and dreams and thoughts into one very simple concept and I'm going to have a crack at building something soon. I'm making a huge effort to not overthink it - of course I have some ideas for where it could go, but I'm keeping these as high-level next steps, and not thinking too much about the details.

Unfortunately I don't think I'm going to have much time in the next few weeks to do much, but I'll be taking a proper holiday for four weeks after Christmas and I'd really like to have something to show for it after that.

And no, I'm not telling you the idea yet. Its probably not that amazing or exciting to most people, but its been mine for years and years and not willing to share it until I have something to show for it :)

I've recently picked up maintainer duties for Net::OpenID::Consumer and Net::OpenID::Server which have needed some love for a while. I'm starting to get them into shape and have today released developer versions to CPAN. If you're using these modules to implement an OpenID provider or relying party, then I would reatly appreciate you taking the new versions out for a spin to make sure nothing breaks.

All the details are in this post to the openid-perl list: Net-OpenID-* 1.030099_001 now on CPAN

Just a reminder that I'm speaking at the Melbourne chapter of the Google Technology Users Group tonight about the details of Monash's move to Google Apps. If you're in Melbourne and without plans you should come along!

• Details: melbourne-gtug.org
• Register: bit.ly/GoogleAppsAPIMonashUni

Well almost. I'm speaking at the Melbourne chapter of the Google Technology Users Group next week about the details of Monash's move to Google Apps. I don't have a lot of speaking experience so I'm a bit nervous but its all coming together nicely and I think its going to be a lot of fun. If you're in the area do come along and support me and learn some stuff too :)

• Details: melbourne-gtug.org
• Register: bit.ly/GoogleAppsAPIMonashUni

So Australia goes to the polls tomorrow to elect a new Federal Parliament for the next few years. My primary vote this year will go to the Australian Greens. A few people have asked me why, so I thought I'd write a little bit about my thinking. This isn't particularly coherent, just a series of random thoughts that point roughly in the same direction.

I think a solid third choice party is extremely important for a healthy democracy in this country. Without it the two major parties squabble like spoiled children. We need someone to step in and calm things down a bit. There's nothing particularly special about the Greens for this purpose; the Democrats used to fill the gap fairly well before they imploded. I'm also not under any delusions that the Greens wouldn't act the same way if they were one of the major parties. You want someone big enough that their voice and vote can make a difference, but small enough that they can't do anything useful by themselves.

This is even more important in the Senate. If there were only two parties then we'd be faced with either a friendly Senate where the government can do whatever they want without question, or a hostile Senate where the Opposition parade about and flex their muscles and prevent any work getting done. Neither are healthy. Its imperative that there be a solid third party in the Senate to force discussion and compromise to happen.

On the Greens as a party I don't agree with everything they stand for, but I'm not self-important enough to believe that a party concerned with the needs and wants of an entire country should cater to my every whim and fancy. For the most part though their vibe seems to be "look after the planet and each other", and thats something I can support. To their credit they have details of what they stand for on their website all the time, as opposed to the other parties who appear to change their tune depending on what the flavour of the momth is.

I've also thoroughly enjoyed listening to a couple of Greens members speak over the course of the campaign. Scott Ludlam was involved in the recent Communications Forum and I found him to be articulate and engaging while Conroy and Smith resorted to name-calling more than once. Bob Brown I've heard speak numerous times, and he was the same as always - engaging, interested and passionate.

The passion is contagious too. One of the problems with the big parties is they spend too much time looking for power instead of worrying about the right thing. The smaller parties tend to be far more passionate about their interests because you don't join a small party looking to gain power; you join because you care.

I've mostly talked about the Senate here. I live in the relatively safe seat of Menzies in Victoria. Its one of those seats that a few parties run a candidate in but no campaigning actually happens. I think its pretty sad really, because I'd love to vote for a local candidate that actually seemed interested. I had no hesitation in voting for the Labor candidate (Andrew Campbell) in 2007 because went doorknocking in our street during the campaign and was happy to come inside and have a coffee and chat with us. Like I said above, I don't need you to give me everything I want, but make me feel like you care and I'll happily follow you.

This time around there's nothing like that, so I'm left to other means to figure out where my vote is going. For me its more a case of working out who I'm not voting for. The reigning champion is the delightful Kevin Andrews, a man that I frankly find sickening. He has a long history of controversy during his tenure, and I can't in any good consciense give him my support. It doesn't help his cause that he's the only candidate that has sent out any kind of information during this campaign and its all been negative. Of course, I'd still rate him higher than the Family First candidate. I'm not even going to go into why I consider them to be complete nutbags; lets just say that if you're reading this and you disagree then you're probably not in my target audience :)

So I have a Labor and a Green candidate left over. Since I don't know anything about them I'm left to choosing by party. Voting Green first effectively results in a Labor two-party-preferred vote, but with the important point that the Greens get a primary vote data point. While that kind of statistic does not affect the outcome of the election, it does inform party strategists when gauging the opinions of the populace. I want my vote to register the fact that although I'm notionally voting Labor, Green issues are important to me.

So that's my vote. A few other points:

• If its not already clear, my general preference is towards Labor over Liberal. Tony Abbott strikes me as being a very nice fellow and I'd love to buy him a beer and have a chat with him, but he seems a bit random and kneejerky for a leader. But not just that, some of his crew seem worringly incomptent. I'm looking in particular at Andrew Robb and Tony Smith. At least Joe Hockey gives me a chuckle. But generally, the Liberals feel tired and old. Another term or two in opposition might give them a bit of a chance to regroup.
• On the Labor side I'm less interested in the personalities and more about what they've done in the last term. They don't seem to have made any particularly huge mistakes, despite all the carry on about the insulation programme and the school halls. Yeah, it wasn't perfect, but nothing ever is. For the most part the approach has seemed right. As for them dropping Kevin Rudd at the first sign of trouble, I don't particularly agree with it but I also know enough about the Labor Party to know that its not going to do much to change policy. As I read back over this I think that I might not be being hard enough on them, but really I just don't see anything of note, good or bad.
• I'm really confused by Steven Conroy. In the aforementioned Communications Forum I thought he did a pretty good job arguing the merits of the National Broadband Network, but then fell apart completely when talking about the filter. He's been trotting out the same tired arguments for two years now and is borders on foaming at the mouth when confronted with the quite rational arguments to the contrary. I just don't understand how he can be insightful and confident on one issue and a blubbering idiot on another one. He's either unstable or he's in someone's pocket, both of which I consider dangerous. For this reason I'm dropping him way down the list on my Senate ballot.
• I loathe the Liberals focus on "stopping the boats". Its both a non-issue in the larger debate about immigration as well as just going against my "be excellent to each other" vibe. Australia is a good place to live. For many of the people looking for a way in, their country sucks. We've got an insane amount of room and a crazy amount of spare food. You're no threat to my way of life. Come on in, enjoy the country and add whatever you're good at to make it even more awesome. And yes, there's lots of people who will make a big song and dance about a fat pile of related issues but at the end of the day these are people who need a hand, and we're in a position to help. Why shouldn't we? So anyway yeah, that loses the Libs a few points.

I'm kinda getting tired of typing now, so I'll stop. Ultimately your choice is your own, and I'd never try to tell you that you're making the wrong choice. All I ask is that you make an informed choice, consideringly carefully what you personally stand for and how the people and the parties involved will align with that. Don't fall for the sound bites, they don't tell you anything.

And if you still don't know, or think your vote won't count, then here's a nice tidbit from The Conscience Vote:

And when you go to the polls tomorrow, don't - don't, I beg you - cast an informal vote. If you can't stand either of the major parties, put your vote where your heart is - and don’t let anyone tell you that it won't count. Because you can bet that when the figures finally come in from the Electoral Commission, strategists and analysts from both sides will be going over the fine detail. Every vote that bleeds to the Greens or a minor party is a signal of discontent with the status quo.

And you're not "sending a message", regardless of what Mark Latham tells you. You're just lumped in with every ballot paper that was incorrectly filled in, illegible or just plain doodled on. If you want to send a message, do it with a valid vote.

Every single vote matters.

And remember to watch the ABC coverage of the count tomorrow night. There's graphs and stuff - its geeky and cool :)

Disclaimer: I work for Monash University, but I don't speak for them. Anything I write here is my own opinion and shouldn't be relied upon for anything.

So Google Wave has been put out to pasture. That makes it a good time for me to write a bit about what I've been working on in the last few months and what I think about the whole thing.

For those that don't know, I work for Monash University as a web developer and Google Apps specialist. We've spent the last ten months rolling out the Google Apps suite to our staff and students. We completed our rollout to some ~150K students last month, and so far have about 10% of our staff across. A big part of the reason we've been able to move that fast is that for the most part, people are extremely excited about Google technologies and how they might use them for education. That excitement goes to the highest levels of the University (one of our Deputy Vice-Chancellors was the first production user to go) and has seen Google Apps being included in our recently-announced strategy for online and technology-assisted learning and teaching, the Virtual Learning Environment.

The interest from our users in Google extends beyond the Apps suite of products to pretty much every product that Google offers, and perhaps none more so than Wave. Through the eEducation centre Monash has already been doing a lot of research into how teachers and students can teach and learn from each other (instead of the traditional top-down lecture style) and how technology can assist with that. Groups sharing and building information together is really what Wave excels at, so it wasn't long before we started seriously considering whether or not Wave was something we could deploy to all of our users.

There were three main issues that needed to be addressed before this could happen:

• Wave doesn't have enough features to allow a lecturer or tutor to control access and guide the conversation flow.
• The sharing model opens some potential legal issues surrounding exposure of confidential information, particularly to third-party robots.
• The Wave UI does not meet the stringent accessibility requirements that University services must meet

Over the last few months we've been working with the Google Wave team to address these issues.

The first is simply a case of the Wave team writing more code. Its well known that they have been thinking and working on access control stuff. Plans exist for limiting access to a wave to a set group of users, allowing robots to better manage the participants in the wave, locking the conversation root down so that users can only reply, and so on. In many ways its the easiest thing to fix, and given the commitment from the Wave team to talk to us and do something to help with what we needed we were never particularly concerned about this stuff.

I won't comment much on the legal side of things, mostly because I don't understand most of it. I do know that its a serious issue (eg Victorian privacy law is perhaps the strictest in the world) but its something that our solicitors have been working on and it probably would have come out ok in the end, if for no other reason than if it didn't people would just use the public Wave service with no protection at all. Users are notoriously bad at looking after themselves :)

The accessibility issues are where my interest in Wave came from so I'll spend a little time there.

I'll be the first to admit that I don't really get accessibility. I am in the happy position of having my whole body working as designed and to my knowledge all my close friends and family are the same, so I really have very little exposure to the needs of those who are perhaps not so fortunate. What I do understand though is that its critically important that information be available to everyone equally and achieving that is far more complicated than the old tired lines of "add alt attributes to your images" and "don't use Javascript". So I'm very happy to follow the lead of those who do know what they're talking about.

Not far away from me in my building we have a wonderfully competent team of usability and accessibility experts. They were asked to do an accessibility review of the Wave client and perhaps not surprisingly, it failed hard. Most of it comes from the difficulty of expressing to assistive technologies (eg screen readers) that something in a page has changed, particularly with proper context. The Wave client builds a complex UI on the fly (eg as the wave is incrementally loaded) and of course has realtime updates. At a more basic level though the static parts of the interface are constructed without using semantically-correct markup. A user agent (eg a screen reader) that scans the page looking for interesting things like links pretty much comes up with nothing.

The accessibility team presented their findings to some people from the Wave team and the response from where I sat appeared to be equal parts of surprise and dismay. They were receptive to the issues raised though. I travelled to Sydney shortly afterwards for DevFest and had the opportunity to chat to some of the team and they all had seen or heard of the report, so it would appear that it was taken seriously.

For me though, I could see that this had the potential to be a real showstopper to our deployment and I didn't want that as I could see the potential for Wave to be a game-changer. Since at the time I knew very little about accessibility, I started work on answering more technical but somewhat related question: "can Wave work without Javascript?". The Wave team had just released a data access API so I set to work trying to build a client using it. That work grew into the (still unfinished) ripple which more or less answers the question in the affirmative. This type of client doesn't solve the accessibility issues but its definitely a step in the right direction.

The part of ripple that I'm most proud of is the renderer. Rendering a wave well is actually quite a complicated prospect. Styles and links are represented as ranges over the static textual content. Its possible for these ranges to overlap it complex ways that make it difficult to produce semantically-correct HTML. It took three rewrites to get it there, and there's still a couple of little nits that I would have addressed sometime if this code had a future, but I mostly got there and I was happy with it :)

Anyway, these problems were being addressed, a few areas around the university started doing research and small pilots usng Wave, and it all seemed to be only a matter of time. I started work on a robot/data API client library for Perl for two reasons, one being that ripple really needed its server comms stuff abstracted properly and two being that we're a Perl shop and we would soon want to host our own robots and integrate them properly into our environment.

This was a great opportunity for me to learn Moose and my suspicions have been confirmed - Moose is awesome and I'll use it for pretty much everything I do with Perl moving forward. A few of weeks later and we get to Wednesday night and I've got things to the point where you could have a nice conversation with a Perl robot. And then I got up Thursday morning and heard that Wave was going away and all my code just got obsoleted.

I was shocked initially, but I was surprised that I didn't feel angry or sad or anything. I can hardly call the time I spent on it a waste as I learned so much (Moose, tricky HTML, accessibility, operational transform) and met some incredibly smart an awesome people, both at Monash, at Google, and elsewhere. I think for the most part though I was ok with it because its probably the right decision.

We (as in the Wave users and developers everywhere) have been playing with Wave for over a year, and we still don't know what it is and what its for. Unless you have the ability to build your own extensions it doesn't really do much for you. The interface is painful, the concepts within don't match anything else we're used to and despite various mechanisms for exposing internal data, you're still pretty much confined to the Wave client if you want to get anything useful done.

The technical issues would have been addressed with time. We would have gotten enough functionality to write a full-blown replacement client. It would have gotten much easier to expose not only data but the structure of data in other applications. But if you take that to its conclusion, Wave becomes a backing store for whatever frontend applications you build on top of it.

But what of the interface? By having lots of different ways to structure and manipulate data Wave tries to let you focus on the task at hand rather than the structure of the data. Traditional applications (web-based or desktop) are tailored to their own specific data models, so we have seperate apps for email, calendars, spreadsheets, etc. Wave wanted to pull all that information together so you could work on all the pieces of your puzzle in the same space. You start a document then realise you need some help, so you bring in friends and talk about the doc as you build it together. You need to process some data, so you drag in a spreadsheet gadget. You embed images or videos or whatever else you need to add to the discussion. Robots can help out by mining external databases and manipulating in-wave data to present even more rich information and even allow feedback via forms. Its all a nice idea, but how do you represent the different kinds of data and structure effectively? Wave tried, and we tried, but I'm not convinced anyone really had a clear idea of how to build an interface that makes sense.

It might not have been an interface issue. It might be that people want to have seperate loosely-integrated applications, one for each of the different types of data they want to manipulate. I don't think thats the case, but I think that a clearer migration path from those other applications would have helped a lot. People first came to Wave wanting to do their email in it. What if from the outset they could have easily pulled mail into Wave and if there was a "mail mode" that allowed some manipulation of Wave data in a way that they were familiar with? What about doing similar things for other data types? I'm don't know how much difference that sort of thing would have made, but something, anything to answer the "what do I do with this" question that everyone had that the start couldn't have hurt.

Wave's failure may also just be a problem of timing and circumstance. The Wave team have regularly acknowledged that they were surprised by the response. The message was supposed to be "we made something different, what do you think?". Unfortunately it was painted in the tech media as an "email killer", which of course it wasn't, but of course that's going to get everyone interested. Being such an early preview Wave was naturally buggy and slow and couldn't accomodate the load caused by the droves of users that wanted to play. So you got swarms of people banging down the door to see what all the fuss is about, and the few that got in found that it wasn't what they'd been led to believe it was and none of their friends could get in so they couldn't try it for what it was. So naturally they disappeared, disappointed, and even later when the bugs were fixed and the system was stable the first impression stuck and those users couldn't be lured back. And although there was a bit of a second wind a couple of months ago after I/O 2010, the same "what to do I now?" question came up.

From what I've seen of Google in the past, they're willing to take a risks if they see a likely or even possible positive outcome. But looking at Wave, how much future did it really have? We loved it, and we saw that it could do things better than existing services (though with some effort), but was it really going to displace them for the casual user? Was it going to make any serious money for Google? Was it ever even going to break even (remember that it takes plenty of infrastructure and manpower to develop and maintain things like this).

Based on all of this, you can totally understand an executive saying "guys, I see what you're trying to do, and thanks for trying, but the numbers just don't add up". Its not like its been a complete waste - there's some awesome technology thats already finding its way into other Google applications (eg Docs now has live typing just like Wave).

So is Wave dead? The product is, but as a concept it lives on. We're fortunate that Google and others have given us plenty of docs and code and their pledge to open-source everything remains. Then there's the third-party protocol implementations that already exist, both open-source (eg PyGoWave, Ruby on Sails) and commercial (eg Novell Pulse, SAP StreamWork). It will take some work, but any one of us could build and deploy another Wave. The question is, would you want to? I think its more likely that we'll see people incorporating bits of the technologies and concepts into new products. And maybe, just maybe, in a few years time some of the work that Wave pioneered will be commonplace and people will be amazed and we'll be those old curmudgeons saying "eh, Wave did that years ago".

So for Monash, we'll continue working on our existing plans. We've mostly been looking at Wave as a delivery platform for what we wanted to do. Not having it available means we'll have to look elsewhere for the technology we need (whether thats buying or building), but our direction won't change.

And for me? I won't continue work on ripple and Google::Wave::Robot code but they'll live on in GitHub should anyone want to rip them off for anything. My next project is building an OpenSocial container in Perl with a view to integrating it into the Monash portal (my.monash, which is where my "Web Developer" duties lie); hopefully I'll write something about it! I will however be hanging around Wave until the bitter end and I would like to do something with operational transforms in the future as they look really cool and interesting. See, its not dead, really!

And to any Wave team reading this, thanks guys. You've kept my interest and my enthusiasm alive, you've put up with my incessant questioning and harassment and you've contributed more good ideas and happiness to me and my colleagues than you're probably aware of. For the few of you that I've met and worked with already, I really hope that this isn't the end and that we get to work together in the future. I'll probably stalk you for a while to see where you end up because frankly, people are far more interesting than technology and you've all proven yourselves. Cheers :)

Today I declared myself officially on the Google Wave bandwagon when I released a tiny Wave client called ripple. I wrote it to see if it would be possible to make Wave work in only HTML, something we may soon want for work if we're to provide an accessible alternative interface to Wave for our users.

From what I'm hearing from the Wave crew, this is also the first example of doing something significant with Wave in Perl. That's exciting.

There's more detail on the splash page, but here's some quicky links to get you started:

• Try the client
• Get the code
• Discuss

Another little program from my toolbox. This one I'm quite proud of. Its a tiny little file transfer tool I call otfile (that is, one time file).

The idea is this. Quite often I need to send a file to someone on the work network. These can vary from small data files or images to multiple gigabytes of raw data or confidential documents. Our network is fast and the network and servers themselves are considered secure so I don't have to worry about eavesdropping, but there's a real problem with the transport mechanisms - they all suck.

I can put the file in an email, but there are transmission and storage size restrictions. Its also fiddly - create message, attach file, send.

I can put the file on a web server, but the only ones I have ready access to are publically-accessible, so I have to set up an access control. If its a large file then I have to think about disk space on the server (usually an issue) and then I have to wait while the file copies before sending the recipient a link. Oh, and I have to test that link myself because invariably I've screwed up file permissions or something else.

Probably the closest to what I want is file transfer via IM, but for various reasons that's currently blocked at the network level. I could probably get that block changed but it'd mean a bunch of negotiations for something that isn't actually related to my job. Its not worth my time.

So I wrote otfile. You run it with a single file as an argument, and it creates a web server on your machine with a randomised url for the file. You paste the url into an instant messaging session (I'm chatting with my team all day long). They click it, file downloads directly from the source, and then crucially, the script exits and the web server goes away. That url, while open for anyone to connect to, is near impossible to guess and only works once. That's secure enough for me.

The major thing I think this is missing right now is the ability to do multiple files at once. Its not that big of an issue because its pretty easy to run multiple instances - just a shell loop. If I went for multiple files I'd have to decide if I want to make it produce multiple urls (a pain to paste and to then require someone to click on them all), produce a directory listing (what are the semantics? when do the files disappear? when does the server shut down?) or build some kind of archive on the fly (cute, but is that painful for the receiver?). I'll probably just dodge it until I use it like that enough to be able to ask the receiver what they would have expected.

#!/usr/bin/env perl

use 5.010;

use warnings;
use strict;

use autodie;

use File::MMagic;
use File::stat;
use UUID::Tiny;
use Sys::HostIP;
use URI::Escape;
use Term::ProgressBar;

use base qw(HTTP::Server::Simple);

my @preferered_interfaces = qw(eth0 wlan0);

say "usage: otfile <file>" and exit 1 if @ARGV != 1;

my ($file) = @ARGV;

open my $fh, "<", $file; close $fh;

my $mm = File::MMagic->new;
my $type = $mm->checktype_filename($file);

my $size = (stat $file)->size;

my ($fileonly) = $file =~ m{/?([^/]+)$};

my $uuid = create_UUID_as_string(UUID_V4);

print "I: serving '$file' as '$fileonly', size $size, type $type\n";

my $server = __PACKAGE__->new;

my $interfaces = Sys::HostIP->interfaces;
my ($ip) = grep { defined } (@{$interfaces}{@preferered_interfaces}, Sys::HostIP->ip);

my $port = $server->port;
my $path = "/$uuid/".uri_escape($fileonly);
my $url = "http://$ip:$port$path";

print "I: url is: $url\n";

$server->run;

my $error;

sub setup {
    my ($self, %args) = @_;

    print STDERR "I: request from $args{peername}\n";

    if ($args{path} ne $path) {
        $error = "403 Forbidden";
        print STDERR "E: invalid request for $args{path}\n";
    }
}

sub handler {
    my ($self) = @_;

    if ($error) {
        print "HTTP/1.0 $error\n";
        print "Pragma: no-cache\n";
        print "\n";
        return;
    }

    open my $fh, "<", $file;

    print "HTTP/1.0 200 OK\n";
    print "Pragma: no-cache\n";
    print "Content-type: $type\n";
    print "Content-length: $size\n";
    print "Content-disposition: inline; filename=\"$fileonly\"\n";
    print "\n";

    my $p = Term::ProgressBar->new({
        name => $fileonly,
        count => $size,
        ETA => "linear",
    });
    $p->minor(0);

    my $total = 0;
    while (my $len = sysread $fh, my $buf, 4096) {
        print $buf;
        $total += $len;
        $p->update($total);
    }

    $p->update($size);

    close $fh;

    exit;
}

sub print_banner {}

I really need to set up a repository for things like this. Not hard to do of course, I'm just not sure if I should have one repository per tool, even if its just a single file, or all these unrelated things in one repository. I'll probably just do the latter; its way easier to manage.

Its kind of hilarious that out of everything I've done in the last couple of months this is the thing I decide to come up for air with, but its been that kind of a day. This is the result of three hours of study and hacking. Its using the new IMAP OAUTH mechanism implemented by Gmail to let me login as one of my users via IMAP.

#!/usr/bin/env perl

use warnings;
use strict;

use Net::OAuth;
use URI::Escape;
use MIME::Base64;
use Mail::IMAPClient;

# user to connect as
my $username = q{some.user};
# apps domain
my $domain   = q{some.domain.com};
# oauth consumer secret. dig it out of the "advanced settings" area of the apps dashboard
my $secret   = q{abcdefghijklmnopqrstuvwx};

my $url = 'https://mail.google.com/mail/b/'.$username.'@'.$domain.'/imap/';

my $oauth = Net::OAuth->request('consumer')->new(
    consumer_key     => $domain,
    consumer_secret  => $secret,
    request_url      => $url,
    request_method   => 'GET',
    signature_method => 'HMAC-SHA1',
    timestamp        => time,
    nonce            => int(rand(99999999)),
    extra_params => {
        'xoauth_requestor_id' => $username.'@'.$domain,
    },
);
$oauth->sign;

my $sig = $oauth->to_authorization_header;
$sig =~ s/^OAuth/'GET '.$oauth->request_url.'?xoauth_requestor_id='.uri_escape($username.'@'.$domain)/e;
$sig = encode_base64($sig, '');

my $imap = Mail::IMAPClient->new(
    Server        => 'imap.gmail.com',
    Port          => 993,
    Ssl           => 1,
    Uid           => 1,
);
$imap->authenticate('XOAUTH', sub { $sig }) or die "auth failed: ".$imap->LastError;

print "$_\n" for $imap->folders;

I guess three-legged OAuth would be pretty similar to get going, but I don't have a particular need for it right now.

Nothing big to report, just work plodding along, so here's an update in pictures.

Here's the fully assembled N64 RGB DAC board:

It's not currently working though, so some debugging is required. When I hook it up I get no picture, and occassionally it seems to short out the whole console. I haven't really had time to diagnose it properly yet. I'm mostly waiting to figure out a systematic approach, since its all a bit confusing right now.

My other project is the USB-to-N64 controller bridge. I've written a lot of AVR assembly for this so far but haven't done any actual hardware work. Its coming very soon though, so I very carefully removed the cable from one of my controllers (so I have the option of putting it back together later) and added some pins to the end so I can connect up a breadboard:

This is my first outing with shrinkwrap tube and it was a breeze. My wife has a hot air gun that she uses for her craft work, mostly with embossing inks, so I borrowed it and it worked brilliantly. I was surprised at how much physical strength it actually gives.

This is the insides of my N64:

The large ribbon is the digital signal tap for the RGB DAC, soldered to the inputs on the existing composite DAC chip (easier to solder to that chip than to the much narrower output pins on the video chip. The brown/black pair in the left is the 3.3v power feed for the RGB DAC. Over there on the right under everything is a DE9 D-sub connector with lines for the outputs from the RGB DAC (the narrower ribbon), audio from under the multi-out (the purple/gray/white ribbon) and a 5v line that's needed for some SCART signalling (the fat red wire). Right now its actually hooked to a 3.3v line under the board because I was testing something. Soon I'll hook it instead to the 5V regulator you see just to the right of the composite DAC.

Finally, some recent ebay loot:

Clockwise from top left: a pack of 78L33 3.3v voltage regulators; a sheet of 6 74HC374 8-bit latches and 4 74HC04 inverters; an anti-static pack containing two ATmega88 microcontrollers; a giant roll of 560-ohm 1% resistors (190 left on the roll); a tube of 74HC04 inverters; and a pack of 10n ceramic capacitors (which I use for IC bypass caps).

As I've mentioned before, ebay is an incredible source of cheap parts. There's less than $30 of parts in this picture, and that's not everything I've bought recently. I love getting home every second day and there's a little parcel waiting for me!

My laptop has had an interesting couple of days. The main filesystem went read-only a couple of nights ago after a couple of random journal errors. After being fsck'd and cleaned up it did it again, so I reinstalled it and restored it from backup yesterday. Then last night it overheated, leading me to open the case and clean the wall of dust out of the fan. Its back together now, but a couple of lost of tiny parts means I have no indicator lights and no trackpoint. Fortunately the trackpad still works, but its taking a little getting used to. On the other hand, its not burning my lap or my hands anymore, so its probably an overall victory though its not quite feeling that way yet.

One of the things I did lose in the rebuild, due to it not living in one of my backup locations (which is /etc/ and /home/rob) is my cute little mobile roaming script. I rewrote it on the bus on the way home yesterday and thought that perhaps its interesting enough to post here.

The basic idea is that every day I switch between at least two networks. My home network has a PC in the hall cupboard which among other things runs a web proxy and a mail server. Its also the firewall, so web and mail traffic can't go out directly. Work on the other hand, implements transparent proxying (with some network authentication) and has a SMTP server, but naturally it has a different address. I also occassionally use other networks (friend's places, coffee shops, etc) which usually have no facilities at all, requiring me to fend for myself.

My laptop runs a SMTP server (Postfix) of course, because that's just what you do on Unix. I also run a Squid proxy which I point all my local HTTP clients at. This way, when I move networks, I only have to reconfigure the local proxy rather than tweak every web client I have.

I spent a long time looking for a decent roaming reconfiguration package, but I never managed to find one. Some would try to do network detection and too often get it wrong. Some would have overly complicated and/or feature deficient configuration languages. I vaguely recall that I really liked one of them but it was tightly integrated with NetworkManager, which I don't use because it could never seem to keep the network alive for more than a few minutes (and it appears to be pretty much tied to the GUI, which is painful when I need network on the console).

So, in the finest open source tradition, I rolled my own. The script itself is trivial; its just a tiny template expander. I'll list the script in a moment, but first I'll talk about its operation.

The script, which I call location, takes a location name on the command line (like home or work), runs over a (hardcoded) list of config files, reads them in, modifies them, and spits them out to the same file. It makes modifications according to templates that may exist in the file. If the file has no template, then location ends up emitting the unchanged file.

In any file you want it to modify, you add an appropriate template. This is the template I have in my /etc/postfix/main.cf:

### START-LOCATION-TEMPLATE
##@ home relayhost = lookout.home
##@ work relayhost = smtp.monash.edu.au
##! /etc/init.d/postfix restart
##! sleep 1
##! /usr/bin/mailq -q
### END-LOCATION-TEMPLATE

When it finds itself inside a template, location stops its normal operation of outputting the lines of the file as-is and instead starts parsing. Interesting lines begin with ##, anything else is ignored. Its the third character that determines how the line is interpreted. So far I have the following functions:

• #: do nothing, just output the line
• @: emit if at location. If the location specified on the command line matches the first argument to @, then the rest of the line is added to the file as-is.
• !: run command. Calls a shell to run the specified command after the file has been generated.
• >: interpolate line. Include the rest of the line in the file, but expand any %variable%-type markers. So far only %location% is defined, and is replaced with the location specified on the command line.

(I'll provide an example of that last one in a moment).

So in the case of main.cf, lets say we ran location with home as the location. This would result in the template section being written to the output file as:

### START-LOCATION-TEMPLATE
##@ home relayhost = lookout.home
relayhost = lookout.home
##@ work relayhost = smtp.monash.edu.au
##! /etc/init.d/postfix restart
##! sleep 1
##! /usr/bin/mailq -q
### END-LOCATION-TEMPLATE

The listed commands are then run, which cause Postfix to be restarted and the mail queue to be flushed:

/etc/init.d/postfix restart
sleep 1
/usr/bin/mailq -q

Naturally Postfix interprets the template parts of the file as comments, so nothing to worry about. The next time location is run, the "bare" relayhost line is ignored, so it doesn't get in the way.

The config for Squid is similar. Because Squid's config file is huge, I don't quite trust my script to handle the whole thing sanely, so at the bottom of squid.conf I've added:

include /etc/squid/location.conf

And in location.conf I have:

### START-LOCATION-TEMPLATE
##@ home cache_peer lookout.home parent 8080 0 default
##@ home never_direct allow all
##! /etc/init.d/squid restart
### END-LOCATION-TEMPLATE

By default Squid will try and hit the internet directly, which is fine for work and unknown locations. For home, i need to force it to always go to an upstream proxy, which is what those the cache_peer and never_direct directives will achieve.

The proxy at work used to be an authenticating proxy, so I had to specify both a peer and a username/password combination. This made the required amount of variable config a little unwieldy to be include in a template, which is where the > function came from. location.conf used to have this:

##> include /etc/squid/upstream.%location%.conf

Which would arrange for upstream.home.conf, upstream.work.conf, etc to be included depending on the location. There's every chance this will come in useful again one day, so I've left the code in there for now.

Here's the script in its entirety:

#!/usr/bin/env perl

use 5.010;

use warnings;
use strict;

my @files = qw(
    /etc/squid/location.conf
    /etc/postfix/main.cf
);

use autodie qw(:default exec);

use FindBin;

if ($< != 0) {
    exec "/usr/bin/sudo", "$FindBin::Bin/$FindBin::Script", @ARGV;
}

say "usage: location <where>" and exit 1 if @ARGV != 1;

my ($location) = @ARGV;

for my $file (@files) {
    say "building: $file";

    my @out;
    my @cmd;

    open my $in, "<", $file;

    my $in_template = 0;
    while (my $line = <$in>) {
        chomp $line;

        if ($line =~ m/^### START-LOCATION-TEMPLATE/) {
            $in_template = 1;
            push @out, $line;
            next;
        }

        if ($line =~ m/^### END-LOCATION-TEMPLATE/) {
            $in_template = 0;
            push @out, $line;
            next;
        }

        if (!$in_template) {
            push @out, $line;
            next;
        }

        my ($tag) = $line =~ m/^##([#@!>])/;
        if (!$tag) {
            next;
        }

        given ($tag) {
            when ('#') {
                push @out, $line;
                next;
            }

            when ('@') {
                push @out, $line;

                my ($want, $rest) = $line =~ m/^##@ (\w+) (.*)/;
                if ($want eq $location) {
                    push @out, $rest;
                }

                next;
            }

            when ('!') {
                push @out, $line;

                my ($cmd) = $line =~ m/^##! (.*)/;
                push @cmd, $cmd;

                next;
            }

            when ('>') {
                push @out, $line;

                my ($rest) = $line =~ m/^##> (.*)/;

                $rest =~ s/%location%/$location/g;

                push @out, $rest;
            }
        }
    }

    die "$file: unclosed location template" if $in_template;

    close $in;

    open my $out, ">", $file;
    say $out $_ for @out;
    close $out;

    for my $cmd (@cmd) {
        say "running: $cmd";
        system $cmd;
    }
}

Because its so trivial and I only run it a couple of times a day, I just run it when I get to work (location work) or when I get home (location home). If I felt inclined I could probably hook it up to my network stuff but I think that would be more trouble than its worth.

On occassion I have to use Windows on the same machine. I have no idea how to achieve something similar there, so I just reconfigure my browser. Fortunately I don't go there often, and almost never from work. This is why I like open source. I can make my system work in exactly the way I want and usually with a minimum of fuss.

A couple of weeks ago I placed an order with BatchPCB for the N64 RGB DAC board. Today I received two of them!

I'm quite excited to see the design all professional looking. If I'm really lucky they might even work!

As I understand it sometimes the fabs will make extras of a board in case something goes wrong. If they all come out fine then there's not much to do with the extras, so they just chuck them in as a bonus. That's good; now I have a spare if something goes wrong and I don't feel like I got such a bad deal with the insane postage cost.

eBay is the best source of cheap components in bulk. There's hundreds of stores all selling manner of things in huge quantities for mere pennies and half of the time with free shipping. This time around I've picked these up:

• 250x 1/4W 1% 560 ohm resistors
• 100x 0.01uF ceramic capacitors
• 10x 74HC04 hex inverter ICs
• 10x 74HC374 octal D-type flip-flop ICs

All for the bargain basement price of $22.50. Sure, I have to wait a couple of weeks, but I'm not in any hurry here. If I'd been really smart I would've ordered the parts a couple of weeks ago. Oh well :)

Ok, so what am I working on. Part two of my N64 modernisation project is to arrange it so that USB controllers can be used with the Nintendo 64.

The reasoning here is pretty simple. N64 controllers have a design flaw. I don't fully understand it, but the gist is that there's a magic powder inside the analog sticks that gives them their "springiness" and makes them return to centre. As the controllers wear, the powder escapes and it gets to the point where the sticks won't return to centre anymore as well as losing their sensitivity. Mine have held up pretty well, mostly because I've gone to great pains to take care of them, but they're fifteen years old now and they're starting to show it.

Obviously these controllers aren't manufactured anymore. Its not enough to buy used ones, for obvious reasons, and new ones are difficult to come by. Not impossible - I've seen them on eBay and in shops like Gametraders and Cash Converters, and I guess I wouldn't need to buy many of them, but still, they aren't exactly cheap or plentiful.

A better option is USB gamepads. As you'd expect from any PC peripheral, they're common as dirt and the good ones (eg Logitech or XBox) are comfortable, sturdy and responsive. So my thought has been to arrange it so that these sticks can be used with the N64.

First stop: Google. There's lots of projects where people have interfaced N64 controllers to something else (USB, parallel, Gamecube/Wii) or interfaced similar controllers (Gamecube) to the N64. As far as I've been able to tell though, nobody has ever got a USB stick going on a N64. Great, new territory - seems I can't avoid it.

I've spent the last couple of weeks researching and thinking and while I haven't yet done any testing with real hardware, I think I have a rough concept for how an interface might work. There's three aspects to it:

• Acting as a USB host controller and HID class implementation.
• Translating USB gamepad data into equivalent N64 button/position data
• Speaking the proprietary N64 controller protocol.

The translation is fairly straight forward. The gamepads I'm interested in (I'll be using a Logitech Dual Action pad for my testing) have (at least) two sticks, a direction pad, four buttons and two shoulder buttons. There's enough here to map to the N64 layout, which is one stick, a direction pad, two buttons (A and B), a second directional pad (C) and two shoulder buttons. The left shoulder maps to the Z trigger or the real left shoulder, as they were never used together on the N64 due to the structure of the controller. The second stick on the Logitech pad will map to the C buttons, with some threshold to determine if the analog stick is considered "pushed" or not.

The USB side is interesting. Its pretty easy to build a AVR-based USB device. V-USB is a very good software stack to turn an AVR into a USB device controller, or you can use one of the numerous chips from FTDI. For a USB host however, the options are far less compelling. As far as I'm able to tell, V-USB does not implement a USB host controller at all. I had intended to use the FTDI Vinculum VDIP1, but as I mentioned previously, the cost of shipping is prohibitive. After some more searching yesterday I found SIAM32, a software USB host controller implemented by a student project team at Cornell. I think some combination of their code and the V-USB code should be enough to implement a minimal host controller and HID class, which is all I need.

On the other side is the N64 controller protocol. Although its proprietary, its long been studied and is pretty well understood. The most useful sources of information have been tzanger's n64dev page and Micah Dowty's Cube64 project.

Its a command based protocol. The N64 sends a command, and the controller sends the response. There's no provision for the controller to initiate a data send - the N64 regularly polls the controller by sending a "get status" command, to which the controller response by sending a data packet containing the current state of the buttons and stick. There's also commands in there ask the controller if its there and what peripherals it has attached (such as a rumble or memory pak), as well as reading and writing to the memory card. Its pretty simple really, which is good - I like simple.

The connection to the N64 has three lines - +3.3V, ground and data. Obviously the power lines play no part in the communication. All that happens on the data line.

The most difficult thing about the data protocol itself is its strict timing requirements, as it typical of a serial protocol without a seperate clock. The line begins in a high (logical 1) state, held there by a pull-up resistor in the controller itself (the line is never explicitly driven high by either end).

A single byte consists of eight data bits and one stop bit. A single bit is 4 microseconds wide. To start the bit, the sender pulls the line low for 1us. The next two microseconds are either high or low, depending on the value of the bit. The final microsecond is high, then it goes again. After all eight bits are sent, the final 4us are all high to signal the end of the byte. See tzanger's page - it has some diagrams that make it easier to follow.

What this means is that however I implement this I need to be able to sample or transition the data line every microsecond. At 16MHz, that means I need to do something every sixteen cycles. Most instructions on the AVR take a single cycle to execute, so there's plenty of time to do things in between, but because I need to be able to respond to the N64 sending data within 2us, its pretty much impossible to run the USB host out of the same AVR.

So my interface has two AVRs - one doing the N64 comms, the other managing USB. This complicates things as now some mechanism is required for the two AVRs to communicate with each other.

This is the bit I'm not quite sure about. I originally thought to have the AVR ports tied together such that the USB AVR could just chuck the current state on the port and the N64 AVR could read it whenever it wanted. This is no good though because the entire controller state is 32 bits wide - sixteen for the button state, eight for the analog stick X axis and eight more for the Y axis. I don't really have the bandwidth available to do it that way, not even with the larger AVRs, which would be overkill in every other way anyway.

I've been thinking about perhaps using eight lines and sending the data a byte at a time, but at that point I've now got the two AVRs needing to coordinate communication when they could both be interrupted at any moment, breaking the whole thing. It might work if I allowed the transfer to be interrupted and in that case the N64 AVR will just use the last button state, but then this means that the USB AVR would have to be constantly streaming the current state rather than just sending updates when transitions occur. If it didn't then a state transition could be lost if the transfer is interrupted.

There's always the option of putting four eight-bit latches in between the two AVRs and storing the state there, as they can effectively be thought of as memories with seperate read and write channels. This however means slower access (external memories access instructions take two cycles in most cases instead of one) which might present timing problems, as well as requiring more board space.

I need to study the AVR datasheets to figure out if any of the peripherals it comes with can help me out. I'm sure a simple solution will present itself, I just have to find it. Fortunately the need for it is quite a way off. The USB and N64 comms need to be developed first, and they need to be done in isolation to ensure they work correctly.

So that's where I'm at. So far I'm just getting my development environment setup. This week I've built myself an AVR programmer which is working nicely, so next I need to write a few basic programs and make sure my laptop is setup properly and I know what I'm doing. Then the real work can begin :)

I was supposed to blog more often, but as usual I missed it. I'm not just going to dump everything though, but rather try and break things up a bit into more logical chunks. So today lets talk about shopping!

I've been looking around for local suppliers of parts and toys. In the past I've always just taken a trip down to my local Jaycar for whatever I needed, but I'm increasingly coming to understand that their range is actually quite limited for what I want and the prices are quite expensive. So I've naturally turned to the internet for help.

The first stop was RS Electronics. They're one of the big industry suppliers, and have a warehouse in Port Melbourne (which means fast shipping). Somehow they also offer free delivery. I placed an order with a couple of weeks ago as a tester, though still for stuff I needed: a couple of AVRs, a PIC, and some 20Mhz crystals. They had the chips in stock and they arrived by courier the next day. The crystals were only available in their UK warehouse, so they didn't arrive, but neither did they bill for them straight away either. I was a little confused as to whether or not they were still on order or if I had to do something else, so I sent an email. A couple of hours later I got a phone call from a lovely lady who apologised for the confusion and said she'd make sure the order was still good. Obviously it worked, because a week later my parts arrived. So +1 to RS. Mid-range prices, fast free shipping, excellent customer support.

Now I had 20Mhz crystals for the PIC, but I was still looking for 16Mhz crystals for the AVRs. I couldn't initially find anyone that stocked them locally, but eBay had the answer. The ALLEPARTS store operates out of China and has bulk components for a pittance and free shipping (how do folks afford this, I don't get it). The paltry sum of $8 resulted in a pack of 20 crystals arriving a couple of weeks later. That's an insane price - Jaycar charge $5 PER CRYSTAL! There are other very similar stores on eBay, but I'll probably end up back there because now I know them.

At this point I had enough to get on with the first stages of my next project, which I'll write about soon. While waiting for parts I spent a lot of time trawling for other suppliers, and found all sorts of stuff along the way.

First, the venerable SparkFun. They seem to be near the centre of the hobbyist electronic world, supplying lots of common and uncommon parts and kits, and working hard to make parts that are difficult to get or to work with accessible to mortals (eg by building breakout boards). There's tutorials and forums and all manner of things. I've spent a lot of time here reading things (particularly the Eagle tutorials) and generally lusting after things, and I probably would have laid down a chunk of cash pretty quickly if it wasn't for the insane price of shipping to Australia, a topic I'll rant about soon. Not being able to just buy stuff immediately forced me to continue looking locally for suppliers, but also to really consider what I actually need.

For example: I'm mesmerised by the Arduino, and I'm of course not the only one. My first thought was that it would be the perfect platform for learning AVR stuff, and so I determined that this was what I needed. Upon further study, I started to realise that while it is very very cool, its not at all what I want. Its a great tool for rapid development, and its been positioned so that non-technical people can use it too, but from what I can gather, its has way more overheard than I want. I'll get into it more when I talk about my new project next time, but I need some very specific hardware with insanely quick response times. The code is likely going to need to be cycle-counted to work properly.

I could do this with the Arduino by bypassing its firmware and development environment and else and just using it as an AVR board, but by the time I do that I've removed all the things that make it special and worth the extra cash. Its not hideously expensive but if you hadn't already gathered, I am (for a variety of reasons) working on less than a shoestring budget. I can't justify the cost. But I'm still very interested in the platform, and I'm thinking about buying the starter kit for a tech-savvy friend that hasn't done any kind of electronics stuff before.

Lets talk about international shipping. Its insane. There's a world of interesting and reasonably priced parts that become inaccessible because the price of shipping is often more than the price of the part itself! My current hate is the FTDI Vinculum VDIP1. Its a brilliant little module that acts as a USB host controller. This week I could just find $40 for the part itself, but I can't justify $80 once shipping from the UK is factored in. I haven't yet found a local supplier that prices it reasonably; RS have it for $70 which is slightly better but still more than I want to spend. Honestly, the freight plane pilot could put this thing in his pocket; that's how small it is. How can that kind of cost be justified?

PCB manufacture is another thing that's going to hurt. I've been spending a lot of time in Eagle and it makes preparing PCBs a snap. As I mentioned previously, I planned to try BatchPCB to get my DAC board fabricated. I sent the order in a couple of weeks ago. The board itself came to a quite reasonable US$15. By the time handling and shipping was factored in, it blew out to a cool US$50. I made the order anyway, because I want/need the board and I'm treating it as a trial, but its only going to be something I can do for complex designs. This seems to just be the nature of the industry though; board manufacture isn't cheap on small scales. I have heard good things about Seeed Studio and their efforts to make this sort of thing more accessible, but I haven't quite figured out how they work yet.

That's the shipping news. Lets get back on to buying things.

So local stores! I've managed to get a few recommendations from the local HackerSpace group (what an awesome idea, can't wait to get more involved with this). Little Bird Electronics appear to be a local SparkFun reseller, though I think they have a few other bits. I'm intending to buy a Bus Pirate from them soon, as it looks like an incredibly useful bit of kit to have on the desk.

Via Jon Oxer at Practical Arduino I found ProtoStack. They don't have a huge range, but they do have all the "essentials" for microcontroller hacking. Here's a tip though - make it clear on your website which country you're in. If I'd come across this site on my own, I probably wouldn't have looked at it in any great depth because the prices are in US dollars, the site is a .com, etc - I would have gone "crap range, US = crazy shipping, ignore". It was only because Jon's video mentioned that they're in Australia that I took the time to look at them in depth. Shipping is only $5 so on the occassion that I need something they have, I will be buying from them.

Anyway, that's about all I have about shopping for now. I have a bunch of other local sites bookmarked, but I haven't done anything more than a cursory search on most of them. I'm starting to get a list of affordable places to buy things together, though there's still a couple of bits I need to find. Fortunately I now have enough parts to do a good portion of my prototyping. That's a story for next time!

Here it is, first post of the new year on the last day of my almost-four-week break from work. The time off has been awesome because I've gotten so much down. Apart from various parties and outings and other festivities, most of the first two weeks were spent organising the garage in a pretty serious way. I took almost all of my old computer gear to the local tip (who are participating in the Byteback program, making my culling reasonably environmentally-friendly). This is a pretty big thing for me, as I'm a hoarder and had kept all sorts of stuff (mostly for sentimental reasons) dating back to 1982. I've kept one working model of every computer and console I had for posterity and/or hacking, but have thrown all the extras and all the PC stuff I had that I'll likely never use. I've kept a few things that might have some actual monetary or other value as collectables, and I'll put those up on eBay when I get around to it.

As a result of all this, my garage has just about nothing in it, so I've set up a desk and sorted all my various electronics bits, tools and whatever so they're all nicely labeled and accessible. I even put in some halogen lights so the whole place is extremely well lit and I can see what I'm doing. And no computer in sight, though I am dragging the laptop out there with me.

All of this is to support this weeks' new hobby, which is getting back into hardware hacking in a pretty serious way. I'm not sure if its as a result of general burnout or because I'm now write code for my job rather than hacking to support my job as I was previously, but I found towards the end of last year that I just had no brain for code by the time I got home. I'm thinking that perhaps hardware is close enough to what I know to hold my interest and not be completely impossible, but different enough that there's room in my brain for it. Time shall tell.

Anyway, back in May I bought a RGB-to-HDMI converter and did some work to get my Amiga going on my LCD TV. As I mentioned then, my next project was to get RGB out of my Nintendo 64 so that I could play it without it looking horrible. I began work on what seemed like a fairly modest project: to build Tim Worthington's N64 RGB DAC using discrete logic rather than a CPLD (which at the time seemed way to complicated).

At the time I didn't really want to commit any money to this project as I didn't know if it was something that I was actually capable of doing. Since I had some stripboard, connectors and most of the other parts I'd need I opted to just build the thing on stripboard and buy the few chips and resistors that I'd need.

In hindsight this turned out to be the wrong decision. Routing data buses on stripboard means a lot of bits of wire flying around, and it doesn't help that the board has to be small to fit inside the case. Over the course of the next couple of months I got perhaps three-quarters of the way there, and after a big effort in the last two weeks I produced this monstrosity:

Yeah, I know. There's more pics on Flickr but it doesn't get any prettier.

There's not much to it. Its four data latches (one for each of red, green and blue and one for the control lines), a couple of support chips and three R2R ladders.

In spite of the mess I still had high hopes for it, so I hooked it up and to my great surprise it (sorta) worked. Here's what Super Mario 64 looks like on my TV with the standard composite cable:

The major thing I'm trying to fix here is the weird "hatching" effect on high-contrast edges (like the life/star counters). Its not bad in static screens, but once things start moving its a horror on the eyes; its pretty much impossible to play.

But, with the magic of wires, we get this:

As you can see, everything is nice and crispy, which is was the desired result. Some of the colours are off though, which obviously isn't right.

Another example, this time from Yoshi's Story:

Composite:

RGB:

I haven't had the chance to really think about it in depth but with the way that the colours are generally brighter and Mario's clothes are washed out, and the way the other colours appear, my gut feeling is that I've wired the inputs to the R2R ladders wrong in such a way that they're off by one bit. With the board being so insane though I figured I have pretty much no chance of debugging it and even if I do figure it out its going to kill my fingers to try and make any changes to the board.

Actually getting the damn thing to work though has given me a lot of confidence and so I've decided to build it again, but this time done right, which means a real PCB. So over the last week I've been teaching myself how to use Eagle, a circuit and PCB designer that seems to be pretty popular. The learning curve is pretty steep, but I've made some good progress with it.

The first thing you do is draw the circuit. I've pretty much just copied Tim's design, getting this:

Next comes the board layout. Its pretty straightforward: setup the board size, place the components, then add all the wire routes to the board. The last bit is made simple using Eagle's autorouter. Various forums and whatnot suggest that real board designers don't use the autorouter, but I don't care - it seems like it will work well enough and I'm just a noob here so I'll take all the help I can get.

I also found a wonderful little program called Eagle3D which produced 3D renders of Eagle boards, including components. I ran mine through it to see what it would look like and got this:

Top side:

Bottom side:

I'm feeling pretty good about this! I'll sit on this for a couple of days just to make sure I've got it right, then I'll send it off to BatchPCB, a PCB fabrication service that will do short runs (even one) for reasonable prices.

I've no doubt that I've missed something, and it won't work properly the first time, but at least this board can be debugged. I see some good looking games in my future :)

Day two of training today. I had to run off early, but not before getting a crash course in Moose. I've been watching Moose for a couple of years, and have a project in mind for it, but haven't got around to doing anything with it yet. Doing a few basic exercises with it was awesome just to see what it can do, but I did manage to get frustrated by three things within the first ten minutes.

1. The first thing I noticed is that every attribute accessor created is publically available. There isn't really a way to make an attribute that is read/write from inside the class but read-only from outside. The fine manual suggests this:

   has 'attr' => (
       is     => 'ro',
       writer => '_set_attr',
   );
   

   This works ok, but its still possible for code outside the class to call _set_attr directly. Until we get lexical subs its impossible to make the writer method invisible to the outside world, but until then I'd still like it to be possible for Moose to produce an accessor that can check its caller.

   In a similar vein, its not possible to create proper protected or private attributes. Private attributes can sort of be done by assigning directly to the object hash:

   $self->{attr} = 1;
   

   I don't like this because it because it makes assumptions about the internal implementation of the objects (and with Moose I'd like to remain as ignorant as possible on this point), but also because it provides no type or constraint checking.

   Protected attributes (that is, attributes private to a class and its subclass) seem to be completely impossible.

2. By default, a Moose class quietly accept any and all parameters passed to its constructor, regardless of whether or not they correspond to an attribute in the class or its parents. This confused me for a moment as I've come from Params::Validate which allows you to declare parameter types and constraints much like Moose attribute declarations, but dies if you provide a parameter that is not declared. The fine inhabitants of #moose on irc.perl.org pointed me at MooseX::StrictConstructor, which does what I want - dies if undefined parameters are provided.

   It gets better though. I was declaring an attribute that I wanted to be impossible to initialise via the constructor as I planned to set its initial value in BUILD, and to allow the user to provide a value only to ignore it is confusing. The manual explains that specifying init_arg => undef in the attribute definition will arrange for that parameter passed to the constructor to be ignored, but again, it does it quietly.

   It turns out (again via #moose) that combining MooseX::StrictConstructor with init_arg => undef yields the desired results. I can live with that, but I would never have anticipated that result from the documentation. Hmph.

3. Moose doesn't provide any syntactic sugar for class attributes/methods. A quick search just now turns up MooseX::ClassAttribute which will probably be as much as I'll need, at least initially, but I was surprised that core Moose didn't have anything for this. Are class attributes so uncommon?

At the end of the day though, these are all pretty minor nits. Moose is awesome. Its very actively maintained and developed by a number of incredibly smart people, so its not going away any time soon. I'm looking forward to having the time to do something serious with it.

Work is sending me on a Perl Training Australia course this week, so I'm getting to hang out with Paul and Jacinta and get a good refresher on Perl OO. I wouldn't say I needed it, but I've been enjoying the discussion and it never hurts to make sure that your accumulated understanding matches the current reality.

One of the exercises involved a class representing a coin with methods to flip the coin. One of the things we were asked to do at one point was to create an array of coins and do various things to them. My first instinct to create the array was to do this:

my @coins = (Coin->new) x 10;

I was saddened but not surprised to find that this doesn't work. As the following test demonstrates the left hand side is only evaluated once and then just copied, so I ended up with an array containing ten references to the same object:

$ perl -E '$c = 0; @x = ($c++) x 10; say @x'
0000000000

The best I could come up with is this, which I don't think reads anywhere near as well:

my @coins = map { Coin->new } (1..10);

We briefly discussed whether it would be worth developing a core patch to do something like it, but realistically the only option that preserves a reasonable amount of backward compatibility is to only reevaluate the left side for a very specific set of types, namely code references, giving something like this:

my @coins = (sub { Coin->new }) x 10;

Given that that really doesn't read particularly better than the version using map, and not knowing if anything smarter is possble (and how to do it if it is), and knowing that the core developers aren't particularly keen on new features to existing constructs at the best of times, I've opted to leave it for now but keep my eyes open for things like this.

One part of another exercise had me dealing with decks of cards. Internally I represented suits as integers, with the following list to assist with the as_string method:

my @suits = qw(hearts spades clubs diamonds);

When I got to adding the initialiser for the class, naturally I wanted to be able to specify a string. The usual thing I'd do here is create a hash from @suits with the values as the integer array indexes. This time I came up with this one-liner to determine the index of a value in an array:

my $index = do { my $found; grep { $found = 1 if $_ eq $needle; $found ? 0 : 1 } @haystack };

It plays on the fact the grep in scalar context returns the number of matches; that is, the number of times the code block evaluates true. All this does is arranges it such that the block is true for every array index before the wanted value but false for every index after (and including) the wanted index. If $index == @haystack, then it wasn't found.

Its certainly not optimal - a binary search is always going to be quicker, and you'd nearly always want to use the hash method if you were doing it many times, but it was certainly fun to write a cute oneliner to do it.

A couple of months ago I started work in a new job, doing various programmery things for the Monash student intranet thing The job description says "web programmer", but there hasn't been much web yet. At this point I'm mostly concentrating on the glue code needed to hook up various Google services to our environment. 99% of the code I'm writing and working on is Perl.

I've been using Perl for ages though. For the last nine-and-a-bit years I've been a mail sysadmin at Monash, and while the "core" of our systems has always been full-on professional mail software packages (both proprietary and open-source), all the bits in between have always been Perl. We've written all sorts of stuff, from full web environments and workflow packages to all the traditional sysadmin tools like log parsers, report generators, config builders and everything in between. There's never been any question for us - Perl is just so obvious for this kind of work.

Previously though, Perl was merely a tool that I used to get the job done. In many ways though its now become the job itself. For any given day I can be reasonably confident that most of it will be reading or writing Perl code, whereas before I'd only bust it out when I needed it. Additionally, I haven't really done this type of work before so I'm getting lots of ideas for stuff I want to play with on my own time and also finding gaps in my knowledge that I want to fill out. So now I'm finding just about every moment I'm at the computer I'm doing something with Perl, far more than ever before.

So, I've decided that it would be really good to finally join the mob rather than just hang around the edges looking in. I'm signing up for the Iron Man challenge to keep me honest, and I'm moving my code to Github for a bit more visibility. This is going be an interesting change for me, as in both blogging and coding I'm used to producing something large and fully-formed before showing the world, but obviously that doesn't work if you need to post once a week. I've started making a list of little Perl things I can write about in a couple of paragraphs, so hopefully I'll be able to keep it fresh.

Additionally, I've committed myself to writing everything I possibly can in Perl. I have a long history with C as well, and for the longest time always reached for it for anything closer to the hardware/OS (a fuzzy line, but typically that means server-type things). No more. From now on unless there's a very specific reason why Perl is unsuitable, I'll be choosing Perl for my code.

So that's it. Hi :)

As is often the case with me, what started as a small hack to blosxom to make it do tags and per-tag feeds turned into me rewriting it from the bottom up. I quite like what its become, though I doubt its of much use to anybody but myself. Give me a yell if you want the code.

Anyway, now the whole site has proper tags, and you can use them to subscribe to just bits of my ramblings rather than the whole lot, which should make a huge difference considering how much I don't actually write. Oh, and there Atom feeds too if you like that sort of thing.

To celebrate this momentous occasion, I've moved the whole mess to a new and somewhat relevant domain, eatenbyagrue.org. I think I got all the redirects right, so existing subscriptions should work ok.

That's all. Back to work shortly.

I love blosxom, so I persist with it, but its a nightmare to refactor. It could have done what it does just as well without being quite as clever. I wonder if that's why development on it is mostly dead. In any case, the refactoring goes well, and I hope soon to have something that does exactly what I want (which I'll talk about more soon; better to spend my time on code right now).

Tonight I wrote a simple tag plugin for blosxom (the blog engine I use), imported all the categories from the old Wordpress blog as tags, and wrote some CSS to make it work properly. There's still a bit more to do, notably making the tags into links that take you to ther posts tagged the same thing as well as getting per-tag feeds going (though the RSS plugin needs a lot of work anyway).

The plugin, for the curious:

package tags;

use vars qw($tags);

sub start { 1 }

sub story {
    my ($pkg, $currentdir, $head_ref) = @_;

    $tags = '';
    if ($meta::tags) {
        my @tags = split /\s+/, $meta::tags;
        $tags = "<div class='tags'>tags: <ul>";
        $tags .= "<li>$_</li>" for @tags;
        $tags .= "</ul></div>";
    }
}

1;

Obviously, its really the meta plugin that does most of the heavy lifting.

So now at the top of a post I write something like:

meta-tags: site perl

and tags pop out. Lovely!

Ooh, it has been a while, and a lot has happened. If I wrote properly about everything, I'd be here all day, so here's my usual long list of stuff I've been up to. Much of it is interrelated in strange ways, so this is not necessarily chronological. You'll get over it.

• pyro, my trusty laptop, suffered a massive hard drive crash about a month ago after a bungled gparted session. I have backups of all the really critical stuff, so it wasn't a complete tragedy, but there's plenty of stuff that I may have lost. I say "may", because via the magic of e2extract I seem to have determined that while the directory structure is completely destroyed, most of the data is intact. I currently have a 120GB filesystem image sitting on an external hard drive waiting for me to get around to sifting through it for data.
• as a result, I haven't really touched any AROS code since then. Unless I can save my git pack files, I actually have lost quite a few of my private development branches (anything not saved to repo.or.cz. The hosted rewrite is still there at least, should I ever decide to get back to.
• so, laptop rebuild. Its actually much better put together now, I think. The Linux side has less warts than it did before, and most of the hardware seems to be working just fine. On the Windows side I installed Windows 7, and I'm actually quite impressed with it (and I've never been impressed with Windows in the past). It seems to perform much better than XP ever did (I never tried Vista).
• I bought my wife a new laptop, and installed 7 there too, and she's very happy with it too.
• not one to make the same mistake twice, the very next day after the crash I went out and bought a 1TB external drive and bolted it to our home server and setup automatic backup regimes for both our laptops. This is particularly good for the wife, who has several gigabytes of photos from the last five years of our life (since the kids have been around).
• without much motiviation to salvage my code, and with work proving rather taxing on my brain, I started spending my evenings loading up Windows and playing Left 4 Dead. A lot. They recently released support for third-party maps and stuff, so my brothers and I have been trying some really great new campaigns and having a lot of fun with it.
• then the really interesting bit happened. I was approached by another area within my division and was asked if I was interested in coming across to work on an extremely interesting new project. I can't go into details yet (the whole thing is still tied up in legals), but essentially I'm getting paid the same to cut code and design systems, with no management responsibility of any kind. This is something close to my dream job and something I never thought I'd be able to get at Monash, so after a token amount of consideration I left my job of nine years and took the plunge. Today is my fifth day and so far I'm loving it - the work is interesting, the enviroment is comfortable and the mood among my coworkers is very happy and light, which makes a huge difference from the doom and gloom that permeated my former life. I do miss my team though - I've worked with some of them so long that they've become like family to me. At least they're only a short walk away, so I can still get a coffee with them from time to time.
• for various reasons I'm not allowed to keep my laptop, which belongs to my old department (though they're kindly letting me continue to use it for a little while), so my nice new job have bought me a nice new laptop. Christened junai, it arrived yesterday and I should be able to bring it home and switch to using it full-time today. Cloning a Debian system is trivial, by the way.
• the new job is all Perl, which I unfortunately haven't had much time for lately, so as well as enjoying the day I'm finding I'm getting home and wanting to work on that too. Since it fits in with work doing it on my own time is actually just more of the same rather than being yet another thing I have to try and fit into my life. As such, I'm thinking I might spend some of my spare time getting a pile of unreleased code into shape and onto the CPAN, and maybe, just maybe, getting along to the occassional Melbourne.pm meeting.

I think that's everything. The other thing I'm going to try and do is blog a little more often, probably by lowering my standards a little and not writing an epic tome each time :)

The last couple of months have been busy but I've managed to find bits of time here and there to hack on the new AROS hosted port. Last week I really got the guts of the task switching and interrupt code working the way I wanted, which is what I'm here to tell you about today.

Task switching in a typical multitasking system is very simple in concept. Imagine a computer running a single task. There's a big pile of instructions in memory somewhere, and the processor just runs them in sequence. It will keep doing that until something stops it. That something is the most important requirement to make preemptive multitasking work.

What usually happens (again in very simple terms) is that there's an extra bit of circuitry somewhere in the computer that works as a timer. Every now and again (though tens or hundreds of times a seconds), it will prod the CPU. In response, the CPU will stop what its doing and go and run a different bit of code somewhere else in memory. The "prod" is known as an interrupt (or Interrupt Request (IRQ)), and the bit of code that runs is the interrupt handler (or more formally, the Interrupt Service Routine (ISR)). Its the handler's job to arrange for a different task to run.

Something the CPU will do when responding to the interrupt is to save its complete state (known as the context) before it calls the handler. That is, somewhere in memory (typically on the stack) it will save a copy of all its registers, the stack pointer, the program counter and everything else it needs to continue running the program from where it was stopped. This is necessary as the handler will need to use those registers in order to do its work. Many CPUs provide a single instruction to restore the entire CPU state in one go.

To make task switching work, the interrupt handler will take a copy of the context and store it inside the OS task state, which usually contains lots of other info about the running task, such as memory it has allocated, files it has open, etc. Then, the handler chooses another task based on some criteria (this is the scheduler). Finally, it copies the saved context from the state of the task to wherever the CPU needs it, then tells the CPU to reload the context and leave the handler. The handler "returns" to running the newly selected task. This process contiues ad infinitum and you get the illusion that your computer is doing lots of things at the same time.

The existing Unix-hosted version of AROS does fundamentally the same thing, but in a highly convoluted way. The main thing to note is all tasks run inside a single Unix process, which then does some deep magic with Unix signals to make interrupts and task switches are happening. The kind of magic employed is highly OS-specific, and although I don't know exactly why it was done the way it was, I can guess that it was one of:

• The facilities for user-space task switching weren't available or were incomplete when it was first written (I know this was the case for Linux)
• Originally AROS was much more tightly integrated with the Linux desktop (eg one AROS window per X11 window, etc)

Times have changed though, and so what I'm trying to do is make a new port that is designed to be much closer structurally to its native cousins. I'm realising this through a number of mechanisms provided by POSIX: threads, signals and the ucontext set of functions (though somewhat ironically these have been removed from the latest versions of POSIX and SUS).

What I do is this. I create a thread to mimic the function of the timer interrupt delivery circuit. It sits in a tight loop, waiting a little while then sending a signal to the "main" thread. This obviously mimics the the interrupt that would exist on a real system, and causes the main thread to stop what its doing and jump to a signal handler.

When a signal is delivered to a Unix process, the kernel saves the current process state (context) onto the stack and then calls a signal handler function. When the handler returns, the kernel reloads the state from the stack and continues from where it was. This sounds like almost exactly what we want, except Unix typically doesn't provide a portable way to get at the saved state on the stack. The existing hosted AROS implementation for Linux uses a bunch of Linux-specific knowledge to dig into the stack and get the data it needs, but thats obviously not portable. These days however, we have the ucontext functions which, while not without their quirks, are far more useful.

The prototypes look like this:

• int getcontext(ucontext_t *ucp);
• int setcontext(const ucontext_t *ucp);
• void makecontext(ucontext_t *ucp, void (*func)(), int argc, ...);
• int swapcontext(ucontext_t *oucp, ucontext_t *ucp);

For those who've seen setjmp() and longjmp() before, getcontext() and setcontext() will be quite familiar in function. getcontext() takes a copy of the current process state, including the CPU context, and drops it into the memory pointed to by ucp. setcontext() restores the process state and CPU context from whatever is saved in in ucp, effectively causing a direct jump to the point just after the getcontext(). What this means is that you get the appearance of setcontext() never returning, whereas getcontext() can return multiple times. Interesting times indeed.

makecontext() takes an existing context and modifies it such that when setcontext() is called on it it will jump to func with the arguments specified on the on the stack. You actually need to do a bit of fiddling inside ucp before calling it, to setup an alternate stack for the context to run on and so forth. For the most part this call is not particularly useful except when setting up.

Finally, swapcontext() is an atomic context get-and-set. That is, it does this:

getcontext(oucp);
setcontext(ucp);

except that a later setcontext(oucp) will return to the point after the call to swapcontext().

Armed with this knowledge, we can now take a look at the (slightly simplified) implementation. The task switch "interrupt" handler, is a two-stage process. The first part, which as far as the Unix kernel is concerned is the actual signal handler, looks like this:

ucontext_t irq_ctx;
char irq_stack[SIGSTKSZ];

void irq_trampoline (int signo, siginfo_t *si, void *vctx) {
    getcontext(&irq_ctx);
    irq_ctx.uc_stack.ss_sp = (void *) irq_stack;
    irq_ctx.uc_stack.ss_size = SIGSTKSZ;
    irq_ctx.uc_stack.ss_flags = 0;
    makecontext(&irq_ctx, (void (*)()) irq_handler, 0);

    swapcontext((ucontext_t *) GetIntETask(SysBase->ThisTask)->iet_Context, &irq_ctx);
}   

(irq_stack is initialised during startup as irq_stack = malloc(SIGSTKSZ))

So the signal from the timer thread arrives, and the current task gets interrupted and we arrive here. The getcontext() and makecontext() bit sets up a new context that, when called, will call the actual interrupt handler (ie the scheduler etc) and select a new task.

Its the call to swapcontext() that is most interesting. What this does is save the current context into the current task structure, and switch to the interrupt handler proper. The handler calls into the scheduler to choose another task then calls setcontext() on its saved context to start it up. The subtlety is in the fact that when the saved context is later used to start the task up again, it will return to the point just after the call to swapcontext(), immediately drop off the end of the signal handler and head back to where it was.

You might wonder why the more obvious method of using getcontext() to save the context then calling the scheduler directly isn't used. The problem comes from the fact that when getcontext() "returns", the caller has no way of knowing if it was the initial call to save the context, or if it was as a result of setcontext() being called. Without this knowledge, we're left to this kind of trickery so that the only time we end up after the context being save is when the context is reloaded.

(This is the opposite of setjmp(), which returns zero from its initial call and non-zero after a call to longjmp(). It perhaps makes the code easier to read to just have a call and test to determine what to do next, but its slightly slower and it would also result in the handler being run on the task stack, which means making the handler more complicated to make sure it rewinds correctly when the task is switched back. Or tricks can be played with sigaltstack(), which further complicates things.

The actual implementation is naturally a little more complicated, mostly because it has to deal with so-called "system calls", which is what happens when an application triggers a task switch (eg by calling Wait()). To allow that, each interrupt signal carries a numeric id that allows the trampoline and handler to determine what type of interrupt was requested. Then, when Exec wants to force a task switch, it will trigger the interrupt requesting it, which will make the scheduler with the main task "stopped", as above, but with slightly different semantics. It doesn't add much code though, and the technique is identical.

There's still lots to be done to clean up the scheduler, which so far is a hack job of the hack job already present in the mingw32 port. The next thing to do is continue to work on the boot sequence, which is almost there but is just a tiny bit finicky at the moment (that's a technical term). Next time I think I'll write about the new host module setup which blows hostlib.resource out of the water (if you know what that is)!

My current bus activity is AROS hacking. I've actually been doing at least an hour a day for the last couple of months, so I'm making plenty of progress, but I'm off on a long and exciting tangent so it all seems quite different to what I was doing before.

I started thinking about what it would take to make cairo a "first-class" graphics system, sitting directly on top of the graphics drivers, bypassing graphics.library completely. This isn't a crazy idea - a major part of graphics.library is providing drawing and font rendering primitives, similar conceptually to what cairo does (though cairo is of course far more advanced). My thought is that we make the graphics system at the bottom of the stack for apps do all sorts of crazy compositing and whatever other eyecandy effects, and the whole desktop benefits. Initially it could operate alongside graphics.library, but it'd also probably be reasonable to implement graphics.library functions on top of cairo at some later time.

From there I started looking at the graphics driver API. What we have works well enough (despite the deficiencies that I've complained about in the past), but its not a particularly good fit to the cairo backend API, and from what I understand, not a great match for a modern 2D hardware interface either. So the next thing I started thinging about was to change the graphics drivers to have the exact same interface as the cairo backend API. From there, a driver and/or the hardware could directly accelerate cairo drawing operations. The cairo software fallbacks are pretty heavily tested and optimised (including some tight assembly versions of things where necessary), so I'd expect that even a graphics card or whatever that doesn't offer a lot of function could still go faster than, say, the current SDL driver (which uses the graphics.hidd fallbacks for just about everything currently).

So now I'm looking at drivers. As you know, I work in hosted, so my two examples are the X11 and SDL drivers. Something I hate about the X11 driver is how closely tied to the underlying kernel implementation. I took some steps to deal with this when I wrote the SDL driver with hostlib.resource, but its not perfect, and lately something has changed in the X11 driver to require it to be linked with the kernel once again. Besides that, the X11 driver is ancient, hailing from a time where AROS windows were X11 windows, and it retains a lot of that structure even though its no longer the way the world works. Also, it relies on the X11 "backing store" feature, which is usually disabled and will shortly be removed from Xorg. In short, the thing needs a rewrite.

So yay, rewriting one, maybe two, graphics drivers. Down a level to figure out what's going on the core, and sure enough, more work required there. In the last few years the structure of an AROS kernel has changed to be a minimal kernel.resource which implements the absolute minimum required to initialise the memory and task-switching hardware and hand control to exec.library. The loader (typically GRUB) can optionally get whatever modules (libraries, resources, devices, etc) into memory and make them available to exec when it starts. This is the basic idea behind the so-called "modular kernel", which has been implemented in the x86_64, Efika, SAM (both PPC), and more recently, mingw32 ports. The only ports that don't do this are the first two - Linux hosted and i386-pc.

The mingw32 port is particularly interesting. Its a hosted port to Windows, and in essence uses the OS threading system to implement a minimal virtual machine, all within kernel.resource. It has a small bootloader that loads an ELF kernel, making it so that stock AROS i386 code can be used even on Windows which doesn't use ELF itself. The other thing it does is neatly split modules into host-side and AROS-side parts. The AROS parts are handled as normal modules, but in their initialisation they call into hostlib.resource (which is now contained within kernel.resource) to load and link the host-side part. These are standard shared libraries (ie DLLs) which can bring in any library dependencies they need, neatly avoiding the problem contained within the X11 and SDL drivers in that its kinda painful to find the needed libraries at runtime. This way, you just find what you need at link time.

And so, after all this, I'm doing a new port of AROS to Linux, based on the structure used for the mingw32 port. I'm improving on it a bit though. There's still too much arch-specific code in exec.library (like thread context manipulation) which I'm hiding inside kernel.resource. I'm also adding a host.resource which will provide ways for modules to hook into the system main loop inside kernel.resource to do things like "virtual" hardware and the like (ie faking interrupts and such). The mingw32 port did this via special architecture-specific calls in kernel.resource, but I want to try to make kernel.resource have a standard interface across all ports, so they can all run an exec.library that is substantially the same.

So that's some kind of plan. I'm currently at the point where the kernel.resource boots and gets exec.library online. The next thing I need to do is reimplement my task switching and interrupt core which I never tested. If you feel like googling something, it turns out that ucontext_t is not particularly easy to copy or cache on Linux due to the Linux people messing up the way they store the floating point state. I need to rewrite it based on the wonderful context_demo.c example, which never requires an explicit context copy and should do much better. After that I should be able to hook DOS up and get something interesting happening.

I'll keep working and maybe let you know some more in another month or two :)

Yeah, its been a while. I'm still here, and I've done heaps of stuff since last time, but I just haven't gotten around to writing about it yet. I'll get there.

What I'm here for tonight is to tell you about something new. I know there's people out there blogging about AROS. I'm subscribed to a few of them myself. I'm sure I haven't got all of them though. So I'm putting together a planet to list them all:

• http://planet-aros.cataclysm.cx/

If you're trying to follow what's going on with AROS, it'll be good for you to subscribe to this planet, as you'll find out everything that's going on. If you're blogging about AROS, it'll be good for you to be on this planet, as you make sure that everyone is reading your stuff and you benefit from other people's popularity.

If you write about AROS, email me (rob@cataclysm.cx) or ping me on IRC (fce2 on irc.freenode.org). Let me know the location of your RSS or Atom feed, and I'll add you. Its cool if you have non-AROS stuff in there, this is about AROS people as well as AROS itself.

If this gets big and popular, I'll see what I can do to get a better URL. How does planet.aros.org sound? :)

Oh, and I need to do something to pretty it up a bit. If you feel like doing something there, drop me a line.

Gub and Penny went out with a friend last night, so with the girls in bed I got a solid five hour block all to myself. At the end of the night I'd achieved something I'm quite proud of:

This is my ancient Amiga 500 hooked up to my nice tv via the RGB-to-HDMI converter I got for my birthday.

I built two cables. The first is the main one that I'll need for all the things I intend to do with this converter. It turns the somewhat unwieldy SCART input on the converter into a rather more convenient DE9 connector. I've done my own version of the Game Station X connector - I've left out the Luma and Chroma lines but added a ground line. So in short, this cable is carrying the red, green and blue video lines, left and right audio, ground and a +5V line - everything I need.

There's not much to it. The only tricky bit is that SCART has a line (BLNK on pin 16) that selects whether the input is composite or RGB. The line needs to be fed 1-3V to select RGB; leaving it unconnected gives composite. Connecting it to the +5V line via a 180-ohm resistor makes a nice voltage divider and provides the needed signal.

The second cable is one that takes the video off the Amiga and turns it into my custom format. I originally intended to modify the cable I built years ago to connect the Amiga to my 1084S monitor (which died long ago), but on opening it I found that the cable was a five-core cable and wasn't carrying the +5V line. A quick dig around in the garage revealed a short length of six-core phone cable, so I used this to build a new cable. I also had to cannibalise a 2xRCA-2xRCA cable to provide the audio (which doesn't come through the video port on the Amiga but instead via two RCA ports), so I now have this rather peculiar looking plug with two distinct cables coming out of it.

The most surprising thing about all this is that it worked first time. I dabble but I am most certainly not an electronics guy. There was a few tricky bits where I just took the option that seemed most obvious, but I really expected it not to work because I hadn't understood some obscure detail. I'm excited that I'm able to do this! I keep old game systems around because I like to play the games from time to time, but most people when hooking their old machines up to a new tv and seeing it looking crap would be powerless to do anything about it. I like that I know enough to be able to buy or build things that can make it work!

It does seem that my Amiga has suffered from its long storage. Half the time it doesn't start at all, and sometimes it crashes at boot:

But damn, that text is crisp!

Next is the Nintendo 64 mod to get the RGB lines out. I'm hoping to find some time for it this weekend.

I'm pretty sick right now and was just getting ready to go to bed when someone knocked on the door and handed me a package:

Its my birthday present!

Inside, a plain unbranded box. China's very finest:

Inside that, a few things: a single page "manual", the converter unit itself, a custom breakout cable for component & audio input (eg from a DVD player) and the power supply with an awesome giant UK plug, egads.

Front and back sides of the converter unit:

This afternoon I'll go around to Dick Smith and grab a power converter so I can plug it in and then give it a test with the DVD player. If I get some time tonight and its not too cold I'll camp out in the garage and start trying to build some cables.

Long ago I wrote a SDL driver for AROS hosted. Back then I wrote about it being slow because of deficiencies in the driver interface that require flushing the output to the screen for every pixel plotted by software fallbacks. Go and read that first.

I never did finish my implementation originally, but in the last week I've resurrected the branch and completed it. Its taken adding an UpdateRect method to the bitmap class and then modifying graphics.library to call it after ever operation. If its running a tight loop to paint a rectangle or something, it will call this once when its finished to push its output.

To test, I removed all the "accelerated" methods in the SDL bitmap class, leaving only GetPixel and PutPixel. Back when I first writing sdl.hidd this was all I had implemented, and it worked fine, but was slow enough that you could watch the individual pixels appear on the screen. With the UpdateRect stuff its now very usable. Its not blinding fast, but its snappy enough to be comfortable.

And the best thing is that no changes are required to existing graphics drivers. For those, the call to UpdateRect will just use the baseclass version, which is a no-op. I've confirmed this is actually the case with the X11 driver, so yay.

I'm not sure what's next for my hacking. I'm really just studying graphics.library and graphics.hidd at the moment, trying to get my head around how it all fits together. Something big is coming, I'm just not sure what it looks like yet :)

It would appear I'm back in the AROS game for a little while. I got a nice email asking for some help with fat.handler so I decided that I'd look into it. In the last 18 months a few things have been broken in things that I care about which were causing my particular configuration to fail to build, so I had to get into the code to fix them. While doing this I started to remember that I actually quite like hacking on AROS and miss it. That and my brain seems ready for a challenge again.

Of course this time around, I'd like to avoid the frustrations that contributed to me quitting last time. So this is my plan:

• I will only work on things that interest me
• I will not work for money
• I will not take on significant commitments (ie "sure, I can take a look at that bug" is ok, but "sure, I'll write you a browser" is not)
• I will not get involved in any political stuff like arguments about project governance, goals (backward compatibility) or anything else

The last point is key. There was a few times previously that I had to do things the wrong way just so that backwards compatibility would be maintained, a goal that I never agreed with. This time, I won't be arguing about it, I'll just be doing what I want to do. Its a light fork, if you like.

I've got a new repository set up over at repo.or.cz. "cake" is what I'm calling my mini-project for now. I'll be committing everything I do there, as well as keeping the AROS mainline there (manually updated as necessary). I will commit things to the AROS Subversion repository as appropriate, but when I do something that causes significant breakage then it will live here. In true open source fashion, anyone who wants my stuff can get it from me and build their own, or if demand gets high, maybe I'll provide some builds or something. We'll see.

So here we go, the brave new world. I'm great at changing my mind, so we'll see how long this lasts :)

It was my birthday last week, so on Saturday a bunch of friends and family joined me for lunch at a nice little café near my house. I had an awesome time and felt very special, yay :) The big surprise though was people's overwhelming generosity. Gub had let everyone know about my project and so I'm now sitting on a fat wad of cash to support it, more than I'm probably going to need to get everything to make this happen. So yeah, pretty amazed.

So yesterday I ordered the SCART-to-HDMI converter that is the critical piece in all these. I'm expecting it to arrive late this week or early next. While I'm waiting I need to make a list of parts I'll need to make the necessary cables and connectors. I'm thinking on Thursday I'll head to RS to get what I need. I'd usually go to Jaycar but they don't appear to have SCART connectors. On the other hand, they do have the cheapest HDMI cables I'm likely to get, so maybe I'll just order a single SCART connector online and then build a single adaptor or something. Haven't quite figured it all out yet.

I've booked a hard rubbish collection with the local council for next week, so I've spent the last few days piling stuff on the nature strip. Its been very purging, and the garage is looking great. As usual a few things have been "stolen" from the pile, which is fine with me - better someone gets some use out of it. What has me curious is why people have chosen the things they have. For example:

• I put out the six pieces of a plastic kids cubby (four walls and two roof pieces) that broke in a storm. Someone has taken two walls and two roof pieces, but left the other two walls.
• I put out a laser printer which as far as I know works fine except for needing toner. Fair enough to nab it, except that whoever took it left the paper tray.
• I put out two computers. One was an old Pentium II in perfect working order (even has a disk and memory in it), but the case was in terrible condition. The other was an ancient HP Vectra that I gutted for parts such that all I threw is the box and motherbord. The first is still there, the second is gone.
• I put out a box containing three broken CD drives, four broken floppy drives and a bunch of random cables. It got taken within half an hour of going out.
• I put out a fully working G3 iMac (one of the original blue ones). It got taken, and in its place was a fully working G3 PowerMac (again, the blue one). That one I grabbed myself for contemplation and hacking.

So I'm not quite sure what people are thinking. I suppose if you can find a use for my junk then more power to you.

She's here!

Penelope (Penny) Norris arrived just before 2pm Thursday by caesarian section. Compared to her sisters she's tiny at 3.4kg (7.5lbs), but she's still got the patented Norris large head. I've already poked fun at her with things like "orange on a toothpick" and "sputnik", but I encourage you to join in too.

Her head is cute actually. Its almost spherical with a squished old-man face right in the middle. She frowns a lot too so she really does look like a grumpy old man.

We're still trying to work out who she looks like. At first she appeared to be a clone of Beth (her 18-month-old sister), but having spent the day with her she really seems to be taking on a bit of a look of her own. So far I think she might look a bit like my eldest younger brother.

Anyway its late and I'm tired, so off to bed. Tomorrow I take Francesca in to meet her. Thats going to be insane.

If all goes according to plan then in just a few short hours I'll be the father of three girls. I'm actually rather nervous which seems strange considering I've done this twice before. If its the kind of thing you go in for then please send up a prayer for Gub, bub, the girls and the army of family looking after them for the next few days. I'll be online tonight with photos and the like.

I had a bit of time to today to read about how to get older consoles to make nicer pictures on the big screen and found that things are much simpler than I thought and this shouldn't be that hard and quite a bit less expensive than expected.

There's two parts to it. Lets assume that we can get raw analog red, green and blue signals out of the console. From this we build a SCART cable and plug it into a SCART-to-HDMI converter. That should be all thats necessary, for about $150 plus whatever shipping from the UK costs for a small box. Unless I can find a local supplier, but it would probably cost about the same.

The tricky bit is getting RGB off the console. In PAL land, the SNES and the Gamecube have RGB right on the MultiAV connector. Its good to know its there, but its not particularly useful since the MultiAV connector isn't a standard plug I can get hold of, and hacking an extra one won't help since the cable doesn't expose all the pins.

For the N64 I'll have to grab the RGB lines directly from the video DAC. It ends up being about the same amount of work because I'll be exposing a new port on the back with some kind of common connector.

So it should actually be doable, assuming the converter works. Yeah, I'm not really nutting anything out, its all documented and tested already, I just have to put all the pieces together.

Oh boy, long time. Probably about to get longer given that in just nine days, assuming all goes to plan, I'll be the proud father of three girls, up from the two I have today. So with that in mind I though it best to do the big dump list of everything that's been happening lately so I don't have it swimming around in my head too long.

Homewise:

• As mentioned, I have a new kid arriving next week. I'm spending a great deal of time getting stuff done around the house in preparation for that and generally supporting my wife, who remains in good spirits despite spending being exhausted all day long and still somehow managing to take care of the other two, both very demanding in their own way.
• The upcoming larger family has required upgrades, so we now have a larger car, larger couch and larger tv.
• The family won't be getting any larger after this, courtesy of Dr. Walters. I can't recommend him enough for this type of thing; he was fast, non-threatening and completely transparent. The only downside of the whole experience is that I didn't get to be number 13000 - I had to settle for 12980. Ok, the only other downside is that I don't have access to the totally insanely awesome sleep drugs he uses. I was out chasing space shuttles and I liked it that way!
• I've been seeing a psychologist to help with my brain problems. She's been awesome, though I'm not very good at doing my homework which I think may be annoying her a bit.

Workwise:

• Work has been crazy, the same project I've bene working on for the last two years continuing to kill me.
• But, there's been a big change in the last two weeks. In the hopes of making the project go faster, the entire project team has been whisked out from under me and reassigned to a new manager they've imported for just this purpose. At the time I had some pretty serious reservations about it all, but as me and my boss work through all the transition and handover stuff with them we're increasingly finding ourselves with plenty of time with which to contemplate all the work that we've wanted to do in the last couple of years but couldn't ever get priority for. Work is suddenly relaxing again, and in a little while might even be fun! Imagine that!

Techwise:

• I've been hacking on OpenTTD a lot in the last couple of weeks, finding my way around the codebase by implementing a new kind of depot. Great fun, highly motivating.
• Gub bought me a new DS to replace the one I destroyed, and I've been liking the new GTA game, so much so that I may actually buy it soon (yarr).
• Given the aforementioned new tv, I've been looking for ways to hook up my older game consoles to it via something other than composite, which while not bad from the Gamecube is a complete disaster from the N64 with a tv that can see every flaw in the produced image. Whatever I do is going to end up being a big job but in the course of my searching for details I've found the magnificent GameSX.com and their RGB+Video forums. I'm slowly working my way through just about everything here, but its becoming increasingly obvious that I require a XRGB-3, but $600 is a bit out of my price range. Lots of study required to figure out what I can do myself, though I suspect I'll run into the same problem I always have with hardware hacking in that I don't have the equipment required to make it happen. Its fun to think about it at least!

So yeah, thats where I'm at. I guess you won't hear from me for a while again, except next week to post some pics. Lucky you!

After my lack of motivation I've had a couple more interesting ideas and so I've started very slowly poking at them, being very careful to not overdo it in an attempt to avoid the burnout. So far it seems to be working!

The short is that my mate Sam and I have been pondering for a year or more the idea of building an arcade cabinet that runs emulators for various old systems (MAME and such). He's a high school woodworking teacher and cabinetmaker by trade so he's perfectly qualified to build the box. I know a thing or two about computers, so I can do that bit. The problem so far is that we've never really had any good place to build it - I have a garage but no tools, Sam had no space at all.

He's just recently moved house and is finishing getting his own garage all kitted out with workbenches and drop saws and drill presses and other things that scare me and he's now ready to build something, so we're taking another look at it. I've managed to scrounge enough parts to build a reasonably good rig, though the ridiculous weather is making me reluctant to go out to the shed to work on it. Arcade Gaming Australia have all the buttons, joysticks and other bits that we'll need. There's only one thing left - an awesome UI for choosing games and things. Yay software!

So tonight I've been sitting under the air conditioner fiddling with Clutter. Its a library for making fancy interfaces by using lots of 3D stuff under the hood. As far as I can tell the most well known example of the type of thing its for is Apple's Cover Flow. Just from playing with some of the samples I already have some idea of how I'd like a game selector to look, so I've started experimenting using the Perl bindings.

The basic idea is that you set up a bunch of actors, which are basic visual elements - some text or an image for example. You can specify various transformations for an actor, eg scaling, rotating, etc. After that, you place your actors somewhere on the stage, which is roughly analogous to a window.

Next is where I get a little confused, but not so much that I can't get something done. You setup a timeline, which has two paramaters - number of frames, and frames per second. You hook up an "alpha" to the timeline, which is a function that gets called every frame, and returns a number that I don't fully understand the purpose of yet. The number is used to drive "behaviours" attached to each actor, which makes them do something depending on the current distance through the timeline. A behaviour might be to move the actor around the stage, rotate it, or something more clever.

There's also an input layer, but I haven't really started looking at that yet.

So here's the fruits of my evening. It takes a random image and rolls it around a window.

#!/usr/bin/env perl

use 5.10.0;

use strict;
use warnings;

use Glib qw( :constants );
use Clutter qw( :init );

say "usage: roll image" and exit -1 if !@ARGV;

my $stage = Clutter::Stage->get_default;
$stage->set_color(Clutter::Color->parse("DarkSlateGray"));
$stage->signal_connect('key-press-event' => sub { Clutter->main_quit });
$stage->set_size(800, 600);

my $actor = Clutter::Texture->new($ARGV[0]);
$actor->set_anchor_point($actor->get_width / 2, $actor->get_height / 2);
$actor->set_position($stage->get_width / 2, $stage->get_height / 2);
$stage->add($actor);

my $timeline = Clutter::Timeline->new(100, 26);
$timeline->set(loop => TRUE);

my $alpha = Clutter::Alpha->new($timeline, sub {
    my ($alpha) = @_;
    return int($alpha->get_timeline->get_progress * Clutter::Alpha->MAX_ALPHA);
});

my $rotate = Clutter::Behaviour::Rotate->new($alpha, "z-axis", "cw", 0.0, 359.0);
$rotate->apply($actor);

my $path = Clutter::Behaviour::Path->new($alpha, [ $actor->get_width,                     $actor->get_height                      ],
                                                 [ $actor->get_width,                     $stage->get_height - $actor->get_height ],
                                                 [ $stage->get_width - $actor->get_width, $stage->get_height - $actor->get_height ],
                                                 [ $stage->get_width - $actor->get_width, $actor->get_height                      ],
                                                 [ $actor->get_width,                     $actor->get_height                      ]);
$path->apply($actor);

$timeline->start;

$stage->show;

Clutter->main;

Hard to show it here, but here you go:

Of course I have no idea if this is the "right" way to do it, but it seems to perform well enough so it will do for now. Next is to make a little photo thumbnail viewer, using the arrow keys to scroll through the photos and a little zooming magic.

I haven't written any code for ages and ages. I'm having a lot of troubled getting motivated, and although I've had a few idea I haven't really come across anything sufficiently challenging or world-changing to really get excited about it. Some ideas I've had that haven't got anywhere, in no particular order:

• A group chat client (IRC, Jabber MUC, etc) that allows images, videos, etc from the web to be embedded directly in it so that when we share links I don't have to click everything. I started prototyping this in Perl using WebKitGtk, but it doesn't have API that allows a page to be added to incrementally, and I don't really want to get back into hacking on WebKit right now.

• A version of Settlers of Catan with a really sweet AJAX/DHTML/Web 2.0/buzzwordish interface. Currently the options for playing Settlers on a computer appear to on your local machine with Pioneers or on the web through Java applets. I got as far as thinking about the AI and poking at the Pioneers code a bit before realising that the only interesting bit is too hard for me which means borrowing the AI code and then writing the web interface, which is totally boring.

• A browser-based sprite engine. THe idea is you'd throw in a bunch of images, specify coordinates and any filters/transformations you might want, and it takes care of their display. Again though, its hacking browser stuff which I'm pretty much over at this point in my career.

• A web-based forum that doesn't suck. I'm quite over every bit of web software requiring a database and having insanely complicated interfaces. I had a thought of some kind of cross between Digg, the Joel On Software forums (design notes) and an image board. Basically no mad hierarchy of forums and posts, no threads, no userpic/sig/meta clutter on every post, and no login requirement. Internally its very much like blosxom, just a simple one-file script that you can drop into your cgi-bin and it just goes, and that you can extend in any way you like with plugins. The whole thing would be backed by git and just be sweetness. I made a good bit of progress on this but at the end of the day its basically a repeat of the code I've just written to get this blog updated, and its for web forums which I hate and avoid as much as I can anyway.

• A database diff tool for Lotus Notes databases. This is kind of interesting, and it would help a lot at work and dodge the need for us to buy any more stupidly expensive licenses for a commercial product, but the Notes API really really sucks and I haven't been able to get a working development environment working on Debian yet, which means I'm going to have to use a Redhat VM if I want to make this work. That amount of effort for what is really part of my job in my spare time is just too much.

So there you go. A few reasonably good ideas, but no motivation. As a result I've been killing time playing games, but that's starting to wear a bit thin. I'm not sure what to do right now, because I really feel like I want to write something. Motivation came easily when I was working on jabberd2 - we were changing the world! It was easy for AROS because I'd never worked on an so many different facets of an operating system before and everywhere I turned there was something new to be done. None of the above so far fit into one of those categories. Every existing project I've considered so far doesn't seem to be either groundbreaking enough or have enough for me to learn to make it worth the effort. I'm not sure what to do now.

Its likely I'm growing old and curmudgeonly. I hope I can get interested soon, if for no other reason than my bus trips are getting boring.