tuesday, 6 november 2012

posted at 23:21

So today Frontier Developments started a Kickstarter campaign for the next chapter in the Elite game series: Elite: Dangerous. I heard this from half a dozen people within the first hour of it appearing. The most common question is what "what does this mean for Pioneer?". The quick answer is "nothing", but I have some thoughts about the whole thing, so I thought I'd write something.

So what's going on?

Lets have a short history lesson.

Long ago, David Braben (along with Ian Bell) made Elite. Some years later its sequel, Frontier appeared, and shortly afterwards it gained a sequel of its own, Fronter: First Encounters. Reaction to FFE was mixed, but everyone agreed Frontier was pretty fantastic. Life was good.

Braben, via his company Frontier Developments, announced that there would be an Elite 4, but nothing happened for years. Frontier got old, and the community wanted something to fill the void. Tom Morton stepped in with Pioneer, an open-source clone of Frontier. I turned up a few years later and nowadays we're trying to put legs on a game that we now like to say is "inspired by" Frontier.

A couple of people refused to give up, but most people never really thought that Elite 4 would happen. But today we saw the announcment for Elite: Dangerous, the next Elite game.

Is Pioneer now obsolete?

Nah. For a few reasons.

  • Elite: Dangerous looks more like a modern remake of Elite rather than Frontier. One particularly interested tidbit came from David Walsh over at the Frontier forums: Elite: Dangerous will not have time acceleration. Now we've thought long and hard about what the key features of Frontier are that we must maintain for Pioneer, and one of them is the ability to speed up the clock. Its also the feature that makes multiplayer infeasible. So whatever they're making, its not Frontier, and because of that, we still have a niche.
  • We have some ideas that we haven't seen done in any game yet. We can still bring something to the genre.
  • We're having too much fun :)

Does this take attention away from what you're doing?

Doubtful. Indie space games are kind of in vogue at the moment - it seems like we see a new announcement every week. Every time it happens, lots of people pop up pining for Elite or its successors, and usually there's at least one person pointing folks in our direction. We've had a steady trickle of new players and contributors for many months, and I don't see it going away. If anything happens, we'll probably see more people as more people become interested in the genre and are reminded of their youth. Or not, and we'll continue as we always have.

Is it going to work?

Honestly, I have no idea. I have my reservations about Kickstarter for software projects - seems like everyone treats it like magic fairy dust, but we still haven't actually seen any output from any of these multi-million-dollar projects.

On the other hand, someone in #pioneer noted Braben promised a $35 general-purpose computer and was then heavily involved in the delivery of the Raspberry Pi. He's got the track record.

If it doesn't work, then its going to be a very public embarrassment for Frontier Development among the Elite player community, one they may not come back from. I will watch with interest.

Need any help with Pioneer?

Only semi-related, but advertising never hurt anyone :)

We always need help. There's lots that needs doing. There's a few resources on the wiki to help you get started, or come and say hi in #pioneer on freenode.

monday, 6 february 2012

posted at 21:56

Nine months isn't too long between posts, right?

This is the contents of a brainstorming post on G+. Its some musings about the Pioneer GUI and where to take it. If you've only been reading this blog then you've missed all the work I've done on this in the last few months. Ask if you're interested and I'll throw some pointers your way. Also there's updates on the G+ post, so keep reading that. The modern world of blogging at its finest..

Anyway, I know visually what I want, and its basically what jQuery gives you out of the box - nice swishy transitions, zooms, etc. I think the model is appropriate since we want to expose some parts of this to scripts, but even in core it'd be wonderful to just say "slide this widget onto the screen and once its there, call this method".

In the long-term I'm intending to move Pioneer away from its 800x600 virtual screen, mostly because it looks like shit on a really high-res widescreen monitor. So my intention is that positioning will be done as 0.0-1.0 in x, 0.0-1.0 in y, measured against a chosen container edge. Scaling by default will be aspect-aware, so I can say that a particular image (quad) should use 10% of the horizontal width and its height will do the right thing.

I'm pretty sure I want a fluid layout, where widgets ask for a minimum/ideal/maximum amount of space and are then handed out an amount depending on the layout strategy of the container they're in. This is not so different to what we have now, but we don't use it in a lot of places. I want fixed position to be used very very sparingly, mostly where precise positioning is required, like control panel buttons.

Scrollbars should be automatic at the container level. If scrolling is enabled for a container then it will give out the maximum possible width/height to each component (based on its requested dimensions).

I like the model Unity uses for its UI where each widget has a corresponding style component. This also gives a convenient place for transitions to hook - they simply modify styles. To do positoning or zooming-type animations I think I don't want to try to modify the widget's metrics, but rather apply a transform at draw time.

I'm still not sure how font metrics feed into widget metrics. I've been assuming for a while that I need a way for a widget to say it wants a height of N lines of text, but whenever I try to think of a clear example I can't. Perhaps I don't care - perhaps it just needs to say "I'll take all of the vertical space you can give me, and please scroll me if I blow it". Though I can see that text layout needs to know whether it should prioritise width over height or vice-versa when doing wrapping.

I've done a good amount of research and experimentation with various GUI systems, both for Pioneer itself (Rocket, GWEN, Guichan, etc) and elsewhere (GTK+, jQuery, Zune, etc). They all have things I like and don't like, which is why I think this has gone on so long.

So basically I think I want to do the lightest and most obvious design possible and see what falls out. That's proving a little more painful than I'd like because Pioneer's GUI has way too much global state which has to be fixed first. It will happen though - I'm quite determined :)

wednesday, 18 may 2011

posted at 11:47

I've been learning a lot about fonts in the last week as I work on the foundations needed to turn this:

into this:

I'm not going into detail right now, because I could write reams and still not end up saying much - font rendering is a real dark art. I mostly just wanted to share a little hack that I put together that might be useful to anyone experimenting in the same space.

The short of it is that Pioneer uses FreeType. Since I was getting into working on the font code I took the opportunity to see if there's any lighter alternatives so that we could remove a dependency. The answer is that there really isn't for the kind of things we need, but there is one worthy contender: stb_truetype.h

Its a very simple TrueType renderer in ~1800 lines of C (of which ~500 of that is comments and documentation). Its missing support for a lot of things, but its a single file to include in your project and does a good job of the common fonts that most people have.

The only trouble I had with it is that the very few examples I could find assume you're using it directly with OpenGL. Its probably not an unreasonable assumption, but it made things a little difficult for me because I'm still not great with GL but I'm much better with SDL. What I really wanted was a simply example for SDL so I could get a feel for the API and check its output without having having to wrestle with GL and then wonder if I was getting odd results because I'd done something wrong.

Alas, no such example existed, so I wrote one. Here it is, for internet's sake:

 * sdl_stbtt - stb_truetype demo using SDL
 * Robert Norris, May 2011
 * Public Domain
 * Compile:
 *   gcc --std=c99 -o sdl_stbtt sdl_stbtt.c `sdl-config --cflags --libs` -lm
 * Run:
 *   ./sdl_stbtt <path-to-ttf-file> <text-to-render>

#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/mman.h>
#include <SDL.h>

#define FONT_HEIGHT 32

#include "stb_truetype.h"

int main(int argc, char **argv) {
    if (argc != 3) {
        printf("usage: sdl_stbtt <path-to-ttf-file> <text-to-render>\n");

    /* getting the font into memory. this uses mmap, but a fread variant would
     * work fine */
    int fontfd = open(argv[1], O_RDONLY);
    if (fontfd < 0) {
        perror("couldn't open font file");

    struct stat st;
    if (fstat(fontfd, &st) < 0) {
        perror("couldn't stat font file");

    void *fontdata = mmap(NULL, st.st_size, PROT_READ, MAP_SHARED, fontfd, 0);
    if (!fontdata) {
        perror("couldn't map font file");

    if (SDL_Init(SDL_INIT_VIDEO) < 0) {
        fprintf(stderr, "sdl init failed: %s\n", SDL_GetError());
        munmap(fontdata, st.st_size);

    /* creating an off-screen surface to render the glyphs into. stbtt outputs
     * the glyphs in 8-bit greyscale, so we want a 8-bit surface to match */
    SDL_Surface *glyphdata = SDL_CreateRGBSurface(SDL_SWSURFACE, 512, 512, 8, 0, 0, 0, 0);
    if (!glyphdata) {
        fprintf(stderr, "couldn't create sdl buffer: %s\n", SDL_GetError());
        munmap(fontdata, st.st_size);

    /* 8-bit sdl surfaces are indexed (palletised), so setup a pallete with
     * 256 shades of grey. this is needed so the sdl blitter has something to
     * convert from when blitting to a direct colour surface */
    SDL_Color colors[256];
    for(int i = 0; i < 256; i++){
        colors[i].r = i;
        colors[i].g = i;
        colors[i].b = i;
    SDL_SetPalette(glyphdata, SDL_LOGPAL|SDL_PHYSPAL, colors, 0, 256);

    /* "bake" (render) lots of interesting glyphs into the bitmap. the cdata
     * array ends up with metrics for each glyph */
    stbtt_bakedchar cdata[96];
    stbtt_BakeFontBitmap(fontdata, stbtt_GetFontOffsetForIndex(fontdata, 0), FONT_HEIGHT, glyphdata->pixels, 512, 512, 32, 96, cdata);

    /* done with the raw font data now */
    munmap(fontdata, st.st_size);

    /* create a direct colour on-screen surface */
    SDL_Surface *s = SDL_SetVideoMode(640, 480, 32, 0);
    if (!s) {
        fprintf(stderr, "sdl video mode init failed: %s\n", SDL_GetError());

    /* the actual text draw. we loop over the characters, find the
     * corresponding glyph and blit it to the correct place in the on-screen
     * surface */

    /* x and y are the position in the dest surface to blit the next glyph to */
    float x = 0, y = 0;
    for (char *c = argv[2]; *c; c++) {
        /* stbtt_aligned_quad effectively holds a source and destination
         * rectangle for the glyph. we get one for the current char */
        stbtt_aligned_quad q;
        stbtt_GetBakedQuad(cdata, 512, 512, *c-32, &x, &y, &q, 1);

        /* now convert from stbtt_aligned_quad to source/dest SDL_Rects */

        /* width and height are simple */
        int w = q.x1-q.x0;
        int h = q.y1-q.y0;

        /* t0,s0 and t1,s1 are texture-space coordinates, that is floats from
         * 0.0-1.0. we have to scale them back to the pixel space used in the
         * glyph data bitmap. its a simple as multiplying by the glyph bitmap
         * dimensions */
        SDL_Rect src  = { .x = q.s0*512, .y = q.t0*512, .w = w, .h = h };

        /* in gl/d3d the y value is inverted compared to what sdl expects. y0
         * is negative here. we add (subtract) it to the baseline to get the
         * correct "top" position to blit to */
        SDL_Rect dest = { .x = q.x0, .y = FONT_HEIGHT+q.y0, .w = w, .h = h };

        /* draw it */
        SDL_BlitSurface(glyphdata, &src, s, &dest);

    /* done with the glyphdata now */

    /* wait for escape */
    SDL_Event e;
    while(SDL_WaitEvent(&e) && e.type != SDL_KEYDOWN && e.key.keysym.sym != SDLK_ESCAPE);



This only uses the "simple" API, so its results aren't as good as stb_truetype is capable of, but it was enough to play and see what the output is like (very good).

As it is, I've settled on sticking with FreeType for a number of reason, but that doesn't take anything away from stb_truetype. If you're looking for basic font rendering without much overhead, do give it a try!

sunday, 20 february 2011

posted at 21:58

Like many games, Pioneer uses Lua for its content generation and dynamic world stuff. It doesn't expose enough of its lucky charms to the world this way though (in my opinion), which is why I'm currently doing a major overhaul of everything Lua related.

From looking at the code there's a couple of ways that Lua has been integrated over time. All the models in the game (ships, space stations, buildings) consist of static data (textures, object definitions) and a Lua script to hook it all together. The script can define a function that gets called as the model is rendered that can actually modify the model. This is how flashing running lights, rotating radar dishes and other things are done. Its quite a clever system. The Lua style used is pretty much plain calls to the Lua C API, with some hand-coded Lua classes for matrices, vectors, and so on.

The other place Lua is used is in the modules. These are essentially plugins that add stuff to the game based on triggers. Its how missions and smugglers appear on the bulletin board and pirates appear in the system when you enter it, to name but two functions. This interface uses a combination of normal Lua and OOLua.

OOLua is a C++ binding for Lua, which provides lots of macro and template magic to make it easy to expose your C++ classes as Lua classes. It automatically handles unpacking the Lua stack and argument typing and whatnot so that when in Lua do something like o = SomeClass:new("abc", 1) and o:thing(23) it'll arrange calls to SomeClass::SomeClass(const char* a, int b) followed by void SomeClass::thing(int a). I'll leave it to you to go and read the theory and code linkes from the OOLua homepage. Its quite interesting, though it took me quite a bit of use and misuse before I really started to get my head around it.

My plan, which of course you read because I linked it above, is to expose pretty much everything that might be useful to people wanting to add to the Pioneer universe as Lua, since that's theoretically easier for non-programmers to get to grips with. A good start has already been made to getting OOLua hooked up, so I decided after a few experiments that my first steps should be to convert all remaining non-OOLua stuff to use it. The big one here is all the code in LmrModel.cpp, which is where all the model magic mentioned above happens (indeed, LMR stands for "Lua Model Renderer", which should give some idea of just how central Lua is to all of this).

The way a Lua model definition works is pretty straightforward. At boot Pioneer loads all the model Lua files, which typically contain calls to define_model(). The arguments to define_model() contain lots of call to functions that define the model. Examples are cylinder(), texture() and so on. As you'd expect, Lua calls all of these functions to assemble the arguments before calling define_model() with the whole lot. LmrModel turns this inside out. When a call is made to eg cylinder(), it actually pushes a bunch of commands like "draw triangle", "draw billboard", "use light" and so on onto a stack. When the final call to define_model() is made, LmrModel attaches that stack to a global model definition. Its a bit unusual and can be prone to errors (eg currently if you call one of the element functions outside of define_model(), you'll usually get a segfault), but it also simplifies the code a great deal because it greatly reduces the amount of data that needs to be passed back and forth between C++ and Lua.

The difficult thing about converting all of this to OOLua is that its all the element functions are static functions, not object methods. OOLua is really only built for proper classes, and has only the most minimal support for calling an unadorned function from Lua. That minimal support does do the stack and type signature handling that I described above, so I've built an entire layer on top of it to sanely handle calling static functions from Lua. Its still under heavy development (you can follow my lua-overhaul branch if you're interested) but its already very functional. Here I want to describe a bit about how it works, because I'm quite proud of what I've been able to do. The details are at the bottom of LmrModel.cpp and in OOLuaStatic.h if you want to follow along.

OOLua requires that all static functions be registered against a Lua class (causing them to appear under its namespace). What that means is that we have to define a class, even its empty. Sucks, but lets do it.

Once there's a class in place, its trivial to register functions against it. A typical call is:

OOLUA::register_class_static<pi_model>(l, "call_model", &static_model::call_model);

pi_model is the already-registered class we're hooking the function to. call_model is the name of the function that appears on the Lua side (so in this case we've just registered pi_model.call_model). The final arg is a pointer to the C function that will be called when the function is invoked.

The function is a standard Lua callback with the signature int func(lua_State*). If you like you can use this as-is, but OOLua provides some extra magic here to use this function as a thunk that does stack unpacking and type checking before passing the call on to a real handling function. A typical callback function for OOLua looks likes:

int static_model::call_model(lua_State *l) {
    OOLUA_C_FUNCTION_5(void, call_model, const std::string&, const pi_vector&, const pi_vector&, const pi_vector&, float)

The arguments are straightforward - the return type, the name of the function to call, and the types of its arguments. This will result in a call to:

static void call_model(const std::string& obj_name, const pi_vector& pos, const pi_vector& _xaxis, const pi_vector& _yaxis, float scale)

If the types or number of arguments are wrong, then instead a Lua error will be generated.

So this is all very nice, but has some shortcomings. The simplest is the amount of boilerplate that needs to be written to set up a function. Some simple start/end macros to define the thunk function are all thats necessary.

The next thing I stumbled on is the need for a form of multiple dispatch. OOLua already does this for constructors, but not for method calls, which I find a little odd. What it meant is that I had to impelement it myself. Since Lua is typeless there's really no way short of some educated guessing to make a choice based on types, but choosing the function is possible based on the number of arguments. This is expected in a few places in the existing model interface. For example, texture() in its simplest form requires the name of a texture only, but its also possible to call it with extra args specifying position and transformation of the texture. So we now have two possible functions that we could call. OOLua can't support this directly, so I wrote some macros that when used, expand to (slightly simplified):

int static_model::texture(lua_State *l) {
    const int n = lua_gettop(l)-1;
    if (n == 1) {
        OOLUA_C_FUNC_1(void, texture, const std::string&)
    if (n == 4) {
        OOLUA_C_FUNC_4(void, texture, const std::string&, const pi_vector&, const pi_vector&, const pi_vector&)
    return 0;

Using the macros, this gets written as:

    STATIC_FUNC_1(void, texture, const std::string&)        
    STATIC_FUNC_4(void, texture, const std::string&, const pi_vector&, const pi_vector&, const pi_vector&)

That works well. Later I found another problem, which was a bit trickier to solve. The function extrusion() looks like this in Lua:

function extrusion (start, end, updir, radius, ...)

The first three args are vectors, the fourth is a number (float). That's not important though. What's important here is that following the required args comes an arbitrary number of vectors to define points for the shape to extrude. This posed a problem - its easy to create a macro that says "expect 4 or more arguments), but OOLua's function call mechanism fails outright if the number of args on the Lua stack don't match the number of arguments called for.

The solution I settled on was to define a seperate STATIC_FUNC_4_VA macro. When this appears in the thunk definition, it looks for extra arguments on the stack and puts them into a Lua table. It then pushes the table and the number of items in it onto the stack and calls the function with two extra arguments. All this gives the following:

    STATIC_FUNC_4_VA(void, extrusion, const pi_vector&, const pi_vector&, const pi_vector&, float)

static void extrusion(const pi_vector& start, const pi_vector& end, const pi_vector& updir, float radius, OOLUA::Lua_table t, int nt)

STATIC_FUNC_4_VA expands like so:

if (n >= 4) {
    OOLUA_C_FUNC_6(void, texture, const std::string&, const pi_vector&, const pi_vector&, const pi_vector&, OOLUA::Lua_table, int)

You can go read _static_varargs_table() too. Its interesting but not really relevant to this discussion.

So right now this is all working wonderfully well. I'm not quite finished refactoring all the functions, but its only a short hop away. But there is one fatal flaw in all this which I'm really struggling with right now. The problem is that all calls made by OOLua via its method/function macros don't have ready access to the lua_State representing the interpreter, which means if at any point OOLua can't do something for you (which is often) and you need to drop back down to the standard Lua API, you're stuck.

In LmrModel this is not a problem as the state is held in a global. In LuaUtilFuncs however there's no such global and indeed, you wouldn't want one, as these functions as used by several different interpreter contexts throughout the codebase.

Its actually tricky to solve this one. Obviously when the registered function is called its called with the Lua context in the args, so we do know it. But we lose it as soon as we ask OOLua to call our function with its fancy typechecks and stuff. We can store it in a global just for the duration of that call, but then we aren't re-entrant which could be a real problem down the track. I don't want to do that.

The only idea I have at this point is to push a pointer to the context onto the Lua stack so that OOLua can unpack it and pass it, but these seems rather heavy. Its not just a pointer either; due to the way OOLua does its type handling I have to push a full object instance. Thats a slight lie; a few primitive types like int and float don't need an object, but I don't want to do crazy stuff like casting pointers to integers to make this work.

I will try that option, but I'm keen to find something else. A thought that occured to me is that perhaps this is all wrong; perhaps it should always be the case that these functions are actually called as object methods. It makes a certain amount of sense. The model definitions could be built up in an object rather than in globals, which paves the way for object loading to be done in parallel in the future. The difficulty with this however is that the pseudo multiple-dispatch that I've implement for functions is not available for method calls, so I've undone a good amount of work.

I think at some point I'm going to need to take all this over to the OOLua author and discuss getting all this implemented properly. Its a fantastic system and I don't want to have to move away from it, but its starting to run out of steam. The author has been very responsive and helpful so I expect there's lots that can be done, which is hopeful.

That'll do for now.

saturday, 5 february 2011

posted at 09:13

Hmm, I don't write here much anymore. As is the case for lots of people, my blogging has suffered in favour of spewing random crap into Twitter, Facebook and elsewhere. I'm actually doing a lot more stuff via "social" sites in general, so I think a redesign of this site might be necessary soon to bring all that stuff into one place so I don't look dormant to anyone that just follows here.

Anyway. Christmas and holidays have come and gone. I spent a good amount of time on the big idea that I mentioned last time, and got it close to finished, but then in performance tests found that that the naive old-school Ajax implementation I'd done wouldn't scale much past 30 users. Thats unacceptable, so I started to read up WebSockets, Comet and other things to reduce network and processing on a web application. I settled on using Tatsumaki to implement a long-polling version, but that meant a rewrite of much of the server and the client. At this point I was well and truly on holiday and my brain had shut off, so I threw the project on the backburner.

This can be a dangerous thing for me, because I inevitably change my mind and do something else. I started watching Babylon 5, a show I'd somehow missed back in the day. Anyone that's read for a while knows my long infatuation with space games, so of course I started looking around for something spacey to do or play. And last week, I found Pioneer.

Pioneer is a clone of Frontier, the sequel to Elite. I always much preferred it to Elite. I think it was mostly because of the scale - I could fly forever, land on planets, and just bum around in my spaceship. So I grabbed the code, built it and had a play. And I got sold on it quickly because its awesome, but has a giant stack of things that need work still. In the spirit of the new "just do it" style I'm trying to live up to, I decided first that I wanted to hack on it and then started playing and figuring out what I wanted to hack on. After a couple of hours play I found a heap of little bugs and tweaks that needed fixing, and because the main coder is an awesome guy, lots of my stuff has already been merged.

Not much else to write. This is mostly a "here's what I'm up to" kind of post, so now you know. If you like space games do grab Pioneer (and pull my integration branch if you just want to see what I'm doing). Cheers :)