tuesday, 31 march 2009

posted at 22:16
  • mood: positive
  • music: regurgitator

Oh boy, long time. Probably about to get longer given that in just nine days, assuming all goes to plan, I'll be the proud father of three girls, up from the two I have today. So with that in mind I though it best to do the big dump list of everything that's been happening lately so I don't have it swimming around in my head too long.


  • As mentioned, I have a new kid arriving next week. I'm spending a great deal of time getting stuff done around the house in preparation for that and generally supporting my wife, who remains in good spirits despite spending being exhausted all day long and still somehow managing to take care of the other two, both very demanding in their own way.
  • The upcoming larger family has required upgrades, so we now have a larger car, larger couch and larger tv.
  • The family won't be getting any larger after this, courtesy of Dr. Walters. I can't recommend him enough for this type of thing; he was fast, non-threatening and completely transparent. The only downside of the whole experience is that I didn't get to be number 13000 - I had to settle for 12980. Ok, the only other downside is that I don't have access to the totally insanely awesome sleep drugs he uses. I was out chasing space shuttles and I liked it that way!
  • I've been seeing a psychologist to help with my brain problems. She's been awesome, though I'm not very good at doing my homework which I think may be annoying her a bit.


  • Work has been crazy, the same project I've bene working on for the last two years continuing to kill me.
  • But, there's been a big change in the last two weeks. In the hopes of making the project go faster, the entire project team has been whisked out from under me and reassigned to a new manager they've imported for just this purpose. At the time I had some pretty serious reservations about it all, but as me and my boss work through all the transition and handover stuff with them we're increasingly finding ourselves with plenty of time with which to contemplate all the work that we've wanted to do in the last couple of years but couldn't ever get priority for. Work is suddenly relaxing again, and in a little while might even be fun! Imagine that!


  • I've been hacking on OpenTTD a lot in the last couple of weeks, finding my way around the codebase by implementing a new kind of depot. Great fun, highly motivating.
  • Gub bought me a new DS to replace the one I destroyed, and I've been liking the new GTA game, so much so that I may actually buy it soon (yarr).
  • Given the aforementioned new tv, I've been looking for ways to hook up my older game consoles to it via something other than composite, which while not bad from the Gamecube is a complete disaster from the N64 with a tv that can see every flaw in the produced image. Whatever I do is going to end up being a big job but in the course of my searching for details I've found the magnificent GameSX.com and their RGB+Video forums. I'm slowly working my way through just about everything here, but its becoming increasingly obvious that I require a XRGB-3, but $600 is a bit out of my price range. Lots of study required to figure out what I can do myself, though I suspect I'll run into the same problem I always have with hardware hacking in that I don't have the equipment required to make it happen. Its fun to think about it at least!

So yeah, thats where I'm at. I guess you won't hear from me for a while again, except next week to post some pics. Lucky you!

thursday, 17 july 2008

posted at 09:07

So, long time huh. Time to write something, I guess.

Its hard to write again after a break, not least because so much as happened in the last three weeks that unless I write for another three weeks I'm never going to cover it all. So right now I'm not going to try. i'll just write about what I've been working on, get caught up to some degree, and maybe come back to any other interesting stuff down the track. So here we go!

I've long threatened picking up a DS homebrew kit and doing something interesting with it. I finally snagged a M3 DS Real from the Monash post office, of all places. Its a cute little thing. I got a 4GB microSD card with it, so I should have enough grunt to do anything I could ever hope to do with it. The next thing is just to work out what that is. I have an idea of course, one probably rather predictable if you know me.

I have an interest bordering on an infatuation with the game Frontier: First Encounters; specifically the JJFFE variant. Its a great game, and the effort and style that went into making an old Windows game come to life on more modern systems still totally imprresses me. John Jordan took the game binary and run it through a custom disassembler that produced an assembler source file that would compile on whatever platform he chose. His disassembler also identified operating system calls for graphics, sound, input and filesystem access, which he then abstracted and reimplemented for DirectX, SDL, and whatever else. So now the game runs on (i386) Linux without issue as well as on modern Windows systems. He even fixed a heap of bugs. Thats great!

I've messed with this code at various times. I implemented the OS abstraction for AROS a couple of years before I got involved in AROS proper. (That work later led to me working on some graphics subsystem speedups and a SDL graphics driver for AROS). I've also long dreamed of somehow converting it to pure C so that it could be hacked on properly. I've dabbled with this at various times, both using automatic and manual techniques, but haven't really got very far mostly because of because of the limited success others have had with the general problem of decompiling assembly back to C.

So anyway, I got a DS kit, and of course started to think about how cool it would be to play FFE on it, and also about how to take advantage of the dual screen and touch screen. I've been dreaming of interesting UI changes that would make the game work much better on the DS, but of course first I have to get the game working there. That is not a trivial task, and has been the subject of inquiry for the last couple of weeks.

The problem is obvious. The DS has a pair of ARM CPUs. The JJFFE source is in i386 assembly. So there are really only two options - some sort of emulation, or converting the code to a higher-level language and then recompiling it for ARM.

Emulation would only really require a processor emulator for the game core since all the systems interaction could be done in C, and perhaps would have been the easier option, it doesn't help much with my eventual goal (or "hope", rather) of making significant modifications to the code to support the DS hardware properly. So instead, I've again returned to converting the assembler code to C.

As mentioned above though, this is something I'd pretty much given up on as being too difficult. I thought about it for a while and realised as a first cut, I don't need to convert it back to anything resembling its original C. Instead, what if I was write an assmbler that produced C code that implemented the processor instructions, rather than producing raw machine code. The result would look much like C - we'd essentially have a kind of static CPU emulator built into the program code itself, with global variables representing the processor stack and registers. But, it could be recompiled for another CPU, which is the point of the exercise.

This seemed like a reasonable approach, but writing an assembler is insanely complicated. After attempting a hacky parser in Perl, I decided that nothing short of a full assembler would be able to do the job. NASM proved too complicated to penetrate, but then I found YASM, which is a highly modular clone of NASM.

So I took YASM and started writing a custom object format, one that would output C code. However, after experimenting to gain some experience with the interface, I realised that I was just getting the raw machine code and then converting it to C with a little bit of symbol table gymnastics to identify and produce simple unadorned C functions. This reminded me of a project I worked on for a while in 2004 that turns out to be much better suited. That project is a custom disassembler/decompiler of the same kind of was used to produce JJFFE in the first place! Let me explain.

Another old game that I love is Transport Tycoon (actually its sequel, Transport Tycoon Deluxe). At the time, it was Windows-only. There was a project called TTDPatch which would take the server binary and hook all sorts of stuff into it to add new features and fix bugs and whatever else. This worked well, but it was still Windows-only. Wine did a reasonable job with it, but it was still less than ideal. So I decided that I'd give it the same treatment as FFE got, and produce a disassembly and system abstraction that could be run anywhere.

I spent a lot of time studying JJFFE and Jordan's decompiler and even had a series of email discussions with him to get a feel for just how to do this. After several weeks I managed to get my decompiler to the stage where it produced a viable disassembly and C file of OS call stubs. But, as fate would have it, the day it compiled and ran for the first time (segfaulting of course, as I hadn't yet learnt about how Windows uses the %fs segment register), OpenTTD was announced, which was essentially a conversion of the original game back to C. My decompiler had no further reason to exist, and so I abandoned it.

The way it worked was pretty straightforward. It implemented what is essentially a Portable Executable (ie the Windows binary format, like ELF for Unix) loader with calls into the NASM disassembler to analyse the code and produce a disassembly and a stub file. Simplified, it does the following:

  • Inspect the program binary and find the code, data, bss, import and relocation segments.
  • Load the program binary into RAM.
  • Apply the relocations to produce a complete program image, additionally creating a "label" for each relocation.
  • Inspect the import section to build a list of all the external libraries and functions that the program wants.
  • Disassemble the code segments to find all the relocation labels that are in use and what they point to. From the instruction type, we can determine whether the target is code, data, bss (ie unitialised data), a vector table, etc.
  • Disassemble from each code label to the next to find any other labels missed in the first disassmble run. That might have happened, for example, if there were "garbage" bytes in between the end of one function and the start of another that caused the wrong disassembly to be produced crossing the function boundary.
  • Do this dissambly over and over until no new labels are produced.
  • Run through any relocation labels that have not been processed yet, and make them data labels. This works off the assumption that if the linker thought it important enough to include a relocation, we should probably include whatever that relocation points to in the output, even though we couldn't actually find it in the code.
  • Output EXTERN declarations for each external function name.
  • Disassemble from each code label again, this time producing actual output. Any memory references in the output (ie things beginning with 0x) get replaced with their corresponding label, if there is one.
  • "Disassemble" from each data label, producing raw byte output (ie db or dd). For any data that was referenced via a relocation, produce the corresponding label.
  • "Disassemble" from each bss label, producing a memory reservation in the output (ie `resb

Theoretically, the resulting output from that is just about usable. There's a bit of manual cleanup that has to happen (like the mentioned deal with the %fs register), but this output should at least compile and link, which is most of the fun. Theoretically you implement the stubs for your platform and you're away.

So, back to our original problem of producing C from a binary. I realised that in this code, I'd already done most of what I needed. I know where all the function boundaries, jump points, vector tables and data are. All that needs to happen is instead of producing formatted assembly, all I need to do is produce some equivalent bit of C code. There's some complication of course, like the fact that sometimes several instructions map to a single C construct (like if or while) but I figure I'm most of the way there.

So right now, I'm working on cleaning up and generalise the decompiler, which I've christened opsoup. It was pretty heavily tied to the PE format before, which of course is no good for me - I need ELF. I'm not bothering with trying to keep it compatible with PE at this point, as I have a pretty specific purpose. I can always add that back in later if I ever need it.

I have absolutely no idea how this is going to go, but its fun finding out. In adition to playing my game, I'm hoping that having the code in C, even horrible not-quite-C, will make it much easier to gradually convert some of the code in actual nice C (due to the availability of things like gdb and printf). I don't expect it to happen fast, but I've been hacking at this code on-and-off for the last five years, so messing with it for another five doesn't really concern me that much.

tuesday, 16 october 2007

posted at 21:21
  • mood: fightery

This morning I was greeted with a PayPal email informing me that as a result of my recent SDL work Team AROS have seen fit to award me the princely sum of $38.92. This brought my latest object of my affections well into range of this weeks budget, so I went down to EB today and picked up a copy of Phantom Hourglass.

As expected, I'm loving it. The graphics are awesome and the control system is crazy good. Yep, descriptive, I know, Just buy it, its good.

So I might be quiet for a few days while I get stuck into it. Don't be alarmed, I'll be back soon :)

monday, 3 september 2007

posted at 15:43
  • mood: shiny

Had a great Father's day weekend. Saturday I went out and bought my AVR and a 74HC573 for the memory latch. I have a couple of 8K RAMs that I picked up on eBay last year and some "ladder" LEDs and other interesting lights, so I should now have everything I need to start experimenting. I'm short a power supply though: it'll be a race to see whether I hack up an old plugpack or drive over to Rosanna to pick up my bench supply from my mate's place.

Sunday I awoke to Francesca awkwardly trying to climb into bed holding her Father's Day loot. I helped her up and she helped me unwrap a copy of Settlers DS (a port of Settlers II). Its got some pretty lousy reviews, and I can see why - the interface is clunky, the gameplay is sluggish and there's obvious bugs. Its still Settlers though, which was a game I was addicted to back in the day, so I'm happy. The girl also gave me a nice picture book about a Daddy bear and his kid bear and all the things they do together, and we had a great time reading it together. I do like being a Dad :) Today I found this presentation about Git, which I've been hearing lots about but decided was too much of a leap away from Subversion for my brain to handle. At the time I opted for SVK instead, and I love it, but lately I've found its starting to run out of steam which seems to be tracable back to its Subversion roots. The presentation was fascinating and enough to convince me that Git is worth my effort, so right now I have git-svn running to pull in the AROS repository. It won't be done before I go home so it'll probably be tomorrow before I can really experiment with it properly. I hope its a good as everyone claims.

friday, 3 august 2007

posted at 10:47
  • mood: tactical
  • music: pearl jam - rearviewmirror

Long time hey. I'm having a week off. So far I've been away for a couple of nights alone with my wife, been to the zoo, visited my mum and my in-laws, had dinner with some C64ers from years gone by and done several hundred other odd jobs. Its all been lots of fun and very relaxing.

I haven't written any code and have barely even touched my laptop, all because of a highly addictive game that I picked up last week. With the DS being as portable as it is, I've had a lot of time to play - on holiday, in the car, before bed, etc. I typically play the DS for maybe an hour a day and the battery lasts a couple of weeks. Its currently in the middle of its fourth charge since Saturday. That should be telling.

So far I've nearly finished the campaign, then I'll dive into the standalone maps and start trying to unlock stuff. I'm not going to review it or anything like that, but if you care at all about my opinion (and you don't) you will go and buy this game today. You'll buy a DS to go with it if you don't already have one.

I'll probably try and get hold of the earlier GBA versions too. From what I can tell they're just as good, and the story ties in too. And there might be extra unlockable things in the DS version if you have the GBA carts inserted. Oh, and there's another one coming out in December, supposedly. I may never write any code again.

thursday, 26 july 2007

posted at 09:11
  • mood: toys
  • music: katie melua - piece by piece

Gub got a nice tax return a few days ago (love family tax benefits) and insisted that I spent some of it on myself. So I looked around and found the RAM upgrade for my laptop that I priced at $169 a couple of months ago was now down to $99. I'm not passing that up, so as of yesterday at lunchtime my laptop has a nice new 1GB stick in it, taking me up to 1.5GB (ish, you know how inaccurate all these numbers are). Its noticably faster.

The spending spree isn't over yet - I've also been instructed to buy a DS game. Advance Wars: Dual Strike looks like it will be the winner. I'll go looking for a pre-owned version from EB tonight. If I can get it cheaper, then I'll have a little bit over to buy a C64 DTV, which I notice Target had squillions of at $25 a pop.

New toys make us powerful and happy :)

friday, 23 march 2007

posted at 13:12

I got paid from the bounty, so I went out looting last night. I had planned to buy a 1GB stick of RAM for my laptop, but the wife told me that I had to spend it on something frivolous instead. She gave me the choice - I either had to buy a Nintendo DS or a Wii. I chose the DS, because the Wii games I'll want to play (Metroid Prime 3 and Super Mario Galaxy) aren't out yet and I still have Gamecube stuff to finish. Gub says she'll get me a Wii for Christmas :P

I've been pining for a DS ever since Daniel got one and regaled me with his tales of derring-do every fricking morning when I got in to work. I bought MarioKart DS so that I can have a clue of what he's talking about (although I already get some of it having played previous MarioKart games), and picked up New Super Mario Bros as well.

So far its fun. MarioKart is always entertaining, and I've always liked Mario in 2D, though this one doesn't yet seem to have the depth that Super Mario Bros 3 did, back in the day). My brain is not quite in the right place to play lots of games yet - still full of work and code. It'll be fun for ten minute breaks though.

It could also be fun for homebrew stuff. This card looks like a great way to get started all in one go. A bit pricey though - nearly the cost of the console! AROS could be fun on this thing, haha. Maybe later :)