Hoi Zäme!

This post is a tribute to the Emerging Languages conference that I just missed. I’ve started a new project called Snabb Switch and this is a braindump on how I’m choosing which programming language to use.

The Snabb Switch project is a low-level networking stack with emphasis on hypervisors. The #1 technical requirement is that you’re receiving a packet every microsecond or so and you have to do something clever with it and ship it back out quickly. I have a background of real programming in a bunch of languages: C, Erlang, Common Lisp, Scheme, Forth, Smalltalk, Emacs Lisp, Java, and the usual Unix suspects. I have the engineering philosophy of a firmware developer: I appreciate minimalism and I don’t want to depend on software that I’m not willing to understand and debug (less is more). I want to faithfully follow my own engineering values and simultaneously make the project accessible to other interested people.

Here are the languages on my radar for this project and a raw dump of my thoughts on them:

C. In my mind some code just is morally C code, no matter what language it’s written in, and other code morally is not C. I like to use C for code that’s morally C, like device drivers and packet shuffling, but I don’t want to use C for stuff that’s morally high-level, like configuration and status reporting and scripting. So I’m only happy to use C for problems that are 100% low-level or otherwise in a cocktail with a higher-level language.

C++. I am impressed when I see people use C++ to overcome the limitations of C and find clever ways to write whole applications. They win the ICFP programming contest pretty often and that impresses me. My gut feeling though is that this works well in tight organisations like Google but that C++ is a barrier to entry in more loosely coupled open source projects. The language also has a bit of a corporate feel to me – I’ve never picked up a C++ book just for fun. The C hacker in me is jealous the good collection of basic data structures though.

Objective-C. This is another chance to keep what’s good about C and overcome the limitations. I don’t want to commit to OSX as the target platform though – I’m focused on Linux and open to Windows – and I’m not confident that it’s portable in practice.

Erlang. I have done a lot of Erlang programming over the years and I know that it’s really excellent for a wide range of problems. I don’t want to use it for this project though. Erlang isn’t the right tool for writing morally-C code, and I haven’t really enjoyed using the FFI (linked-in drivers) in the past. I also feel that the Erlang runtime system is a really scary program these days – I don’t want to have to chase through it with gdb when hunting really tough bugs.

SBCL. Common Lisp is an extremely interesting option and one that I’ve explored in depth previously. You can write morally-C code in Common Lisp, but I feel like you have to work really hard when you do this, and I prefer to do my bit-bashing in C. CFFI is not quite convenient enough to make this fun for me in day to day usage. The SBCL runtime system is also of a similar complexity level to Erlang and I’ve found that it interacts badly with tools like strace and gdb. I don’t want to be chasing weird bugs involving e.g. signal handlers waking up when I write to memory that the GC has write-protected etc.

Embedded Common Lisp (ECL). Is this a Lisp with a dream-like FFI and minimalist runtime system? I’m not sure, it sounds a bit too good to be true, and I haven’t investigated because I don’t have a specific agenda to use Lisp. I would love to hear from people who have experience building systems comparable to Snabb Switch in ECL.

Openfirmware. Mitch Bradley’s Openfirmware Forth system is a masterpiece and in many ways very well suited to my project. Compact size, good mix of high-level (bytecode) and low-level (inline assembler), minimal and predictable runtime system, suitable for writing morally-C code. I reckon that Openfirmware based network equipment would work extremely well, but I also reckon that very few people could build such systems in a realistic amount of time. Forth is also naturally a barrier to adoption: it’s tricky to get the hang of and the study of Forth feels more like advanced computer science than basic engineering. So rather than use Forth I will seek out other tools that are acceptable to my inner Forth programmer.

LuaJIT. Wow! Lots of positive points. The whole implementation is small and simple. The runtime system is minimal and leaves me space to make important decisions like whether to use threads and how. The FFI is the best I’ve ever seen – it makes it easy to talk directly to hardware via shared memory and easy to call into C libraries. I can use a lovely workflow where morally-C code is initially written in LuaJIT and be selectively rewritten in C over time. The JIT is creating its own machine code in Forth-hacker real programmer style – I don’t understand it but I want to, and its small implementation makes this seem realistic. And there’s a reasonable story on soft realtime.

So: I’ve started off using LuaJIT and C. Initially the code is mostly LuaJIT but I do feel like I’m comfortably able to move the LuaJIT-C border around to see where it feels the best. This feels comfortable for the parts of my soul that love C, Forth, Scheme, and Emacs Lisp. The bits of my soul that love Erlang, Common Lisp, and Smalltalk will have to be patient :-).

I will blog about how it goes over time. If you want to discuss this stuff in more depth feel free to email me on luke@snabb.co.