Lobster Design Philosophy, History, and Future

This document tries to clarify why certain design choices were made in the Lobster language and system.

NOTE: this may be a little out of date. Lobster has evolved a lot since I wrote this.

Primary Motivation

The most important over-arching reason behind everything in Lobster was to make it a pleasant language to use for one single person: me, the author. I've designed a fair number of languages in the past that had a particular goal or should appeal to a particular target group, or were even commercial. Lobster is not such a language.

Unlike past work, its goal from the start wasn't to try to appeal to anyone in particular. If you happen to like Lobster, then congratulations, your taste in programming is probably similar to mine.

Besides being useful to me, creating Lobster in itself was merely meant to be a fun project. I love creating programming languages, but there's really no money in it, and generally to most people new programming languages are not desirable things, which is what had stopped me for a while creating any more of them. Lobster was the "screw it, I am making one because it's fun" project. That, and I had some new ideas about some subtle features that would make a language suitable for my style of programming (heavily refactoring-based).

What Lobster is not:

What Lobster is:

What Lobster was (History):

Lobster development started in October of 2010.

Unlike code, normally a programming language design is so intricate, you benefit from designing almost all of it up-front, before you write the first line of code. Lobster wasn't designed like that, its design was constantly refactored much like its implementation. In fact, normally the first thing I write before I start writing a parser is the grammar, but Lobster's grammar was instead derived from the parser code. Because initially it was simply a platform for fun and experimentation of language features, it started out very different, both in syntax and semantics.

Lobster started out fully dynamically scoped. I like the idea of dynamic scoping, and wanted to experiment with it, to see if all the people on the net claiming it is evil are a bunch of whiners. I found out very quickly they are not whiners and reverted to default lexical scoping.

My other early experiment was a complete implementation of Icon-style backtracking. For example, you could write things like:

print #3 < 2 | #3 * 2

which would print 0 1 0 2 4. In short, whereas in most programming language an expression returns exactly 1 result, in Icon an EarlyLobster it would 0..N results. # was an iteration operator that would generate values much like for does nowadays, and | would simply concatenate streams of values. The concept of something being false just meant no values were generated, i.e. the < operator returns 0 or 1 results (the lhs). This backtracking would even work across function definitions.

If that sounds complicated, it is certainly hard to get right, though it was helped by the fantastic paper The Implementation of Generators and Goal-Directed-Evaluation in Icon by Janalee O'Bagy whose implementation technique I followed closely.

Sadly, though examples like the above look super terse and very powerful, more general examples turned out less readable, less general, and not much shorter than the equivalent code using higher order functions, which is when I got inspired to instead focus on making higher order functions terser and more similar to built-iteration, and make sure they have the same shortcutting power as backtracking (using Lobster's cross-function return). I ripped out all backtracking code at this point. I still have a copy of the last version that has this functionality in, which at some point I can release if there's interest.

Syntax wise, EarlyLobster had a more haskell-y / smalltalk-y bracketed syntax. See for example:

factorial: [ n => 1 > n | (factorial n - 1) * n ]
print "factorial of 7: " + (factorial 7)

Function blocks were [], and lists were {}. One of the first changes I made post EarlyLobster was to switch those two, and make function calls more C-style, as I had to admit I found I liked that better anyway, and made it easier to have function blocks follow the function call instead of being inside of it. Then I noticed I could denote function blocks entirely without brackets and just with indentation, and those were removed too.

The implementation still had the C-stack based function calling implied by the abovementioned paper, which now wasn't necessary anymore, and Lobster switched to a more traditional VM stack.

Later, multi-methods, and more recently, coroutines, got added.

Most recently, Lobster transitioned from being a dynamically typed language to a statically typed language. I prefer programming using dynamic typing, but I had an idea how I could use specialization in combination with type inference and flow-sensitive typing to get very far in being able to type-check all existing Lobster code without having to litter it with types. That challenge took it much further than I anticipated. The end result is impressively powerful, even though I made it a bit more strict than I planned (i.e. making nil its own type). More here: type system.

Another big change is moving to compile-time reference counts (lifetime analysis) and in-line structs, about which you can read more in memory management.

This signals the move of Lobster to a more efficiency/strong typing oriented language. Strongly typed enums and bools got added, and multi-methods got replaced by a very flexible overloading / dispatching system. See the language reference.

In July 2020, co-routines were removed, along with the lesser known "frame log" feature. A full explanation of why is here The summary: it is a feature that seems very attractive for game programming, but in practice never got used (and in most cases works just as well with classes), and they put a big complexity strain on the implementation that makes it harder to make the language faster. The last version of the language that does contain co-routines is marked with the label last_coroutine in git, and the diff showing exactly what was all removed (and thus gives a good picture on how it used to work) is here. Similarly, the last version that has frame log functionality is last_frame_log and the diff is here

What Lobster will be (Future Plans)

I've collected many random scribbles about what I think Lobster still needs in dev/TODO.txt.