Today I’m kicking out version 0.11 of the Haskell implementation of Cap’n Proto. This includes a few bug fixes, performance improvements, and some minor changes to existing APIs, but the big ticket item is a next-generation API that will eventually replace much of the existing API surface. This post is mainly about the why of this change, and a brief roadmap for where this is going. If you want to get a handle on how to use the new API, Capnp.Tutorial has been updated, as have the most examples in the examples directory
The Problem
The root of the problem is a design oversight I made very early on.
Since haskell-capnp
allows the user to both read immutable messages
and manipulate mutable messages in-place, much of the code needs to be
parametrized over this distinction in some way. Early on, we had a
Message
type class, which contained all of the message-level read
operations, and basically everything was parametrized over this.
The typed wrappers emitted by the scheme compiler plugin looked like
this:
newtype Node msg = Node'newtype_ (Untyped.Struct msg)
At some point, I refactored this a bit so that instead of a type
class defined for messages themselves, Message was parametrized
over a mutability, using the DataKinds
extension:
import Data.Kind (Type)
data Mutability = Const | Mut Type
data Message (mut :: Mutability) where
-- ...
Which improves clarity a bit, but is ultimately not a radical change to the design.
In any case, this mutability parameter shows up everywhere; it shows
up in the types in Capnp.Untyped
(for working with data without a
schema), which forces it to show up in the newtype wrappers generated
by the schema compiler. In turn, and this is where it first started to
be A Problem, this forces it to show up in a lot of the type classes in
Capnp.Classes
. For example:
-- | Types which may be stored in a capnproto message, and have a fixed
size.
--
-- This applies to typed structs, but not e.g. lists, because the length
-- must be known to allocate a list.
class Allocate s e | e -> s where
-- @'new' msg@ allocates a new value of type @e@ inside @msg@.
new :: M.WriteCtx m s => M.Message ('Mut s) -> m e
The above type class is implemented by all struct wrappers generated by
the schema compiler, but because these newtype wrappers are
parameterized over mutability, Allocate
must be as well. In the case
of Allocate
, you’re always working with mutable messages, so it’s
actually parametrized over the state token, rather than the mutability,
but in any case, you get instances like:
instance Allocate s (Node ('Mut s)) where
new msg = Node'newtype_ <$> Untyped.allocStruct msg 5 6
This works, but it makes it really hard to write code that operates
generically on things that implement Allocate
(which kinda defeats the
point of having a type class), since the state token has to be part of
the instance. It feels like you want QuantifiedConstraints
in a lot of
places, but I got nowhere trying to put this into practice, so a lot of
my application code ended up just saying screw it and hard-coding s
as
RealWorld
, and only being able to operate in IO
. That’s awful.
It seems like there’s a better solution to be had here, which is to
define the instance on Node
, rather than Node ('Mut s)
, by making
the type class take a type of kind Mutability -> *
. This seems
promising. But this gets incredibly gnarly as soon as you add generic
capnproto types to the mix, since those don’t actually have kind
Mutability -> *
, they have kind Mutability -> * -> *
or the like
(more -> *
for more parameters). So the mutability ends up showing
up in arguments anyway, and the more I messed with this kind of thing
the worse it seemed to get.
This kind of thing ends up showing up in parts of the library that shouldn’t even care about mutability, since they’re not actually working with messages directly, but because they are working with types from the code generator output which have this type parameter, they are tainted by it too.
For a long time I just lived with this ugliness, but eventually I hit
some use cases where I really couldn’t hack around it, so it was time
to rethink, and find a way to free ourselves from that pervasive mut
parameter once and for all.
Back To Basics
The root of the problem is that we really need a way to talk about a capnproto type in the haskell type system – rather than define instances on the view into a message, there should be some stand-alone, uninhabited type that is used as a phantom in many places.
This is exactly what the new API does. The code generator now emits:
data Node
No mut
parameter, but also no data constructors. The type that was
previously called Node mut
is now Raw mut Node
, where Raw
is
defined in the capnp package.
Once I made this simple change, not only did it mostly solve the type-class hell I’d found myself in, it actually made doing type level programming with this stuff really, really easy. As a result, the new API involves a lot less generated code, is much easier to work with in a generic way, and (especially for the low-level API) much more ergonomic.
What’s Next
I’d originally planned to keep the old API support around for a while,
but as it turns out, haskell-capnp has some serious performance
problems (I hadn’t benchmarked it until recently), and to solve them
without re-introducing type-class hell I need to push this API deeper
into the library (into Capnp.Untyped
in particular, rather than
sitting on top of it), which is going to break the old API in
non-trivial ways. Furthermore, the code generator is currently spitting
out a lot of code, and most of it is in the old API. This really hurts
compile times, which kinda sucks.
So, I’ll be removing support for the old API rather quickly, but not without providing users a stepping stone.
Version 0.12 will fill out the remaining functionality supported by the old API but not the new. If I miss a few minor things, I may make releases 0.12.1 etc. to backport the additions. This release will support both APIs.
Version 0.13 will remove support for the old API.
So, upgrade path will be: update to 0.12, move to the new API, then update to 0.13.
0.13 will likely also include substantial performance improvements enabled by removing the old API.