Today I’m kicking out version 0.11 of the Haskell implementation of Cap’n Proto. This includes a few bug fixes, performance improvements, and some minor changes to existing APIs, but the big ticket item is a next-generation API that will eventually replace much of the existing API surface. This post is mainly about the why of this change, and a brief roadmap for where this is going. If you want to get a handle on how to use the new API, Capnp.Tutorial has been updated, as have the most examples in the examples directory

The Problem

The root of the problem is a design oversight I made very early on. Since haskell-capnp allows the user to both read immutable messages and manipulate mutable messages in-place, much of the code needs to be parametrized over this distinction in some way. Early on, we had a Message type class, which contained all of the message-level read operations, and basically everything was parametrized over this. The typed wrappers emitted by the scheme compiler plugin looked like this:

newtype Node msg = Node'newtype_ (Untyped.Struct msg)

At some point, I refactored this a bit so that instead of a type class defined for messages themselves, Message was parametrized over a mutability, using the DataKinds extension:

import Data.Kind (Type)
data Mutability = Const | Mut Type

data Message (mut :: Mutability) where
    -- ...

Which improves clarity a bit, but is ultimately not a radical change to the design.

In any case, this mutability parameter shows up everywhere; it shows up in the types in Capnp.Untyped (for working with data without a schema), which forces it to show up in the newtype wrappers generated by the schema compiler. In turn, and this is where it first started to be A Problem, this forces it to show up in a lot of the type classes in Capnp.Classes. For example:

-- | Types which may be stored in a capnproto message, and have a fixed
size.
--
-- This applies to typed structs, but not e.g. lists, because the length
-- must be known to allocate a list.
class Allocate s e | e -> s where
    -- @'new' msg@ allocates a new value of type @e@ inside @msg@.
    new :: M.WriteCtx m s => M.Message ('Mut s) -> m e

The above type class is implemented by all struct wrappers generated by the schema compiler, but because these newtype wrappers are parameterized over mutability, Allocate must be as well. In the case of Allocate, you’re always working with mutable messages, so it’s actually parametrized over the state token, rather than the mutability, but in any case, you get instances like:

instance Allocate s (Node ('Mut s)) where
    new msg = Node'newtype_ <$> Untyped.allocStruct msg 5 6

This works, but it makes it really hard to write code that operates generically on things that implement Allocate (which kinda defeats the point of having a type class), since the state token has to be part of the instance. It feels like you want QuantifiedConstraints in a lot of places, but I got nowhere trying to put this into practice, so a lot of my application code ended up just saying screw it and hard-coding s as RealWorld, and only being able to operate in IO. That’s awful.

It seems like there’s a better solution to be had here, which is to define the instance on Node, rather than Node ('Mut s), by making the type class take a type of kind Mutability -> *. This seems promising. But this gets incredibly gnarly as soon as you add generic capnproto types to the mix, since those don’t actually have kind Mutability -> *, they have kind Mutability -> * -> * or the like (more -> * for more parameters). So the mutability ends up showing up in arguments anyway, and the more I messed with this kind of thing the worse it seemed to get.

This kind of thing ends up showing up in parts of the library that shouldn’t even care about mutability, since they’re not actually working with messages directly, but because they are working with types from the code generator output which have this type parameter, they are tainted by it too.

For a long time I just lived with this ugliness, but eventually I hit some use cases where I really couldn’t hack around it, so it was time to rethink, and find a way to free ourselves from that pervasive mut parameter once and for all.

Back To Basics

The root of the problem is that we really need a way to talk about a capnproto type in the haskell type system – rather than define instances on the view into a message, there should be some stand-alone, uninhabited type that is used as a phantom in many places.

This is exactly what the new API does. The code generator now emits:

data Node

No mut parameter, but also no data constructors. The type that was previously called Node mut is now Raw mut Node, where Raw is defined in the capnp package.

Once I made this simple change, not only did it mostly solve the type-class hell I’d found myself in, it actually made doing type level programming with this stuff really, really easy. As a result, the new API involves a lot less generated code, is much easier to work with in a generic way, and (especially for the low-level API) much more ergonomic.

What’s Next

I’d originally planned to keep the old API support around for a while, but as it turns out, haskell-capnp has some serious performance problems (I hadn’t benchmarked it until recently), and to solve them without re-introducing type-class hell I need to push this API deeper into the library (into Capnp.Untyped in particular, rather than sitting on top of it), which is going to break the old API in non-trivial ways. Furthermore, the code generator is currently spitting out a lot of code, and most of it is in the old API. This really hurts compile times, which kinda sucks.

So, I’ll be removing support for the old API rather quickly, but not without providing users a stepping stone.

Version 0.12 will fill out the remaining functionality supported by the old API but not the new. If I miss a few minor things, I may make releases 0.12.1 etc. to backport the additions. This release will support both APIs.

Version 0.13 will remove support for the old API.

So, upgrade path will be: update to 0.12, move to the new API, then update to 0.13.

0.13 will likely also include substantial performance improvements enabled by removing the old API.