API scribblings

Before I leave Hypothesis (😢) I wanted to get a couple of thoughts out of my head about the evolution of the software we’ve been building together, specifically about the API.

Why are we talking about the API specifically?

Good question! I want to emphasise that this isn’t about the API because the API is the most important part of Hypothesis. It’s not.

Instead, it’s about the API because talking about the API forces me to be specific and concrete in my suggestions, which in turn hopefully means that poor ideas will become apparent more quickly. Some of the suggestions below have impact that stretches far above the API, but the API is still a useful layer at which to discuss them. If the low-level domain model of Hypothesis doesn’t make sense, it’s not fair to expect a regular user to form an implicit mental model of it.

Ok, so what problem are we trying to solve here?

Honestly, it’s kind of a jumble of different but related things. The changes I propose are supposed to:

• make it easier for developers to create annotations using the API
• help Hypothesis to deprecate the single, open “Public” group¹
• lower the barrier for us to support W3C standards for open annotation
• simplify the conceptual modelling of “groups,” and hence the user experience around publishing in one group or another

Let’s pick one. Why is using the API right now hard for developers?

Two main reasons². The first is that our API doesn’t support OAuth, which makes it hard for people to build tools against our API that other people can then use. This is in the process of being fixed, so I won’t say more about it.

The second reason is that two of the most important actions in the API – creating annotations and finding annotations – are, relatively speaking, quite complicated and quite poorly documented. A surprising amount of functionality is hidden behind each of these endpoints, in sometimes non-obvious ways.

Like what?

Like the fact that when you create an annotation using the API, you include:

• Metadata scraped from the page you’re on, in a mostly undocumented format in the document field. This is entirely optional, but given that this could be entirely independent of the action of creating an annotation it’s weird that it’s in here.
• The group that the annotation is to be published in, in the group field. But there’s no (easy-to-use) API that allows a user to navigate the available groups. Representing this as a “field” on the annotation is also misleading. It’s a fundamental decision about where the annotation goes which can’t be changed later.
• Whether or not to share this annotation with other people in the group, in a verbose expanded ACL-like object in the permissions field.

There’s nothing fundamentally wrong with this stuff, it’s just not necessarily needed. There is a way to keep group and permissions information entirely out of the body of the annotation.

So what are you proposing?

First and foremost, that we make groups a first-class object exposed by the API. And second, that most (but not necessarily all) access to annotations is mediated through groups.

To get concrete, for the first part I’m imagining stuff like:

# Get a list of the user's groups
GET /api/profile/groups

# Get a list of all groups in creation order
GET /api/groups

# Get information about a specific group
GET /api/groups/:groupid

# Create a group
POST /api/groups

# Change group settings (for group admins)
PUT /api/groups/:groupid

# Join a group (for groups with membership)
PUT /api/groups/:groupid/members/:username

# Leave a group (for groups with membership)
DELETE /api/groups/:groupid/members/:username

And, for the second:

# List all annotations in a group
GET /api/groups/:groupid/annotations

# List annotations in a group for a specific page
GET /api/groups/:groupid/annotations?uri=...

# Create a new annotation in a group
POST /api/groups/:groupid/annotations

And so on, and so on.

I haven’t thought carefully about whether we’d want to go all in and have all RESTful access to annotations under this namespace, so I’m not quite sure if

# Fetch information about a specific annotation
GET /api/groups/:groupid/annotations/:annotationid

is the logical extension of these ideas.

Cool, so we can access groups through the API. So what?

I think there is a pretty long list of benefits that flow from doing things this way. I’m not going to try and enumerate all of them, but here are a few:

• Creating annotations via the API doesn’t require specifying any permissions or group information in the POST payload: it’s all implied by the URL. As a result, creating annotations is programmatically simpler. The JSON representation of an annotation is shorter and clearer.

• Fetching annotations for a particular group and page is a dedicated URL, rather than an abuse of a “search” endpoint. This potentially simplifies client implementation, and is likely to make caching these lookups on the server much easier. (It provides a very clear point at which to invalidate the cache: when a new annotation is written to the group.)

• A group is a natural analogue for the W3C Annotation Protocol’s Annotation Containers, and could be implemented in a way that conforms to that specification.

• Groups as Annotation Containers extends naturally to “virtual groups” such as the “Only Me” group (see below) as well as groups populated by feeds from external sources. That is: not all groups have to contain annotations made by humans.

• It seems likely that implementing annotation lookup and creation in this way would also result in some significant simplification of the code which does access control checking for annotations/groups.

But what about private annotations?

Right. Most of the improvements I mention above assume that we’ve made another change at the same time, which is to turn “Only Me” into a group rather than a status that an annotation can have within a group.

We could quite easily give each user their own virtual “Only Me” group, or simply leave it up to them to create a group which only they can see.

As I’ve written about this elsewhere I won’t expand on this much more here.

I’m sold! So why wouldn’t we do this?

On the face of it, there aren’t too many downsides to this way of doing things, at least as far as I can see. That said, it’s potentially quite a big change to the way the API works, and could result in substantial additional complexity if the old way of doing things has to work forever, too.

I’d suggest doing a serious evaluation of whether this could be a clean break with the existing API, with an accelerated deprecation plan for the old one. We’re lucky, in that the number of people making serious use of our current API probably numbers in the low dozens at the moment, so now might be the right time to do something like that.