Without boats, dreams dry up

Ringbahn: a safe, ergonomic API for io-uring in Rust

In my previous post, I discussed the new io-uring interface for Linux, and how to create a safe API for using io-uring from Rust. In the time since that post, I have implemented a prototype of such an API. The crate is called ringbahn, and it is intended to enable users to perform IO on io-uring without any risk of memory unsafety.

io-uring is going to be majorly important to the development of async IO on Linux in the future, and Linux is the most used platform for the kinds of high performance network services that are a major user of Rust. Rust needs to not only have a solution for using io-uring, but should also have the best solution. Users should be able to easily access the full power of io-uring from ergonomic, memory safe, well abstracted APIs. ringbahn is not production ready, but it is a step toward that goal.

Here’s a code sample, showing how easy it can be to perform IO on ringbahn:

use std::fs::File;
use futures::io::AsyncRead;

use ringbahn::Ring;

async fn read_on_ringbahn(file: File, buf: &mut [u8]) -> io::Result<usize> {
    // the `Ring` adapter takes a std IO object and runs its IO on io-uring
    let mut file: Ring<File> = Ring::new(file);
    file.read(buf).await
}

I hope to write a few posts documenting the internal structure and design of ringbahn.

Events, drivers, and submissions

Ringbahn’s core concept is the Submission future, which allows users to safely run an IO event on an io-uring instance managed by a driver. Submission provides a contract between two types:

  • Events: are IO events (like reads and writes) which can be scheduled on an io-uring instance.
  • Drivers: implement the Drive trait and manage an io-uring instance onto which events can be scheduled.

The Submission type handles the exchange between discrete IO events and the driver that manages the io-uring instance so that the interface is memory safe and straight forward. Once the driver has proffered up space to submit an event, the Submission future manages submitting the event and preparing an internal Completion to wake itself when the event completes.

What is a driver?

One of the most important innovations in ringbahn is the notion of drivers, which manage an io-uring instance. Rather than simply providing an interface to io-uring, ringbahn actually allows end users to replace the default driver and take low-level control over how their IO is scheduled. A driver is responsible for three things:

  • Readying a spot in the submission queue to submit an event. Drivers can apply backpressure if there is not space to submit another event.
  • Submitting all of the prepared events in the submission queue. Again, drivers can apply backpressure here if needed.
  • Processing all of the events in the completion queue so that they will be awoken by passing them to the complete function.

There are many different patterns for implementing drivers and determining how they actually submit events and process their completions. By making this an abstraction point, users will be able to experiment with different driver patterns and benchmark how they perform under different kinds of loads, without rewriting all of the memory-safety aspects themselves.

The current version of ringbahn contains a demo driver which is suitable for testing purposes, but is not the most performant possible implementation. In a future post I’ll explore different designs for driving io-uring and how they could be implemented.

The event contract and ownership lifecycle

On the other side of the contract are event types, which represent a single IO event to be scheduled on the io-uring. The Event trait has an interesting quirk: it contains an unsafe method, Event::prepare.

It’s important to understand what an unsafe methods means: unsafe methods are unsafe to call - meaning that callers must guarantee additional invariants when calling them. However, they are safe to implement (as long as you don’t do anything unsafe inside of them). So when a user implements the Event trait, they are given additional guarantees about how this method will be called that normally they could not assume.

Specifically, they are given a guarantee that after this method is called, the event type will not be accessed again until after the event that’s been prepared is completed or cancelled. In other words, it acts as if Event::prepare “passes ownership” of the event to the kernel. That way, implementers of Event can safely give the kernel ownership of buffers without worrying about them being accessed by other parts of the program.

The Event API also supports a cancellation API, which constructs a “cancellation callback.” At a high level, the concept is this: if a user cancels interest in an event, the cancellation callback is constructed. When the event completes, instead of waking the user’s task (which no longer cares about this event), the cancellation callback is called to clean up any objects the kernel had taken ownership of.

What this means is that the Event API provides a memory-safe abstraction for scheduling IO events which fully supports cancellation without leaking any resources.

The Ring interface and buffer management

The most high-level interface in ringbahn is the Ring type. Inspired by the Async type from smol, it wraps a standard IO object and performs IO on it using io-uring, instead of blocking IO. The Ring type acts like a type that repeatedly submits events to its io-uring driver (internally, it doesn’t use the Submission and Event APIs directly, but the code mirrors those APIs fundamentaly). Ring implements AsyncRead, AsyncWrite, and AsyncBufRead.

Currently, the Ring type submits events using its own managed buffers. However, in the longer term Ring would ideally be abstract over multiple different kinds of buffer management strategies - especially including ones which pre-register buffers with the kernel, which would be the most optimal approach.

Ring will provide a simple and straightforward way to perform async IO using io-uring, and will always be the most accessible, high level API. Users can choose between that and constructing more complex IO patterns using the lower level Event API.

Next steps

Ringbahn is currently just a prototype, which is why I’ve released it to crates.io as 0.0.0-experimental.1. It contains a lot of unsafe code which has not been sufficiently audited or hardened, and users should not use it in production yet.

In the immediate future, I want to blog in more depth about every aspect of ringbahn’s design. I hope this will help steer the future of Rust’s io-uring support in a productive direction. Expect to see a series of posts on the completion state machine, the driver interface, buffer management, and so forth over the next few weeks.

In the longer term, I would like for there to exist a production ready, best-in-class io-uring interface in Rust. However, I don’t know how much time I will be able to commit to this project, or whether it will continue to be developed over the long term. If anyone is interested in funding this work on a serious basis, I would be interested in hearing from them privately.