In the previous post, I described the goal of Rust’s Pin type and the history of how it came to exist. When we were initially developing this API in 2018, one of our explicit goals was the limit the number of changes we would make to Rust, because we wanted to ship a “minimum viable product” of async/await syntax as soon as possible. This meant that Pin is a type defined in the standard library, without any syntactic or language support except for the ability to use it as a method receiver. As I wrote in my previous post, in my opinion this is the source of a “complexity cliff” when users have to interact with Pin.

We knew when we made this choice that pinned references would be harder to use and more confusing than ordinary references, though I think we did underestimate just how much more challenging they would be for most users. Our initial hope was that with async/await, pinning would disappear into the background, because the await operator and the runtime’s spawn function would pin your futures for you and you wouldn’t have to encounter it directly. As things played out, there are still some cases where users must interact with pinned references, even when using async/await. And sometimes users do need to “drop down” into a lower-level register to implement Future themselves; this is when they truly encounter a huge complexity cliff: both the essential complexity of implementing a state machine “by hand” and the additional complexity of understanding the APIs to do with Pin.

My contention in my previous post was that the difficulties involved in this have very little to do with the complexity inherent in the pinned typestate as a concept, or in pinned references as a way of representing it, but instead arises from the fact that Pin is a pure library type without support from the language. Users who deal with Pin are almost always doing something that is totally memory safe, the problem is just that the idioms to do so with Pin are different from and less clear than the idioms for doing so with ordinary references.

In this post, I want to propose a set of language changes - completely backward compatible with the language as it exists and the async ecosystem built on Pin - which will make interacting with pinned references much more similar to interacting with ordinary references.

Places and values

We are often taught to think of expressions as evaluating to values. For example, the expression 2 + 2 evaluates to the same value as the expression 4. But in imperative languages with mutable state, there is a category of expressions which also evaluate to places. These places are locations in memory in which values can be stored and later loaded. For example, a variable name is a place; so is a dereference operator and a field access. When you use a place expression in a value context, it evaluates to the value which is currently at that place. Because they are places, these kinds of expressions can also be used in ways values can’t, such as by assigning values to them. For example, you can’t write 4 = x but you can write x = 4.

This distinction between places and values is the same as the distinction between what are called “lvalues” (places) and “rvalues” (values) in other languages like C++. The rvalue/lvalue terminology has its origin in Christopher Strachey’s excellent 1967 Fundamental Concepts in Programming Languages, which gives a very clear and thorough presentation of the conceptual difference between the two kinds of expressions. The Rust project chose to use “place” and “value” instead of “lvalue” and “rvalue” because, contrary to Strachey, they believe its clearer and less confusing (I agree with them). The Rust reference also has a fully enumerated list of the different place expressions and value expressions in Rust.

Properly understood, imperative programming involves code operating over a world of objects, these objects have types, which enumerate the set of possible values the object could have, and these objects exist at places; as a result they also have a notion of identity, in that while two different objects at different places may have the same value, they do not have the same identity. Any programmer knows these facts intuitively but we often elide them when we speak about programming and often cannot clearly explain them, in the same way we successfully use our native languages without being able to explain their grammar.

The reason I raise all of this is that in Rust it is places that are, for example, mutable or immutable. We often talk about “mutable values” when we talk about mutability and immutability, but the truth is that values as such are not mutable at all .The value of 4 does not change! Mutability is a property of a place which controls operations on the object which could change that object’s value at a given point in time; in essence, an immutable place cannot be assigned to. Remember that places are not only variables, but can also be other things like (for example) the target of a reference.

Mutability is not the only attribute of places in Rust; for example, a place can be borrowed, which limits the other operations you can perform on that place while it is borrowed. They can also be moved, so that they are no longer valid for any operation. And which of these states the place is in can change over the lifetime for which the place exists.

Many languages do not allow for such controls on the use of places; increasingly some give programmers the option to restrict mutability, but very few give them the power to restrict the number of times an object is used. In fact, these restrictions on places are really the core value proposition of Rust. There is another interesting comment from Christopher Strachey on the subject of what we would now call functional and imperative programming languages, in the discussion of Peter Landin’s 1965 “The next 700 programming languages”; here “DLs” means what we would today call “functional programming languages”:

The important characteristic of DLs is that it is possible to produce equivalence relations, particularly the rule for substitution which Peter Landin describes as (β) in his paper. That equivalence relation, which appears to be essential in almost every proof, does not hold if you allow assignment statements. The great advantage then of DLs is that they give you some hope of proving the equivalence of program transformations and to begin to have a calculus for combining and manipulating them, which at the moment we haven’t got.

…DLs form a subset of all languages. They are an interesting subset, but one which is inconvenient to use unless you are used to it. We need them because at the moment we don’t know how to construct proofs with languages which include imperatives and jumps. [emphasis added]

(I owe the discovery of this quotation to Uday Reddy’s excellent answers (1 and 2) to the Stack Overflow question “What is referential transparency?”)

Rust is, put simply, an attempt to improve our ability to analyze the behavior of programs with assignment by introducing controls on the behavior of the places which can be assigned to. This is somewhat an aside; the important thing for the remainder of this post is that in Rust there is a notion of a place, and controls on mutability and movement are really applied to places, not to values.

Pinned places

The key idea, then, is to also distinguish in the language between places that are pinned and places that are unpinned, in the same way that we distinguish between places that are mutable and places that are immutable. (I owe this conceptual framework to Jules Bertholet, though some aspects of my treatment will be different). An object at a pinned place is in the pinned typestate; this is how the pinned typestate is represented in the surface language.

Today, we use library APIs built on pinned pointers to emulate the notion of pinned place: the target of a pinned pointer has the same semantics that a pinned place would have. The fact that this property of places is only achieved by emulating it indirectly is the source of much of the confusion around Pin. Instead, Rust could enforce the rules of pinned places directly in the language in a similar manner to how it enforces mutability. This would be completely backward compatible and would make the user experience similar to the current user experience with ordinary references.

An object at a pinned place cannot be moved from and cannot be borrowed with the &mut operator. However, it can be assigned to (if it is also mutable) and it can be borrowed with the & operator. This is the behavior Pin currently emulates by its implementation Deref and its Pin::set method. There will also be pinned reference operators (introduced in the next section) which can only be applied to pinned places.

How do you create a pinned place? First, any place referenced by a pinned pointer must be a pinned place. So when you have a Pin<&mut T>, for example, that place you dereference to is a pinned place (it’s also a mutable place and a borrowed place, but I digress). The new language feature, though, is that you can also create pinned places on the stack by creating pinned bindings, just like you can do with mutable bindings:

// `stream` is a pinned, mutable place:
let pinned mut stream = make_stream();

If the type of a pinned place implements Unpin, the restrictions on that place don’t apply: you are free to move out of it and take mutable references to it. This could be modeled by saying these places with Unpin types aren’t pinned, or that the restriction on pinned places don’t apply to places with Unpin types; these mean the same thing from the user’s perspective.

Native pinned references

Once Rust has pinned places, it also needs native syntax for pinned references, similar to the ordinary references. For example:

// `stream` is a pinned, mutable place:
let pinned mut stream: Stream = make_stream();

// `stream_ref` is a pinned, mutable reference to stream:
let stream_ref: &pinned mut Stream = &pinned mut stream;

The &pinned T and &pinned mut T types would just be syntactic sugar for the already existing Pin<&T> and Pin<&mut T>; that way, any code using the new syntax is completely interoperable with code which currently uses the library type. But the pinned reference operators would be a new way of constructing a pinned reference which is more idiomatic and powerful than the existing methods.

The pinned reference operators can only be applied to pinned places. A difference from the pin! macro, which moves the value into a pinned place and evaluates to a pinned reference, is that the pinned reference operators can be applied repeatedly. This makes them behave much more similarly to mutable references: you create a pinned reference to any pinned place in the same way you can create a mutable reference to any mutable place, and later on you can do so again.

The compiler will also implement the same re-borrowing behavior for pinned mutable references as it does for unpinned mutable references, so users no longer need to use Pin::as_mut to reborrow them. This makes pinned references have semantics just like ordinary references.

Pinning in method resolution

There is one last piece to make this work well. When you call a method on a type that takes a reference, you don’t have to explicitly insert the reference. For example, you don’t have to do this every time you want to call a borrowed method:

let capacity = (&vec).capacity();

The same should be true of pinned methods. When performing method resolution, the compiler should insert pinned reference operators in the same way that it inserts the existing reference operators when evaluating possible candidates for method lookup.

As an example, consider that the Stream trait has a next adapter, which is a future that evaluates to its next element. Today, this requires pinning the stream in place, such as with a pin! macro:

// Today: must pin the stream with macro and explicitly reborrow it with as_mut:
let mut stream = pin!(make_stream());
stream.as_mut().next().await;
stream.as_mut().next().await;

With the proposal from the previous section, this could instead be written like this:

let pinned mut stream = make_stream();
(&pinned mut stream).next().await;
(&pinned mut stream).next().await;

But if the compiler included pinned references in its method resolution, and automatically inserted them, pinning disappears except for the marker on the binding:

let pinned mut stream = make_stream();

// Inserts `&pinned mut` operator for each call to next:
stream.next().await;
stream.next().await;

This is exactly how an ordinary mutable method work, but applied to pinning. And remember that if the stream does implement Unpin, you can feel free to move it again after calling these methods.

Making Pin methods easier to implement

Adding native enforcement and syntax for pinned places to the language would make dealing with pinned objects much easier in the “higher-level” register of Rust that uses async/await and so on. However, sometimes you do need to drop into the “lower-level” register and implement a pinned trait like Future or Stream by hand, instead of using async/await. These features would make that easier as well.

Pinned method receivers

The first is a little bit more syntactic sugar to bring pinned references up to par with unpinned references. There should be special syntax for pinned method receivers in the same way their is for other reference receivers, rather than having to write self: Pin<&mut Self>:

trait Future {
    type Output;

    fn poll(&pinned mut self, cx: &mut Context<'_>) -> Poll<Self::Output>;
}

trait Stream {
    type Item;

    fn poll_next(&pinned mut self, cx: &mut Context<'_>) -> Poll<Option<Self::Item>>;
}

Safe pinned projections

The biggest improvement to pinned methods would be to support safe pinned projections to fields without requiring on crates like pin-project-lite. Unfortunately, pinned projections cannot be safe by default the way normal reference projections are because of the issue with Drop discussed in my previous post, so you need some kind of annotation on the type to indicate that it “opts into” pinned projections.

There’s a second reason that pinned projections require special annotations: sometimes you don’t want a pinned reference to a type to also imply a pinned reference to a field; maybe that field is meant to stay in the unpinned typestate for the lifetime of this type, and later it may become pinned elsewhere.

For this reason, fields of types can now gain an annotation which says that if the object they are a field of is pinned, they are also pinned:

struct Foo {
    pinned bar: Bar,
    baz: Baz,
}

This means that if a Foo is pinned, its bar field is also pinned. A projection to that field is treated as a pinned place, just like the place containing the Foo, and has all the same allowances and restrictions: you can call pinned methods and shared methods on it and assign to it, but you can’t call mutable methods or move out of it.

However, if a type contains a pinned field, its other fields are also understood to be “unpinned.” This means that a projection to those fields does not maintain the pinned state, and those places are unpinned. As a result, you cannot get a pinned reference to those places, but you can get an ordinary mutable reference and move out of them.

For this to be sound, there would also need to be two checks on any type with a field marked pinned. The first would be that the type containing the annotation cannot have an explicit implementation of Unpin; it can only implement Unpin via the auto trait mechanism (so it is only Unpin if all of its fields are also Unpin). The second would be that its destructor (if it has one) must take a pinned reference:

impl Drop for Foo {
    fn drop(&pinned mut self) {
        ...
    }
}

This prevents users from moving out of the pinned field reference in the destructor, the problem that originally resulted in pin projections being unsound. You can still treat unpinned fields (like the baz field) normally in the destructor, because they are not marked as pinned.

One enabling factor for this is that the Drop::drop method cannot be called directly, so its fine to change the signature of the drop method to use a pinned reference for types that have explicitly opted into it by adding a pinned field. You don’t need to add a new kind of Drop trait, you can allow types with pinned fields to use the same method with a different signature.

Bringing it together

I want to conclude this enumeration of features with an example of just how nice this code could look by implementing the Join future using this set of features. First, I will rely on a MaybeDone helper, which represents a future or (if the future is finished) its result:

enum MaybeDone<F: Future> {
    Polling(pinned F),
    Done(Option<F::Output>),
}

impl<F: Future> MaybeDone<F> {
    // This performs a pinned projection to the future to poll it,
    // then re-assign self if its done:
    fn maybe_poll(&pinned mut self, cx: &mut Context<'_>) {
        if let MaybeDone::Polling(fut) = self {
            if let Poll::Ready(res) = fut.poll(cx) {
                *self = MaybeDone::Done(Some(res));
            }
        }
    }

    // This immutable method can be called on pinned references:
    fn is_done(&self) -> bool {
        matches!(self, &MaybeDone::Done(_))
    }

    // This performs an unpinned projection to the output and then
    // moves out of it:
    fn take_output(&pinned mut self) -> Option<F::Output> {
        if let MaybeDone::Done(res) = self {
            res.take()
        } else {
            None
        }
    }
}

You’ll note that the pinned keyword only explicitly appears thrice: when we declare which field supports pinned projections and when we declare that the methods are pinned methods. Throughout the body of the code, the compiler uses these annotations to support projecting, reborrowing, and assigning through pinned references just like it supports normal references.

MaybeDone also provides a great example of an unpinned field: the Option<F::Output> field of the Done variant is not pinned. This is because you will want to move the output out of this later on; if that output is a self-referential type, you don’t want it to immediately enter the pinned type state, because you plan to move it later in take_output.

Now, I will use that helper to implement Join:

struct Join<F1: Future, F2: Future> {
    pinned fut1: MaybeDone<F1>,
    pinned fut2: MaybeDone<F2>,
}

impl<F1: Future, F2: Future> Future for Join<F1, F2> {
    type Output = (F1::Output, F2::Output);

    fn poll(&pinned mut self, cx: &mut Context<'_>) -> Poll<Self::Output> {
        // Poll the two futures:
        self.fut1.maybe_poll(cx);
        self.fut2.maybe_poll(cx);
        
        // If they're both done, take their results:
        if self.fut1.is_done() && self.fut2.is_done() {
            let res1 = self.fut1.take_output().unwrap();
            let res2 = self.fut2.take_output().unwrap();
            Poll::Ready((res1, res2))
        } else {
            Poll::Pending
        }
    }
}

Here again, the only place pinned explicitly appears is in the field annotations and the method receiver. The compiler uses this to safely project through Join to its fields, and we are able to implement this fundamental combinator in code which looks very similar to the implementation if pinning didn’t exist at all.

Comparison to immovable types

There’s been a proposal floated recently to re-introduce the idea of a Move trait, but with a different definition from before. In this new definition, a type that does not implement Move is always in the pinned typestate; as soon as it exists, you cannot move it. You can read more about this idea here.

Conceptually, I think this proposal is misguided because I believe that “being pinned” is best represented as a property of a place, rather than a property of a type, in the same way that being mutable is so represented. On the other hand, I do believe that “being pinnable” (in the sense that the pinned typestate has any meaning) is properly a property of a type. Pinning represent this distinction well by having pinned places and an Unpin trait that opts out of pinned places having any meaning. This is really very similar to the behavior of, for example, the Copy trait: “being moved” is a property of a place, but opting out of the restrictions on move is a property of some types, which implement Copy.

By representing “being pinned” as a property of a type, this proposal needs to emulate the stateful aspect of pinning by having two types: one which implements Move (and is the type of the object until it becomes pinned) and one which does not (which is the type the object is converted to as it is pinned). This is additional complexity that arises from emulating with library code what could be a native feature of the language. It also requires more language features to support: the normal way to convert an object from one type to another is to call a function, and returning an object from a function is a move: if the type you want to return is supposed to be immovable, this is not possible. So this modeling of the pinned typestate also introduces a requirement to support some form of emplacement, where objects are constructed at their final location in memory instead of moving them there. Previous attempts to add emplacement to Rust have been unsuccessful.

Of course, the biggest problem with this proposal is how dramatically backward incompatible it is. As I wrote in my previous post, it is not backward compatible to add either a new auto trait or a new ?Trait to control the ability to move a value. This is a hard, possibly unsolvable problem with any proposal that requires adding such a trait. And this new method of expressing the pinned typestate would be completely incompatible with the existing ecosystem of async Rust which expresses the pinned typestate through pinning. To me, this seems like an extremely bitter pill to swallow.

As I’ve written on this blog in the past, I was sympathetic to the idea that maybe Rust should reconsider a Move trait until I actually sat down and wrote out what would be possible by integrating pinned references into the language just like ordinary references. But now that I’ve actually written out this set of changes and analyzed the core concepts more closely, I think a much better approach would be to integrate this concept of “pinned places” into the language.

Everything in this post is also completely backward compatible to add to Rust and also totally interoperable with any code which uses the current Pin APIs. It requires reserving a keyword for pinned in an edition (I chose “pinned” instead of “pin” because “pin” is already in std APIs), with an interesting beneficial caveat: the sequence pinned mut is never valid Rust right now, so the project could add all of these features for pinned mutable references without even an edition using a contextual keyword. Since the primary use case for pinned references is pinned mutable references, this is quite nice. This could be added in any point release and code could move to the new idioms without breaking compatibility with the existing async ecosystem; then pinned could become a non-contextual keyword in the next edition to support the far less used pinned immutable place and reference.

Finally, I should make clear that the post linked above contains ideas other than adding a Move trait; in particular, it contains ideas for adding safe self-referential types to Rust. I haven’t carefully evaluated those ideas and I express no opinion about them; the point is that once you do have a self-referential value, it needs to be in the pinned typestate, and instead of providing that by having it not implement a Move trait, in my view the more backward compatible and theoretically sound way to represent that is by requiring it to be at a pinned place and having it not implement Unpin.

For the next language

I do think there is a better design than Pin in the design space, but one that is not at all backward compatible with Rust as it exists. (And maybe it doesn’t work: I’ve only considered the idea, never implemented it!) It’s worth keeping it in mind, but not something I think Rust could ever move to.

When we were trying to solve the problem of self-referential structs, the way that Aaron Turon put it was that mutable references are “too powerful;” they always give the reference the ability not only to assign to the value but also to move out of it. One could imagine an alternative design in which instead of places being unpinned by default and opting into pinning, places are pinned (or perhaps “immovable”) by default, and have to opt into supporting the ability to move out of them. This would make it so that by default places have the least power (can only access via shared reference) and they gain a monotonically increasing set of powers (can assign to them, can move out of them).

In addition to places having to opt into moving, there would be three reference types instead of two: immutable, mutable, and movable references. APIs like mem::swap or Option::take would take the third kind of reference. There would be some kind of wrapper type which only provides a mutable reference to support pinning an object in place. And how would you support use cases like the MaybeDone case above, in which you want to be able to move out of a mutable but immovable reference? The same way Rust handles mutating values through shared references: by having some sort of “interior moveability” feature in the same way it has cells for “interior mutability.”

At first glance this design seems more elegant and internally consistent than where Rust ended up, but I do not think it superior in a way that would be worth breaking compatibility with the existing language. Fields of user-defined combinators on Future and Stream will need to have pinned modifiers when they want to perform a pinned projection; so be it. Given the constraints of backward compatibility, I think Pin once integrated into the language becomes a perfectly serviceable feature, and the current difficulties can be easily resolved without losing compatibility with the existing and thriving async Rust ecosystem. We exist in the context of all in which we live and what came before us.