Pinned places
In the previous post, I described the goal of Rust’s Pin
type and the history of how it
came to exist. When we were initially developing this API in 2018, one of our explicit goals was the
limit the number of changes we would make to Rust, because we wanted to ship a “minimum viable
product” of async/await syntax as soon as possible. This meant that Pin
is a type defined in the
standard library, without any syntactic or language support except for the ability to use it as a
method receiver. As I wrote in my previous post, in my opinion this is the source of a “complexity
cliff” when users have to interact with Pin
.
We knew when we made this choice that pinned references would be harder to use and more confusing
than ordinary references, though I think we did underestimate just how much more challenging they
would be for most users. Our initial hope was that with async/await, pinning would disappear into
the background, because the await
operator and the runtime’s spawn
function would pin your
futures for you and you wouldn’t have to encounter it directly. As things played out, there are
still some cases where users must interact with pinned references, even when using async/await. And
sometimes users do need to “drop down” into a lower-level register to implement
Future
themselves; this is when they truly encounter a huge complexity cliff: both the essential
complexity of implementing a state machine “by hand” and the additional complexity of understanding
the APIs to do with Pin
.
My contention in my previous post was that the difficulties involved in this have very little to do
with the complexity inherent in the pinned typestate as a concept, or in pinned references as a way
of representing it, but instead arises from the fact that Pin
is a pure library type without
support from the language. Users who deal with Pin
are almost always doing something that is
totally memory safe, the problem is just that the idioms to do so with Pin
are different from and
less clear than the idioms for doing so with ordinary references.
In this post, I want to propose a set of language changes - completely backward compatible with the
language as it exists and the async ecosystem built on Pin
- which will make interacting with
pinned references much more similar to interacting with ordinary references.
Places and values
We are often taught to think of expressions as evaluating to values. For example, the expression
2 + 2
evaluates to the same value as the expression 4
. But in imperative languages with mutable
state, there is a category of expressions which also evaluate to places. These places are
locations in memory in which values can be stored and later loaded. For example, a variable name is
a place; so is a dereference operator and a field access. When you use a place expression in a value
context, it evaluates to the value which is currently at that place. Because they are places, these
kinds of expressions can also be used in ways values can’t, such as by assigning values to them. For
example, you can’t write 4 = x
but you can write x = 4
.
This distinction between places and values is the same as the distinction between what are called “lvalues” (places) and “rvalues” (values) in other languages like C++. The rvalue/lvalue terminology has its origin in Christopher Strachey’s excellent 1967 Fundamental Concepts in Programming Languages, which gives a very clear and thorough presentation of the conceptual difference between the two kinds of expressions. The Rust project chose to use “place” and “value” instead of “lvalue” and “rvalue” because, contrary to Strachey, they believe its clearer and less confusing (I agree with them). The Rust reference also has a fully enumerated list of the different place expressions and value expressions in Rust.
Properly understood, imperative programming involves code operating over a world of objects, these objects have types, which enumerate the set of possible values the object could have, and these objects exist at places; as a result they also have a notion of identity, in that while two different objects at different places may have the same value, they do not have the same identity. Any programmer knows these facts intuitively but we often elide them when we speak about programming and often cannot clearly explain them, in the same way we successfully use our native languages without being able to explain their grammar.
The reason I raise all of this is that in Rust it is places that are, for example, mutable or
immutable. We often talk about “mutable values” when we talk about mutability and immutability, but
the truth is that values as such are not mutable at all .The value of 4
does not change!
Mutability is a property of a place which controls operations on the object which could change that
object’s value at a given point in time; in essence, an immutable place cannot be assigned to.
Remember that places are not only variables, but can also be other things like (for example) the
target of a reference.
Mutability is not the only attribute of places in Rust; for example, a place can be borrowed, which limits the other operations you can perform on that place while it is borrowed. They can also be moved, so that they are no longer valid for any operation. And which of these states the place is in can change over the lifetime for which the place exists.
Many languages do not allow for such controls on the use of places; increasingly some give programmers the option to restrict mutability, but very few give them the power to restrict the number of times an object is used. In fact, these restrictions on places are really the core value proposition of Rust. There is another interesting comment from Christopher Strachey on the subject of what we would now call functional and imperative programming languages, in the discussion of Peter Landin’s 1965 “The next 700 programming languages”; here “DLs” means what we would today call “functional programming languages”:
The important characteristic of DLs is that it is possible to produce equivalence relations, particularly the rule for substitution which Peter Landin describes as (β) in his paper. That equivalence relation, which appears to be essential in almost every proof, does not hold if you allow assignment statements. The great advantage then of DLs is that they give you some hope of proving the equivalence of program transformations and to begin to have a calculus for combining and manipulating them, which at the moment we haven’t got.
…DLs form a subset of all languages. They are an interesting subset, but one which is inconvenient to use unless you are used to it. We need them because at the moment we don’t know how to construct proofs with languages which include imperatives and jumps. [emphasis added]
(I owe the discovery of this quotation to Uday Reddy’s excellent answers (1 and 2) to the Stack Overflow question “What is referential transparency?”)
Rust is, put simply, an attempt to improve our ability to analyze the behavior of programs with assignment by introducing controls on the behavior of the places which can be assigned to. This is somewhat an aside; the important thing for the remainder of this post is that in Rust there is a notion of a place, and controls on mutability and movement are really applied to places, not to values.
Pinned places
The key idea, then, is to also distinguish in the language between places that are pinned and places that are unpinned, in the same way that we distinguish between places that are mutable and places that are immutable. (I owe this conceptual framework to Jules Bertholet, though some aspects of my treatment will be different). An object at a pinned place is in the pinned typestate; this is how the pinned typestate is represented in the surface language.
Today, we use library APIs built on pinned pointers to emulate the notion of pinned place: the
target of a pinned pointer has the same semantics that a pinned place would have. The fact that this
property of places is only achieved by emulating it indirectly is the source of much of the
confusion around Pin
. Instead, Rust could enforce the rules of pinned places directly in the
language in a similar manner to how it enforces mutability. This would be completely backward
compatible and would make the user experience similar to the current user experience with ordinary
references.
An object at a pinned place cannot be moved from and cannot be borrowed with the &mut
operator.
However, it can be assigned to (if it is also mutable) and it can be borrowed with the &
operator.
This is the behavior Pin
currently emulates by its implementation Deref
and its Pin::set
method. There will also be pinned reference operators (introduced in the next section) which can
only be applied to pinned places.
How do you create a pinned place? First, any place referenced by a pinned pointer must be a pinned
place. So when you have a Pin<&mut T>
, for example, that place you dereference to is a pinned
place (it’s also a mutable place and a borrowed place, but I digress). The new language feature,
though, is that you can also create pinned places on the stack by creating pinned bindings, just
like you can do with mutable bindings:
// `stream` is a pinned, mutable place:
let pinned mut stream = make_stream();
If the type of a pinned place implements Unpin
, the restrictions on that place don’t apply: you
are free to move out of it and take mutable references to it. This could be modeled by saying these
places with Unpin
types aren’t pinned, or that the restriction on pinned places don’t apply to
places with Unpin
types; these mean the same thing from the user’s perspective.
Native pinned references
Once Rust has pinned places, it also needs native syntax for pinned references, similar to the ordinary references. For example:
// `stream` is a pinned, mutable place:
let pinned mut stream: Stream = make_stream();
// `stream_ref` is a pinned, mutable reference to stream:
let stream_ref: &pinned mut Stream = &pinned mut stream;
The &pinned T
and &pinned mut T
types would just be syntactic sugar for the already existing
Pin<&T>
and Pin<&mut T>
; that way, any code using the new syntax is completely interoperable
with code which currently uses the library type. But the pinned reference operators would be a new
way of constructing a pinned reference which is more idiomatic and powerful than the existing
methods.
The pinned reference operators can only be applied to pinned places. A difference from the pin!
macro, which moves the value into a pinned place and evaluates to a pinned reference, is that the
pinned reference operators can be applied repeatedly. This makes them behave much more similarly to
mutable references: you create a pinned reference to any pinned place in the same way you can create
a mutable reference to any mutable place, and later on you can do so again.
The compiler will also implement the same re-borrowing behavior for pinned mutable references as it
does for unpinned mutable references, so users no longer need to use Pin::as_mut
to reborrow them.
This makes pinned references have semantics just like ordinary references.
Pinning in method resolution
There is one last piece to make this work well. When you call a method on a type that takes a reference, you don’t have to explicitly insert the reference. For example, you don’t have to do this every time you want to call a borrowed method:
let capacity = (&vec).capacity();
The same should be true of pinned methods. When performing method resolution, the compiler should insert pinned reference operators in the same way that it inserts the existing reference operators when evaluating possible candidates for method lookup.
As an example, consider that the Stream
trait has a next
adapter, which is a future that
evaluates to its next element. Today, this requires pinning the stream in place, such as with a
pin!
macro:
// Today: must pin the stream with macro and explicitly reborrow it with as_mut:
let mut stream = pin!(make_stream());
stream.as_mut().next().await;
stream.as_mut().next().await;
With the proposal from the previous section, this could instead be written like this:
let pinned mut stream = make_stream();
(&pinned mut stream).next().await;
(&pinned mut stream).next().await;
But if the compiler included pinned references in its method resolution, and automatically inserted them, pinning disappears except for the marker on the binding:
let pinned mut stream = make_stream();
// Inserts `&pinned mut` operator for each call to next:
stream.next().await;
stream.next().await;
This is exactly how an ordinary mutable method work, but applied to pinning. And remember that if
the stream does implement Unpin
, you can feel free to move it again after calling these methods.
Making Pin
methods easier to implement
Adding native enforcement and syntax for pinned places to the language would make dealing with
pinned objects much easier in the “higher-level” register of Rust that uses async/await and so on.
However, sometimes you do need to drop into the “lower-level” register and implement a pinned trait
like Future
or Stream
by hand, instead of using async/await. These features would make that
easier as well.
Pinned method receivers
The first is a little bit more syntactic sugar to bring pinned references up to par with unpinned
references. There should be special syntax for pinned method receivers in the same way their is for
other reference receivers, rather than having to write self: Pin<&mut Self>
:
trait Future {
type Output;
fn poll(&pinned mut self, cx: &mut Context<'_>) -> Poll<Self::Output>;
}
trait Stream {
type Item;
fn poll_next(&pinned mut self, cx: &mut Context<'_>) -> Poll<Option<Self::Item>>;
}
Safe pinned projections
The biggest improvement to pinned methods would be to support safe pinned projections to fields
without requiring on crates like pin-project-lite
. Unfortunately, pinned
projections cannot be safe by default the way normal reference projections are because of the issue
with Drop
discussed in my previous post, so you need some kind of annotation on the type to
indicate that it “opts into” pinned projections.
There’s a second reason that pinned projections require special annotations: sometimes you don’t want a pinned reference to a type to also imply a pinned reference to a field; maybe that field is meant to stay in the unpinned typestate for the lifetime of this type, and later it may become pinned elsewhere.
For this reason, fields of types can now gain an annotation which says that if the object they are a field of is pinned, they are also pinned:
struct Foo {
pinned bar: Bar,
baz: Baz,
}
This means that if a Foo
is pinned, its bar
field is also pinned. A projection to that field is
treated as a pinned place, just like the place containing the Foo
, and has all the same allowances
and restrictions: you can call pinned methods and shared methods on it and assign to it, but you
can’t call mutable methods or move out of it.
However, if a type contains a pinned
field, its other fields are also understood to be “unpinned.”
This means that a projection to those fields does not maintain the pinned state, and those places
are unpinned. As a result, you cannot get a pinned reference to those places, but you can get an
ordinary mutable reference and move out of them.
For this to be sound, there would also need to be two checks on any type with a field marked
pinned
. The first would be that the type containing the annotation cannot have an explicit
implementation of Unpin
; it can only implement Unpin
via the auto trait mechanism (so it is only
Unpin
if all of its fields are also Unpin
). The second would be that its destructor (if it has
one) must take a pinned reference:
impl Drop for Foo {
fn drop(&pinned mut self) {
...
}
}
This prevents users from moving out of the pinned field reference in the destructor, the problem
that originally resulted in pin projections being unsound. You can still treat unpinned fields (like
the baz
field) normally in the destructor, because they are not marked as pinned.
One enabling factor for this is that the Drop::drop
method cannot be called directly, so its fine
to change the signature of the drop
method to use a pinned reference for types that have
explicitly opted into it by adding a pinned
field. You don’t need to add a new kind of Drop
trait, you can allow types with pinned fields to use the same method with a different signature.
Bringing it together
I want to conclude this enumeration of features with an example of just how nice this code could
look by implementing the Join
future using this set of features. First, I will rely on a
MaybeDone
helper, which represents a future or (if the future is finished) its result:
enum MaybeDone<F: Future> {
Polling(pinned F),
Done(Option<F::Output>),
}
impl<F: Future> MaybeDone<F> {
// This performs a pinned projection to the future to poll it,
// then re-assign self if its done:
fn maybe_poll(&pinned mut self, cx: &mut Context<'_>) {
if let MaybeDone::Polling(fut) = self {
if let Poll::Ready(res) = fut.poll(cx) {
*self = MaybeDone::Done(Some(res));
}
}
}
// This immutable method can be called on pinned references:
fn is_done(&self) -> bool {
matches!(self, &MaybeDone::Done(_))
}
// This performs an unpinned projection to the output and then
// moves out of it:
fn take_output(&pinned mut self) -> Option<F::Output> {
if let MaybeDone::Done(res) = self {
res.take()
} else {
None
}
}
}
You’ll note that the pinned
keyword only explicitly appears thrice: when we declare which field
supports pinned projections and when we declare that the methods are pinned methods. Throughout the
body of the code, the compiler uses these annotations to support projecting, reborrowing, and
assigning through pinned references just like it supports normal references.
MaybeDone
also provides a great example of an unpinned field: the Option<F::Output>
field of the
Done
variant is not pinned. This is because you will want to move the output out of this later on;
if that output is a self-referential type, you don’t want it to immediately enter the pinned type
state, because you plan to move it later in take_output
.
Now, I will use that helper to implement Join
:
struct Join<F1: Future, F2: Future> {
pinned fut1: MaybeDone<F1>,
pinned fut2: MaybeDone<F2>,
}
impl<F1: Future, F2: Future> Future for Join<F1, F2> {
type Output = (F1::Output, F2::Output);
fn poll(&pinned mut self, cx: &mut Context<'_>) -> Poll<Self::Output> {
// Poll the two futures:
self.fut1.maybe_poll(cx);
self.fut2.maybe_poll(cx);
// If they're both done, take their results:
if self.fut1.is_done() && self.fut2.is_done() {
let res1 = self.fut1.take_output().unwrap();
let res2 = self.fut2.take_output().unwrap();
Poll::Ready((res1, res2))
} else {
Poll::Pending
}
}
}
Here again, the only place pinned
explicitly appears is in the field annotations and the method
receiver. The compiler uses this to safely project through Join
to its fields, and we are able to
implement this fundamental combinator in code which looks very similar to the implementation if
pinning didn’t exist at all.
Comparison to immovable types
There’s been a proposal floated recently to re-introduce the idea of a Move
trait, but with a
different definition from before. In this new definition, a type that does not implement Move
is
always in the pinned typestate; as soon as it exists, you cannot move it. You can read more about
this idea here.
Conceptually, I think this proposal is misguided because I believe that “being pinned” is best
represented as a property of a place, rather than a property of a type, in the same way that
being mutable is so represented. On the other hand, I do believe that “being pinnable” (in the
sense that the pinned typestate has any meaning) is properly a property of a type. Pinning represent
this distinction well by having pinned places and an Unpin
trait that opts out of pinned places
having any meaning. This is really very similar to the behavior of, for example, the Copy
trait:
“being moved” is a property of a place, but opting out of the restrictions on move is a property of
some types, which implement Copy
.
By representing “being pinned” as a property of a type, this proposal needs to emulate the stateful
aspect of pinning by having two types: one which implements Move
(and is the type of the object
until it becomes pinned) and one which does not (which is the type the object is converted to as it
is pinned). This is additional complexity that arises from emulating with library code what could be
a native feature of the language. It also requires more language features to support: the normal way
to convert an object from one type to another is to call a function, and returning an object from a
function is a move: if the type you want to return is supposed to be immovable, this is not
possible. So this modeling of the pinned typestate also introduces a requirement to support some
form of emplacement, where objects are constructed at their final location in memory instead of
moving them there. Previous attempts to add emplacement to Rust have been unsuccessful.
Of course, the biggest problem with this proposal is how dramatically backward incompatible it is. As I wrote in my previous post, it is not backward compatible to add either a new auto trait or a new ?Trait to control the ability to move a value. This is a hard, possibly unsolvable problem with any proposal that requires adding such a trait. And this new method of expressing the pinned typestate would be completely incompatible with the existing ecosystem of async Rust which expresses the pinned typestate through pinning. To me, this seems like an extremely bitter pill to swallow.
As I’ve written on this blog in the past, I was sympathetic to the idea that maybe Rust should
reconsider a Move
trait until I actually sat down and wrote out what would be possible by
integrating pinned references into the language just like ordinary references. But now that I’ve
actually written out this set of changes and analyzed the core concepts more closely, I think a much
better approach would be to integrate this concept of “pinned places” into the language.
Everything in this post is also completely backward compatible to add to Rust and also totally
interoperable with any code which uses the current Pin
APIs. It requires reserving a keyword for
pinned
in an edition (I chose “pinned” instead of “pin” because “pin” is already in std APIs),
with an interesting beneficial caveat: the sequence pinned mut
is never valid Rust right now, so
the project could add all of these features for pinned mutable references without even an edition
using a contextual keyword. Since the primary use case for pinned references is pinned mutable
references, this is quite nice. This could be added in any point release and code could move to the
new idioms without breaking compatibility with the existing async ecosystem; then pinned could
become a non-contextual keyword in the next edition to support the far less used pinned immutable
place and reference.
Finally, I should make clear that the post linked above contains ideas other than adding a Move
trait; in particular, it contains ideas for adding safe self-referential types to Rust. I haven’t
carefully evaluated those ideas and I express no opinion about them; the point is that once you do
have a self-referential value, it needs to be in the pinned typestate, and instead of providing that
by having it not implement a Move
trait, in my view the more backward compatible and theoretically
sound way to represent that is by requiring it to be at a pinned place and having it not implement
Unpin
.
For the next language
I do think there is a better design than Pin
in the design space, but one that is not at all
backward compatible with Rust as it exists. (And maybe it doesn’t work: I’ve only considered the
idea, never implemented it!) It’s worth keeping it in mind, but not something I think Rust could
ever move to.
When we were trying to solve the problem of self-referential structs, the way that Aaron Turon put it was that mutable references are “too powerful;” they always give the reference the ability not only to assign to the value but also to move out of it. One could imagine an alternative design in which instead of places being unpinned by default and opting into pinning, places are pinned (or perhaps “immovable”) by default, and have to opt into supporting the ability to move out of them. This would make it so that by default places have the least power (can only access via shared reference) and they gain a monotonically increasing set of powers (can assign to them, can move out of them).
In addition to places having to opt into moving, there would be three reference types instead of
two: immutable, mutable, and movable references. APIs like mem::swap
or Option::take
would take
the third kind of reference. There would be some kind of wrapper type which only provides a mutable
reference to support pinning an object in place. And how would you support use cases like the
MaybeDone
case above, in which you want to be able to move out of a mutable but immovable
reference? The same way Rust handles mutating values through shared references: by having some sort
of “interior moveability” feature in the same way it has cells for “interior mutability.”
At first glance this design seems more elegant and internally consistent than where Rust ended up,
but I do not think it superior in a way that would be worth breaking compatibility with the existing
language. Fields of user-defined combinators on Future and Stream will need to have pinned modifiers
when they want to perform a pinned projection; so be it. Given the constraints of backward
compatibility, I think Pin
once integrated into the language becomes a perfectly serviceable
feature, and the current difficulties can be easily resolved without losing compatibility with the
existing and thriving async Rust ecosystem. We exist in the context of all in which we live and what
came before us.