for await loops (Part I)
The biggest unresolved question regarding the async/await syntax is the final syntax for the await operator. There’s been an enormous amount of discussion on this question so far; a summary of the present status of that discussion and the positions within the language team is coming soon. Right now I want to separately focus on one question which impacts that decision but hasn’t been considered very much yet: for loops which process streams.
A feature of other languages - like JavaScript for example - and of the macro-based futures-async-await prototype is a syntax for writing for loops that operate over streams instead of iterators. It’s very likely we’ll want to add this feature to Rust someday, and its worthwhile to think a bit about the syntax for it so that we don’t design ourselves into a corner.
The futures-async-await library uses this syntax for the feature:
#[async]
for elem in stream { ... }
Framing this as an “async for loop” and making the syntax something like async for elem in stream
makes some intuitive sense, but I think its actually a
mistake when you investigate further. async modifies things to make them create
a future of something instead of evaluating to that thing directly - this is
not what this syntax does. What this does is yield from the surrounding async
item when the next item in the stream is not ready yet - exactly the same thing
as what the await
operator does to futures already. This is why JavaScript
went with this syntax, which uses await:
for await (elem of stream) { ... }
(In JavaScript, the equivalent of for elem in stream
is for (elem of stream)
.
Proceding from the basis that we this syntax would use the await
keyword, we
should think about what our options for the syntax here are, and how they feed
back into the decision for the initial await expression operator.
Is the await part of the pattern or the loop syntax?
One of the fundamental decisions we need to make in implementing this syntax is this: is the await operator a part of the pattern component of the syntax, or the for loop operation. If its a part of the pattern, then it would be an irrefutable pattern that would match any future and await it, and could be used in other positions (like the branches of a match statement). If its part of the loop, then its just a modified form of the for loop syntax, that takes both a pattern and an expression.
Making it part of the pattern is somewhat appealing: its more generally useful
and makes sense as a feature to have. It also goes hand-in-hand with another
appealing feature: allowing the ?
operator in patterns to unwrap them
irrefutably. This is desirable because this is already a common pattern when
dealing with IO iterators, like iterating over the lines in a file:
for line in file.lines() {
let lines = lines?;
// ...
}
However, making await a part of the pattern has a big problem: for loops take
an iterator, and a stream is not equivalent to an iterator of futures (that
is, you could not implement Iterator for S where S: Stream
) because the
futures returned by a stream are not wholy independent of one another - the
Next
future that Stream::next
returns borrows mutably from the underlying
stream until it resolves. In other words, Streams are more analogous to the
hypothetical trait StreamingIterator
(which can’t be expressed in Rust today)
rather than to the Iterators.
I’ll explore a way we could modify for loops to take streams and make the await part of the pattern in the next post, but for the rest of this post let’s assume we will make await a part of the for loop syntax, and what the implications of that are.
Implications of for await constructs
Based on precedent from other languages, the most obvious syntax for this
feature would be for await elem in stream
. However, if the syntax we use for
the await expression syntax is not await future
(as it is in other
languages), this introduces quite a bit of inconsistency between the two
positions in which the await keyword appears.
On the other hand, the majority of the postfix await syntaxes visually conjoin
the await operator with its operand - that is, foo.await
(for example) is
designed so that foo
and .await
are not intended to be whitespace
separated. It seems confusing and unexpected to me that, using a syntax like
for elem.await in stream
, that .await
is not a part of the pattern that the
loop takes, but is instead part of the for loop.
It also seems especially challenging for syntaxes which mimic other expression
types. For example, some users have argued for foo.await()
so that the
operator looks like a method and can be thought of like one (though it isn’t
a method at all). I think it would be problematic and very inconsistent to
allow for elem.await()
but not allow “other” methods in this position.
I think this favors space-separated syntaxes, either of these, because they clearly separate the operator from the pattern:
for await elem in stream { }
for elem await in stream { }
However, there is one interesting quirk to consider for both of these syntaxes.
The most popular prefix argument favors adding an await?
syntax sugar which
combines the await operator with the ? operator. It would be most consistent
for that to work here as well:
for await? elem in stream { }
As I mentioned earlier, we’ve also long considered adding ?
to patterns.
Because of the inherent order of operations between the for loop syntax and the
pattern syntax, this would make these two syntaxes equivalent:
// Apply ? to the stream item as a part of the for await loop syntax
for await? elem in stream { }
// Run the for await loop syntax, then apply ? to the stream item
// as a part of the pattern
for await elem? in stream { }
The reason that the await?
syntactic sugar is being proposed is that the
order of operations of await elem?
in the expression context is not of this
order. Having both orders work the same way in loops introduces its own small
inconsistency between for await loops and await expressions.
For postfix-space, it introduces a similar but different problem. In
particular, postfix-space does not propose to add a special-case await?
combined operation, so it does not follow that for elem await?
would work.
This means that if we ever add ?
in patterns, the behavior of for elem? await
would be to apply await before ?
(what users probably want), even
though visually they are in the opposite order. This seems quite troubling.
Conclusions
I think that if we make await part of the loop syntax, the choice that makes
the most sense to me is the prefix-await syntax. However, even that syntax
introduces a surprising little inconsistency where the order of operations in
await elem?
in a for loop header is different from the order of operations in
await future?
in an expression context. So none of these syntaxes really
works perfectly.
That’s why in the next post I’ll look at the changes we’d need to allow making await in patterns integrate well with for loops, and also at how that would impact things.