Entries tagged - "generators"

poll_progress


Last week, Tyler Mandry published an interesting post about a problem that the Rust project calls “Barbara battles buffered streams.” Tyler does a good job explaining the issue, but briefly the problem is that the buffering adapters from the futures library (Buffered and BufferUnordered) do not interact well with for await if the processing in the body is asynchronous (i.e. if it contains any await expressions).

I think we can better understand the problem if we examine it visually. First, let’s consider the control flow that occurs when a user processes a normal, non-asynchronous Iterator using a for loop:

                ┌── SOME ────────────────┐ 
        ╔═══════════════╗        ╔═══════▼═══════╗ 
        ║               ║▐▌      ║               ║▐▌
  ──────▶      NEXT     ║▐▌      ║   LOOP BODY   ║▐▌
        ║               ║▐▌      ║               ║▐▌
        ╚════════════▲══╝▐▌      ╚═══════════════╝▐▌
         ▀▀│▀▀▀▀▀▀▀▀▀│▀▀▀▀▘       ▀▀▀▀▀▀▀│▀▀▀▀▀▀▀▀▀▘
           │         └───────────────────┘
           └── NONE ──────────────────────────────▶

The for loop first calls the iterator’s next method, and then passes the resulting item (if there is one) to the loop body. When there are no more items, it exits the loop.

Coroutines, asynchronous and iterative


I wanted to follow up my previous post with a small note elaborating on the use of coroutines for asynchrony and iteration from a more abstract perspective. I realized the point I made about AsyncIterator being the product of Iterator and Future makes a bit more sense if you also consider the “base case” - a block of code that is neither asynchronous nor iterative.

It’s also an excuse to draw another fun ASCII diagram, and I’ve got to put that Berkeley Mono license to good use.

poll_next


In my previous post, I said that the single best thing the Rust project could do for users is stabilize AsyncIterator. I specifically meant the interface that already exists in the standard library, which uses a method called poll_next. Ideally this would have happened years ago, but the second best time would be tomorrow.

The main thing holding up the AsyncIterator stabilization is a commitment by some influential contributors of the project to pursue an alternative design. This design, which I’ll call the “async next” design, proposes to use an async method for the interface instead of the poll method of the “poll next” design implemented today. In my opinion, continuing to pursue this design is a mistake. I’ve written about this before, but I don’t have the sense my post was fully received by the Rust project.

Yosh Wuyts, a leading contributor to the async working group, has written his own post about why the async next design is preferable to poll next. A lot of this is structured as an attempted refutation of points made by me and others about problems with the async next design. I do not find the argument in this post compelling, and my position about what the project should do is unchanged. I’ve written this to attempt to express again, in more detail and more definitively, why I believe the project should accept the poll next design and stabilize AsyncIterator now.

Iterator, Generator


I have been devoting a lot of my free time in the past month to thinking about structured concurrency, and a blog post about that is coming soon, but first I want to revisit iterators and generators.

In a previous post, I wrote about one of the hardest problems for generators: self-referential generators. Unlike the Future trait when we were designing async functions, the Iterator trait is already stable, and it does not take a pinned reference to itself. This means an Iterator cannot be self-referential.

Generators


One of the main emphases of my recent posts has been that I believe shipping generators would solve a lot of user problems by making it easy to write imperative iterative code, and especially to make that iterative code interact well with asynchrony and fallibility as well. One thing that frustrates me about the situation is that generators have been nearly ready to ship for years now, but very little visible progress has been made. In particular, the core compiler transform to take a generator and produce a state machine already exists, because it’s exactly how async functions are implemented.

Generators II: The Question Mark Problem


This is my second post on the design of generators. In the first post, I outlined what an MVP of the feature would look like. In this post, I want to take a look at the first design issue for the feature: how it integrates with the ? operator. To explain exactly what I mean, let’s start with a specific motivating example: // This generator yields the number of alphanumeric characters in every line // in some io::Read'able data // exact sign function declaration syntax left unspecified on purpose |data| { let mut buffered_data = BufReader::new(data); let mut string = String::new(); while buffered_data.…

Generators I: Toward a minimum viable product


We’re still not finished with the design of async/await, but it’s already become clear that it’s time to get the next phases of the feature into the pipeline. There are two extensions to the minimal async/await feature we’ve currently got that seem like the clear high priority: Async methods: allowing async fn to be used in traits. Generators: allowing imperative control flow to create Iterators and Streams the same way async fn allows imperative control flow to create a Future.…