Without boats, dreams dry up

Organizational Debt

We all know that classic aphorism: Year comes to an end, Rust blog post press send. This is mine.

There are lots of cool technical improvements to Rust that I want the project to achieve this year, and a few in particular that I’m definitely going to be putting a lot of time into. But this blog post is going to talk about none of them. Instead, I want to talk about organizational debt, and how badly the Rust project needs to deal with it in 2019.

The Rust project has been growing like a startup for the last several years. This has some good aspect - “hockeystick curves” - but it has some very bad aspects as well. If this project is going to have a chance of sustaining itself in the long term, we need to get real about dealing with the organizational debt we have accumulated. I think we have serious problems at every level of our organization that need to be addressed, and I’m going to enumerate them from my personal perspective. Press send.

Using GitHub issues to discuss design is like drinking from a firehose

I’ve collected some of the GitHub issues and internals threads related to one specific API addition I worked on this year: the pinning APIs. This API is far from one of the most controversial APIs - not even in the top 10. I would guess (with no overall numbers) that its pretty close to the mean, maybe a little above.

Here’s a list of the threads I found about the design of the pin API; none or very little of this discussion is the sort of code review back and forth you get on pull requests, because none of these are pull requests to the rust repo, only design discussion threads:

That’s a total of 770 comments on the design of the API for the Pin type, which has not yet been stabilized; I expect it will pass 800 comments before this is done. This is just one significant but ultimately fairly small std API addition, and it doesn’t isn’t including the discussion that’s gone on around the features that are built on top of it, like async/await, generators and futures. And it doesn’t include discussion outside of official venues, like reddit threads and IRC or Discord chatter.

Rust is my full time job and even I find it impossible to keep up on every design discussion happening related to the teams that I am on (lang, libs, and cargo). Its been a regular event this year that I discover during lang team triage that there’s a new RFC I hadn’t seen that already has more than 50 comments. For someone who isn’t paid to do this, trying to participate productively in any of these discussions seems extremely difficult.

And this compounds itself. When there have already been 770 comments on a subject, you’re obviously not going to read them all. Its been very common for users to make comments that are repetitious of previous discussions, re-opening questions that were previously resolved (sometimes for good reason, but sometimes spuriously), making the conversation even harder to follow as the issue has to be relitigated or a fine and easily confused point explained yet again. Every comment added to a discussion thread is ultimately a form of debt, and its a form of debt with compound interest.

What’s worse is that its become clear that breaking the discussion into smaller components is not a solution to the problem. No matter how many GitHub issues we create, it seems that every single one will grow in length until it becomes an unsustainable conversation. We have a problem of induced demand: just as adding lanes to a highway does not resolve traffic congestion, creating more threads does not resolve comment congestion.

All of this discussion is having several negative consequences:

  1. It becomes overwhelming and exhausting for people whose role is to drive consensus in these discussions.
  2. It becomes harder for users with genuinely new insights to participate because they can’t tell if their insight is new or not and may decide just not to post it.
  3. It creates conflict as miscommunication occurs between people with most or all of the comment context already absorbed and people who are entering the discussion for the first time.

The RFC process we’ve been using has not scaled to our level of participation. It’s hurting the project as a whole and having a toll on many of the contributors who work within that process as one of their primary activities. I personally would feel very unhappy about needing to initiate a new major consensus discussion (e.g, proposing a major new language feature) until we have overhauled this process.

The project is not coordinating smoothly

In order to produce a coherent user experience, Rust needs to have a cohesive design vision across different aspects of the product. It used to be, when the total team membership was under 30 people, that a shared vision could be diffused naturally across the project, as people involved in mutual projects coordinated and discussed and developed their viewpoints together, like a beautifully evolving neural network.

But we’ve long since reached the point where coordinating our design vision by osmosis is not working well. We need an active and intentional circulatory system for information, patterns, and frameworks of decision making related to design. I’ve run into threads repeatedly this year in which decision making felt frought to me because we had no guidelines from which to make decisions. Different teams begin to diverge in product viewpoint, different components become managed by subsets of teams and other contributors without much interaction from the rest of the team, and the project risks becoming scattered and irregular.

In theory, the core team could play this role, but it does not seem well equipped to do so. I think its time that we rethink the organization of a unitary core team in itself and recognize the different kinds of coordination that need to happen between the teams, and create the appropriate cross-team bodies tasked with and capable of providing each specific form of coordination.

The teams are experiencing growing pains

Rust project management is largely performed by the various teams responsible for various areas of the project. I’m a member of three of these teams, and I’ve felt this year like they’ve experienced some serious growing pains:

  • Teams have vague balliwicks, being responsible for vast sections of the project. To some extent we’ve solved this by breaking up teams (the “tools and infrastructure” team of 2016 is now 5 different teams!), but I think there is still difficulty spinning off smaller groups to work on specific issues.
  • At the same time, teams often feel directionless. Because of their “governance” orientation, teams do not have a specific goal, and so the teams I’m on spend the majority of their time triaging issues raised by other people. It feels hard now to imagine the team as a site capable of initializing new projects.
  • The membership of many teams has grown, in some cases stressing the ability the team to meet synchronously - both in terms of coordinating peoples' schedules but also in terms of the effectiveness of synchronous meetings of so many people.
  • As the membership has grown, the institutional memory of the teams has weakened. It’s hard to transfer knowledge from previous team members to new team members naturally, and we have no intentional structure to perform that kind of knowledge transfer.

We need more infrastructure for team organization and an ability to more quickly and easily break off discussion groups on specific matters. The working groups have been a step in the right direction, but the teams themselves need to be more proactive in spinning off working groups to respond proactively to specific needs.

Working groups need a toolkit

In this last year, we’ve experimented with working groups as a way to alleviate some of the pressure on teams. A working group is a group with relatively relaxed membership tasked with iterating on a specific problem, leaving ultimate decision making in the hands of the team. This is a great idea, and some working groups have worked well and produced a lot of good results.

But not every working group has been a success. I’ll own: of the four initial “domain working groups,” the networking working group has been the least functional. The reason is pretty straightforward: both Taylor Cramer and I, the people initially assigned to lead it, focused our time on the design of the futures APIs and async/await feature instead of on building out the working group. Leadership of the group has since been passed on to other people, with somewhat better results, but this initial failure has a lot of important lessons in it.

As Nick Fitzgerald pointed out, we need to abstract the ways working groups have worked into a template that other people can use. But even more importantly, we need more people who are capable and interested in doing the very important coordination and leadership work necessary for working groups to succeed.

Community management is emotionally exhausting

In the television series The Wire, which is about the dysfunctional nature of the modern American city, the newly elected mayor has coffee with a man who had been mayor in the past. The mayor-elect asks the former mayor why he had chosen not to run for re-election, and he tells him a parable:

Let me tell you a story, Tommy. The first day I became mayor, they set me down at the desk, big chair and dark wood, lots of beautiful things. I’m thinking, “how much better can it get?” There’s a knock at the door in the corner of the room, and Pete comes walking in carrying this gorgeous sieve silver bowl, hand chased…

So I think it’s a present, something to commemorate my first day as mayor. And he walks over, puts it on a desk. I look down at it. It’s disgusting. I say, “What the hell is this?”

He said, “What the hell’s it look like?”

I said, “It looks like sh*t. Well, what do you want me to do with it?”

He says, “Eat it.”

“Eat it?”

He says, “Yeah. You’re the mayor. You gotta eat it.”

So, OK. It was my first day, and Pete knows more than I do. So I go at it. And just when I finish, there’s a knock on the door. And in walks Pete carrying another silver bowl…

And you know what, Tommy? That’s what it is. You’re sitting eating sh*t all day long, day after day, year after year.

Being a public leader of a major open source project practicing “radical openness” can feel a lot like this.

Someone, somewhere, is always upset about something you are responsible for, and a part of your job is understanding and responding to their concerns. To them, this is the thing that they are upset about, and probably they experience being upset only occassionally. But to you, life is a stream of different people being upset about different things, and you have an obligation to be empathetic and responsive with their concern in each new instance.

That’s fine as far as it goes: after all, that’s the job. But what is so exhausting for me personally is that there is so often not a reciprocity of empathy from the other side of the discussion. Frankly, what I experience often is a startling lack of professionalism from community members, a standard of conduct which - while not below the bare minimum of appropriate behavior set by our Code of Conduct - I cannot imagine would be acceptable as a manner of communication to a colleague in a workplace.

Unlike the project, the community has no defined boundary of membership: different people engage on different subject matters, community members leave, new members join. And so its much harder to pinpoint specific people who need to modify their behavior in specific ways. But I think that it is time that we adopt higher standards of conduct for engaging in the work of the Rust project, as opposed to the Code of Conduct which sets out standards for merely interacting casually with other members of the community. We need to establish the norm that discussion forums for work on Rust are professional spaces, and professional communication in mandatory.

It’s time to talk about pay

There’s one more important point that I need to address here. We’ve been growing like a start up, and that include a lot of our contributors working for made up internet money. Tons of people are devoting tons of time into Rust and getting paid nothing. A lot of these people are doing it because it is intrinsically interesting and fun, a lot of them are in a life position in which they have an excess of free time, and I’m sure a lot of them have the hope that it will pay off in the long term in developing leads for future employment.

I’m fortunate enough to be one of the very few people who do get paid to work on Rust, and my gig is great, but this is a broken and unsustainable situation that is creating lots of problems. Volunteers driving vitally important projects have dropped out as their life situation changes, leaving us in a situation where feature work is incomplete and hard to resume. The scarcity of money inevitably leads to bad feelings and distrust in regard to how it ultimately gets distributed. And only people who the privilege of a lot of free time and confidence are able to get significantly involved in the project on a volunteer basis.

Open source is the unpaid internship of tech. It’s often suggested that the problem is best modeled with open source contributors as an underpaid proletariat, but that is wishful thinking: right now, open source contributors are bourgeois dilettantes; before we can talk pie in the sky fantasies about organizing open source as some kind of workers’ cooperative or union, we need to become workers at all.

I don’t know what a solution to this issue looks like, and I think proposals for a “Rust Foundation” are largely magical thinking that reorganize the problem statement without solving it.

I can give this anecdote, speaking in regard to my particular situation. I don’t know the numbers of course, but I think the work that has been done by project contributors on async/await this year will create really significant, massive value for just the companies that are already adopting Rust for asynchronous networking. But very little of that value will be captured by Mozilla, the only company paying anyone (me) to work on async/await syntax as their primary task for their full time job. More of the value being generated by the Rust project needs to start getting routed back into contribution on Rust than it is now if Rust is going to be sustainable.

Conclusion

So my proposal for Rust 2019 is not that big of a deal, I guess: we just need to redesign our decision making process, reorganize our governance structures, establish new norms of communication, and find a way to redirect a significant amount of capital toward Rust contributors.

I don’t think that its feasible to imagine the Rust project, or any project accomplishing anything significant, as a smoothly functioning body which satisfies and fullfills everyone involved in it while always making the best decisions with the least organizational overhead. But we have been putting off dealing with our organizational debt for too long now, and we need to acknowledge it and work actively on reducing it before it overflows our stacks and crashes our project.