Building Products at Scale - Release Trains (Part II)

In Part I, we ended up with a well-defined product backlog. Our Engineering Team can go into their Sprint Planning with a list of well-refined list of User Stories and plan their sprint. However, the Engineering team does not live in a vacuum. When building a product at scale, you have other release activities. We engineers tend to "ignore" them - but they do happen. Releases might need marketing materials prepared, trainings built, webcasts recorded, compliance records reviewed, and many more.

Stability and Agile

Agile is built around flexibility - specifically around adapting to change. However, that is often mistakenly taken to understand there is no plan involved at all. Or, that it means there can be no stability. But with external functions like marketing, you want, née need, stability.

The approach I tend to favour, and have seen work well, is a Release Train. This means that you basically create a fixed schedule of releases - like a train schedule. Trains depart a station at a certain time, and they arrive at different stations at very specific times. They also wait for nobody.

Now, everybody who relies on releases to schedule their activities, knows exactly how to plan activities. But how do they know what goes into a release?

Planning a Train

To recap the previous post, we defined several Product Backlog Items (PBIs) that all build on top of each other, and narrow the focus: Themes -> Epics -> Features -> User Stories (-> Tasks).

As I alluded to in the first post, I've seen this approach implemented in several teams I admire. Microsoft Dynamics releases run in waves, specifically, a release happens twice a year. Azure and Windows are on a Semester cadence. .NET sets its release Themes and surfaces them through http://themesof.net/. In fact, the nice thing is teams can pick the PBI that fits their train departure cadence best.

The common denominator will be that it's fairly vague. For this blog post, that would translate to the "bottom part" of the pyramid. This gives teams the ability to commit to the important tasks, but leave enough room to understand them as they go through refinement. They can still adapt to change, but also introduce some stability.

Introducing stability into change

Let's revisit our theoretical example of building an e-commerce solution. With a small team of 10 people, we can only achieve a limited set of goals. Because of this, our leadership goes and speaks to partners. Through these conversations, they identified that the biggest impact we could have is to enable our partners. Specifically, they noticed that partners want to (1) sync their e-commerce systems with on-premises catalogs, (2) want to extend the payment providers themselves to expand into new markets and (3) want to publish simple online shops.

Any engineer looking at the above statements would shudder at the thought of saying "Yup, we'll deliver all of this in the next release.". Because, well, you can't. For starters, nothing is really well understood - what does "publish simple online shop" really mean? Are we talking a full online web shop? Or maybe syncing to a Shopify solution would solve the problem? To answer these questions, we need to refine. But to run our business, we need to commit and plan.

So, how would this look like with trains?

Step 1: Define a Train Schedule

First, let's create a train schedule. For our example, let's mimic Dynamics' approach and run two major releases.

We want to arrive at our destination in April and September. We have two trains departing on a set date, then. But, the fun doesn't stop there. While we have the trains reaching the same destination, they can stop along the way. We can define additional stops (stations) where a release (train) sits. So, our hypothetical train schedule could look something like this:

Step 2: Assign Themes to trains

So, we now know that our train departs on April 1st. But what goes on it? Well, that remains a bit of a mystery for a while, as teams refine the PBIs.

As Leadership already did their job, the Themes are well defined. If the highest priority would be (2), the theme of the next train(s) will be something like "Expanding supported Payment Gateways".

We don't know exactly what that means yet, but we can already commit to this being the big theme. That means Marketing, Sales and other supporting functions can go and start planning around that theme. It also means that PMs can focus on this one PBI (Theme) and refine it into multiple Epics first.

Because we know this is the train's Theme, any new work that might come into the backlog warrants the question: Does it support the train's theme? If the answer is not YES, it goes to the end of the backlog.

Step 3: Narrow the scope

It turns out that marketing isn't really happy with this amount of ambiguity - neither is engineering, to be honest. But keep in mind, as this planning is happening, we're likely still in the process of preparing the previous train. Remember, in the previous post I said activities happen in parallel. They also happen continuously!

That means that the leadership team will work on loading the required information for the upcoming train, and hand it off to the PM team, who will hand it off to engineering, etc. In my original diagram, I also noted that different organizational layers will have a different temporal priorities - leadership will look at several releases ahead, and think about big level items. This converges towards the current release the closer you get to engineering... Engineers are, for most of the time, most focused on the ongoing release.

This approach also means that as refinement sessions happen, we start getting a lot more clarity on what a release would contain. As the broad theme is refined, a narrower scope might emerge - i.e., in the form of 5 more specific Epics. They are again prioritized and fed back to marketing. Now they know more about what might make it into the release and they can also continue to refine what that means for them. As things start getting even more refined, we might now have a list of 1129 user stories.

Here's where it gets "tricky" and where I see teams fail most. Marketing (or other teams) shouldn't care if we'll be implementing "Move button A to Screen X". If they do, there's a problem. This is one of the reasons teams pick a PBI that is sufficiently vague.

This distinction also gives us, as the engineering team, the flexibility to adjust our backlog according to feedback. So, Feature A might be really important, but we know that it is composed of multiple User Stories. Our Product Owner can (and should!) make the distinction that some of them are really important, and some aren't. The team then picks the most important one (i.e., the one at the top of the backlog) and starts working on it. We're already delivering on Feature A, but it might not be fully built. IF planning is done right, however, this means we already delivered (some) value of that Feature.

If we now get signals (i.e., stakeholder feedback) that we need to focus on something else, or change things, we have the flexibility to do that, and not jeopardize our release plan. We are still delivering the big theme, but we are only now figuring out the specifics of it, just in time.

No, we're special, and should plan everything ahead of time...

Great, that's the Waterfall method! What does that mean? Some teams defer to wanting to plan in detail, in advance. They create a deadline, and want an estimation on each feature that is going to be built ahead of time. That means that you have to speed up refinement sessions - well, run them ahead of time. That also means, you cannot run them in parallel to development work. You now have a period of Planning, then Development usually followed by Testing and then Releasing.

Nothing is wrong with using it, but you absolutely have to admit to yourself that you are using it. This is where I see teams fail most frequently. It ends up in attrition and bad morale. Engineers believe there's an Agile methodology being implemented, but in reality, it's just ceremonies happening with no actual effect or ability to enact change. Sprints are almost always planned incorrectly and people need to work long hours, weekends... They get fed up, and quit.

If you cannot stand the ambiguity of slowly narrowing the scope of the upcoming release as you build it, don't kid yourself in being ready for agile. There's no shame in using a waterfall process, but make sure you are not running Agile ceremonies just for the sake of it.

Multiple trains per release

The trains are just a metaphor... So, we can adapt it however we see fit. For example, if we have multiple teams, delivering features (like we did in Visual Studio), we can have independent trains running frequently to an intermediate station - kind of like a holding pen. Teams deliver small increments constantly, and those are evaluated constantly by our product teams. Some features then might get picked and put on the actual release train. Think of these small release trains as running to a Transport Hub. Basically, like small trains (or planes) running between smaller cities where people then hop on the bigger one to get to their destination.

If we look at our theoretical example schedule above, we can adapt it to this new way of thinking - the final station for those releases would be the Transport Hub and they could run every day, or even multiple times a day. They would still stop at each station as defined above, enabling surfacing work to (specific preview) customers early on!

This theory requires a ton of overhead work, though. Releases are hard to do, and require a lot of packaging, testing, etc. So you need an enabler. The best one is a good DevOps approach. I'll touch on this in Part III.