Performance Zone is brought to you in partnership with:

Leigh has been in the technology industry for over 15 years, writing marketing and technical documentation for Sun Microsystems, Wells Fargo, and more. She currently works at New Relic as a Marketing Manager. Leigh is a DZone MVB and is not an employee of DZone and has posted 106 posts at DZone. You can read more from them at their website. View Full User Profile

Long-Running Branches Considered Harmful

11.26.2012
| 6701 views |
  • submit to reddit
Originally posted by Jade Rubick

I work on the Core App team at New Relic. We’re responsible for running the product website and data collectors that collect all the juicy data you see on our charts and graphs. And recently we hit a milestone. Last week, we deployed to our site every weekday. This velocity allows us to be a more responsive organization. But it also makes the pain points that much more obvious. It forces us to continually improve our engineering practices and to have a lot of conversations that we wouldn’t have otherwise.

One of these conversations is about how to deploy more than once a day. We know that this is going to require a lot of changes, so our team is arguing the merits and tradeoffs of such changes. Culturally, New Relic is very fond of experimentation. If we’re unsure of a course of action, we’ll try something and see what the result is. So this week, we’re experimenting with pull requests.

Although pull requests seem like they’re always a great idea, they do have some drawbacks. They make your git history less linear and more confusing to navigate. This is a section of our git history before we use pull requests. (See how nice and clean looking it is to have a linear git history!)

New Relic's git history before pull requests

And this is just a small portion of our git history from last week, when we started using pull requests. (Looks like the information superhighway!)

New Relic's git history after pull requests

There are some things we really like about pull requests, though. They give us a nice mechanism for code review and agile QA before code is merged to master.

But another disadvantage of pull requests can be that this workflow promotes long-lived feature branches, unless you take specific steps to combat them. When the Core App team discussed this issue, there was a lot of disagreement as to whether long-lived feature branches were a good idea or a bad one. So this is my public argument against them and why I think they’re almost never a good idea.

Merging Your Code Back in Frequently == Communication with Your Team
People will often say that as long as you are rebasing your branches frequently from master, it’s not a big deal to have long-lived feature branches.

I disagree.

Long-lived feature branches hide your work from the rest of your team. This is fine if no one else touches any aspect of the work you’re doing. (And having too much overlap may be a sign that it’s time to break up your app.) But the minute two people are working on the same section of the code, you’re each blind to how your work affects the other person.

Think of your work as delta off of master. The size of the delta increases as your branch incorporates more and more work. As the size of your team grows, the amount of work hidden from each other increases. And the chances that your assumptions about the state of the code hold true decreases the more you use long-lived feature branches.

If you merge your code back to master frequently, you demonstrate the direction you’re taking the code. For example, you may add a conditional that shows the new code you’re working on and have it disabled by default. If some else is working on the code, they know what you’re doing with it.

You can argue this makes your code more complex. However the fact is, that even if you haven’t merged you code in, the conditional is in fact there. It’s just invisible to the rest of your team.

Merging Your Code Frequently Avoids Integration Pain
When you merge your code more frequently to master, the pain of integration happens at the beginning instead of the end of your work. You can find issues faster and fix them at the earliest possible moment. This gives you time to react and communicate with colleagues, saving you a lot of pain and time.

Merging Your Code Back In Forces Incremental Development
If you’re continually merging your code back to master, you are forced to do your work in increments that preserve existing functionality. You take smaller steps, which generally breaks less and leads to more stable development. And when you do break something, you can find it sooner and fix it faster, instead of waiting days or weeks. You’ll also have more context about how to fix things and acquire information about your assumptions of the code that much earlier. In general, being forced to take smaller steps is a good thing. A large part of our craft is about breaking work down into manageable chunks.

There are times you have to break code to make it work in a new way. Generally, this is less common if you challenge whether breaking it is really necessary. But often it is absolutely necessary and in those cases, you merge things back in at the first time it is practical to do so.

Feature Branches are Inventory
You can think of feature branches as inventory. In the same way that inventory is necessary to build something, feature branches (and sometimes even long term feature branches) can sometimes be necessary. The longer they run, however, the more cost your company incurs without any benefit being brought into the actual product.

To reduce the cost, merge frequently with feature flags. This has a nice side benefit. You can turn on the feature to a percentage of your customers and deliver that value earlier, or collect earlier feedback.

Merging Your Code Frequently Results in Fewer Merge Conflicts
When we started on our Ruby 1.9 upgrade, our first task was to get the tests running on Ruby 1.9. We began by making a branch for the upgrade and then we would rebase frequently.

We quickly ran into a problem. There was so much work on master that it was a huge pain to rebase our work every day. I believe I’m quite decent with git, but I quickly despaired of being able to keep the branch rebased. I was spending as much time with merge conflicts as I was spending on fixing tests. I merged the code to master and the process was much, much smoother after that.

Some will argue that it isn’t always possible to do work outside a long-lived feature branch. For example, what about an upgrade from Rails 2 to Rails 3, a major architectural change? Although it often seems impossible to make large changes incrementally, I think it’s worthwhile to push really hard to see if it is in fact possible. Two of my colleagues are now embarking on a Rails 3 upgrade and they’re frequently merging their work back to master. I’m sure they’ll have more to share about their experience in a future blog post, but it’s hugely simplifying their work. If they were doing it on a feature branch, they’d be spending much of their time fixing merge conflicts with all the work the rest of the engineers in the organization are making against master. And the rest of us would have no idea we were breaking all their work on the Rails 3 upgrade. How much easier will it be when everyone else can see their work?

Long-Lived Branches Lead to Larger Chunks of Work. Is That a Good Thing?
There is one argument for long-lived branches that I initially sympathized with. Long-lived branches allow you to commit one chunk of work that represents all the work, in one increment. And it’s immediately understandable what the work is because it’s in one commit.

Admittedly, that’s very appealing. It can seem messy to have to deal with a pile of incremental commits. Merging once a day per developer is a lot more commits to wade through. It also means you only need to review and integrate one chunk of work, which sounds great in practice. But it does mean that the size of that chunk is much larger.

Increasing the size of changes increases risk. It’s essentially the same as deploying your code less frequently. The amount of change is larger and the risk is greater. Instead of pipelining your process, you either hold it up and commit a huge block of code or you do it in isolated segments, where people are blind to your changes. The best practice, I believe, is to find ways to pipeline your process so every aspect of your engineering process is pipelined.

Conclusion
So, that’s my argument against long-lived feature branches. What do you think? Did I miss an important point? Let me know if the comments below. I’d love to hear your thoughts and experiences.

Jade works on the Core Team at New Relic. He’s best known for his tweet: “If someone told you you had to kill someone to get a job at New Relic, you should seriously consider it. It is that good here @newrelic”. If you’d like to join our team, no murders are necessary — contact us through newrelic.com/jobs


Published at DZone with permission of Leigh Shevchik, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Brad Appleton replied on Wed, 2013/02/13 - 2:58pm

It's not the length or longevity of the branch that is the problem. It's whether or not all development work is integrated frequently across ALL implemented functionality.

Feature branches are often used to collect together work for a particular feature/team but it's name and use does not automatically imply a lack of cross-feature integration. Both "isolated" feature-branches as well as frequently-integrated feature branches are commonly used and it's not safe to assume that "feature branch" (or even any long-lived branch) somehow automatically implies a lack of full application/system-wide integration.

The problem is "unsynchronized work" (as noted in the Poppendieck's 3rd book "Implementing Lean Software Development") and it can happen not only with feature branches that stay "isolated" (rather than integrated), but even within a developer's "sandbox" where they are working on a particular change (with or without a corresponding developer branch). [NOTE: The Kanban folks might describe it as a function of the "cost of delay" in getting fast+frequent feedback across all the implemented functionality.]

It's time to stop blaming tools and techniques like branches/branching and start blaming our own misuse/misapplication and misunderstanding of them. Then we can start looking at the real culprit, which in this case, is lack of "full" frequent/continuous integration.

Continuous Integration (and frequent integration) encompasses the entire product/project, not just one team or codeline. And if any team or codeline is not being frequently integrated together with all the other work, then you are still doing big-bang integration across features and/or teams.

As far as single codelines (including both short-lived and long-lived branches), Kent Beck's 4 rules of simple code (http://c2.com/cgi/wiki?XpSimplicityRules) apply equally well to codelines with only a little bit of effort. The 4 rules of simple code are:

Simple code should ...

  1. Correctly run (and pass) all the tests
  2. Contain no duplication (The DRY Principle)
  3. Clearly express all the ideas/intentions we needed to express (reveals all intent and intends all it reveals)
  4. Minimize the number of classes and methods (has no superfluous parts)

With just a little bit of effort to translate the context from "code" to "codelines" we get the 4 rules of simple codelines:

Simple codelines should ...

  1. Correctly build, run (and pass) all the tests [don't break the build/codeline]
  2. Contain no duplicate work/work-products
  3. Transparently contain all the changes we needed to make (and none of the ones we didn't)
  4. Minimize the number and length of sub-branches and unsynchronized work/changes

(For a more detailed explanation of these rules see http://blog.bradapp.net/2008/06/four-rules-for-simple-codeline.html)

The last rule above covers the real lesson here and that is to minimize the amount and duration of unsynchronized (un-integrated) work/changes. Integration delayed or deferred is still integration denied and their will be plenty of technical integration "debt" to be paid later (with compounded interest)!

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.