Programmer Musings

BPGB: (Dis-)Integration Branches

This is another post in my intermittent series of Best Practices Gone Bad (BPGB)

Today, we are going to take another side-step into version control. Most development groups use version control of some form. Whether you prefer Subversion, Git, Mercurial, Bazaar, Clear Case, or any of the many others, version control is an important technique for keeping your changes under control. This is especially true if you are maintaining multiple releases concurrently or have more than a couple developers on your team.

Back in BPGB: Feature Branch Fail, we covered what happens when branches live for too long. When you need to merge multiple long-running branches, you increase the probability of conflicts.

Integration Branch

In order to prevent these kinds of conflicts from messing up the main branch, many people discover the idea of an integration branch. You branch from the main line, merge multiple feature or bug fix branches into this integration branch, and fix any conflicts there. When the integration branch is clean and survives the tests, you merge the branch back to main.

This approach seems pretty reasonable and usually solves the first set of problems that people have with branch conflicts. Although the conflicts still exist, the integration branch gives us the time to resolve conflicts without leaving the main branch broken. If the conflicts take time to merge, we don't have main in a broken state. If the conflicts are too overwhelming, we have an easy way to back out. Maybe merging branches in a different order will make the conflict resolution easier. In any case, we have a few more options. Life is good.

Then, someone has the idea that recreating the integration branch each time we want to do this is a waste. The obvious approach is to leave the integration branch around and just keep it in sync with main. Although this seems reasonable, we have just turned the integration branch into the equivalent of a long-lived feature branch. as we found in the previously mentioned post, this tends to result in worse conflicts and pain.

If the integration branch is not kept in sync with main, there is a real possibility of problems when integration is merged to main. I've also seen situations where someone decides that the integration branch is obviously more up-to-date and overwrites (force push) the main branch, potentially posing changes that had already been merged. This becomes the same kind of issue that we were trying to solve with the integration branch in the first place.

Key to making the integration branch strategy work is that this branch starts out identical to your main branch before you begin merging. If there is any difference at all, you court the possibility of doing a bunch of work to get the integration branch functional, only to have the same problems again when you merge to the main line.

Summary

Most version control tools provide methods for maintaining and merging multiple lines of development. Despite the fact that the tools have become increasingly good at recognizing and resolving simple conflicts, human intervention may still be required. Care is needed to make sure that you reduce the effort needed to make changes rather than just move the effort.

One real anti-pattern for version control is long-lived branches. There are a few cases where it makes sense. But, they are a lot rarer than people believe. Don't ever make a long-term integration branch solely to save the time of setting up and tearing down this branch as needed. The pain will quickly outweigh the minor benefit.

Review of Release It!

Release It!
Michael T. Nygard
Pragmatic Bookshelf, 2007

I've had this book on my shelf for a few years, and finally got some time to start reading it. I should not have waited.

Nygard takes the position that the life of a piece of software actually only begins when it is released. He spends a lot of time on the way things go wrong in production that you won't see in a development environment. Anyone who has ever survived a push to release and been surprised that there's no time to relax will really appreciate this book.

The book contains a large number of war stories showing things going wrong in real projects. Some of the failures are obvious, others are surprising. Nygard distills these failures down to some anti-patterns that can cause problems with stability or capacity. Then, he provides design patterns that can mitigate or eliminate some of the problems in release.

Some of these design patterns seem obvious: Use Timeouts or Pool Connections. Others are less familiar: Circuit Breaker or Bulkheads. Like the patterns from the Gang of Four book, a large part of the benefit of the patterns is having common names and descriptions of the patterns. This applies whether you have been using them for years or have seen them for the first time. If you are seeing them for the first time, his descriptions are good enough that you should quickly understand the pattern.

The section on general design principles is very effective, but the part that most developers really need to read is the section on Operations. Too often, those of us developing software forget what the operations people need from the software. When we think of them at all, we try to provide a nice interface for a few admin tasks. Nygard points out that a pretty interface is nowhere near as useful as a scripting or command line interface. He also describes examples of how this kind of approach actually helps operations.

Overall, this is a really great book for anyone that is releasing software that must run in a production environment. If you fall in this camp, you should get the book and read it before your next project.

LCDC: Rising Tide

I began this series on Least Common Denominator Code (LCDC) with The Myth of Code Anyone Can Read. In the posts that followed I attempted to show how this kind of code is not actually possible in a real development shop. I've shown how different code should be written for different audiences, and how real business value drives code that cannot be written for the least common denominator.

In this post, I propose to show why, even if it were possible, LCDC would be a bad idea.

Expertise

Different developers have different kinds and levels of expertise. Unless you only have one developer on staff, you have probably seen that some developers are more productive, or faster, or safer than others. You may have one developer that always works on the critical code because she is the most careful or security conscious. A different developer may be tasked with time-critical issues, because he can always get a 90% solution in place in short order, even if other developers are needed to finish the work.

These developers have different skills and expertise, as well as temperament, that determines how the company can best make use of them.

However, you don't want the developers on your team sitting still. The rest of the industry is continually learning new tools and techniques. They are exploring other ways of solving problems. If your competitors are improving and your team is not, your company will fall behind. This could cause the project (or company) to fail.

On a related note, the kind of people who become software developers usually like to learn. They are probably spending some time looking into new technologies and related fields. The good ones will be exploring on their own to improve their skills. If one of your good developers finds an interesting tool or technique, they are going to want to use it.

On the other hand, there are developers who just code as a job. Just like any field, there is a spectrum of skill and motivation. These are the people that may just be doing the same thing year after year. As a manager or mine used to say:

There a big difference between 5 years of experience and 1 year of experience 5 times.

If you want to keep up or surpass your competition, you need more of the former than the latter.

Developer Motivation

Many people have written about what motivates software developers.

In most of these discussions, you will find some variant of interesting problems and opportunity to learn or improve. The best developers seek out positions with with these features. They also like to be recognized by other developers for their expertise. LCDC fights against all three of these motivational factors. If a really good developer has no or few interesting challenges, is not allowed to learn more effective ways to solve problems, and cannot show off new techniques to the other developers, their main motivation is to find a new job.

I have heard managers argue in the past against training for their developers, because they might learn enough to want to leave. One trainer I met answered with What if your developers don't learn anything, and then stay? As I pointed out above, developers outside your company are learning, if your team is not, they are falling behind.

The Rising Tide

The open source community is fond of the phrase:

A rising tide raises all boats.

The idea is that if we share our expertise the whole industry gets smarter and more capable. You can take advantage of this inside your company or team as well.

Instead of aiming to keep your code written to the level of your most junior developer, aim for just above the middle level of expertise on your team. Anyone below that level should be encouraged to learn. The developers with more expertise will be looking for ways to improve the overall team ability, so that they can show off their skills.

The downside of this approach is that new people will have a lot to learn. But, the really good developers like to learn. That's a large portion of why they are in this field. The upside is that the developers who want to learn and improve will stay with the team. The ones who refuse to learn will fall behind (and might leave). The overall result is a more capable, more experienced team.

Summary

The point is that LCDC is quite likely impossible. Every bit of experience in your code, fights against the idea of LCDC. The more you look, the more likely it is to be bad for your project. Not only is this a bad idea from a technical point of view, but it also tends to chase off your best developers.

This whole series of posts is aimed to convince you that LCDC is a really bad idea.

LCDC: Different Audiences Have Different Needs

In the last few posts, beginning with The Myth of Code Anyone Can Read, I've focused on what you can expect from your programmers in general. Of course, generalizing is what got us into this discussion in the first place, so let's spend a little time not generalizing.

When teaching new programmers, I always tell them that the code had at least two audiences: the computer and the next programmer. This is not actually true, depending on the purpose of the code, it may have many audiences. For instance,

1. Code used as an example to show how to use an API, may be used by almost any level of developer. They may not have any experience with your system and may be relatively junior.

2. Code used as examples for writing plugins for your system can expect that the reader has some level of familiarity with the way your system works, if only at a superficial level. Since plugins normally have a tighter integration with your system than an API call, the developer can be expected to learn a little more before doing development.

3. Most of the code in any given system is non-critical. If non-critical code breaks, it is annoying and possibly embarrassing, but it won't result in loss of data or compromise of a customer. Since it is inside the system, any developer working on it will have at least the level of expertise that the manager hires for. If every new hire is expected to have three years development experience, the code can assume that level of experience.

4. Core business code is the part that makes the money. The programmers that are working on that code should be at least the level of experience you hire for, and probably more. They should be familiar with the business and terminology, so that they can be expected to understand the jargon.

5. Core library code is the foundation upon which all of your system sits. Some of this will be libraries that you have gotten from a third party and just use. Some will be the low-level routines written by your senior developers and used by the rest of the team. This code requires deep knowledge of the domain of that library. The only people working on it will be experts in that particular domain. Only the most senior people should be making changes in this code.

Levels of Developer

Let's start with a fundamental rule:

Expertise is not evenly distributed.

This rule is why programmers (and people in general) are not interchangeable. You cannot expect to replace the senior programmer, who has 5 years of experience in your code base, with an entry-level programmer straight out of a computer science degree and expect the same level of productivity.

I have had some people suggest that this is an elitist approach to development. As a really new programmer, I would have been insulted by someone suggesting that I wasn't qualified to work on a particular piece of code. In actuality, this is really more of a pragmatic view of development than an elitist one.

The truth is that developers in the real world have different levels of experience, along many different dimensions. Some are experts in the language, but know nothing about your business. Some know about your business and have never written more than a 10 line program. Most are somewhere in-between.

API User

Let's take the first category, people who use our APIs are completely outside our control. We have no knowledge of who they are or what they have worked on. For this reason, we pretty much have to assume LCDC as a requirement. This is as close to random person walking in off the street as you should get. Anyone who has ever supported an API has stories of people misusing the API for many reasons including:

• Using the API for the wrong business.
• Trying to perform an action that is not valid for the industry
• Using the API to violate the laws of physics
• Trying to use the API from an unsupported language
• Trying to use the API with unsupported hardware

My favorite example of this was a system I supported that displayed charts of stock price information. We had a customer trying to use our interface that reported a bug. No matter what he did, he could not get the chart to show data later than today. It took a bit of effort to explain the impossibility of displaying tomorrow's stock price.<shrug/>

You can assume nothing about a user of your external API.

Plugin Developer

Even if the plugin developer may start near the same level as someone who uses your API, they will inevitably become more knowledgeable before they get very far into the development of a plugin. By their nature, most plugins require some understanding of the system. Most of the time, someone has used the program itself and possibly any API for a while before deciding that a plugin is the way to go.

Since developing a plugin requires more expertise and knowledge, it serves as kind of a filter on the kind of developer will attempt it. By structuring your plugin interface for more experienced developers, you can further filter for developers with the kinds of skills you are willing to support.

Business Code

The next two categories of code will mostly only be seen by your developers. Most of this code is not business critical. A large amount of any code base is normal input code, validation, data-massaging, and output. Support for templates, language, user interface, etc. are important, but may not harm your customers in the event of a mistake.

Most business, however, have some code that is core to the business. Some examples include:

• makes the money (charging credit cards, making trades, finding oil, etc.)
• handles the consistency of the user's data
• handles legal requirements
• prevents unauthorized access to private user data (credit card number, etc.)

All of that kind of critical code is not immediately turned over to junior developers, because mistakes in those areas could mean failure of the project (or lawsuits or loss of income). The more critical the code is to your business the more likely you are to have it done or, at least, overseen by more senior people. This is simply because your most experienced people are more likely to be aware of the issues that are important in this code.

That Code

Many systems have a piece of code that is core to their business, but is so involved, low-level, or just plain weird that only a few are trusted to touch it. This code normally lives in libraries that any of your developers can use, but few understand. Sometimes it's the result of someone determined to have job security because they are the only one who understands the code. Most of the time the reason is more benign. This code could be:

• the result of someone's PhD thesis and only a handful of people on the planet understand it
• the system that ensures that data is maintained according to the laws governing your industry
• the trade secret that allows your code to do something your competitors can't
• the interface code to a piece of hardware required by your system
• the registration code that makes certain your company is paid for every copy in use

In most cases, this code is important and a small number of developers have become the experts with that code over time. (Hopefully, that small number is not 1 or less.)

This kind of code often has special jargon all its own that no one new to the system is going to understand. There is really no need to make this code generally understandable. You do not want a junior developer making changes here, since there is a large probability that they are going to break it.

Different Styles for Different Audiences

Different portions of the code assume different levels of expertise from their audience. Trying to write all of the code to the least common denominator will make parts of the code unfit for the designed purpose. An approach that cannot assume some knowledge of the part of the developer will be harder to maintain for people who do have that knowledge.

Realizing the audience for a particular piece of code will allow your team to choose an appropriate level at which to write the code, instead of mandating a bad idea. One other side effect is that someone wandering into code that uses more advanced idioms, that they don't understand, has an indication that they should probably know more before making changes here. It's kind of a you must be this tall to ride the ride sort of marker.

If the code is written as if any idiot can change it, likely one will. This approach only works if your developers are comfortable with the idea that code may be written with different idioms and that the idiom is an indication of expertise needed. It also works better if anyone is allowed to read the code, but only allowed to change it once they have convinced the current maintainers that they have the appropriate understanding.

Summary

Different kinds of code require different levels of expertise. Writing everything as LCDC, hides useful markers of different kinds of code. This could tempt less experienced developers into breaking critical code because they touched something they didn't understand.

In the final post of this series, I'll show that this is not a static situation.