Core Committers

I work in Boston, where the average Java programmer makes approximately $100,000 per year. In small companies, coders are often the single highest expense a company will incur.  It is therefore in the best interest of the firm to ensure that this cash ultimately flows toward working software.  As Joel points out, the primary function of a great software company is the conversion of cash into code.

As your team grows in numbers and your system grows in complexity, you are bound to discover that it gets easier and easier to introduce critical bugs.  New team members will repeat mistakes that the original authors made years ago; reintroducing old problems.  Programmers with incomplete knowledge of the code base can make decisions that seem correct on a micro level but do not make sense on a macro level.  And the technical leadership will grow more and more frustrated at seeming to solve the same problems over and over again.

How then, to make sure new team members are not re-introducing old problems into your code base, and thus defeating the purpose of bringing them on in the first place?

Solution 1:  Restrict access to version control to core committers with complete knowledge of the code base.

Let’s assume for the sake of argument that you can identify 2-3 people who can decide, unfailingly, whether or not a given code change will have any adverse affects whatsoever.  Each one will be promoted to a new job title, perhaps technical architect, developer manager, or what-have-you.  All other developers in your organization must submit changes to this core group for review.  The committer then accepts or rejects each submission.  Accepted submissions are checked into the code base, rejected ones are not.  This approach is quite common in open source projects with large communities of contributors, where the volume of submissions is so large that it is impractical to make everyone a committer.

In a smaller development shop, however, this approach has a few drawbacks.  The core committers are also often the strongest developers in the organization.  Rather than writing and re-factoring code, they spend a large part of their day reviewing submissions.  So your strongest coders are not writing code, and conversely, weaker coders write almost all of your new code.  New feature counts decline; bug reports and Daily WTF submissions go up.  More bugs and less features means the cost of each new feature has gone up on the balance.  There’s also a new bottleneck in your organization; features can only be introduced when a committer has the time to review it.

Responsibility for the health of the code base has also changed; at least individuals’ perception of it.  The committers are now completely responsible for the quality of the code base, while the developers are now liberated from it.  Developers that feel no responsibility for the long term affects of their code don’t take the necessary steps for long term code base health like adding comments, re-factoring for legibility, and writing unit tests.  The committers of course can yell and scream and beat the developers with reeds, but again, the overall cost of new code has gone up.

You’ve also just identified the largest internal threat to your company’s survival.  People leave for better offers, start families, get hit by buses; pick your favorite PHB metaphor for losing a key team member.  When it comes time to replace one, you face the onerous task of replacing someone motivated by quality and bug prevention with someone who has spent the past X years writing code with little ownership in the final product.

But I think the largest issue with this solution is the implicit assumption that people with complete knowledge of the code base exist.  Any system of non-trivial size is too large in size in scope for any one person to hold in his or her head all the potential implications of any design decision, even the system’s original authors.  Once more than a few people contribute to a code base, it very quickly becomes impossible to keep track of all the changes that are going on.  The difficulty of accepting that you no longer completely control your own creation is one of the biggest reasons this solution is so popular in mediocre companies.

Solution 2:  Re-factor the system’s design to separate out individual functions and allow a common vocabulary throughout all components.  Divide developers into teams to work on individual functional areas.

Instead of locking down control, you gather the strongest coders together and work to re-factor the system into functional areas.  In a banking system for instance, one team might be responsible for account summaries and transaction reporting, one team might be responsible for transfers and payments, and still another might be responsible for processing loan and account applications.  An interfaces team could design a uniform manner of exchanging results between these systems, and a core model team could own the structure and format of stored data.  Now each strong developer can recruit team for implementing new features.

Now each team is collectively responsible for a functional area together.  In effect, you have divided a large, complex system into smaller, more manageable components.  The team manages quality within their boundaries, and interfaces are designed and tested collectively by the leads.  The key difference now is that developers take more ownership over their decisions; they are the ones who must deal with their own coding decisions.  They can fix mistakes as they discover them, add comments and unit tests, and still roll in new features.  Your money is being converted into code without the control bottleneck of a few elite.

Of course, it’s still possible for a developer to introduce a bug.  I’ll go further than that: every developer will eventually.  So quality must be monitored for he whole system at a level beyond what a human can keep in his or her head.  This is where your unit tests, peer review, and continuous integration system come into play.  Unit tests defeat the O(n^2) problem of regression testing.  Anyone who has worked on a section of code for more than 6 weeks has struggled and learned enough to carry out a worthwhile code read.  Your CI will find more bugs in a week than a core committer will in two months.  But even with all this QC, it is still possible for bugs to leak out.  Deal with it.  You must have an error reporting system and a process in place to roll out critical fixes before releasing any version of your system.  Be honest with your customers about the possibility of problems, and transparent about your resolution processes.

When your programming resources are your largest expense, you want to get the highest output possible from your investment.  You are turning cash into working software.  Re-factoring does more than just improve readability; it can help reduce coupling to the point where individual areas can develop and grow organically.  The smaller in scope each functional area, the higher quality it will become, and as a by-product, the higher quality your entire system will become.  Don’t be afraid to let your fledging system grow into a mature one.  Mature systems are stable, and ultimately more profitable.