Developing Incrementally

Incrementality is a style of development that affects everything in a software company, from “how to structure PRs” at the bottom to “how to release and market products” at the top.
I have a collection of practices that I’ve learned, where general theme is to make software development more incremental. I use them unless I have a good reason not to. I’ve seen more failures from insufficient incrementality than from superfluous incrementality, but I’ve seen a non-zero number of failures of each type.
Starting from the top: it’s good to build MVPs. Every product is, at release time, an experiment testing the hypothesis “people will like this”.
Engineers are often anxious about releasing MVPs because they have visions of being overwhelmed by operational problems. I’ve learned that, typically, no one uses a piece of software on release, and you usually have several weeks to fix things before your software gets any users at all (even with good marketing), and perhaps months before you get more than a handful of users.
At a lower level of abstraction than the software product itself: I usually try to include the ability to release experimental features.
- I usually implement this with a single “experimental mode” feature flag, client library, or beta release series, containing all experimental features to limit combinatorial complexity.
  - I know some projects, e.g. ember.js and Google Chrome include a set of feature flags, one per experimental feature. If you’re confident you can manage the combinatorial complexity, this is better for users because they can use as little experimental code as they need
- This way, you can release features as “experimental” as you develop them, get feedback from 1-2 interested users, iterate, and then release those features as “non-experimental” in the next major release.
- You can greatly reduce value risk and product risk with this approach, and also provide more value directly as experimental users aren’t stuck waiting for “the next big release that fixes everything”.
Finally, at a lower level of abstraction than “features”: I strongly endorse writing code incrementally:
- Write a design doc before writing any code.
  - Even if you don’t show it to anybody (initially) design docs are much shorter than code, but detailed enough to reveal a lot of design problems. Iterating on the design is much, much faster when writing English.
  - They’re also a useful piece of documentation (make sure to include why the project is needed)
  - They can obviate annoying status meetings; just record your implementation progress in the design doc as you go and send it to partners/managers who want to see progress.
  - On teams with limited product vision, a common problem is that there are too many ideas. Design docs serve as a crude triage mechanism by imposing a “proof of work” burden on new ideas. If someone wants to take the product in yet another new direction, you can delay (or sometimes eliminate) debate by asking them to write a design doc first. This is pretty dysfunctional, but it’s better than actually changing direction every day.
- Write any new persistent data structures or schemas next. Whenever writing new code you should always write the data structures first¹²
  - When it’s time to write code, I’m a huge, huge fan of breaking up patches as much as possible. Reviewers are sometimes annoyed by the flood of patches, but in my experience, code gets merged much more quickly and safely this way, because each patch is between easy and trivial to review, so they get reviewed immediately.
  - To do this, I often implement each change twice: once as a monolithic patch that contains a whole prototype of the feature (which I eventually discard), and then again as a series of small patches. I use the monolithic change as a guide for what’s left to merge (by continually rebasing on ’trunk’ and refactoring as I merge patches), and try to factor out:
    - any non-functional changes (e.g. updating comments, renaming variables), merged as separate patches. In general, since I’m trying to make reviews fast, I also try to keep diffs small, and factoring out non-functional changes is critical to that goal.
      - For example, if I move a function, rename it, and change the implementation, I’ll make that three separate patches:
        
        Moving a function is trivial to review (the diff is the size of the function but the lines are the same)
        
        renaming a function is trivial to review (the diff is 1+number of callers, but every diff line is a simple replace)
        
        changing the implementation is nontrivial to review, but because the function has already been moved and renamed, the diff is no larger than the function body and shows exactly what’s different.
    - any new classes or internal data structures, with no implementation. This attracts a lot of design feedback that is much easier to apply before the implementation is written
    - any new methods/APIs (again, with an implementation of “error: not implemented”). The implementation and tests are added in a second, now-smaller followup patch.
    - each API call’s implementation, and tests for just that API

Other writing (specifically about breaking up patches) that I’ve done on this, which I’d like to incorporate:

Small patches are easier to review and will go through review faster, especially if some patches are refactoring*only. If you need to refactor your code in order to make a change, it will be much faster to refactor first, get that change reviewed, and then make a much smaller patch with logical changes and tests.
Bugs are more likely to be caught during the review.
There’s no epic merge that has to be made once the change is done, which is a major source of bugs when not doing things incrementally.
The main branch will not change while the project is in progress or (much more frustratingly) in review.
You’ll have a much better sense of whether the project is behind schedule and

In fact, an important part of working incrementally is separating refactoring (which is not user-visible, does not require tests, and is comparatively fast to review) from logic changes. Typically, you should refactor first (which, again, will be fast to review) such that your subsequent logic-change patch is as small as possible, and then make the logic change, which will be fast to review due to its smallness. when it’s likely to be done.

How to Slice Up Work

If you’re making some big, cross-cutting change, notice when some part of that change could be done as a standalone refactoring patch. Make that change and get it reviewed separately. By the end, your big cross-cutting change may be fairly small.
If adding a new API or codepath, don’t add the whole thing all at once. First make changes to the data layer (e.g. the schema, the protobuf, the jsonspec, etc.), then add the API implementation, then add all the call sites. Reviewing each of these separately minimizes the cost of any changes proposed. Specifically, if you make all changes in on PR, and then a reviewer recommends a different schema, much of your code will have to be rewritten. If you get the schema merged first, you can confidently add an API implementation knowing that it’s approximately correct, and then add callers knowing the API won’t change.
The three-part change: when changing some method and all of its callers, rather than changing everything in one mega-patch, you should copy the method to a new implementation containing the desired changes, then gradually change all callers to use the new method over several patches, then delete the old method. This mostly allows you to minimize the amount of time your change spends in progress and limit the number of new callers of the old method that other engineers add while you’re working.

If you’ve worked for a big SaaS company (Google, Facebook), you know this type of change is common there, to work around mismatched deployment schedules (add new code to server, deploy updated server. Change client to call new code, deploy updated client. Delete old code from server, deploy updated server. Done.), but if you haven’t, it’s good to know about.