Extreme Remote: Code Ownership - The advantages of a TDD mindset

We have proposed a 'relay race' style of development where in-house and remote team members hand off active development on code modules as their work days start and end with the rotation of the earth.  This manner of work necessitates very granular requirements so that individual requirements can be completed within a day.  

Is this, however, just a nice theoretical way of working?  Isn't it common to get into a task and go down a rabbit hole when you discover a deep issue or a refactoring opportunity?  What you thought you could finish in the last two hours of the day turns into something much bigger.  So, then you end up not committing your changes and your counterpart on the other side of the globe doesn't have an accurate picture of your progress before they begin to work.  Duplicate and conflicting work may result.  

Yep.  That's going to happen fairly often.  But, having a TDD mindset can help with this scenario.  

The mantra "red-green-refactor" tells us to do the simplest thing possible to go from a failing test to a passing test, then think about how to make the code more beautiful and robust via refactoring.  With this approach, we can make the simple change that causes the test to pass and immediately commit the change.  Then, if we get deeply involved in some refactoring or deep investigation, we have established a clear indication of our progress.  Even our refactoring can be done incrementally.  At the end of the day, we can commit what we have accomplished so far and leave notes for what is left to do tomorrow. 

When our counterpart starts his/her work day, they will see which requirements have been completed and can move on to the next one.  Good notes in the code may also help indicate what issues you are planning to address the next day to prevent conflict.

This won't solve every issue that causes you not to finish a task within a day, but it is a very helpful manner of working in this sort of project.

Extreme Remote: Code ownership - Daily reviews

The current series of posts have been recommending short-lived feature branches to help in-house and remote developers to work on the same code modules.  Feature toggles and automated tests provide important protections while working in this manner.

Code reviews provide another key protection when code is being merged to the master branch regularly.  (While the relative merits of automated tests versus code reviews have been debated vigorously, the simple truth is that the best results are obtained when both are implemented effectively.)  

At Amplified Development, we recommend that all team members review all code merged to master in the previous 24 hours.  When these reviews are performed at the beginning of the work day, it will usually take about a half hour each day (for a small team of 4-6 developers).

Feature toggles, automated tests, and code reviews work together very effectively.  Feature toggles ensure new code never executes in a given environment until you want it to.  Automated tests ensure there is no regression in system functionality.  And, code reviews counteract the build up of technical debt.

All of this allows teams to work in a tightly integrated manner, achieving genuine code ownership at all times. 

Extreme Remote: Code Ownership - Automated tests

When you work with short-lived feature branches, feature toggles help you sleep at night by quarantining new code for partially implemented features.  Yet, there may still be a worry that somehow there was some lapse in feature toggle implementation and side-effects have crept in.

Automated tests add one more layer of security.  Automated tests cover both unit tests (associated with TDD) and integration/acceptance tests (associated with BDD).  When you are confident that your code has good unit test coverage and integration test coverage, you have sound reason to believe that feature toggles are correctly quarantining new code.  

Automated tests should be run by developers before merging code to the master branch.  In addition, a solid continuous integration process will run the full suite of automated tests and reject the merge if any tests do not pass.  (Note: You should run automated tests with various configurations of feature toggles to simulate the settings for each environment - test, staging, and production.)

Extreme Remote: Code Ownership - Feature toggles

In the previous post, I recommended short-lived feature branches to help keep in-house and remote developers working effectively on the same code modules.  This ensures that your in-house developers maintain genuine code ownership at all times.

But, short-lived feature branches cause incomplete features to be built and deployed to testing, staging, and possibly even production environments.  How can that be a good thing?  It isn't, if the code for the incomplete features is active in those environments.  What we need is a way to quarantine the incomplete feature code from the active code.

Enter feature toggles.

The feature toggle concept is very simple: Have a boolean value that indicates whether a feature should be enabled in the current environment.  Then, surround new code with a simple IF statement checking the feature toggle. In the simplest implementation, this boolean value can be a configuration setting.  There are also more sophisticated implementations where you can change the setting at run-time.

The developer will work with the feature toggle on, so that he/she can see the effects of their work.  The other environments (e.g. test, staging, and production) keep the feature toggle off until the feature is ready.  Once the feature is considered complete by the developers, then the feature toggle can be turned on in the test and/or staging environments, allowing for a testing period.  When it checks out thoroughly, the feature toggle can be turned on in production.

While the concept is simple, using feature toggles requires very thoughtful implementation.  I hope to describe some key considerations in a later post.  But, the key point here is that feature toggles help quarantine incomplete features while incremental implementation is merged into the master branch of the repository.

Extreme Remote: Code Ownership - Short-lived feature branches

We have been discussing the advantages of a 'relay race' style of development.  This approach has in-house and remote developers completing granular requirements and tasks each day, and their counterpart picking up progress as their work day begins.

Success with this approach requires articulating a good strategy for branching in the code repository.  There are two main strategies:

Long-lived feature branches - A feature branch is created when work commences on a significant feature.  Team members working on that feature contribute their changes to the feature branch only.  When the feature is complete, it goes through a process of testing and, when approved, the feature branch is merged to the master branch.  Sometimes, the merge process is challenging since conflicting changes may have been introduced over time from multiple feature branches.  To reduce this risk, many teams will merge changes from master regularly so that any merge issues are resolved in advance.

Short-lived feature branches - With this strategy, a feature branch is created when work on some small segment of work is initiated.  When each chunk of work is completed, it is merged to master - this may happen multiple times a day.  Merging from master regularly is still a good way to avoid merge surprises.   

(Some may observe that there is a third way: commit directly to master.  We view that short-lived feature branches serve essentially the same purpose.  Having a feature branch, however, gives a nice way to collect a coherent set of changes to be reviewed.  More on that later...)

At Amplified Development, we strongly recommend short-lived feature branches.  Applying changes to the master branch is an essential part of a continuous integration strategy.  And, it promotes the sort of team integration that Extreme Remote teams need.  

Applying changes to the master branch before a feature is complete, however, raises some challenges.  You don't want incomplete, unreviewed features appearing in a staging/test environment, much less production.  There are two key strategies to address the challenge:  feature toggles and daily code reviews.  Let's discuss those next...

Extreme Remote: Code Ownership - Granular requirements and tasks

In the last post, we described a kind of 'relay race' development where in-house and remote developers work on the same code modules, but hand off active development as work days start and end with the turning of the globe. In order to achieve this type of development, requirements and tasks must be quite granular - granular enough to be completed within one work day.  

Gherkin-style requirements are ideal for this type of development since each scenario is typically granular enough to be completed in less than a day. Each scenario represents a very specific requirement.  A developer can usually satisfy several requirements each day. In order to 'pass the baton', his/her counterpart on the other side of the globe can simply run the automated tests to see precisely where progress was left off. After a review of the new code, he/she is ready to pick up active development of the module for the day.  

Not all development work maps nicely to Gherkin requirements.  (But, more than you many think!)  Even when you cannot describe the work with Gherkin, the tasks should be skillfully defined so that they are granular enough to be completed within one day.

With granular requirements and tasks, you can run the sort of coding 'relay race' that keeps both in-house and remote team members intimately familiar with the code at all times.

Extreme Remote: Code Ownership - Everybody in the same code pool

When you have a team comprised of in-house and remote members, it is very tempting to organize the work so that each group works on separate sections of the project.  This helps each group work together effectively, have easy communication, etc.  Yet, there is a painful reality to be faced at the end of the project:  there are large portions of the project with which the in-house team is insufficiently familiar.  That tends to create a dependency on the remote team members beyond the planned end of the project. The goal should be to continue the relationship with remote team members because you were delighted with their work rather than feeling trapped by circumstances.

To avoid this, the project process should ensure that both in-house and remote team members are familiar with all parts of the system at all times.  The best way to ensure that is to have both in-house and remote developers working on the same code modules.  When a team is co-located, this can cause frequent conflicts; with an Extreme Remote team, however, the non-overlapping work hours allows for a 'relay race' style of development.  The developers 'pass the baton' at the beginning/end of each work period.

Here's how it can work:

(WH = Western Hemisphere; EH = Eastern Hemisphere)

• WH Day 1 - In-house developer works all day implementing requirements, ensuring that work ends at a well-defined stopping point and changes are committed to the code repository.
• EH Day 1 - Remote developer gets latest code, runs automated tests to ensure code is in a good state, and reviews changes from the in-house developer.  The remote developer identifies where the in-house developer left off in the list of requirements and works all day implementing additional requirements.
• WH Day 2 - In-house developer gets latest code, runs automated tests to ensure code is in a good state, and reviews changes from the remote developer.  The in-house developer identifies where the remote developer left off in the list of requirements and works all day implementing additional requirements.
• EH Day 2 - (You get the picture....)

With this approach, the in-house developer is intimately familiar with the code at all times.  If the remote developer must leave the team for any reason, the rate of progress will suffer, of course; but, it will only reflect the drop in man-power, without an additional learning curve.

To accomplish this style of work, there are specific techniques and processes that can help.  I will discuss these next.

Extreme Remote: Code Ownership

With traditional off-shore development, it’s hard not to end up with a code ownership problem.  When the planned project ends, only the off-shore team is truly familiar with the code base; even with best intentions, in-house developers are rarely ready to provide production support.  Your options?  (1) Continue to retain the off-shore team.  (2) Endure sub-optimal support while the in-house developers gain adequate expertise with the code base.  (And, it is possible to get an unpleasant surprise regarding code quality.)

A far better outcome would be for in-house developers to be deeply familiar with the code base when the planned project is complete and fully capable of production support.  A decision to continue the relationship with the remote team members would be based on a delightful experience rather than feeling trapped.  Of course, you would have the option to scale back to just in-house developers, as originally planned, with negligible transition pain.

To accomplish this, we need to break down the barriers between the remote and in-house team members.  In the next few posts, I will describe some models and techniques to promote deep familiarity with the complete code base for both local and remote developers.  

Extreme Remote: Product Management - Online conversations

Since the product owner is not within hearing distance of the remote team members (or not even awake when they are discussing issues), you absolutely must create a vibrant and effective online conversation.  Many tools such as HipChat and Slack support group chats.  Make this the primary way that issues get discussed.  And, have questions for the product owner clearly labeled so that they can focus on the product issues rather than all of the other technical discussions.

Extreme Remote: Product Management - Iteration Meetings

During the iteration cycle, there are some key meetings you can strongly encourage product owners to attend. 

Feature overview (aka ‘Story time’) – In advance of an iteration, you may wish to have a business-oriented overview of a system feature.  This is a great opportunity to have the product owner describe how an associated facet of the business works.  Many misunderstandings get pre-empted when developers can picture the business context.

Iteration planning – Early in iteration planning, you can review the requirements and any mockups.  The product owner may identify a misunderstanding that can be corrected before development begins.

Iteration review – The product owner should certainly be part of the iteration review, if at all possible.  Here is where any misunderstandings become plain to see.  If the product owner identifies course corrections at the end of every iteration, they can be implemented quickly and efficiently in the next iteration, keeping the project from straying off-course.

Note:  Product owner involvement in iteration meetings becomes even more meaningful when iterations are kept short.  Ideally, one week is best since it allows course correction very often.

For these iteration meetings, you may be able to request some extra flexibility from the remote team members.  Could they come in very early or be available late so that the product owner can meet at a more convenient time?  Depending of the product owner and the company culture, that may be important.

Extreme Remote: Product Management - Daily Meetings

Likely, to keep the team connected, there will be some form of daily meeting.  But, it is likely to be at odd hours that may not be convenient for product owners.  Don’t hesitate to invite them, though.  You may be surprised at what is possible.

On one project, we managed to have two executive stakeholders on a 9:30pm nightly online meeting with the remote team members.  Nice participation if you can get it!  Realistically, I don’t ever enter into a project expecting that level of product owner involvement.  But, take as much interaction as you can get!

Extreme Remote: Product Management - How can they know what I want from the other side of the earth?

Agile development promotes having the product owner deeply integrated into the development process.   That may mean having the product owner easily accessible for questions or to review incremental progress.  Ideally, it means having the product owner actually sitting within a few footsteps of the development team.

With an Extreme Remote team, a product owner is going to be several million steps away from some members of the development team.  Further, the working hours of the product manager and the remote team members do not naturally overlap.  So, how do you achieve the goal of having the product owner deeply integrated into the development process?

You will need to seek the optimal arrangement by being creative and using some key techniques. 

Having excellent requirements is essential.  No, we are not promoting a major requirements gathering phase that results in a massive and meticulously detailed requirements document.  The requirements should be developed iteratively, in step with the development process.  Yet, developing the ability to write clear, meaningful requirements will help offset the gap between the product owner and the most remote members of your development team.  At Amplified Development, we are huge enthusiasts for Gherkin style requirements (aka Cucumber, SpecFlow, Behave, Behat, etc.).  Written properly, the Gherkin requirements are lucid and verifiable by the product owner, while also mapping unambiguously to automated tests.  Such requirements go a long way towards ensuring that the right features are being developed.

There are, however, always issues that need to be discussed.  And, even with excellent requirements, the developers’ understanding of the feature will often differ from the product owner.  So, you need to create as much interaction between the product owner and the remote team members as possible.  The following posts will consider some possible ways to do this:

• Daily meetings

• Iteration meetings

• Online project conversation

Extreme Remote = Off-shore?

In the previous post, I defined Extreme Remote as "having team members so far away that regular overlapping work periods are very difficult to maintain".  And, I referred to countries such as Pakistan and India as sources of technical talent.  So, isn't that just what we typically refer to as "off-shore development"? 

I consciously avoid using the term 'off-shore' because, to most people, it implies a way of working and a set of problems that is exactly what we are trying to avoid.  'Off-shore' typically implies a waterfall process and an 'over-the-wall' mentality.  With Extreme Remote, we want to apply genuinely agile principles and creatively solve/mitigate the challenges posed by distance, time-zones, and culture.

Here are some key differences between off-shore development and Extreme Remote teams:

Off-shore	Extreme Remote
Big Up Front Design	Iterative design
Requirement misunderstandings revealed at time of delivery	Requirements expressed as automated tests; continuous integration allows early evaluation and course correction
Lack of code ownership	In-house team members intimately familiar with code at all times
Code quality issues revealed at time of delivery	Daily code reviews and regular group code reviews preserve code quality continuously
Separate teams	Integrated team

The Case for Extreme Remote

Let’s start by accepting a well-established fact: 

The ideal environment for software development is a team physically co-located in the same room in close proximity to key stakeholders.  Nothing can replace the ambient knowledge, interpersonal dynamics, and streamlined access to stakeholders. 

So, why consider anything else?

The book Remote: Office Not Required by 37 Signals skillfully articulates the case for remote teams.  (It might be worth your while to get the book and read it before continuing.   At 256 pages, it’s a fairly quick read.)  Remote highlights the ability to access talent and to reduce overhead such as office space and commute costs.  Technology offsets many of the obstacles of remoteness via on-line chat, video conferences, screen sharing, etc.

One of the key tenets of Remote is:  Thou Shalt Overlap.  In other words, team members should make adjustments so that they are working at the same time for a few hours, ideally every day.

What if your team includes some remote members whose time zone makes it extremely difficult to overlap regularly?  If there is a 12 hour time difference, there is just no way to avoid having someone start very early or stay up very late in order to have some overlapping hours.  It is very challenging to maintain that sort of schedule day after day, over the course of a project.

For the sake of this discussion, let’s call that Extreme Remote: having team members so far away that regular overlapping work periods is very difficult to maintain. 

So, again, why would you ever consider such a team arrangement?

As with so many things in business:  Cost.

Consider the relative cost of living for developers in the Extreme Remote time zones (i.e. 10-14 hour time difference).  Here is a chart comparing several cities to Los Angeles (where I live and do most of my business).  

City*	Time difference**	Relative cost of living***
Almaty, Khazakstan	13 hours	-67%
Hyderabad, India	12 hours	-73%
Karachi, Pakistan	12 hours	-62%
Minsk, Belarus	10 hours	-71%
Nizny Novogorod, Russia	10 hours	-70%

* These are all cities where I have worked with skilled remote team members or are well known sources of technical talent.

** The time difference will vary through the year as daylight savings times start and end.

*** Cost of living comparisons are taken from www.expatistan.com.  Other sources may differ somewhat, but they still support the fundamental conclusion.

The cost advantage is clearly very compelling. Even after accounting for generous pay and various overhead costs, developers from these cities will cost less than half of what a developer of comparable talent in Los Angeles will cost.

So, the question really becomes:  How can we address the challenges of Extreme Remote teams sufficiently so that we can produce a high-quality product without losing the extraordinary value?  I will try to address this question in the next series of blog posts.