Enterprise Software Development
<a href="/Solutions/tabid/86/LiveTabId/5470/Default.aspx">Innovate Anywhere</a>
<a href="/Solutions/tabid/86/LiveTabId/5473/Default.aspx">Reuse Best Practices</a>
<a href="/Solutions/tabid/86/LiveTabId/5472/Default.aspx">Build Better Teams</a>
The Best of Software<br/>Development Platforms

Black Diamond Software, Inc.


Blog for Black Diamond Software, Inc.

William Rojas
William Rojas
William Rojas's Blog

Four DevOps Patterns Essential to Reaping DevOps Benefits William Rojas
Four DevOps Patterns Essential to Reaping DevOps Benefits

Four Patterns Essential to Reaping DevOps Gains

DevOps stresses on communication, collaboration and integration between the development and operations sides of your organization. As organizations embark onto the DevOps transformation they will quickly realize that without a whole new approach that changes the organization ‘from the ground up’, the benefits of DevOps will remain untapped. It is simply not enough to increase the number of releases while calling for greater communication and collaboration. Systems must be (re)architected with a DevOps strategy in mind. This new interaction strategy calls for new architectural, development, version control and release patterns. While patterns are still newly accepted and emerging, nonetheless some of these techniques have been around for a few years and may even have been first developed for other intended benefits. These patterns are proving to be essential to organizations undergoing a DevOps transformation as they move to greater agility and lean production.

This paper will highlight of benefits of these patterns to DevOps and how these interact to provide a more beneficial solution. References are provided for further reading on each individual pattern –for those interested in more detail.

Branch by Abstraction[i]: is a development and version control technique that enables to incrementally produce and deliver large-scale changes to a system from the source control mainline; updates can be release regularly while the large change is still in-progress. The large-scale function is gradually released to production without ever having any down time in that feature due to unpredicted delays in feature branch merges or to destabilized code or breakages in areas that are still undergoing changes due to the large size and/or impact of the new large-scale update.

Feature Toggles[ii]: is an architecture/design and development technique that allows partially completed features to be developed and included in a regular release, but not necessarily yet enabled in the production environment. Feature Toggles enable a continuous flow of delivered capability in a Scrum or Kanban (i.e Agile methodologies) initiative via Continuous Integration (CI)/Continuous Delivery (CD) techniques. Basically, you can send new features to production in the off state, and then in an admin console turn the feature on when proven to be production ready - or off if you discover it’s broken. Feature Toggles are designed and coded into the system to route traffic away from the new functionality until it is fully available. The system is (re)designed to be configurable at build or at run-time to enable/disable the access to the new functions. Once the functionality is fully enabled in production the feature toggle can be removed from the code. Alternatively these can be preserved if the system requires dynamic configuration capabilities allowing control of which features are available to users at any point in the future. This pattern is being employed by many large organizations today: Facebook, LinkedIn, Google and others.

A proper implementation of feature toggles in a system requires strict discipline given the release of ‘broken’ or ‘dark code’ to production therefore these prerequisites are put in place along with it: continuous delivery/deployment, continuous integration, automated unit testing, automated integration testing, automated stress & performance testing, system automation. An alternative to Feature Toggles is a Feature Branch, but this is not without pros/cons

Dark Launch[iii]: is an approach for launching new features by which the new functionality is deployed and enabled in your production environment but not made available to users through the user interface. The term was originally coined by Facebook development during the launch of the Chat feature. Dark Launching allows verifying the new functionality and stabilizing the infrastructure before the feature is made live to the end users. Functionality can be gradually enabled to the user base to validate and stabilize it before turning it on for the entire end user base. As the new functionality proves to be stable and, if applicable, the infrastructure is proven to be able to handle the new load demand of the new function, then the new features are enabled to a larger and larger audience; but the functionality had been deployed to production all along.

Blue-Green Deployment[iv]: is an approach for releasing to production that aims to reduce the fail-over between the final stages of testing and going live. The approach calls for two, as close as possible, parallel production environments; while one environment is considered to be live call it the Blue environment, the new functionality is released to the Green environment. Once the functionality has passed all the final stage testing then live production traffic is switch from the Blue to the Green environment. The Green environment is considered to be live production going forward. The switching from one production environment to another is often accomplished via network switches or routers.

DevOps Enables the Objective of Agile and Lean

Organizations more and more are realizing that the true benefits of agile development are tightly interconnected with Lean principles. Only those organizations that can deliver gains in productivity, quality, team morale and time to market are the ones that truly reap the benefits claimed by the agile movement. But in order to achieve these gains organization must adopt practices such as DevOps and Continuous Delivery practices as well.

Agile has proven that ‘smaller is better”. Agile developers can very quickly gather true feedback from actual users running actual deployed code in an actual production environment. This feedback in turn influences development in their future creation and delivery of product. This cycle is causing organizations to redefine the ‘conception to measure results’ product lifecycle and adopt more of a ‘test out a hypothesis’ approach to their product definition and requirements. One way we see the results of this movement is with agile teams that now delivery smaller features more often that can then validate via techniques such as A/B Testing.

Embracing these CD Patterns

Case for Branch by Abstraction: With this new approach to requirements –being more of hypothesis rather that hard set fixed ideas, comes the need to develop and check in code that as CI says ‘is ready for release within a day’. This requires that all development occur from a single mainline trunk branch in your version control system. The argument being that if code is not being integrated daily, then there are lurking issues that will not be discovered until the merged of branched code occurs jeopardizing the commitment to releasable code at all times. Thus the need to a Branch by Abstraction is crucial.

This pattern requires that all development occur on the mainline regardless of the size or impact of the change being made; further the team is in constant communication and collaboration as code is integrated daily upon commit. Contrast these results with a Feature Branching pattern where new code can go on without being integrated back into the mainline for days or longer.

While it’s possible to employ feature branching as an alternative to your release and integration methodology, this does not come without challenges. In order to maintain a stable mainline development trunk, the often a feature branch is deployed to a dev/qa environment, certified and then merged back into the mainline trunk so that it can then be pushed to SIT, UAT and eventually production. While this strategy does allow somewhat independent/isolated development, it comes with the price of potentially large merge conflicts with the feature branch is eventually merged back into mainline.

Another important benefit from adopting this type of pattern is the increase in the architecture’s resilience from potentially needing to redesign existing functionality into a set of interacting components and then in defining an abstraction layer between the clients of a component and the to-be replaced component itself. Architectural improvements to legacy systems will provide beneficial results beyond just the immediate update. The abstraction layer can be discarded once the large scale feature is fully deployed if no longer needed, or it can be preserved to allow for different implementations of a common service within you system to coexist.

Case for Feature Toggles: A stronger alternative to Feature Branches above is the use of Feature Toggles instead. In order for a team to concurrently develop, potentially, multiple features in various stages of completion and yet always maintain a release-ready system then a pattern such as Feature Toggles is much more effective. While it is true, that ultimately what are called for are smaller features that can be safely introduced into the product in single release cycles rather than the historical large-well define feature approach; the ability ‘hide’ partially built features and yet validate incrementally is very powerful, providing great flexibility to organizations on what capabilities are available to their user base.

Besides from the obvious benefit of being able to release code that is yet not complete, provided it is stable enough, Feature Toggles also provide for another side benefits. Feature Toggles, when well implemented, will minimize the interface points from the existing system into the code for the new feature, for example having a single UI link from a home page to the new functionality –and all its other UI elements. In this case only that code with that single point of entry is the one that would need to be wrapped with the feature toggle code. This is just plain good old OO design practice.

Case for Dark Launch: Along with being able to gradually deliver large-scale update to production and to configure the availability of these features at run-time often comes the need to deploy these to a production environment but still in a validation and stabilization mode since it is likely the new feature will not be able to be fully tested until it is functioning in the production environment due to the unique nature of the production infrastructure and user base. As they say: ‘there is no test like the production environment’. This release approach thus calls for a Dark Launch pattern.

The new feature can be deployed to production and fully stabilized well ahead of its eventual go-live release.

Case for Blue-Green Deployment: This release pattern allows to bring the previous three patterns together into a scenario where the organization can easily switch from the live-production environment to the alternate one –the one which has been undergoing the final stages of testing for a major release where Branch by Abstraction, Feature Toggles and even Dark Launch needed to be employed, once all production-ready testing passes. Or alternatively revert back to the current live-production if any of the final stage testing fails.

Another use of this technique is to ensure the DR (disaster recovery) environment is functional and indeed can take over production if true disaster were to occur. Again the other usage has other important benefits to the organization.

Putting it all Together

As we’ve seen DevOps influences an organization to approach the creation of product in new ways. DevOps calls for smaller features that are released more often, this requires designers, developers, testers along with operations and infrastructure to break down the traditional roles and silos. Further, this is also changing how business and marketing areas conceive and request for new features

But for this evolution to occur new patterns that influence the entire IT organization are needed. It is not sufficient to have operations deliver more often if the system developed and tested cannot support this. The adoption of these patterns makes great strides in increasing productivity and quality, while reducing lead-time and maintaining high team morale.

Further Reading & Notes

[i] Branch by Abstraction: See Blogs from Martin Fowler , Paul Hammant or Jez Humble for more detail and usage

[ii] Feature Toggles: See bliki from Martin Fowler for a great explanation on Feature Toggles

[iii] Dark Launch: Dare Obasanjo provides useful background on this concept

[iv] Blue-Green Deployment: Refer to CloudBees post from Michael Neale and Martin Fowler for more details