Monday, March 21, 2016

Metrics to keep you on track for a continuous delivery journey

Practices like continuous delivery helps software development teams produce software in short cycles. High quality teams achieve this by continuously integrating the software, building executables, and running automated tests on those executables to detect problems. By following this, teams are aiming to reduce the cost, time, and risk of releasing new versions of software into production. In most of the cases, organizations will make changes to their current processes or tools by following the general guidelines or recommendations as part of a CD/ DevOps adoption cycle. But it’s also important to measure the progress of the approach to know whether you are improving or making progress.
With the support of proper data and metrics, it’s easy for teams to inspect the current way of working and come up with ideas or processes to improve the overall CD/ DevOps movement. Measuring the current capabilities and process helps teams determine the problem areas and defined focus points for making a change.

What are the metrics for continuous delivery?

Well, it depends. There is no standard definition of metrics for continuous delivery. Below are some of the key metrics I've identified and used for measuring our progress and success of our continuous delivery adoption at work. I've seen that these metrics used in combination provides with enough details about our way of working as a software delivery team.

Committed to Done (PBI)

The time taken for the user stories per sprint to be in the done state from the committed state. The scrum team puts a story in the committed state when they have completed the discussion of the PBI/ user story in the sprint planning, created the tasks and agreed to build the story in the sprint. Done is when all the tasks and acceptance criteria’s are met. The product owner has agreed the PBI meets the specifications and quality standards. Teams should try to minimize the number of days it takes them to move the story from committed to done. This indicates whether the team’s reliability and whether the stories are sized properly.

New to Done (Task)

The time taken for the tasks to move from new to done state. Tasks should be ideally estimated to complete within 4-16 hours. Teams should focus on completing a task first and then move to the next one. New to Done metrics shows how the team is working on the tasks. Whether there are too many open tasks that all of them are closed at the last day of the sprint? Are the tasks too big? etc.

Team predictability/ reliability

Measuring how reliable the teams predictions helps business to create a stable product plan. Making the predictability metric available to the teams helps them plan, coordinate, and deliver predictably enough to make a release level commitment. A good way of calculating this is by using the data from sprints based on the number of points committed vs. number of points delivered per sprint. This metric helps teams become predicable in their estimating and sprint planning. It encourages smaller stories and more effort getting work ready prior to the sprint. Teams that use good engineering practices such as test-driven development, continuous integration, automated testing and deployment etc. are more reliable in their predictions and often have very less errors in the estimation


Velocity is a metric that predicts how much work an agile software development team can successfully complete within a sprint. Velocity is calculated by counting the number of units of work completed in a sprint. Velocity is a useful planning tool for estimating how fast work can be completed and how long it will take to complete a project. Once established, compared with the team predictability metric, velocity can be used to plan projects and create reliable release forecasts, product completion dates, de-scoping functionality, resource planning etc.

Regression test duration

Regression tests helps teams verify that software that was previously developed and tested still performs correctly after it was changed or interfaced with other software. The purpose of regression testing is to ensure that changes such as those mentioned above have not introduced new faults. One of the main reasons for regression testing is to determine whether a change in one part of the software affects other parts of the software. Tracking the time it takes to complete a full regression test helps teams to device effective measures to reduce this time, hence helping them make new software releases faster. This means focusing on adding more automated tests than manual tests, creating parallel testing strategies, following good testing practices from the test pyramid etc.

Number of open bugs per sprint

Teams adopt practices like continuous delivery to create faster delivery of quality software to the end users. With shorter releases and automated releases, teams should maintain high quality standards to ensure that bugs are not shipped the customers. It’s important for teams to maintain a very low error ration in the software shipped to customers to ensure that customer satisfaction is high with the releases.

Team and P.O happiness

This is an index measuring the level of happiness in the scrum team per sprints. The trend of happiness says quite a lot about a group and how well everything is going regarding its goals, A happy team and members are productive and are working towards their mutual goal to deliver the product.

Percentage of failed deployments

The goal is to make sure that regardless of how often you deploy, what gets deployed doesn't cause a failure.  Measuring and keeping track of deployments allow teams to make adjustments in their process to achieve small-batch developments to update every few days, or even several times per day.

Check-in frequency vs. broken build time

This is the plot displaying the time taken to fix the broken builds within a  given time period. Continuous integration encourages practices like the whole team drops their work and verifies who should handle the broken build when notified. Check-in code to an already broken build creates a lot of complexity and waste. Teams should focus on integrating code changes frequently, so that integration doesn’t become such a hard process with ensuring that the build is stable.

Code coverage vs. Maintainability index

The metric shows the trend lines for code coverage and maintainability index of the code. With rising lines of code, it’s important for the teams to improve the quality of the code to make it easier for building new features in the product. Practices like improving testability helps you maintain a maintainable code base. These metrics can be easily generated by tools like SonarQube. The trend line displays how creating testable code improves the code maintainability. Also it’s a good indicator of deciding what your code coverage should be (at what point it stops adding value).

Mean time to recover

This measures how long it takes from when an incident is reported to when it is resolved. In general, this number needs to be trending downwards. This indicates both the responsiveness of your team as well as the capability to resolve and deploy solutions. Continuous Delivery automation with a combination of monitoring techniques and tools like Application insights/ ELK stack makes rolling out rapid fixes easier and spot failures quickly through a ‘production immune system’. Based on these practices the ideal DevOps team can produce frequent, rapid deployments with a low (and declining) failure rate and a short (and shrinking) recovery time

Cycle time

According to Dave Farley "Your cycle-time is the time it takes you to release the smallest possible change to your code following your normal release validations and procedures. Your cycle time should be short enough so that when an emergency hits, you can still release your fixes having validated them fully.". Teams should focus on reducing their cycle time by improving their software delivery practices. A reduced cycle time allow them to work in small batches, ensuring that each change is small-enough so that when it does go wrong it is simple to see where the problem lies and how to fix it.


As you can see there are a lot of metrics that you can choose from. Before adopting these in your team, it’s important to find out which of these metrics are important for your team and how they are applicable in your process. The idea is to have the right set of metrics and data that allows you to stay focused on the continuous delivery goal and put it to good use

Sunday, March 6, 2016

Microservices – Why does it matter for continuous delivery

Software delivery organizations across the globe has adopted continuous delivery as a central practice to build high quality software faster, with less stress and effort. One of the goals of continuous delivery is to make deployments as boring as possible. To assure that we need to deploy often and find errors as soon as possible before delivering the software to end user, continuous delivery promotes certain architectural approaches of a platform. Releasing often, with small changes, is extremely difficult task to perform for a complex application with a huge monolithic architecture. The idea of rearranging the software architecture into smaller, focused, domain-specific applications composed of microservices has gained a lot of popularity in the last 2-3 years.
Microservices are a new software development pattern emerging from recent trends in technology and practice. A microservice is an independent, self-contained process that provides a single business capability. Different Microservices act in concert as one larger system by communicating to each other using a language agnostic API. In microservice architecture each service is highly decoupled from each other and are created in mind to focus on one problem domain.

Monolithic and the problem for continuous delivery

Traditional monolithic applications are built as a single unit, where all the functionality is pulled into a single process and is replicated across multiple servers. The single application is responsible for all functionality including handling requests, authentication, executing business/ domain logic, persistence, logging, exception handling etc. Due to this behavior its makes it hard to add a small change in the system that involves building, testing and deploying the entire application. Scalability and availability involves running multiple instances of this application behind a load balancer on multiple servers even though the bottleneck lays in a single component in the application. As a result, updating monolithic applications become time-consuming activity that needs to have a lot of coordination between people involved in multiple functions and departments.
Apart from the issues related to deployment and scalability, there are some other challenges that this kind of architecture brings when the application grows in size.
A large code base makes it hard for all the developers to understand and modify the application. This makes the code base harder to manage and difficult to add new features which also results in bad quality creeping in as time progress. The team is forced to stick on the technology stack which was chosen at the time of starting the development and commit to the stack for a long-term. This also introduces dividing engineering teams at an organization level to focus on specific functional areas, like UI team , database teams etc. The coordination, development and deployment processes becomes harder with this structure which takes the teams a step backward in their journey towards practices like DevOps, continuous delivery etc.

How microservices helps?

In contrast to Monolithic applications, microservices makes independent management of business functionality by separating the functionality into smaller services that can be independently deployed, tested and scaled. With microservices scaling and deploying each service can be done independently without affecting the other parts of the system, by creating instances of these services across containers or virtual machines. In short a Microservice can be summarized as
“Applications composed of small, independently versioned, and scalable customer-focused services that provides a unique business capability and communicate with each other over standard protocols with well-defined interfaces.”
In the microservice approach the application is divided into smaller parts that communicate with each other with lightweight mechanisms like REST, message brokers etc.
Due to the highly evolving business needs, software applications needs to be build considering high scalability and availability to respond to the needs of customers in an agile way. This is only possible if, making change to a service is less expensive and risky. Microservices make this possible due to the fact that each service is operationally encapsulated by other services and are independently deployable without affecting the other services in the infrastructure. This makes easier for development teams to apply the changes and test them independently faster than ever.

Free white paper

The white paper addresses the common characteristics and challenges with solutions to your journey towards refactoring a monolithic to a microservice. You can download the complete article from this link.