Practices like continuous delivery helps software development teams produce software in short cycles. High quality teams achieve this by continuously integrating the software, building executables, and running automated tests on those executables to detect problems. By following this, teams are aiming to reduce the cost, time, and risk of releasing new versions of software into production. In most of the cases, organizations will make changes to their current processes or tools by following the general guidelines or recommendations as part of a CD/ DevOps adoption cycle. But it’s also important to measure the progress of the approach to know whether you are improving or making progress.
With the support of proper data and metrics, it’s easy for teams to inspect the current way of working and come up with ideas or processes to improve the overall CD/ DevOps movement. Measuring the current capabilities and process helps teams determine the problem areas and defined focus points for making a change.
What are the metrics for continuous delivery?
Well, it depends. There is no standard definition of metrics for continuous delivery. Below are some of the key metrics I've identified and used for measuring our progress and success of our continuous delivery adoption at work. I've seen that these metrics used in combination provides with enough details about our way of working as a software delivery team.
Committed to Done (PBI)
The time taken for the user stories per sprint to be in the done state from the committed state. The scrum team puts a story in the committed state when they have completed the discussion of the PBI/ user story in the sprint planning, created the tasks and agreed to build the story in the sprint. Done is when all the tasks and acceptance criteria’s are met. The product owner has agreed the PBI meets the specifications and quality standards. Teams should try to minimize the number of days it takes them to move the story from committed to done. This indicates whether the team’s reliability and whether the stories are sized properly.
New to Done (Task)
The time taken for the tasks to move from new to done state. Tasks should be ideally estimated to complete within 4-16 hours. Teams should focus on completing a task first and then move to the next one. New to Done metrics shows how the team is working on the tasks. Whether there are too many open tasks that all of them are closed at the last day of the sprint? Are the tasks too big? etc.
Team predictability/ reliability
Measuring how reliable the teams predictions helps business to create a stable product plan. Making the predictability metric available to the teams helps them plan, coordinate, and deliver predictably enough to make a release level commitment. A good way of calculating this is by using the data from sprints based on the number of points committed vs. number of points delivered per sprint. This metric helps teams become predicable in their estimating and sprint planning. It encourages smaller stories and more effort getting work ready prior to the sprint. Teams that use good engineering practices such as test-driven development, continuous integration, automated testing and deployment etc. are more reliable in their predictions and often have very less errors in the estimation
Velocity is a metric that predicts how much work an agile software development team can successfully complete within a sprint. Velocity is calculated by counting the number of units of work completed in a sprint. Velocity is a useful planning tool for estimating how fast work can be completed and how long it will take to complete a project. Once established, compared with the team predictability metric, velocity can be used to plan projects and create reliable release forecasts, product completion dates, de-scoping functionality, resource planning etc.
Regression test duration
Regression tests helps teams verify that software that was previously developed and tested still performs correctly after it was changed or interfaced with other software. The purpose of regression testing is to ensure that changes such as those mentioned above have not introduced new faults. One of the main reasons for regression testing is to determine whether a change in one part of the software affects other parts of the software. Tracking the time it takes to complete a full regression test helps teams to device effective measures to reduce this time, hence helping them make new software releases faster. This means focusing on adding more automated tests than manual tests, creating parallel testing strategies, following good testing practices from the test pyramid etc.
Number of open bugs per sprint
Teams adopt practices like continuous delivery to create faster delivery of quality software to the end users. With shorter releases and automated releases, teams should maintain high quality standards to ensure that bugs are not shipped the customers. It’s important for teams to maintain a very low error ration in the software shipped to customers to ensure that customer satisfaction is high with the releases.
Team and P.O happiness
This is an index measuring the level of happiness in the scrum team per sprints. The trend of happiness says quite a lot about a group and how well everything is going regarding its goals, A happy team and members are productive and are working towards their mutual goal to deliver the product.
Percentage of failed deployments
The goal is to make sure that regardless of how often you deploy, what gets deployed doesn't cause a failure. Measuring and keeping track of deployments allow teams to make adjustments in their process to achieve small-batch developments to update every few days, or even several times per day.
Check-in frequency vs. broken build time
This is the plot displaying the time taken to fix the broken builds within a given time period. Continuous integration encourages practices like the whole team drops their work and verifies who should handle the broken build when notified. Check-in code to an already broken build creates a lot of complexity and waste. Teams should focus on integrating code changes frequently, so that integration doesn’t become such a hard process with ensuring that the build is stable.
Code coverage vs. Maintainability index
The metric shows the trend lines for code coverage and maintainability index of the code. With rising lines of code, it’s important for the teams to improve the quality of the code to make it easier for building new features in the product. Practices like improving testability helps you maintain a maintainable code base. These metrics can be easily generated by tools like SonarQube. The trend line displays how creating testable code improves the code maintainability. Also it’s a good indicator of deciding what your code coverage should be (at what point it stops adding value).
Mean time to recover
This measures how long it takes from when an incident is reported to when it is resolved. In general, this number needs to be trending downwards. This indicates both the responsiveness of your team as well as the capability to resolve and deploy solutions. Continuous Delivery automation with a combination of monitoring techniques and tools like Application insights/ ELK stack makes rolling out rapid fixes easier and spot failures quickly through a ‘production immune system’. Based on these practices the ideal DevOps team can produce frequent, rapid deployments with a low (and declining) failure rate and a short (and shrinking) recovery time
According to Dave Farley "Your cycle-time is the time it takes you to release the smallest possible change to your code following your normal release validations and procedures. Your cycle time should be short enough so that when an emergency hits, you can still release your fixes having validated them fully.". Teams should focus on reducing their cycle time by improving their software delivery practices. A reduced cycle time allow them to work in small batches, ensuring that each change is small-enough so that when it does go wrong it is simple to see where the problem lies and how to fix it.
As you can see there are a lot of metrics that you can choose from. Before adopting these in your team, it’s important to find out which of these metrics are important for your team and how they are applicable in your process. The idea is to have the right set of metrics and data that allows you to stay focused on the continuous delivery goal and put it to good use