Practices like
continuous delivery helps software development teams produce software in short
cycles. High quality teams achieve this by continuously integrating the
software, building executables, and running automated tests on those
executables to detect problems. By following this, teams are aiming to reduce
the cost, time, and risk of releasing new versions of software into production.
In most of the cases, organizations will make changes to their current
processes or tools by following the general guidelines or recommendations as
part of a CD/ DevOps adoption cycle. But it’s also important to measure the
progress of the approach to know whether you are improving or making progress.
With the support of proper
data and metrics, it’s easy for teams to inspect the current way of working and
come up with ideas or processes to improve the overall CD/ DevOps movement.
Measuring the current capabilities and process helps teams determine the
problem areas and defined focus points for making a change.
What are the metrics for continuous
delivery?
Well, it depends.
There is no standard definition of metrics for continuous delivery. Below are
some of the key metrics I've identified and used for measuring our progress and
success of our continuous delivery adoption at work. I've seen that these
metrics used in combination provides with enough details about our way of
working as a software delivery team.
Committed to Done (PBI)
The time taken for the user stories per sprint
to be in the done state from the committed state. The scrum team puts a story
in the committed state when they have completed the discussion of the PBI/ user
story in the sprint planning, created the tasks and agreed to build the story
in the sprint. Done is when all the tasks and acceptance criteria’s are met.
The product owner has agreed the PBI meets the specifications and quality
standards. Teams should try to minimize the number of days it takes them to
move the story from committed to done. This indicates whether the team’s
reliability and whether the stories are sized properly.
New to Done (Task)
The time taken for the tasks to move from new
to done state. Tasks should be ideally estimated to complete within 4-16 hours.
Teams should focus on completing a task first and then move to the next one.
New to Done metrics shows how the team is working on the tasks. Whether there
are too many open tasks that all of them are closed at the last day of the
sprint? Are the tasks too big? etc.
Team predictability/ reliability
Measuring how reliable the teams predictions
helps business to create a stable product plan. Making the predictability
metric available to the teams helps them plan, coordinate, and deliver
predictably enough to make a release level commitment. A good way of
calculating this is by using the data from sprints based on the number of
points committed vs. number of points delivered per sprint. This metric helps
teams become predicable in their estimating and sprint planning. It encourages
smaller stories and more effort getting work ready prior to the sprint. Teams
that use good engineering practices such as test-driven development, continuous
integration, automated testing and deployment etc. are more reliable in their
predictions and often have very less errors in the estimation
Velocity
Velocity is a metric that predicts how much
work an agile software development team can successfully complete within a
sprint. Velocity is calculated by counting the number of units of work
completed in a sprint. Velocity is a useful planning tool for estimating how
fast work can be completed and how long it will take to complete a project.
Once established, compared with the team predictability metric, velocity can be
used to plan projects and create reliable release forecasts, product completion
dates, de-scoping functionality, resource planning etc.
Regression test duration
Regression tests helps teams verify that
software that was previously developed and tested still performs correctly
after it was changed or interfaced with other software. The purpose of
regression testing is to ensure that changes such as those mentioned above have
not introduced new faults. One of the main reasons for regression testing is to
determine whether a change in one part of the software affects other parts of
the software. Tracking the time it takes to complete a full regression test
helps teams to device effective measures to reduce this time, hence helping
them make new software releases faster. This means focusing on adding more
automated tests than manual tests, creating parallel testing strategies,
following good testing practices from the test pyramid etc.
Number of open bugs per sprint
Teams adopt practices like continuous delivery
to create faster delivery of quality software to the end users. With shorter
releases and automated releases, teams should maintain high quality standards
to ensure that bugs are not shipped the customers. It’s important for teams to
maintain a very low error ration in the software shipped to customers to ensure
that customer satisfaction is high with the releases.
Team and P.O happiness
This is an index measuring the level of
happiness in the scrum team per sprints. The trend of happiness says quite a
lot about a group and how well everything is going regarding its goals, A happy
team and members are productive and are working towards their mutual goal to
deliver the product.
Percentage of failed deployments
The goal is to make sure that regardless of how
often you deploy, what gets deployed doesn't cause a failure. Measuring and keeping track of deployments
allow teams to make adjustments in their process to achieve small-batch
developments to update every few days, or even several times per day.
Check-in frequency vs. broken build
time
This is the plot displaying the time taken to
fix the broken builds within a given
time period. Continuous integration encourages practices like the whole team
drops their work and verifies who should handle the broken build when notified.
Check-in code to an already broken build creates a lot of complexity and waste.
Teams should focus on integrating code changes frequently, so that integration
doesn’t become such a hard process with ensuring that the build is stable.
Code coverage vs. Maintainability
index
The metric shows the trend lines for code
coverage and maintainability index of the code. With rising lines of code, it’s
important for the teams to improve the quality of the code to make it easier
for building new features in the product. Practices like improving testability
helps you maintain a maintainable code base. These metrics can be easily
generated by tools like SonarQube. The trend line displays how creating
testable code improves the code maintainability. Also it’s a good indicator of
deciding what your code coverage should be (at what point it stops adding
value).
Mean time to recover
This measures how long it takes from when an
incident is reported to when it is resolved. In general, this number needs to
be trending downwards. This indicates both the responsiveness of your team as
well as the capability to resolve and deploy solutions. Continuous Delivery
automation with a combination of monitoring techniques and tools like
Application insights/ ELK stack makes rolling out rapid fixes easier and spot
failures quickly through a ‘production immune system’. Based on these practices
the ideal DevOps team can produce frequent, rapid deployments with a low (and
declining) failure rate and a short (and shrinking) recovery time
Cycle time
According to Dave Farley "Your cycle-time
is the time it takes you to release the smallest possible change to your code
following your normal release validations and procedures. Your cycle time
should be short enough so that when an emergency hits, you can still release
your fixes having validated them fully.". Teams should focus on reducing
their cycle time by improving their software delivery practices. A reduced
cycle time allow them to work in small batches, ensuring that each change is
small-enough so that when it does go wrong it is simple to see where the
problem lies and how to fix it.
Conclusion
As you can see there
are a lot of metrics that you can choose from. Before adopting these in your
team, it’s important to find out which of these metrics are important for your
team and how they are applicable in your process. The idea is to have the right
set of metrics and data that allows you to stay focused on the continuous
delivery goal and put it to good use
