Continuous Delivery Metrics and Goals

#1

I am the Product Owner for a systems team that helps support 18 teams. The systems team’s goal is to make it easy to rapidly deliver value to our users through our continuous delivery systems (build, automated tests, on demand test envs, deploy, operate, validate).

I’d like to gather some metrics / stories of how other companies / teams approach continuous delivery to help us consider options and goals for our journey of continuously improving our ability to continuously delivery. For example, it would be great if other could share some of the following:

  • How many teams / people are working on your product?
  • How frequently do you deploy to production?
  • What’s the average cycle time for a change deployed to production (from pulling card to available to users)?
  • How long does it take to run your master pipelines to build / test / deploy to production?
  • What is the success rate of running changes through your master pipelines to build / test / deploy to production?
  • Other interesting stories / aspects of your approach to continuous delivery?
#2

I am just hitting my 90 day mark with my new company, SunTrust Bank. My accountability is head of the DevOps Center of Enablement. All of what you are doing is what I am out to facilitate bringing to life. We are highly regulated and there are many who doubt that we will get to where you have gotten, but it is totally possible. One day at a time for now.

I would add two measures to what you have stated above (all excellent by the way).

Mean Time To Recovery - Mistakes and failures will happen. How often do they happen, and when they do, how long does it take to roll back?

Small Correct Change To Production Time - Credit to my friend, Kyle Griffin, for coming up with this name. This is sort of like cycle time under ideal circumstances. What this means is, make some really small and minor change to the last production release, say changing the words or labels on some web page, even in some way that is not humanly perceptible, like adding an HTML comment. How long does it take to go from committing to make this change to getting it out the door to the first production stage, and then to all production stages (for when you roll out in stages)? This is the best you can do under ideal circumstances. You want to keep an eye on this metric and endeavor to optimize it, so that when you really need to move this fast, you can.

#3

Thanks Paul! Both of those are great additions!

#4

Hi,

  • How many teams / people are working on your product?

Our teams are based on the actual service (e.G. there is one team operating the infrastructure/platform/network while another (my) team is working on the services to our customers). We are 9 people working on the product (PO, SM, Software Tester, 3 Software Engineers and 3 Developers)

  • How frequently do you deploy to production?

Min. once a day - up to 4x a day

  • What’s the average cycle time for a change deployed to production (from pulling card to available to users)?

That depends on how you define change. Incidents at the same day. In average around 3 days.

  • How long does it take to run your master pipelines to build / test / deploy to production?

Approx. 2 hours

  • What is the success rate of running changes through your master pipelines to build / test / deploy to production?

Approx. 70%

  • Other interesting stories / aspects of your approach to continuous delivery?

Do not have multiple teams use the same deployment pipeline. Frequently Team A cannot deploy because a build of Team B failed.

Regards
Sascha

#5

Thanks so much for sharing Sascha!

Would love to get some other perspectives / data from others. Anyone else willing to share?