Measuring Transformation Progress?

#1

What are some examples of metrics you use to show how the agile transformation is progressing? I’m curious to see what other practitioners utilize to show forward progress. I know trying to boil a culture change into some numbers is a bit of analog thinking, but it’s an unfortunate reality and I wanted to see how others communicate their delivery.

#2

Jay I like to measure Agile Transfermation as well as delivery. With five metrics areas business value, lead time to production/ cycle times, happiness, quality, and Agile Fluency. Looking at trends will show success and fail your trends.

Become focused on outcomes and Use experiment to get validated learning on the outcomes.

#3

Agree with @Scrummando above. To add on though, whenever you start a transformation - understanding the items you are transforming away from is critical to measure success. Changing and measuring your delivery system when it may have been fine before is irrational. Key in on what is meant to change, plan to change it and agree on how you will measure, then measure, learn from measurement and act on those learnings.

To measure the happiness of the teams, I always point people to Officevibe.com. I use it. I love it. Snap a baseline of team happiness and engagement day 0 - then see what direction the measurements go.

#4

In my experience, “agile metrics” become most useful when they help us have conversations about behaviors/outcomes, rather than manage behaviors/outcomes.

IMO there are two important measurements which James already highlighted: lead time and [value] (value here represents the variety of ways–from micro to macro–you’re using (leading) indicators of life-cycle profits to guide decision making). The others (happiness, quality, etc.) feel like proxy variables to me… they’re important, but aren’t systemic.

Where I’ve found the most impressive use of “measuring transformation” is surprisingly simple: sliders. No, not the “risk slider” tradeoffs from PMBOK, nor quite exactly like the success sliders from “radical project management.”

Instead, by using sliders somewhat akin to verbiage you might use in a Dreyfus model, you can create a simple visual “dashboard” that enables conversations about whether you’re leaning left (towards existing/old/undesired behavior) or right (towards the new behavior(s)). This helps us explore experiments and ideas to influence new behavior, rather than design a desired state and manage to it.

For being so simple, it’s difficult to explain in text :slight_smile: I’ll be writing about this technique in the lead-up to SGCAL and speaking about it at the conference.

#5

Interesting thoughts on the sliders Zach. Usually, I used sliders to set expectations at the beginning of a body or work, but having team based sliders that can dynamically move sounds cool.

I’m not a fan of using metrics like “value” and lead time to measure transformation because I don’t consider productivity the main driver of transformation. That said, I’m not necessarily disagreeing with what you and James are saying. Have just struggled with performance metrics being the main lens through with “transformation” is viewed.

So maybe I’m just a bit defensive of that approach. :slight_smile:

#6

I agree with much of the above. Here’s some of my own thoughts:

  1. “Working software is the primary measure of progress.” With that in mind, are you delivering working every sprint? If yes, work on delivering it once a week. Then once a day. Then continuously. If no, forget all other measures and make this the only metric that matters. Measuring this is easy and straight forward and nothing shows value like software the customer can interact with (and spend money on).
  2. Once #1 is a yes, understand the rights and wrongs of metrics. Martin Fowler says it best in “An Appropriate Use of Metrics.”
  3. Once you’ve done #1 and #2, check out Troy Magennis’ video about a balanced metric. It inspired me to do much the same in my organization.
#7

Re: “value” & lead time as a transformation metric

Oops. I didn’t mean to align them as transformation metrics… I was reiterating the concepts are important for software development. We often measure a lot of things. I think just a few are useful :slight_smile:

With that said…

I have experienced change (transformation?) as a result of actually paying attention to things like value and the time it takes for customers to experience value.

But I do agree with you - an “agile transformation” for the purpose of improving performance is questionable (and likely unnecessary).

z

#8

In our ‘Back to Basics’ Agile Kata program, we are measuring transformation success of each team with 3 main indicators - Agile Maturity, Stakeholder Satisfaction and Predictability.

Agile Maturity levels (Shu-Ha-Ri) are based on an inspection worksheet that teams use to self inspect at least once a quarter, and coaches use to validate with teams at least 2x a year.

Stakeholder Satisfaction is based on NPS scores collected via a survey of internal stakeholders on a quarterly basis.

Predictability is a combination of backlog readiness and carryover rates - we look for at least 2 sprints worth of ‘ready’ stories in the backlog and 0 carryover sprint to sprint on a consistent basis.

#9

We’re debating putting together a podcast to discuss this topic…would anyone who’s commented on this thread be interested in joining? We would love to not only get our users involved but also welcome some new opinions…

#10

“Agile Maturity”

Man, is that ever a dangerous precedent to set… let alone to measure someone for.

z

#11

I would consider it; we’d have to chat a bit about the format and intent, though.

#12

Thats our typical process for all podcasts. There is usually a working google doc with talking points etc.

#13

Thumbs Up for Troy Magennis work in this area! I also like the simplicity of NPS. I have used a Employee NPS (“so to speak”) early in Agile Adoptions to help us gauge the pulse of the the stakeholders that have been impacted by the Agile adoption, and whether or not they feel it is an improvement from legacy behaviors.

#14

I’m game.

#15

We’re beginning to use NPS ourselves. (When used for employees though, it’s commonly referred to as eNPS.) We just recently put together a balanced metric for our teams to use. It just recently went live, and I’m very curious how the teams will respond to the data and how they’ll use the data. Here’s what it looks like.

agile.PNG1542×873 72.1 KB

#16

Like the dashboard, but I worry about mixing things like delivery performance and happiness together. Do you think both are required to measure a “transformation” as we have mentioned in this thread?

#17

I have concerns with what I see.

I’ll make no judgment of course, as I have zero context. And I’m not foolish enough to think I have correct opinions or answers.

However, I believe the four metrics in use are either proxy variables or harmful to measure.

#18

I may have delved too far down the hole of “how can one measure the results of an agile team?” instead of “how can one measure an agile transformation?” I suppose I’m having a hard time with measuring a transformation because, to me, you’re attempting to measure how a person or group thinks.

#19

YES!!! Exactly. :slight_smile:

There’s a strange juxtaposition we face in the community of trying to not focus on delivery metrics while at the same time still prioritizing delivering working software. My favorite phrase is “shipping cures all”.

Zach and were alluding to this, but delivery metrics are fine. Happiness metrics are great. When they are mixed I just wonder how the numbers are perceived by execs.

Lean change management by Jason Little encourages measuring change based upon the outcomes you want teams to achieve. If you can start by what behaviors ar desired, you can then run experiments to achieve them. More or less. :slight_smile:

#20

I know the concern well, and I myself have baggage about metrics. They can be abused by those in power and gamed by those in the trenches. Just as user stories shouldn’t be requirements but the start of a conversation, the intent of the metrics is the beginning of a conversation as well. Judgement made purely from the data is missing the point and sometimes dangerous. Additionally, the idea is if you excel in one area, another will likely suffer. For example, the team doubled their speed from one sprint to the next, but their happiness plummeted. Why? They doubled their speed by working 12 hour days, and now they’re burnt out.

For predictability and happiness, I think we have good measures. For quality, it’s abysmal, and we’re looking for better ways to measure it. For speed, that depends on how you feel about the no estimates movement. Velocity versus issue count both have their merits.

Bear in mind that this is v1 created with just a few days of work but pulling data from a myriad of systems. We’re now curious how teams will use the data to determine how/where to adjust.