Measuring Transformation Progress?

#8

In our ‘Back to Basics’ Agile Kata program, we are measuring transformation success of each team with 3 main indicators - Agile Maturity, Stakeholder Satisfaction and Predictability.

Agile Maturity levels (Shu-Ha-Ri) are based on an inspection worksheet that teams use to self inspect at least once a quarter, and coaches use to validate with teams at least 2x a year.

Stakeholder Satisfaction is based on NPS scores collected via a survey of internal stakeholders on a quarterly basis.

Predictability is a combination of backlog readiness and carryover rates - we look for at least 2 sprints worth of ‘ready’ stories in the backlog and 0 carryover sprint to sprint on a consistent basis.

#9

We’re debating putting together a podcast to discuss this topic…would anyone who’s commented on this thread be interested in joining? We would love to not only get our users involved but also welcome some new opinions…

#10

“Agile Maturity”

Man, is that ever a dangerous precedent to set… let alone to measure someone for.

z

#11

I would consider it; we’d have to chat a bit about the format and intent, though.

#12

Thats our typical process for all podcasts. There is usually a working google doc with talking points etc.

#13

Thumbs Up for Troy Magennis work in this area! I also like the simplicity of NPS. I have used a Employee NPS (“so to speak”) early in Agile Adoptions to help us gauge the pulse of the the stakeholders that have been impacted by the Agile adoption, and whether or not they feel it is an improvement from legacy behaviors.

#14

I’m game.

#15

We’re beginning to use NPS ourselves. (When used for employees though, it’s commonly referred to as eNPS.) We just recently put together a balanced metric for our teams to use. It just recently went live, and I’m very curious how the teams will respond to the data and how they’ll use the data. Here’s what it looks like.

agile.PNG1542×873 72.1 KB

#16

Like the dashboard, but I worry about mixing things like delivery performance and happiness together. Do you think both are required to measure a “transformation” as we have mentioned in this thread?

#17

I have concerns with what I see.

I’ll make no judgment of course, as I have zero context. And I’m not foolish enough to think I have correct opinions or answers.

However, I believe the four metrics in use are either proxy variables or harmful to measure.

#18

I may have delved too far down the hole of “how can one measure the results of an agile team?” instead of “how can one measure an agile transformation?” I suppose I’m having a hard time with measuring a transformation because, to me, you’re attempting to measure how a person or group thinks.

#19

YES!!! Exactly. :slight_smile:

There’s a strange juxtaposition we face in the community of trying to not focus on delivery metrics while at the same time still prioritizing delivering working software. My favorite phrase is “shipping cures all”.

Zach and were alluding to this, but delivery metrics are fine. Happiness metrics are great. When they are mixed I just wonder how the numbers are perceived by execs.

Lean change management by Jason Little encourages measuring change based upon the outcomes you want teams to achieve. If you can start by what behaviors ar desired, you can then run experiments to achieve them. More or less. :slight_smile:

#20

I know the concern well, and I myself have baggage about metrics. They can be abused by those in power and gamed by those in the trenches. Just as user stories shouldn’t be requirements but the start of a conversation, the intent of the metrics is the beginning of a conversation as well. Judgement made purely from the data is missing the point and sometimes dangerous. Additionally, the idea is if you excel in one area, another will likely suffer. For example, the team doubled their speed from one sprint to the next, but their happiness plummeted. Why? They doubled their speed by working 12 hour days, and now they’re burnt out.

For predictability and happiness, I think we have good measures. For quality, it’s abysmal, and we’re looking for better ways to measure it. For speed, that depends on how you feel about the no estimates movement. Velocity versus issue count both have their merits.

Bear in mind that this is v1 created with just a few days of work but pulling data from a myriad of systems. We’re now curious how teams will use the data to determine how/where to adjust.

#21

One more thought. The data is for the teams, not for execs. Oftentimes, we measure changes in our teams as positive or negative based entirely in an emotion. Wouldn’t it be better to temper that with some amount of data?

#22

Definitely interested in participating in a podcast! Let me know how I can help. Thanks!

#23

Hey Zach - can you explain more why you feel that measuring maturity is dangerous? We’ve only been running our program for about a year, and we’re just now really ramping up the rollout to more of our teams, so if there’s some watchouts we should be on the lookout for, I’d definitely appreciate the insight.

#24

For those that said they’d be interested in a podcast (Zach, Tanner, Jen Meyers)…let me put together a high-level agenda and post for review and then we can schedule the conversation. Judging by this thread alone this is a hot-button topic so I’m curious get everyone on a call (potentially arguing with each other :slight_smile: ). Look for an agenda early next week in this thread.

#25

Jen,

I won’t speak for Zach, but from my perspective, the dangerous part isn’t wanting more insight into maturity. The dangerous part is using questionable metrics to determine how mature a team is.

At my company, we use the Agile Maturity Model, which is a self-assessment tool to measure how mature a team is on the journey. It doesn’t take into consideration anything like velocity, cycle time, happiness or anything of that nature.

Those items are all valuable to measure and can help with things like capacity/release planning (I know, no duh), for the record. I also don’t want to knock Tanner’s dashboard, because those are again awesome things to put together for the team and leadership. Maybe just to tell a different story.

Make sense?

#26

That definitely helps! We use a similar self-inspection spreadsheet that’s focused mainly on measuring the team’s progress towards practicing and exhibiting the principles. It also doesn’t look at anything like velocity or happiness - I can see how those types of metrics can be tough to really measure accurately. Thanks for helping to clarify - I appreciate the additional insight!

#27

It’s dangerous because measuring a concept like “agile maturity” is open to many interpretations, definitions, and milestones. Even things like measuring principles… I question the benefit. Mindset shift isn’t a goal or destination.

I look at the people I admire (like Chris, for example) and realize I’m an agile novice. I gain new insights daily from practitioners around the world. I find myself lost in thought for days after listening to conference talks or reading blogs from our community.

What’s my agile maturity? What really matters here?

For “maturity” I tend to favor methods like a Dreyfus model, where “stages” are primarily described by behaviors. But even then, just because a team might be higher on the scale, doesn’t mean they’re necessarily higher performing.

“Agile maturity” models that I’ve been exposed to tend to treat people where “higher on the scale is better.” I don’t think that’s fair, appropriate, or remotely useful towards the goal of improving working conditions and results with agile.