As a tester, I never liked measurements. But that was before.

I never liked measurements because I’ve always seen them being used in a goal of surveillance or to blame someone or something.

I’ve been asked to track my time, just to be forced then to justify why I took so much time to do this or that or not enough on this. I’ve seen them used to show that we’ve missed deadlines, were not good at this or that but I’ve never saw them followed by advices and actions.

I avoided them as much as I could in the past. Defending myself to have to make reports I won’t understand the use or meaning.

I change company quite recently. Here, metrics are everywhere. Numbers and graphs are shown at every large meetings. Measurements were tapping on my shoulder again and I knew I won’t be able to escape it this time.
On a monthly basis the QA Weather is shown with numbers telling us things about incidents and bugs we had or how many bugs were directly reported by a client.

Last month, unfortunately, willingly to innovate and certainly with no bad intentions, those numbers were used to point at the teams with the highest number of open bugs at a precise moment in the different team dashboards. Discussing with the person who showed those numbers later, he explained to me that it was humor. Yeah, but it was a slack post and the humor tone was not that obvious. And still, what was the point of it? Why would you do that? My team was part of the one to be blamed according his statistics. And oh boy it made me angry!

I had put a lot of efforts building trust and gaining my place into the team. Even if my biggest effort was not to make them focus about bugs because they already had great habits on this one. But instill trust in the QA team was a hard thing to do. AND, in the past 2 sprints before this blaming QA Weather post, we closed 21 bugs. The 5 bugs remaining in our backlog at the moment were at a lower priority and under investigation. It was really not fair to be pointed at. 

Seeing this number put me into Disney’s Hades anger mode. With just a little slack post saying we were to be blamed because we had 5 bugs opened he could have just crash months of work of building trust and discourage the whole team to continue their efforts.

That’s why I didn’t like metrics. They hurt, they destroy trust, they depressed you. That’s what I thought.

But then, I attended AgileTesting Days 2022. Back to Unicornland baby!

The first keynote was Gwen Diagram’s on Happiness is Quality. A promising program. But wow! was I surprised when she told us that she likes metrics. Like a lot. And she did this with such enthusiasm that I had to know why. How could it be possible to show some love to those evil graphs, numbers and tools of surveillance? Especially when the one telling you that seems to be such a happy and fun person!

My curiosity was picked. I was intrigued.

So I chose to give metrics a second chance and investigate on why some of us love them so much.

That’s the real reason why I attended Michael Kutz talk titled “How (not) to measure quality”. Seems to be exactly what I needed, right?

At the beginning of his talk, Michael explained that, as I thought, metrics could be a dangerous thing to use if done the wrong way. He quoted The Goodhart’s laws which says “When a measure becomes a target, it ceases to be a good measure.” I learned more about it in this article from formplus which gives practical examples of it.

So, according to Michael, or more exactly according to what I understood from his talk, measures are just there to give informations. They are indicators for you to know if you’re heading to the direction you want or not.

And that’s why you don’t start with the metrics themselves. You need to start with the goal you want to achieve. What do you want to know? Then asks the questions that could help you have the right answer to this goal. And only then creates the appropriate metrics.

For example, if you want to know if you have a good product useful questions might be:

• Do our users like the product?
• Do they find it useful?
• Is it defective?

Then, and only then, you can find what would be a good measurement to answer those questions and with the answers you know where to investigate and take action.

He convinced me. And throught his talk I understood that metrics should help you, not disserve you. They should not be used to prove something or someone is wrong but to give informations and help you shape an answer to your problem.

Moreover, I recall now that Gwen told us that in order to know if her team produces good quality which would mean that her team is happy, she made a lot of surveys. Asking directly to people how they feel, what they think about their work, the product, their well-being in general, that an idea that seduces me. Take the info where it is. And it makes the metrics more human friendly.

I was still stunned by how measurements have been misused around and by me for all those years when I entered in Maaike Brinkhof’s talk on the Bullshitisation of Software Testing. And she drove the point home talking about how we’ve been asked to do things with no inner sense just for the sake of doing something.

I thought at this moment that’s where the bad and ugly measurements came from. Showing graphs helps to convince that you did something. Graphs are not fun and sometimes difficult to build. They imply some boring stuff like fighting with a spreadsheet. If you make them, that should mean you have an important job, right?

This is maturing for a week now. And what I’m sure about is that they convinced me that measurement are surely a great thing, but only when used with the right intentions. They have to help you answer the right questions. They are there to give you informations and help you improve. Misused it can bring stress, pressure and break trust within your teams and company.

They are a precious tool that should be used with precaution by the ones who know what they’re doing.

Addendum: On Friday we just had the recap of the last month with all the tech team and graphs were everywhere. It was then obvious to me that we are measuring things not really knowing why. Just because in Lean or in Scrum we have to measure this or that but we didn’t put any real goals behind it for ourselves. We’re just doing it because it’s written somewhere that we have to. But, I’ve understand that you have to measure what really serves you. Make the metrics your own. Not just applying something you saw in a book or course. So thank you for saving this for me Gwen, Michael and Maaike!