The GQM Approach to Metrics

Back to catching up with my SEETest 2008 posts.

The 2nd tutorial I attended was “Essentials of Test Managmement”, given by Geoff Thompson from Experimentus in the UK.

I have to admit to being slightly over-experienced for this tutorial, but I did give Geoff a bit of a hard time with my questions

There were about 80 or so people at the tutorial, which was way more than had been expected, so the hands-on part of the afternoon had to be dumped.

Instead, Geoff gave a 5-minute talk about GQM (Goal, Question, Metric) together with an exercise.

GQM isn’t new, but I don’t think it’s very well known, so in a nutshell, here it is:

The idea is that you first need to defined your Goals (the G).

Next you need to ask yourself “how will I know if I’ve reached my Goals?”, or “what are the Questions (that’s the Q) I need to ask in order to know if I’ve reached my Goals?

And lastly, you ask “what are the Metrics and Measurements (the M) that I need to gather in order to be able to answer the Questions?”

There are 3 phases after that, like gathering the data, but these 1st 3 phases are the important ones, since they ensure that you collect the correct data.

Sounds easy … but in reality it’s not quite that simple.

So here’s an example

It’s from the exercise that we did during the tutorial. In the exercise we had to devise a Goal, and then develop the Qs and Ms for it. We had 10 minutes for the whole thing.

The Goal: To reduce the cost of maintenance as a results of defects

The Questions:

1. What is the cost of maintenance per Quarter, over the past 2 years?
2. How many defects were detected per Quarter, and fixed per Quarter, over the past 2 years?
3. What % of these defects were due to new development, as opposed to old development?
4. How much does it cost to find and fix a new defect (per severity)?
5. What efforts have been made to attempt to reduce defects?
6. What % of defects could have been detected in an eralier phase to that in which they were detected?
7. What is the minimum ROI (return on investment) for reducing maintenance costs?

The Metrics and Measurements (note that some metrics can be used for multiple questions):

1. Number of defects detected, fixed, re-opened, etc. per Quarter
2. Average time taken to find, fix, retest (re-fix, re-retest !!), per severity
3. Cost of #2
4. List of efforts to reduce defects, and the results of these efforts
5. … (ran out of time)

So, I hope that this has given you some insight into the GQM approach.

Would you have written the same set of Questions for this Goal?  What Metrics would you have gathered to answer these Questions?

And if you’re already using GQM, send me some examples of what you’re doing, and I’ll put them up as a Guest Post on Debi’Z Blog.

If you enjoyed this post, please consider subscribing to my RSS Feed, to keep up with all future updates

Bookmark