I have been thinking about this question for a while now and after reading this interesting article called Stop Using Story Points I finally decided to put my thoughts in a post. I have worked at half a dozen IT shops over the last three decades and have seen a wide variance in what companies measure. I have seen companies that measure nothing, companies that measure everything, and those in between. There is no secret formula for how much one should measure but the lesson I have learned through the years is that metrics don’t always lead to the desired outcomes. We all know about metrics in the help desk area where mean time to close leads to faster ticket closing but not necessarily to customer satisfaction and actual problem resolution. Often metrics focus on getting stuff done or checked off a list but is that really our desired outcome at the end of the day?
I posted a question on Twitter last week
I received a number of responses on Facebook and Twitter. Many focused on story points and velocity which is what many companies focus their metrics on these days. One colleague of mine posted the answer I was looking for:
Productivity, quality, value, team/individual (team health, peer 360 feedback)
Quality, value, feedback! Those were the things I was looking for. Many metrics focus on output or on how fast we can get a product out the door. Focusing on those metrics alone (velocity, function points, lines of code, story points) without measuring things like customer satisfaction, quality, value can create outcomes equivalent to a customer buying a fake Rolex watch like the ones you find in China Town in New York City. They are cheap and look good on the surface but they are of low quality, break easy, and leave the customer feeling ripped off.
Purpose of metrics
There are many metrics which have value but should not be driving strategy. Anybody who has worked at a company of any size has likely run into the chore of measuring capital vs expense. The purpose of this metric is to differentiate between innovation type work which can be depreciated on the books versus maintenance which is pure cost. This is a meaningful metric for accounting and can paint a telling story for the leadership team if maintenance is alarmingly high, but this metric should not be driving business or technical strategy. Too much focus on this metric as a core business driver can force an organization to under commit resources to fixing issues and maintaining systems. The end result might lead to an undesirable outcome of poor quality and reduced customer satisfaction.
It is important to differentiate between metrics that provide insights and metrics that drive strategy. If a company’s strategy is build the best quality products, focusing too much on Capex vs Expense will drive the opposite outcome. Another one of my favorite examples stems from back in the 90′s. Back then metrics like LOC (lines of code) and FP (function points) were core metrics that many IT shops were measured on. At one company we were charged with increasing FP by 1.5X per year. This drove some of the most insane behavior I have ever seen in an organization. The way function points worked was value was assigned to work units based on visibility to customers. Backend processes like optimizing a VSAM processing job that took 12 hours down to 5 minutes was worth very little if any FP. On the flip side, UI development was worth big points because it was perceived as valuable since users could see it. This drove a behavior that minimized performance tuning and maximized UI changes whether they were needed or not. The client put so much pressure on my company to increase FP each month that most business and technical decisions held little concern for quality and customer satisfaction and was all about FP generating features. Entire teams were devalued if their skillset didn’t align with UI development. This was just pure insanity. Our customers would say we want feature XYZ and we would brainstorm on how to generate the most FP instead of building the best software for the customer.
That is an extreme example but many modern day metrics drive the wrong outcomes too. At my friend’s company, if they measured only productivity (story points or velocity) they would likely drive outcomes similar to my FP example to a lesser degree. The fact that they include metrics on quality, value and feedback means they are focusing not only on shipping product out the door but also on the quality and the customer experience. At the end of the day, isn’t that the real outcome we are looking for?
I would add one metric to my friend’s list and that is customer feedback. It is one thing for us to give each other great feedback internally, it is another to get real customer feedback. That is the feedback that matters the most because those are the people paying your company’s bills. It is important to note that gathering, reporting on, and reviewing metrics can be time consuming and takes away from actual development time. My recommendation is to focus on metrics that drive desired outcomes but don’t consume the team’s time collecting the data. Use tools to make it easy to collect and report the data and don’t let metrics collection interfere with developer output. Remember, every second a developer spends gathering metrics he or she is not being productive. And finally, make sure the desired outcomes are the goal, not the metrics.