A few days ago starting from a comment to my post I found a very interesting discussion of metrics on 360° Vendor Management. In his post Tony covered a few golden rules that are important to consider when introducing metrics in vendor management. I agree in general with most of the principles covered in the post and highly recommend looking at it as well as other materials publish in the blog.
While reading the post I came across of one phrase that triggered a serious of thoughts that I want to cover today. (Metrics allow you “Moderate expensive overperformance that the vendor need not do. Remember – you pay for the extra quality, which may not be tangible.” )
There are multiple negative aspects for vendor over-performance that you should consider:
- There is a direct, potentially a substantial cost to additional effort that was undertaken to deliver against benchmarks that are higher than expectations. Imagine that a round of regression test with expected coverage costs $50K increasing functional coverage (not code coverage) delivers exponential cost increase.
- Someone said that perfectionism (over-performance) is little but a reason for procrastination. Consider an ESL example – learning 500 words of English (which would be essential and sufficient for basic communication) would take an average person 3 month. Moving that 3,000 that would institute a decent conversational ability would take about a year. In 5 years of intensive immersion study one can get to 25K-30K words… English has over 600,000 words, learning those would take several lifetimes… So if you wait till you get to 100K vocabulary before you start you life in the country you might never have a chance to do it
- There is law of diminishing return on effort put against achieving higher benchmarks. Continuing with ESL example – most of educated people in this country get by with less than 30K words… There is a substantial difference between paper written by someone who mastered 500 words and someone with 3,000; there is very little difference between 20,000 and 30,000 and… none what so ever between 300,000 and 600,000 word masters… Diminishing return translates into cost in rather dramatic manner. Consider application of 80/20 rule to QA – it takes 20% of the test cases to find 80% of the bugs associated with a particular round of testing. So cost of finding last 20% is 80% of the total. (Of course I do not mean 100% of bugs existing in the system – for any decent size system getting to 100% is as futile as attempt to reach absolute zero)
There is another potentially lethal aspect to over-performance: putting an unnecessary strain on the vendor. Consider work of QA team working against stable team beyond of point where the benchmarks are reached – there is tons of serious work and zero appreciation from the client. Brain-numbing and de-motivating, with guaranteed negative impact on the overall performance and at customer expense.
If I could have a dollar for every time I heard from offshore vendors about how they continuously exceed customer expectations! Well, who pays for that – it’s either the client or vendor’s team – and that means that in the end client pays anyway.
Over-performance is common and can happen in pretty much any area of vendor activity / deliverables, here are just a few of common examples:
- Technical design – documenting things that are much better illustrated by comments in the code or code itself.
- System documentation that goes stale and almost never used by intended audience.
- Excessive QA coverage, especially in QA automation resulting in gigabytes of scripts that have to be rewritten every time when the system undergoes UI change.
- Meetings, e.g. status reviews with 50 participants dozing off on conference calls.
- Developing, tracking and reporting on metrics that are never used by the customer….