First Metrics

For some reason this month turned out the highest traffic in the history of this blog. It is particular interesting considering that I have not been doing a good job keeping it updated. So I feel obligated to add at least one post before December.

Let me cover some progress I made with my offshore provider I inherited through the M&A process. In particular let me cover some metrics we have established in order to control the relationship. This set of metrics is a preliminary set based primarily on what my vendor could easily retrieve from the tracking systems they already have in place.

A couple months ago we agreed on several metrics and established high level benchmarks. I believe that you can not enforce benchmarks on day one. You should setup goals and see whether they are reasonable to achieve considering multiple factors ranging from abilities of vendor and your own organization to quality of the tools that are used to collect the info.

Here are the metrics we have collected so far:

For sustenance / maintenance tasks –

  • Average turn around time for issues by priority. That metric helps us control compliance with SLA we have in place.
  • End-to-end response time for issues by priority. That metrics also helps us control compliance with SLA.
  • Resource utilization. Helps both parties understand resource utilization / allocation / team effectiveness.
  • Defect removal efficiency. Somewhat generic QA metric helping understand the quality of the deliverables.
  • Effort / Actuals variance. Metric designed to control estimating process and SLA compliance as well.

For development tasks the metrics are slightly different –

  • On time delivery index. Obvious SLA relevant metric.
  • Effort Variance. Same as above just substantially more important for development projects.
  • Number of Code Review Comments. That one was easily available so we decided to track it. I do not have expectations related to its utilization though. The data behind the metric is just too ambiguous.
  • Requirements clarity index. I am interested in how this one pans out. It could be a good metric for measuring the quality of requirements process.

The metrics delivered to on-site team by the vendor once a month at executive reviews. There will be more metrics that we will track, but that’s not bad for the beginning.  I will keep you posted, in meanwhile here is how these metrics look in the monthly report (one of the slides for a specific project).

9 thoughts on “First Metrics

  1. Wow! That’s a great review and it takes some ideas from the post to a completely different level. Very interesting post, way deeper than the original… Thank you so much! It’s been a while since I looked at your blog and I regret it now ;) Oh, well, I guess I have a lot to catch up. Thanks again, nick

  2. This Blog was interesting… In my previous Experience we have been quite successful in managing multimillion $ deals as offshore. The way we deal matters than the metrics that has been shared. We have got great stories where we have been able to accomplish a revamp of an entire online system in 75 days… All days and night out by team… And I was amazed.
    If we are ready for looking at areas to improve and trying to see outsourcing as a profitable entity, here are my 2 cents:
    1 – Make vendors own our deliverables [I don’t see folks in US to slog beyond hours to make us GOOD to GREAT and that’s a survival point]
    2- Working on a T&M mode is always a wrong way that eats away our money and also the vendors’ don’t feel ownership. If your project is a T&M, I swear the team at US end also is responsible for not efficiently utilizing the efforts. It becomes more of one on one tracking than innovation capitalization which made my programs successful.
    BTW, your Graphs are good. But it could be good if you can publish Month on Month for past 6 months to see if we have really succeeded or model to change.

    Chan Tyong

  3. Thanks Chan, interesting comments. I am planning on covering progress with this vendor and metrics going forward, so stay tuned. Best regards, Nick

  4. Hi Nick

    I went through your blog and it was very useful (Lot of knowledge expressed). From the vendor management perspective and looking at your company lists (Top 10) in one of your blogs, I presume you are referring to India.

    The best win-win situation is Fixed Pricing than T&M as the accountability clause is very clear and 95% will lie on the vendor. The 5% is the risk that we need to cover from our end on Requirements, Acceptance Testing etc. [Ensuring Change Management from our end is minimal and that’s were we loose the cost in case we do so]

    Advantage: We don’t have to hunt for resources. In T&M, we always wanted to see the candidate and in Indian scenario, the service delivery practice would govern us and we can worry on deliverable than T&M management. The accountability is purely on the PM who manages the project for you.

    I think I have made too many comments in front of your blog :-). I am sorry to take this space once again.


  5. How do you calculate Requirements Clarity Index. Am looking for the formulae and how did it work to know the quality of requirements.

  6. thanks – very good question. The approach I’ve been using so far is based on couple elements that we added to ours sdlc – first one is ambiguity review, this is a process performed commonly by qa team. they analyze the requirements and identify all kind of ambiguities, e.g. the requirements say “in case if user doesn’t have X do Z” but doesn’t cover what to do in case “user has X”. the number of discovered ambiguities can be easily translated into req. clarity metric. the second element is “reason” attribute of a bug in your bug tracking, you can define a value “unclear requirements”; the number of bugs attributed to lack of clarity in requirements over some metric of release volume (e.g. man*month effort associated with the release) would be another good and relatively easy to track metric.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s