Pros and Cons of Outsourcing QA

I saw a question on LinkedIn early this morning on a topic I was planning to cover “Pros and Cons of QA Outsourcing” – I jumped to the answer and typed up an answer while on BART, ironically it turned out to be too big for LinkedIn and so I decided to put it here. The answer is not complete and I am planning to come back to it some time. In meanwhile here are my thoughts:

1) Outsourcing QA is often a meaningful thing to do, an easy way to start and a potentially very dangerous trap. In particular companies that shed internal QA resources and move their QA operation abroad typically pay the price in knowledge loss and ultimately in degradation of the product quality. I have seen it on numerous occasions. QA engineers in well run teams often have better product knowledge than any other part of the organization, and offshoring that knowledge falls in a category of “outsourcing crown jewels”.

2) Companies in US often consider outsourcing QA for all wrong reasons, like for example “Cost”.  Cost advantage is just a myth, see my earlier post for details Outsourcing Myths: cost advantage. And going offshore for wrong reasons is guaranteed to give your wrong results.

3) QA as a subset of IT field offers a few interesting dynamics

a. It’s one of the most important areas of SDLC which typically is given the least attention.

b. The average quality of talent pool is dreadful, independently of geography. There are many reasons for that, for example here in SF Bay Area one of the main reasons is huge pollution of the pool by fraud and mediocrity – I met 100s of people who fake their QA background or think that a couple months of homegrown education makes them top notch professionals.

c. A perception of QA skills/occupation as a substandard one. It takes as much IQ to become a solid QA engineer as a Java developer, maybe more, but what can you do about perception? As one the best QA guys I’ve eve met put it talking about his resume “going to QA is like going to morgue – there is no way back”. And why don’t developers enjoy testing? Because it’s hard, it takes serious effort to put together a decent test harness, to organize your code, etc. Yet look at the average salaries, at layoff patterns, offshore dynamics – the trends are obvious.

4) Good QA engineers, automation specialists, functional testers, etc. like any other good resources are not easy to find, considering the quality of the pool, much harder. So it’s no surprise that that many organizations after interviewing a couple dozens of key-punchers who can’t tell the difference between priority and severity and ask for $90K move to offshore. The problem with such decision is obvious – it is as difficult to find good testers in Bangalore, Kiev or Beijing as it is in Boston. Of course a large supply of IT talent in countries like India and China makes it so much easier, yet the vendors in these countries have a plenty of own issues and challenges. Outsourcing the problem will not eliminate the problem, it only passes it to a different organization which is hopefully is better equipped to deal with it… But is it? Is your perspective vendor better equipped to do your job? How do you know it? By their brochures? Having interviewed hundreds of engineers in Asia, Eastern Europe, and Latin America I can tell you with certainty – far not ever vendor is equipped to do so, many of them are squarely in a business of selling mediocrity in bulk, and the quality of the resources is far less impressive comparing to what you can extrapolate based on what you see locally.

5) Having said all that I have to state that I still outsource a lot of my development and QA activities and get great results from my partners. There are a few ingredients to that success story, here are the most important:

a. Rigorous vendor selection process with the focus on “the match” between my organization and vendors’. Search for the match on multiple dimensions.

b. Resource augmentation and joined teams rather than complete outsourcing.

c. Abundant communications in all forms with fair portion of face-to-face meetings and on-site / offshore swaps

d. Control and ongoing preventive maintenance in all aspects of the engagement.

e. Adjusting SDLC to accommodate for idiosyncrasies introduced by offshore.

f. A “disposal outsourcing” model that I worked out with my one my partners – augmentsoft panned out quite well in QA arena.

g. Working with nearshore partners was much easier in many aspects especially when running agile projects.

Offshore Destinations: Russia

I was born in Moscow, USSR and the word “Russia” in my mind associates with a large empire of 15 republics. Things since than have changed dramatically and referring to some of the parts of ex-USSR as Russia is not just politically incorrect. Yet you are likely to hear about Russian outsourcing even if the ODC is located in Minsk, or Kiev. As a matter of fact in many respects outsourcing landscape of Byelorussia, Ukraine, and Russia has a lot in common. More so, large outsourcing organization such as ePAM, Luxsoft, and others have offices in these countries. Most of other countries of ex Soviet Union do not play significant role in offshore market, some due to low density of IT talent, some due to high cost. While offering in these countries exist, and you may find great providers in Estonia, Moldova, and others, in terms of outsourcing statistics these countries would be a rounding error. With that in mind let me cover some Pros and Cons of doing business in Russia.

  • Infrastructure. IT infrastructure in large cities of Russia is very good; smaller, second tier cities lag behind, the difference if pretty dramatic. Generally today you will find sufficient network bandwidth, stable connectivity, and solid pool of Sys Admin talent that would allow you stay in touch with your ODC. The cost of it will be not inconsequential though and needs to be taken into consideration. A very important aspect of infrastructure which you need to asses is vendor facilities – it is difficult to find well equipped offices with quality server rooms, etc. that is especially serious for companies with offices in second tier cities. In my view Pros here outweigh the Cons.
  • Operating Environment. Running offshore engagement with Russian ODC will offer many operating challenges even if you stick to tier one cities (for the purpose of this discussion that’s Moscow, St. Petersburg, Kiev, Minsk). Getting to these cities is fairly easy, they offer great selection of hotels, solid municipal infrastructure, and … mind boggling prices. As I heard Moscow has been recently awarded with a title of the most expensive city in the world with St. Petersburg following it closely. Second tier cities are substantially cheaper but you get what you paid for in terms of quality of hotels, food, transportation, etc. Another issue to be aware of is high crime rate (accidental traveler be aware!) and very high rate of corruption. Corruption could become a very serious obstacle for models involving ownership of the resources such as BOT. With caveats considered I would still put Operating Environment as a Pro of doing business with Russia.
  • Skills Availability. That is in my view is one of the weakest traits of the region. First at a very high level, Russia produces IT resources at a fraction of speed of the countries such as China and India. This problem is exacerbated by fairly consistent internal demand for IT resources and high geographical dispersal of the talent pool. In large degree Russia talent pull is already exhausted. Pretty much everyone who is interested in working in offshore organization is already working for some client, often for several, as many of talented engineers work several jobs, moonlight or find other ways of get themselves reasonably compensated. Finding software aces is challenging even in second-tier cities, in the first tier cities it’s practically impossible.
  • Cultural Compatibility. My experience in that arena has been surprising to say the least. I left Russia in ‘91 as an accomplished technology professional with almost 10 years of experience under my belt. I had not expected to have any problems in dealing with companies in Russia and yet I found it easier to work with companies in India instead. Some of my greatest pains came from several areas of communication / work related behaviors.
  • Customer is always right… Maybe, but not in Russia. As a matter of fact the vendor seems to always know what I want better than I do.
  • Being “Politically Correct” is not a Russian way. However, while I prefer straight forward communications I do not enjoy when my vendor is rude to me or more so to some of my employees.
  • Work ethics. Very sensitive topic, I have seen many great, hardworking developers in Russia, but unfortunately they seem to be outnumbered by short-timers with “get money and run” attitude.
  • There is one interesting aspect of Russia’s culture which while “positive” contributes to the difference – attitude towards education. It is amazing how many highly educated people you find among Russian developers and even QA engineers. I am not talking BS, I mean Ph.D. and above. While by all means commendable quality the negative impact of it is actually multifold: theoretical approach to problem solving, abandoning career for the sake of education, investment in education at cost of work skills, etc.
  • English Skills. In my opinion English skills of Russian outsourcing community are at the level you would expect them to be with a typical bell curve distribution and the median being at acceptable level.  Chances are you won’t have problems understanding developers and would be able to carry on a rich conversation with account managers and other client facing resources.
  • Rates. Rates of Russian development workforce vary greatly depending on location. Rates in T1 cities are very high, often making Russian outsourcing to be cost prohibitive. To deal with this issue many T1-city based vendors diversify by opening locations in small cities.  Rates for smaller city are as the standards of living in those cities – they fall off the cliff as soon as you move 100 miles outside of the tier one city boundaries.  However resulting rates continue to stay on a high side comparing to India’s.
  • Resource Turnover. Turnover tends to be on a low side comparing to India especially in a T2-T3 cities. The trend is however discouraging – according to what I hear from my network the turnover rate has bean steadily growing correlating to growing demand and increase in expected standard of living.

Let me close this post on a positive note covering one of the most important Pros of Russian outsourcing community – its Technical Capability. For many reasons Russia IT community in many cities in Russia offers above average technical capacity, innovation and creativity. That is particular notable for boutique vendors from St Petersburg, Moscow, Kiev, Minsk and Novosibirsk as well in the top echelone resources from lagre Russian outsourcers.

Offshore Vendor Selection: Choosing the Destination

Selecting a destination could be quite simple if you have strong drivers pushing you towards certain geography. Maybe your engineering team is predominantly Chinese, or maybe the board of Directors is firmly set on India, or maybe you have vast network of industry connections in Romania… Almost any geography can offer a variety of companies among which you could find that great match for your needs. And what if you do not have strong bias towards a particular location, where should you go? Let me start with a few quick tips here:

  • If your risk tolerance is low and/or your organization is new to outsourcing go to India, you can not get fired for hiring IBM.
  • Go to India if you have to choose on a spot, or have little knowledge of outsourcing, or have to deal with large scope ERP implementation, or … as a matter of fact if you have to ask this question chances are you should consider India as your top destination.
  • If you are not an outsourcing neophyte and ready to invest time in research and vendor selection a whole new world opens up to you, with some decision shortcuts:
    • If you are looking for partner that operates as extension of your team on an agile project consider near shore, e.g. Latin America
    • If you are looking to outsource large volume of manual regression testing consider China or Philippines.
    • If you have IP intensive engagement consider Israel
    • Are you looking for resources for mobile development? – Consider Eastern Europe.
    • Are you very cost sensitive and ready to deal with immature vendors, consider nascent territories: the countries such as Pakistan, Vietnam, Macedonia, etc.

Of course those tips are just ideas, the country selection needs to be taken exceptionally serious as it is one of the most decisions you make in the vendor selection process. While selecting the country I recommend using weighed selection criteria approach:

  1. Identify your selection criteria; keep in mind that the criteria should be country specific rather than company specific.
  2. Give each of the criteria weight – the rating of its relative importance for your organization.
  3. Define an initial list of countries you are willing to consider. Keep it relatively short to limit the rating efforts.
  4. Rate each of the countries using latest information you can find. Industry analysts’ reports would be the most helpful, consider Internet research and outsourcing associations, as well as the common sense.
  5. The rating for the country is derived as a sum of ratings for each criterion times the weight of the criterion.

Below is an example of such analysis performed for a mid-sized software development company in Washington, DC area. The company was looking for a partner to perform substantial on-going customizations of their SaS product. Scope of each customization project would keep a small team of Java developers and QA engineers busy for ~3 months.

After the list of the countries has been agreed upon we started with a very rough analysis and then detailed ranking for a shorter list.

Initial List


Country

Infrastructure

Operating Environment

Skills Availability

Cultural Compatibility

English Skills

Rates

Resource Turnover

Political Climate
Time Zone

Brazil
Med Med Med High Med Low Med Med High

China
Med Low Med Low Low High Low – Med Med Low

India
High High High High High Med High High Low

Israel
High High Med High High Low Low Med Low
Pakistan Med Low High Med High High Med – High Low Low

Philippines
Med Low Low Med High Med Med Low – Med Low
Romania Low Med Low Med Med Low Med Med Med

Russia
Med Low Med Med Med Low Med – High Low Med

Ukraine
Low Low Low Med Med Med Low – Med Low Med

Refined detailed rating


Country

Infrastructure

Operating Environment

Skills Availability

Cultural Compatibility

English Skills

Rates

Resource Turnover

Political Climate
Time Zone Country Rating

Weight:
4 8 10 5 8 6 9 4 7

Brazil
7 8 6 8 7 6 8 8 10 458

China
6 6 7 5 3 10 10 7 6 411

India
10 10 10 10 10 7 4 10 6 510

Russia
7 5 6 8 6 4 6 7 7 371

Ukraine
6 4 5 8 5 5 7 6 7 352

Based on this analysis we narrowed our search to 3 countries (Brazil, India, China), which in my view was still excessively broad and could result in a very expensive vendor selection process.  By the way the winner of the engagement was a vendor from Brazil, and it remains to be seen how well the project works out.

Dealing with Turnover

Turnover impact on the cost for an offshore engagement could be dramatic.  It’s fairly obvious: any change in resources on a team triggers changes in the team dynamics, new resources need to be ramped up on technology, project specific and domain knowledge, etc.

Depending on a type of the assignment new resources will display lower productivity for weeks or even months. For a regular full time employee on development task the cost of replacement is typically estimated at 3 months of fully burden salary plus recruitment fees. The cost of the replacement in offshore scenario depends dramatically on contract arrangement and vendor capabilities. Interestingly enough it could be substantially lower than in captive resources scenario. In my view outsourcing companies could turn the issue of turnover into a competitive advantage. I have not seen too many that managed to so yet.

To deal with the turnover you need to start early during vendor selection stage and do not stop working on it till the engagement is closed.

Vendor Selection. Pick the vendors that have turnover under control; do not just take their word on it, research all aspects of employment lifecycle and derive your own conclusions. Here are some of the questions you need to get the answers to:

  • What is the vendor’s employee sourcing strategy and tactics?
  • Does vendor have any advantages in local market in terms of acquiring and retaining employees vs. competition? Or same question from a different angle – what are the employee choices in local job market?
  • What employee retention mechanisms the vendor has in place?
  • What the vendor does to counter results of the turnover? In particular what are the knowledge retention mechanisms? Cross training? Etc.
  • What were actual turnover ratios for reference (especially unsolicited) accounts? How did vendor acted upon reducing the negative impact of the turnover?

Contract Negotiation. Now it’s time to ask the vendor to put their money where their mouth is. Here are a few elements you may consider for including into MSA:

  • Clear definition of the turnover ratio with a penalty for exceeding specific benchmark.
  • Key employee clauses. You need to cover definition of the key employee, specify minimum retention period for them, and identify process of replacement in case of their departure. I would consider covering two scenarios – force major (vendor can’t do much about loosing the employee) and transfers (the vendor elects to move the employee to a different engagement) with much more considerable penalties for transfers.
  • Regular team member clauses. Those would be similar to key employees with focus on the process of replacement and much softer penalties.
  • Transparency clauses. You want to know about employee departure as much ahead of time as possible, you wan to be able to get to the bottom. Maybe you want to be able to do the exit interviews. You might be interested in right to connect with team members directly, but be prepared for serious fight here unless you are working with small vendors.

Managing the Engagement. The main point is not to rely on your vendor’s adherence to the contract but take an active role in increasing retention of the team members at multiple levels. Of course that must be done in a concert or at least not in a conflict with the activities of the vendor. Most reasonable organizations do not intend on breaking the contract and in a large degree are interested in minimizing the turnover the same way you do, maybe with just a few exceptions. These exceptions come from market pressures and internal constraint, the better you understand them the more you can do to counter the issue. Here are a few reasons for these exceptions and some tips on dealing with them:

  • The concept of seniority in an offshore organization is likely to be different from what you have in-house. For example a java developer with 6 years experience would be considered very senior in China and expected to act as a tech lead on the project with 20 developers. So forcing that developer to be a junior member on a team of 5 would be an insult which is likely to lead for him to quit one way or another. You need to be cognizant of that and other similar trends and form the team which would create a favorable environment for the team members not on your terms but on their terms.
  • Same problem has a different angle: let consider a vendor’s viewpoint. As a practice manager I need to leverage my resources well. If I have a senior tech lead he is expected to run a team of 20 people. In that case I can afford to charge my clients some reasonable rate for his time as my losses would be more than offset with profits I make on junior members. As you can imagine a client’s request to form a small team with several senior members is not going to make me happy… One of the ways to deal with it is to be much more flexible on rates and find alternative / additional methods of compensation. More important if small team with very experienced team members is your preferred scenario – you need to pick vendors who are ready and interested to work in that model (and not just because they told you so!). Consider boutique consulting organization, Eastern Europe and Latin America.
  • Another factor to consider is employee engagement. Long term development projects, especially large scale maintenance ones offer different challenges over time. Initially they could be interesting for people motivated by technical complexity and vast scope, but after a while these challenges disappear replaced by mundane repetitive tasks. So it’s no surprise that some of the team members are ready to fly when you just starting to rip the benefit of their experience and knowledge. In my experience the best way to deal with it is by selecting resources with personality / mind set match for the project. There are plenty of people who a great and maintenance projects and enjoy that work as well, they might be a bit slower in uptake, but that is the bullet you need to bite.

In a large degree to reduce turnover of your offshore team you can do many of the same things you would do for your own staff, often by the vendor’s hands. Here are just a few things you can consider:

  • Compensation & Gifts. Remember a $1K bonus goes much further in India than in Indiana. A few technical books sent directly to a developer would earn disproportional value in loyalty.
  • Classic motivation factors: advancement, recognition, and achievement. Recognition is particular simple and pays off incredibly well.
  • Team building. A few things with huge impact: joined development activities like SCRUM style meetings, offshore visits, especially for local team members with specific tangible objectives such as training or k-transfer, bringing best off-shore team members on-site, etc.

Outsourcing Myths: Turnover Ratio

The impact of turnover on the total cost of outsourcing is difficult to overstate. Of course you know that and put a turnover question as one of the most important ones in the beginning of your RFP. You look at the proposal that just came back from your vendor and see 18% as the response, “whew, I think we found our guys!”… Welcome to the murky world of turnover ratios. The sad part is that this answer may mean very little; being the most infamous curse of offshore engagements the turnover ratio comes with a few extra traps.

The most frequently overlooked issue strangely enough is the meaning of the “turnover ratio”. When your prospect vendor tells you that their turnover ratio is lower than the country’s average (high changes that’s exactly what you are going to hear) what does the vendor mean?

On one of my recent engagements with a reputable company in Noida, India the staff on the project changed at an amazing rate – while working with 10 member team for about 1 year we saw over 20 people, and only one person stayed on the project from the beginning to the end. No matter what formula I tried apply to that situation it did not seem to align with 18% stated in vendors proposal. And yet every account review my vendor pushed the idea that the turnover ratio was not out of bounds. Ah? ‘Well, Nick:

  • We moved Rajiv and Venkat off the project because they were not performing job well enough;
  • Ramki’s mother got sick and he had to quit to do the right thing;
  • Shushma got married and moved to Hyderabad…

And so on and on and on…

As it turned out my valued partner had a completely different view of the turnover ratio. My guess is that they  calculated the ratio based only on the number people who’d left the company for competitors.

Another trap worth mentioning is an internal transfers.  What difference the company’s average turnover rate makes if your project turnover exceeds it by two or three times?  I’ve seen that numerous times and in a large degree it’s unavoidable. The vendor will move people around to increase their utilization, to appease the loudest customer, and to keep employees motivated.

And one more trap to mention is key resource turnover.   If your team has an average turnover of 20% (you lose and have to retrain 2 people a year on a 10 member team) it might not be so bad if these two are junior QA engineers. What if these two spots both belong to the tech lead on the project?  You find a great TL, he comes to your site for knowledge transfer and after two months go back to India just to resign the next week, two months and countless meetings later another one comes to your office, goes through K-transfer and goes back, and then gets hit by a typhoid fever?

Recognizing that there is much more to turnover than just a percentage sign is a huge step forward.  Dealing with turnover is a much more complex issue and a rather large topic, so I’ll cover it in the next post.

Ready to Outsource?

Organization outsourcing maturity is one of the most important ingredients for an outsourcing initiative. Attempts to force outsourcing to an organization that is not prepared / not ready for it are likely to fail and chances are with a lot of collateral damage. That is true for any outsourcing initiative, not only for offshore; well, it applies to pretty much any organizational change, but my focus is on specifics of offshore though. What does it mean to have your organization ready for offshore outsourcing? Here is a high-level checklist to consider:

  • Solid justification / objective reasons for outsourcing. Jumping into outsourcing following the lemming instinct would end up with pretty much the proverbial result. The industry is full of examples when organizations went after outsourcing just because it was “the best practice” and ended up with massive losses – financial, customer satisfaction, knowledge, etc. Take a look at “reasons for outsourcing” or “my reasons”, go through your own list, and make sure that you have solid objective reasons to even consider offshore. Go through a thorough and very conservative what-if analysis and unless you see a substantial ROI set the idea aside.
  • Sufficient budget. You most likely heard about a hockey-stick or a J-curve – a curve / a pattern of success in business. Look at the budgets you have in-hand with the same perspective. Success requires initial investment and ability to sustain negative cash flow for some time, you need to survive that dip, before you start realizing ROI.
  • Executive commitment / sponsorship. Lack of executive commitment is certain to ruin your offshore strategy. If your executives do not accept the realities of offshore savings, if they are not prepared to wait through a negative stage of the J-curve you are certain run out of budgets prior to getting offshore initiative on the path to success. It’s even worth if your executive team doesn’t buy into offshore idea due to political or personal preferences. The last thing you want to do is spear head an initiative that’s only purpose is to show a bad example.
  • Team understanding, support and commitment. Chances are you can’t do it alone and through out all stages of outsourcing you will need to rely on support of your team. I do not think I need any mountaineering metaphors here. However, strangely enough, I have seen many times when offshore initiatives were driven down the thought of a core team, at the expense of the employee morale and wellbeing, without support and consideration. In my view that results at best in malicious compliance, more frequently in clever or blatant sabotage.
  • Processes and procedures. Often undermined in small organizations immature processes and procedures, in particular in SDLC / project management, are almost certain to result in offshore failure. Large process-savvy organizations are also not immune as the need for the processes is proportional to organization’s size. The most important aspect of the processes is communication channels and their efficiency, inter-departmental handovers, and internal roles and responsibilities definitions.

Add to FacebookAdd to DiggAdd to Del.icio.usAdd to StumbleuponAdd to RedditAdd to BlinklistAdd to Ma.gnoliaAdd to TechnoratiAdd to FurlAdd to Newsvine

Fundamental Laws of Outsourcing

A while ago when answering a question on LinkedIn I suggested applying three well-known principals to managing offshore engagements. I call them Fundamental Laws of Outsourcing a.k.a. FLAWs of Outsourcing (FLOs for short). These laws in my view are as strong as the law of gravity, and as you know if you attempt to ignore it you will at best end up with your face in the dirt…

  • The first FLO is The First Murphy ’s Law: “Nothing is as easy as it looks.”
  • The second FLO is The Second Law of Thermodynamics (Entropy Always Increases)
  • And the Third FLO is the First Law of Military Communications: “If an order could be misinterpreted, it will be”.

That really sounds encouraging you may say. How can you ever consider working with offshore under these laws? Well, back to gravity – it doesn’t seem to prevent us from being ultimately successful in our lives and have fun in the process. So here are just a few basic tips…

1) To deal with FLO # 1consider a few things –

a. Starting with a stellar contract (MSA) which addresses typical offshore traps (quality benchmarks, attrition remediation, knowledge transfer / retention guarantee, etc.).

b. Setting expectations right (your management, your team, and your own). “Right” in this context means as low as possible. For example: there will be no cost savings, the work load will increase, everything that could go wrong will go wrong, things that just can’t go wrong still will, and no matter how low your set the expectations your vendor will surprise you.

c. Setting an “exit water mark”. No matter how long you march down a wrong road you will need stop and go back. In case you made a wrong decision there is a point where you are better off by cutting your losses short.

2) The second law of thermodynamics applies to any organization or project. You might remember these quotes:

The uninspected deteriorates.
– Dwight David Eisenhower

The only things that evolve by themselves in an organization are disorder, friction, and malperformance.
– Peter F. Drucker

In offshore outsourcing the second law of thermodynamic exhibits itself as consistent degradation of quality of services in absence of non-stop energy applied from the on-shore. Consider uninterrupted control from a dedicated resource on your side (make sure it’s someone you trust and who has your company – not the vendor’s – interests in heart). Everything from timesheets to hard core deliverables to be scrutinized and verified. The only way to stay ahead / prevent quality decay is to apply pressure on the vendor even when things are still going (seemingly) well.

3) The FLO # 3 applies to all aspects of communications and in particular to the project/product requirements. Any ambiguity in your documentations, specifications, processes, procedures, and especially verbal instructions will be (innocently) exploited to create maximum damage and cost increase. Consider crystal clear communications and small scope of controlled deliverables. For example, very short project phases, interim builds, etc. Work as many control / feedback points in your process (SDLC) as you can possibly afford.