Using Contracts to Mitigate Offshore Risks

MSA – a “horizontal” component of an offshore contract can become a powerful tool in managing an offshore engagement and mitigating its risks. My approach to turning MSA in such tool includes several main steps:

1. Identify specific risks associated with the engagement. See my earlier post Top outsourcing risks as an example.

2. Rank the risks and select top ones; limit the selection to 5-10 items. The reason I recommend limiting the list is the cost / length of negotiation process.

3. Find out the reasons the risk mitigation is not in place / insufficient. You need to understand why this presents the problem for the vendor; without that knowledge negotiations are likely to hit an impasse.

4. Identify your preferred risk mitigation plan(s). The plan should include what both parties should do to reduce / eliminate / mitigate the risk

5. Insert and negotiate corresponding language in the MSA. Keep in mind that negotiating each of the topics may require multiple revisions and some give and take on both sides. Taking a win-win approach to the negotiation from early on is essential.

Let’s consider a greatly simplified example: Let’s assume that you are negotiating an MSA with Indian outsourcing company and after second step arrived with top two risk items: “Excessive resource turnover” and “Technical capability of the resources”.

Why is excessive turnover so common? Could it be avoided? Why don’t they (the vendor) just fix it? Well, they can not. The employment situation in India when it comes to IT resource is similar to what we’ve seen in Silicon Valley during the peak of DOT COM.  Can you spell Java? Hired! Inevitably job hopping becomes common… So, facing the facts, you know that there will be turnover on the project, and it will be higher than the 20% average you vendor told you about (see my post Outsourcing Myths: Turnover Ratio).

What can you do to deal with inexorable? Here are just a few options – maintain ongoing recruiting efforts, keeping staff on stand by, continues investment in crosspollination, knowledge management, documenting everything, etc. The list of mitigating techniques goes on and on. Your vendor is probably has a bunch of them in place. Well, it’s a perfect opportunity to ask the vendor to put the money where their mouth is.

For example you can ask for guaranteed replacement of the resource in two weeks. You can consider overlap of the resources in order to perform knowledge transfer for minimum of two weeks. You can ask for periodic audits of knowledge related documentation.

An important consideration to keep in mind: some of the turnover mitigation techniques employed by the vendor do not work in your favor. The most obvious one is moving resources from project to project or client to client in order to keep the resource engaged. I would recommend consider counter measures for example if the resources are moved off your project but retained within vendor’s organization some harsher penalties / longer overlaps applied. But you do not want to push your vendor against the wall making it financially unreasonable or preventing them from doing basically a right thing.

Here is a small example of MSA language:

Vendor shall not reassign any key resource providing Services for a period of 12 months after their respective start date of providing Services without prior approval from Client, provided that Client commits to the resource ramp up outlined in Section 5 of this Agreement. Key resources shall consist of resources critical to the Statement of Work and unless otherwise agreed, will be the Project Manager, Technical Lead, Business Analyst, Architect, and Quality Assurance Lead.

Let’s now cover the technical capability of the resources. Why that could be a problem? Well, try to find good developers in Silicon Valley even today – not easy by any stretch of imagination. Your vendor faces exactly the same issues exacerbated by several factors with huge competition from multiple dimensions – multinational corps, product companies, large offshore companies, etc.

This particular issue fall’s in a category “that is a fact, it is not my problem” but if I ignore the fact it will become a problem. In any case, the quality of resources is not something I am prepared to compromise on. So what could be my mitigation techniques here?

I typically ask for direct access to resources, right to interview and approve / disapprove, etc. That is a huge issue for many vendors though, most of the vendors do not want you to handpick the resources, for obvious reasons. So, it’s likely that you would have to offset it in some way, for example ask for interviews / etc. process for named key resources and allow vendor to deal with the rest of the team. You may consider some compensation (rate, T&C). Another approach could be setting performance benchmarks and holding vendor to those.

Here is a small example of MSA language:

For Statements of Work undertaken by Vendor on a time and material basis, Vendor shall obtain Client’s approval prior to adding any resources to such Statement of Work. Client will have the option of interviewing Vendor’s resources prior to their providing Services under a Statement of Work.

Offshore Contracts Basics

In general the language you put in contracts will not change the nature of the business, will not counter the Fundamental Laws of Outsourcing, and won’t prevent things going south. Yet it is impossible to overstate the importance of a well-written contract. The goal is to develop the contract in a way that it encourages / enforces desired behaviors and provides a framework for dealing with issues, complications, and disputes. That applies to both parties – the contract has to work for you and your vendor, in that light, considering the nature of the engagement, nothing is as important when developing a contract as keeping a win-win mind.

A typical offshore contract includes two major components: a Master Service Agreement (MSA) that acts as an umbrella document covering specific terms and conditions of the engagement, and series of documents covering the specifics. Individual tasks and assignments are usually covered by documents such as Task Order, Statement of Work, Work Order, etc.

An MSA typically is negotiated once and stays in power through the life of the relationship. There are many important elements of an MSA that define the fabric of the relationship. To some degree they are vendor and even offshore agnostic. These elements fall in a “vanilla” category and typically require just basic template and a lawyer. However even these items can create a serious obstacles and require tooth and nail negotiations. Here are the main items that fall in that category:

  • Term, renewal and extension
  • Legal framework / Changes in laws and regulations
  • Security and privacy
  • Confidentiality / Audits
  • Proprietary rights / IP ownership
  • Legal responsibilities of parties
  • Indemnitification

The second group of MSA articles defines specific aspects of the relationship and should be in general agreed to prior to getting legal departments involved. One of the reasons that should be done is that there is still a long way from general to specific T&C. For example people rarely discuss the penalty for late payments before MSA is on the table. When the initial draft is presented by the vendor the number is typically 2.5% monthly which is completely ridiculous – that is ~35% on annual basis and beats some of the worst credit cards.

  • Definition of services
  • Responsibilities of parties / roles of the parties as it applies to executing the engagement
  • Payments and other financial aspects, terms and conditions
  • Initiation / Setting up ODC and other (“hidden”) fees
  • Termination

This group of the MSA articles requires very detailed analysis as it will impact Total Cost of Outsourcing in the most dramatic manner. The last article in that list, Termination, requires especial attention and a legal eye. As a buyer of offshore services you want to make sure that you can get out of the contract easily in case anything goes not the way it was planned. This is not a symmetrical clause – the vendor’s right to terminate contract should be limited to legal or financial bridge of the contract on your part.

The last group of MSA articles is not typically found in the initial draft. These are clauses that are specific to your company and nature of the engagement. Developing that list and negotiating each point is not a trivial exercise, and deserves a separate post, which I shall have shortly.

Pragmatic Outsourcing vs. Gartner

You may have seen latest Gartner view on offshore destinations via Gartner rates offshore outsourcing hot spots or variety of other sources. I find Gartner view quite interesting, informative yet sometimes not very relevant to needs of software product companies.  In this particular analysis the information is very good and applicable to a large degree with exception of a few important areas where data is misleading when applied to IT and Software Outsourcing specifically for small to mid-sized companies.

An obvious disclaimer here – my opinion is based on a fairly limited sampling – my own observations, recent experience and analysis plus some supporting data from several people in my network with relevant experience and knowledge.

If review had been based on viewpoint of IT/SW professional the rating for Labor Pool, Infrastructure, Cost & Data and IP Security and Privacy would have been different, for example Labor Pool in Uruguay when it comes to software developers is not nearly as bad as the survey presents, and as a matter of fact it is better than in Costa Rica and Mexico.  Some other ratings seem seriously off even to an uneducated eye, take for example cost of resources in Russia marked by Gartner as Very Good; I’d say they have not been in Moscow recently.

But a single rating that prompted me to make calls, talk with many people and write this article was rating for Data and IP security and Privacy in China as Poor.  Anyone who went through on-site visits with major IT Outsourcing vendors in China would tell you – there have been huge advancements in that arena, the quality of Data and IP security that the vendors could provide is very impressive.  As my friend, a VP Engineering for very successful SF startup, told me – “I’ve seen the Great Wall of China, but what really impressed me was the Great Firewall of China I saw during my trip to Beijing…”

Americas:

Americas Source Argentina Brazil Canada Chile Costa Rica Mexico Uruguay
Labor Pool Gartner F G V G F V P
Pragmatic Outsourcing F F G G F F G
Cost Gartner V G F V G V V
Pragmatic Outsourcing G G P V G G V
Data and IP Security and Privacy Gartner F F E F F V F
Pragmatic Outsourcing F G E G F G G

Europe+

Europe, Israel, S. Africa Source Czech Rep Hungary Ireland Israel N. Ireland Poland Romania Russia Slovakia S. Africa Spain Turkey Ukraine
Labor Pool Gartner G G G G P G G V F F G F F
Pragmatic Outsourcing F G F G P G G G P F G F G
Cost Gartner G G F F F G V V V F G G V
Pragmatic Outsourcing G G P F P G F F G F F G G
Data and IP Security and Privacy Gartner V G E E E G G F V V V G F
Pragmatic Outsourcing V V E E V G G G F G V G G

And Asia

Americas Source Australia China India Malaysia New Zealand Pakistan Philippines Singapore Sri Lanka Vietnam
Labor Pool Gartner G V E G F F G G F F
Pragmatic Outsourcing G V E G F G G G P G
Cost Gartner F V V G F V V F V E
Pragmatic Outsourcing P E V G P V V F E V
Data and IP Security and Privacy Gartner E P G F E P F V P P
Pragmatic Outsourcing E G V F E P F V P F

Downfall Impact: Do you keep your current provider?

Another good question posed by Michael Grebennikov in LinkedIn, when the market is down, the budgets tight and future is more uncertain than usual, what do you do with your outsourced projects? Of course this question can not be dealt with in insulation. Major market events require immediate and aggressive action, all aspects of the technology organization need to be dealt with quickly and in the most judicious manner. The organizations that do not react / change fast enough pay huge penalty. When Cash is getting low and/or P&L is looking grim organization must rationalize its R&D and Project portfolio. On my book that means spreadsheets, metrics, analysis and often concrete and resolute actions. The goal is to quickly reassess what you can still afford to keep and what must go, in all aspects of the organization: projects, initiatives, providers, and sorry to say staff.

The “to keep or not keep” question must be applied to an outsourced portion of your project portfolio. If the projects are not “keepers” there is no other question, if they are the question is whether to continue with the outsourcing… One of the way to answer that question is go back to the decision criteria you used when dealing with the “to outsource or not to” question. Reassessing the answer with the new environment in mind on a project by project basis could be one of the most reliable methodologies. It could be quite laborious though. You may consider a simplified set of criteria based on a few key dimensions:

  • Total Cost of Outsourcing (TCO) / Price performance
  • Relative Productivity
  • Quality of deliverables
  • Overhead / cost to manage relationship
  • Quality of relationship
  • Cost of termination / suspending the partnership
  • Cost of restart (with current or alternative provider)

Rating against these criteria will quickly point out suspects for termination. For remaining projects / partners you can do an in-depth what-if analysis compare status quo to an alternative approach and finally make the decision.

In some cases you will still find yourself on a fence. You are still in a grey zone if your pros / cons balance is 40/60 or narrower. In that case I would consider getting the vendor involved – assuming that your relationship with them allows. The vendor has a lot of leverage in reducing the TCO and thus pushing the odds in their favor. In ideal case – and I have been fortunate enough see those – you can work out something with the partner on a basis.

The bottom line is clear: desperate times call for desperate measures. Not recognizing it on any side of the vendor-consumer relationship is lethal for the relationship and possibly for both parties. Dealing with the issue in timely manner, looking for mutually beneficial solutions and considering “win-win” style negotiations is likely to keep both sides relatively happy.

Outsourcing Impact on Technology Choice

I find LinkedIn to be a good idea generator for blog topics, for example a question from Vinay Joshi “.Net OR Java what technology projects you outsource — Does technology matter for making decision to whether outsource or not? …” deserves substantial discussion, beyond my brief answer on the site; especially considering that the rest of answers are more about religious war of .Net vs. Java rather than about the question itself.

Of course the answer depends on the context, if you are a technology company that already has the technology selected or a vendor that has a large team with specific expertise in place the discussion has little relevance. For those about to outsource it could be quite important decision though.

If you are planning on outsourcing but have not selected the technology yet, here are a few tips to consider:

  • Flexibility offered by technology is not your friend. The more discipline the technology offers / requires the easier it is to control it, the less are the chances on-shore and off-shore teams drift apart. In particular using Java vs. .NET discussion – Java offers great flexibility and far less commonalities in solving even basic development task. There is always 10,000 ways to achieve the same objectives. It offers multiple schools of thought and competing technologies. .NET offers more disciplined approach, while it offers some flexibility it’s far less the focus or the modus operandi, typically in .NET there is “the right” way of dealing with majority of tasks.
  • Emerging technologies are not made for outsourcing. That seems like a no-brainer, yet I’ve seen many companies moving projects using cutting edge technologies offshore, typically with painful consequences. So just in case, there are many reasons not to do so: lack of experienced resources, blind spots in understanding the technology on the both sides of the ocean, insufficient supporting community and documentation, undeveloped best practices, etc. Each of these issues by itself can destroy the engagement, when the issues combined the failure is guaranteed. Both Java and .NET by themselves are established technologies, however there is always something new being pitched by the respective camp.
  • Close doors to Open Source. Well, that might be too strong of a statement. As a matter of fact I did quite well outsourcing development using Open Source technologies and products and so many people I know. Caveat emptor! If you go for Open Source make sure that you do not stray off the beaten track and stick to very stable and mature products with strong development community. Too frequent release cycle, fluctuating quality of products, unstable supporting community can add insult to injury when combined with inevitable issues of outsourcing.
  • Don’t let the tail wag the dog. Some advanced technologies come with very costly or complex development tools. Some technologies require you to invest heavily in workstations or development environment. Some technologies require extremely high investment in training. And so on. Unless you have extremely compelling reason to do so, do not consider such technologies. Investing into a partner or their environment is not what you want to do especially in the early stages of the partnership. What if the partner already has it all in place? Well, do you want to be locked into using a specific partner? I don’t think so…
  • You can only find free cheese in a mouse trap. In development today there are a plenty of “very simple” technologies. Those technologies could be quickly learned, and superficial or even spurious expertise sold to a naïve buyer. That usually attracts gazillions of providers and inevitably drives the price down. Have you heard about PHP freelancers for $4 an hour? Just go to elance.com or guru.com – you will find a plenty. The chances are you will get what you paid for. The main point here is while the technology at question could be extremely solid it doesn’t mean that any code monkey can operate it. Finding good providers in such technologies could be a challenging task due to the high pollution of the field. Unfortunately PHP today falls into that category, and I am certain tomorrow that will be the case with RoR.

Pros and Cons of Outsourcing to India

India offers the most developed, experienced and sophisticated outsourcing community. No surprise – embedded advantage of ESL, huge supply of IT talent, and low standards of living made it a top destination for IT outsourcing long time ago. Y2K and management talent solidified the success creating multi-billion dollar giants and changing ethnic landscape of many cities in the USA. As I mentioned in Offshore Vendor Selection: Choosing the Destination “if your risk tolerance is low and/or your organization is new to outsourcing go to India, you can not get fired for hiring IBM. Go to India if you have to choose on a spot, or have little knowledge of outsourcing, or have to deal with large scope ERP implementation, or … as a matter of fact if you have to ask this question chances are you should consider India as your top destination.” Now let me put a few bullets here supporting my statement:

Infrastructure. Unless your partner is tiny and located in a 3rd tier city you won’t have any problems with infrastructure. Well, you may have to deal with some irregularities in connectivity due to some natural disasters, it gets quite rainy during monsoon season out there, but I tell you that: we use AT&T as our internet provider in our San Francisco office and once in a while they drop connectivity despite blue sky and sun outside. With a huge supply of IT services in India you can find infrastructure that would cater to most ridiculous demands.

Operating Environment. Flying to India is far from fun especially from the west coast, in particular if your company doesn’t cover first class travel. 30 hours in transit plus you arrive there in the middle of the night. Unless you time your trip well the nature would great you with heat and humidity. Flying back could be so much better if you did not need to deal with airport lines and crowds. The good part, that’s pretty much the extent of the adversities. Chances are you will be staying in a good hotel, will have a personal driver, eat in good restaurants, and even corruption is wide spread in India at all levels you most like won’t need to deal with it.

Skills Availability. That’s is one of the strongest Pros of the country. No matter what skill you are looking for there will be at least 10,000 people who have it. Well, more seriously, the supply of IT talent in India is outstanding, some areas more than others of course. Mainstream technologies of today and yesterday – Java, .NET, C/C++, ERP, Cobol, etc. – have substantial oversupply. You also can find a lot of talent even on a cutting edge of the technology. The quality of the talent follows the bell curve and nowadays the median has gone up comparing to late 90th.

English Skills. Well, that’s a hidden gem isn’t it? Of course with English being widely popular in India the main issue you would need to deal with would be an accent. Maybe some idiomatic expressions, some speech forms, etc. but generally it is not an ever a showstopper and forms a huge Pro of the country.

Cultural Compatibility. While there are a plenty of cultural differences between India and USA I would put the Cultural Compatibility in a category of Pros, here are a few reasons:

  • The cultural differences on business side were not so dramatic to begin with considering history of British influence on legal and business system of India.
  • Resources from India have been in this country in large numbers and for a long time. People in the USA learned the differences, behavioral patterns, and idiosyncrasies to a pretty good degree.
  • Many Indian vendors invest a great deal into cross-cultural training as well as in accent training. As a result the gap between cultures is narrowing considerably.

There are of course cultural differences that are deeply embedded in people’s psyche, here are a few most notable:

  • “Never say No” or “Yes to Death” – while working with Indian resources you always need to keep in mind that they might have a very difficult time say “No” in any shape or form. “Can you do that? – Yes, we will do Nick.”, “Do you have access? – Yes we do Nick”. That doesn’t mean that they can cater to any need or demand, they just can’t say NO.
  • No bad news is a no-news. While the times of chopping off bad news barer heads are over, the habit is still there. So if you do not hear about bad news, it doesn’t at all mean that everything is going well, it just simply means that you do not hear / do not know what is going on.
  • Motivational hierarchy. Of course Maslow’s Pyramid rules. But there is a plenty of subtle differences in how its upper levels translate for a specific culture. Not bad / not good – just different. For example, personal success in India outsourcing is often measure in number of people the person supervises. “I have 100 people under me…” That pushes good developers away from the technical track towards managerial with inevitable profound negative impact on technical abilities of the organization.

Rates. India rates fall neither into Pro nor into Con category. They are benchmark against which other rates are compared. And I guess that makes for a nice segue into Cons discussion:

Resource Turnover. Turnover is very high, it is high to a degree that it almost outweighs all pros of the region. See my earlier post Myth for more thoughts on the subject.

Resource Quality / Technical Capability. IT Outsourcing proved to be a rather lucrative business for many social groups in India – entrepreneurs, engineers, education providers, etc. Millions of people moved into the field in the Golden Rush of the century. As a result average quality of resources started going down to a degree that even time-proven trademarks of quality do not work anymore. Not long time ago I was stunned when I had to fire a consultant for incompetence; the stunning part came from the fact that he had a master degree from IIT.

One more Con related to the Golden Rush is worth mentioning: huge number of companies with a large number of low quality fly-by-night vendors makes it extremely difficult to find a right provider. It’s very much like looking for gold – you have to go through the tons of dirt to find the right substance. However, you are looking for gold, and one thing I am certain of is that you can find that gold in India.

Don’t Fall Asleep Behind the Wheel

This post is a rude reminder to all of us involved in outsourcing. Just a few weeks ago I talked about fundamental laws of outsourcing (FLO). And yet I just escaped from being hit hard by one of them, the ominous Second FLO, by the skin of my teeth.  Quick note – the second FLO as the same as the second law of thermodynamic – entropy always increases. In offshore outsourcing the second FLO exhibits itself as consistent degradation of quality of services in absence of non-stop energy applied from the on-shore.

In this particular case it was about sourcing. I have an agreement with my current provider that I can interview any resources prior to them being assigned to the project and I can stop that assignment from happening solely on the results of the interview. I have to say that is a somewhat unusual agreement – most of the vendors would fight tooth and nail against it; that did not stop me on many of my contracts though.

Anyway, my vendor has been good in many respects, with quality of the resources being one of the top. So when I did not see any red flags on the resumes of a couple developers about to be assigned to the project I was ready to give the green light. I am not sure what stopped me and why I decided to interview them. But as soon as I asked for the interview all kinds of red flags stared coming up: scheduling delays, preparation phrases, language and cultural difference discussions… Making the long story short interviews were a complete disaster: the guys were not only exceptionally green, they did not have the foundations I was looking for, no grip on technology, no relevant background… they were good guys ready to learn. Sorry, but I do not pay for on-the job training…

So what happen? Why my trusted partner was about to put these spring chickens on my project? Well, just because. The second fundamental law of outsourcing is as strong as the gravity laws. You know, you can fight the laws of gravity as long as you want and yet you will end up with you face in the dirt… Uninspected deteriorates. [Dwight David Eisenhower]

Here is a metaphor for your consideration. A long time ago I was in the far north of Siberia, in Eskimo country. I saw an amazing event there – sleds led by sixpacks of reindeer were competing in a traditional race. That was indeed a fun race and a very vigorous exercise for the jockeys – they had to use a very long stick to control the deer, and make them run. Interesting thing about those deer – they do not run if you do not hit them, and, unlike horses, they would stop the second you stop hitting them. So the only way for the jockey to win the race is hit them non stop all the way to the finish line. Little did I know that many years later I would have to use the same technique on all my offshore engagements.

Pros and Cons of Outsourcing QA

I saw a question on LinkedIn early this morning on a topic I was planning to cover “Pros and Cons of QA Outsourcing” – I jumped to the answer and typed up an answer while on BART, ironically it turned out to be too big for LinkedIn and so I decided to put it here. The answer is not complete and I am planning to come back to it some time. In meanwhile here are my thoughts:

1) Outsourcing QA is often a meaningful thing to do, an easy way to start and a potentially very dangerous trap. In particular companies that shed internal QA resources and move their QA operation abroad typically pay the price in knowledge loss and ultimately in degradation of the product quality. I have seen it on numerous occasions. QA engineers in well run teams often have better product knowledge than any other part of the organization, and offshoring that knowledge falls in a category of “outsourcing crown jewels”.

2) Companies in US often consider outsourcing QA for all wrong reasons, like for example “Cost”.  Cost advantage is just a myth, see my earlier post for details Outsourcing Myths: cost advantage. And going offshore for wrong reasons is guaranteed to give your wrong results.

3) QA as a subset of IT field offers a few interesting dynamics

a. It’s one of the most important areas of SDLC which typically is given the least attention.

b. The average quality of talent pool is dreadful, independently of geography. There are many reasons for that, for example here in SF Bay Area one of the main reasons is huge pollution of the pool by fraud and mediocrity – I met 100s of people who fake their QA background or think that a couple months of homegrown education makes them top notch professionals.

c. A perception of QA skills/occupation as a substandard one. It takes as much IQ to become a solid QA engineer as a Java developer, maybe more, but what can you do about perception? As one the best QA guys I’ve eve met put it talking about his resume “going to QA is like going to morgue – there is no way back”. And why don’t developers enjoy testing? Because it’s hard, it takes serious effort to put together a decent test harness, to organize your code, etc. Yet look at the average salaries, at layoff patterns, offshore dynamics – the trends are obvious.

4) Good QA engineers, automation specialists, functional testers, etc. like any other good resources are not easy to find, considering the quality of the pool, much harder. So it’s no surprise that that many organizations after interviewing a couple dozens of key-punchers who can’t tell the difference between priority and severity and ask for $90K move to offshore. The problem with such decision is obvious – it is as difficult to find good testers in Bangalore, Kiev or Beijing as it is in Boston. Of course a large supply of IT talent in countries like India and China makes it so much easier, yet the vendors in these countries have a plenty of own issues and challenges. Outsourcing the problem will not eliminate the problem, it only passes it to a different organization which is hopefully is better equipped to deal with it… But is it? Is your perspective vendor better equipped to do your job? How do you know it? By their brochures? Having interviewed hundreds of engineers in Asia, Eastern Europe, and Latin America I can tell you with certainty – far not ever vendor is equipped to do so, many of them are squarely in a business of selling mediocrity in bulk, and the quality of the resources is far less impressive comparing to what you can extrapolate based on what you see locally.

5) Having said all that I have to state that I still outsource a lot of my development and QA activities and get great results from my partners. There are a few ingredients to that success story, here are the most important:

a. Rigorous vendor selection process with the focus on “the match” between my organization and vendors’. Search for the match on multiple dimensions.

b. Resource augmentation and joined teams rather than complete outsourcing.

c. Abundant communications in all forms with fair portion of face-to-face meetings and on-site / offshore swaps

d. Control and ongoing preventive maintenance in all aspects of the engagement.

e. Adjusting SDLC to accommodate for idiosyncrasies introduced by offshore.

f. A “disposal outsourcing” model that I worked out with my one my partners – augmentsoft panned out quite well in QA arena.

g. Working with nearshore partners was much easier in many aspects especially when running agile projects.

Offshore Destinations: Russia

I was born in Moscow, USSR and the word “Russia” in my mind associates with a large empire of 15 republics. Things since than have changed dramatically and referring to some of the parts of ex-USSR as Russia is not just politically incorrect. Yet you are likely to hear about Russian outsourcing even if the ODC is located in Minsk, or Kiev. As a matter of fact in many respects outsourcing landscape of Byelorussia, Ukraine, and Russia has a lot in common. More so, large outsourcing organization such as ePAM, Luxsoft, and others have offices in these countries. Most of other countries of ex Soviet Union do not play significant role in offshore market, some due to low density of IT talent, some due to high cost. While offering in these countries exist, and you may find great providers in Estonia, Moldova, and others, in terms of outsourcing statistics these countries would be a rounding error. With that in mind let me cover some Pros and Cons of doing business in Russia.

  • Infrastructure. IT infrastructure in large cities of Russia is very good; smaller, second tier cities lag behind, the difference if pretty dramatic. Generally today you will find sufficient network bandwidth, stable connectivity, and solid pool of Sys Admin talent that would allow you stay in touch with your ODC. The cost of it will be not inconsequential though and needs to be taken into consideration. A very important aspect of infrastructure which you need to asses is vendor facilities – it is difficult to find well equipped offices with quality server rooms, etc. that is especially serious for companies with offices in second tier cities. In my view Pros here outweigh the Cons.
  • Operating Environment. Running offshore engagement with Russian ODC will offer many operating challenges even if you stick to tier one cities (for the purpose of this discussion that’s Moscow, St. Petersburg, Kiev, Minsk). Getting to these cities is fairly easy, they offer great selection of hotels, solid municipal infrastructure, and … mind boggling prices. As I heard Moscow has been recently awarded with a title of the most expensive city in the world with St. Petersburg following it closely. Second tier cities are substantially cheaper but you get what you paid for in terms of quality of hotels, food, transportation, etc. Another issue to be aware of is high crime rate (accidental traveler be aware!) and very high rate of corruption. Corruption could become a very serious obstacle for models involving ownership of the resources such as BOT. With caveats considered I would still put Operating Environment as a Pro of doing business with Russia.
  • Skills Availability. That is in my view is one of the weakest traits of the region. First at a very high level, Russia produces IT resources at a fraction of speed of the countries such as China and India. This problem is exacerbated by fairly consistent internal demand for IT resources and high geographical dispersal of the talent pool. In large degree Russia talent pull is already exhausted. Pretty much everyone who is interested in working in offshore organization is already working for some client, often for several, as many of talented engineers work several jobs, moonlight or find other ways of get themselves reasonably compensated. Finding software aces is challenging even in second-tier cities, in the first tier cities it’s practically impossible.
  • Cultural Compatibility. My experience in that arena has been surprising to say the least. I left Russia in ‘91 as an accomplished technology professional with almost 10 years of experience under my belt. I had not expected to have any problems in dealing with companies in Russia and yet I found it easier to work with companies in India instead. Some of my greatest pains came from several areas of communication / work related behaviors.
  • Customer is always right… Maybe, but not in Russia. As a matter of fact the vendor seems to always know what I want better than I do.
  • Being “Politically Correct” is not a Russian way. However, while I prefer straight forward communications I do not enjoy when my vendor is rude to me or more so to some of my employees.
  • Work ethics. Very sensitive topic, I have seen many great, hardworking developers in Russia, but unfortunately they seem to be outnumbered by short-timers with “get money and run” attitude.
  • There is one interesting aspect of Russia’s culture which while “positive” contributes to the difference – attitude towards education. It is amazing how many highly educated people you find among Russian developers and even QA engineers. I am not talking BS, I mean Ph.D. and above. While by all means commendable quality the negative impact of it is actually multifold: theoretical approach to problem solving, abandoning career for the sake of education, investment in education at cost of work skills, etc.
  • English Skills. In my opinion English skills of Russian outsourcing community are at the level you would expect them to be with a typical bell curve distribution and the median being at acceptable level.  Chances are you won’t have problems understanding developers and would be able to carry on a rich conversation with account managers and other client facing resources.
  • Rates. Rates of Russian development workforce vary greatly depending on location. Rates in T1 cities are very high, often making Russian outsourcing to be cost prohibitive. To deal with this issue many T1-city based vendors diversify by opening locations in small cities.  Rates for smaller city are as the standards of living in those cities – they fall off the cliff as soon as you move 100 miles outside of the tier one city boundaries.  However resulting rates continue to stay on a high side comparing to India’s.
  • Resource Turnover. Turnover tends to be on a low side comparing to India especially in a T2-T3 cities. The trend is however discouraging – according to what I hear from my network the turnover rate has bean steadily growing correlating to growing demand and increase in expected standard of living.

Let me close this post on a positive note covering one of the most important Pros of Russian outsourcing community – its Technical Capability. For many reasons Russia IT community in many cities in Russia offers above average technical capacity, innovation and creativity. That is particular notable for boutique vendors from St Petersburg, Moscow, Kiev, Minsk and Novosibirsk as well in the top echelone resources from lagre Russian outsourcers.

Offshore Vendor Selection: Choosing the Destination

Selecting a destination could be quite simple if you have strong drivers pushing you towards certain geography. Maybe your engineering team is predominantly Chinese, or maybe the board of Directors is firmly set on India, or maybe you have vast network of industry connections in Romania… Almost any geography can offer a variety of companies among which you could find that great match for your needs. And what if you do not have strong bias towards a particular location, where should you go? Let me start with a few quick tips here:

  • If your risk tolerance is low and/or your organization is new to outsourcing go to India, you can not get fired for hiring IBM.
  • Go to India if you have to choose on a spot, or have little knowledge of outsourcing, or have to deal with large scope ERP implementation, or … as a matter of fact if you have to ask this question chances are you should consider India as your top destination.
  • If you are not an outsourcing neophyte and ready to invest time in research and vendor selection a whole new world opens up to you, with some decision shortcuts:
    • If you are looking for partner that operates as extension of your team on an agile project consider near shore, e.g. Latin America
    • If you are looking to outsource large volume of manual regression testing consider China or Philippines.
    • If you have IP intensive engagement consider Israel
    • Are you looking for resources for mobile development? – Consider Eastern Europe.
    • Are you very cost sensitive and ready to deal with immature vendors, consider nascent territories: the countries such as Pakistan, Vietnam, Macedonia, etc.

Of course those tips are just ideas, the country selection needs to be taken exceptionally serious as it is one of the most decisions you make in the vendor selection process. While selecting the country I recommend using weighed selection criteria approach:

  1. Identify your selection criteria; keep in mind that the criteria should be country specific rather than company specific.
  2. Give each of the criteria weight – the rating of its relative importance for your organization.
  3. Define an initial list of countries you are willing to consider. Keep it relatively short to limit the rating efforts.
  4. Rate each of the countries using latest information you can find. Industry analysts’ reports would be the most helpful, consider Internet research and outsourcing associations, as well as the common sense.
  5. The rating for the country is derived as a sum of ratings for each criterion times the weight of the criterion.

Below is an example of such analysis performed for a mid-sized software development company in Washington, DC area. The company was looking for a partner to perform substantial on-going customizations of their SaS product. Scope of each customization project would keep a small team of Java developers and QA engineers busy for ~3 months.

After the list of the countries has been agreed upon we started with a very rough analysis and then detailed ranking for a shorter list.

Initial List


Country

Infrastructure

Operating Environment

Skills Availability

Cultural Compatibility

English Skills

Rates

Resource Turnover

Political Climate
Time Zone

Brazil
Med Med Med High Med Low Med Med High

China
Med Low Med Low Low High Low – Med Med Low

India
High High High High High Med High High Low

Israel
High High Med High High Low Low Med Low
Pakistan Med Low High Med High High Med – High Low Low

Philippines
Med Low Low Med High Med Med Low – Med Low
Romania Low Med Low Med Med Low Med Med Med

Russia
Med Low Med Med Med Low Med – High Low Med

Ukraine
Low Low Low Med Med Med Low – Med Low Med

Refined detailed rating


Country

Infrastructure

Operating Environment

Skills Availability

Cultural Compatibility

English Skills

Rates

Resource Turnover

Political Climate
Time Zone Country Rating

Weight:
4 8 10 5 8 6 9 4 7

Brazil
7 8 6 8 7 6 8 8 10 458

China
6 6 7 5 3 10 10 7 6 411

India
10 10 10 10 10 7 4 10 6 510

Russia
7 5 6 8 6 4 6 7 7 371

Ukraine
6 4 5 8 5 5 7 6 7 352

Based on this analysis we narrowed our search to 3 countries (Brazil, India, China), which in my view was still excessively broad and could result in a very expensive vendor selection process.  By the way the winner of the engagement was a vendor from Brazil, and it remains to be seen how well the project works out.

Dealing with Turnover

Turnover impact on the cost for an offshore engagement could be dramatic.  It’s fairly obvious: any change in resources on a team triggers changes in the team dynamics, new resources need to be ramped up on technology, project specific and domain knowledge, etc.

Depending on a type of the assignment new resources will display lower productivity for weeks or even months. For a regular full time employee on development task the cost of replacement is typically estimated at 3 months of fully burden salary plus recruitment fees. The cost of the replacement in offshore scenario depends dramatically on contract arrangement and vendor capabilities. Interestingly enough it could be substantially lower than in captive resources scenario. In my view outsourcing companies could turn the issue of turnover into a competitive advantage. I have not seen too many that managed to so yet.

To deal with the turnover you need to start early during vendor selection stage and do not stop working on it till the engagement is closed.

Vendor Selection. Pick the vendors that have turnover under control; do not just take their word on it, research all aspects of employment lifecycle and derive your own conclusions. Here are some of the questions you need to get the answers to:

  • What is the vendor’s employee sourcing strategy and tactics?
  • Does vendor have any advantages in local market in terms of acquiring and retaining employees vs. competition? Or same question from a different angle – what are the employee choices in local job market?
  • What employee retention mechanisms the vendor has in place?
  • What the vendor does to counter results of the turnover? In particular what are the knowledge retention mechanisms? Cross training? Etc.
  • What were actual turnover ratios for reference (especially unsolicited) accounts? How did vendor acted upon reducing the negative impact of the turnover?

Contract Negotiation. Now it’s time to ask the vendor to put their money where their mouth is. Here are a few elements you may consider for including into MSA:

  • Clear definition of the turnover ratio with a penalty for exceeding specific benchmark.
  • Key employee clauses. You need to cover definition of the key employee, specify minimum retention period for them, and identify process of replacement in case of their departure. I would consider covering two scenarios – force major (vendor can’t do much about loosing the employee) and transfers (the vendor elects to move the employee to a different engagement) with much more considerable penalties for transfers.
  • Regular team member clauses. Those would be similar to key employees with focus on the process of replacement and much softer penalties.
  • Transparency clauses. You want to know about employee departure as much ahead of time as possible, you wan to be able to get to the bottom. Maybe you want to be able to do the exit interviews. You might be interested in right to connect with team members directly, but be prepared for serious fight here unless you are working with small vendors.

Managing the Engagement. The main point is not to rely on your vendor’s adherence to the contract but take an active role in increasing retention of the team members at multiple levels. Of course that must be done in a concert or at least not in a conflict with the activities of the vendor. Most reasonable organizations do not intend on breaking the contract and in a large degree are interested in minimizing the turnover the same way you do, maybe with just a few exceptions. These exceptions come from market pressures and internal constraint, the better you understand them the more you can do to counter the issue. Here are a few reasons for these exceptions and some tips on dealing with them:

  • The concept of seniority in an offshore organization is likely to be different from what you have in-house. For example a java developer with 6 years experience would be considered very senior in China and expected to act as a tech lead on the project with 20 developers. So forcing that developer to be a junior member on a team of 5 would be an insult which is likely to lead for him to quit one way or another. You need to be cognizant of that and other similar trends and form the team which would create a favorable environment for the team members not on your terms but on their terms.
  • Same problem has a different angle: let consider a vendor’s viewpoint. As a practice manager I need to leverage my resources well. If I have a senior tech lead he is expected to run a team of 20 people. In that case I can afford to charge my clients some reasonable rate for his time as my losses would be more than offset with profits I make on junior members. As you can imagine a client’s request to form a small team with several senior members is not going to make me happy… One of the ways to deal with it is to be much more flexible on rates and find alternative / additional methods of compensation. More important if small team with very experienced team members is your preferred scenario – you need to pick vendors who are ready and interested to work in that model (and not just because they told you so!). Consider boutique consulting organization, Eastern Europe and Latin America.
  • Another factor to consider is employee engagement. Long term development projects, especially large scale maintenance ones offer different challenges over time. Initially they could be interesting for people motivated by technical complexity and vast scope, but after a while these challenges disappear replaced by mundane repetitive tasks. So it’s no surprise that some of the team members are ready to fly when you just starting to rip the benefit of their experience and knowledge. In my experience the best way to deal with it is by selecting resources with personality / mind set match for the project. There are plenty of people who a great and maintenance projects and enjoy that work as well, they might be a bit slower in uptake, but that is the bullet you need to bite.

In a large degree to reduce turnover of your offshore team you can do many of the same things you would do for your own staff, often by the vendor’s hands. Here are just a few things you can consider:

  • Compensation & Gifts. Remember a $1K bonus goes much further in India than in Indiana. A few technical books sent directly to a developer would earn disproportional value in loyalty.
  • Classic motivation factors: advancement, recognition, and achievement. Recognition is particular simple and pays off incredibly well.
  • Team building. A few things with huge impact: joined development activities like SCRUM style meetings, offshore visits, especially for local team members with specific tangible objectives such as training or k-transfer, bringing best off-shore team members on-site, etc.

Outsourcing Myths: Turnover Ratio

The impact of turnover on the total cost of outsourcing is difficult to overstate. Of course you know that and put a turnover question as one of the most important ones in the beginning of your RFP. You look at the proposal that just came back from your vendor and see 18% as the response, “whew, I think we found our guys!”… Welcome to the murky world of turnover ratios. The sad part is that this answer may mean very little; being the most infamous curse of offshore engagements the turnover ratio comes with a few extra traps.

The most frequently overlooked issue strangely enough is the meaning of the “turnover ratio”. When your prospect vendor tells you that their turnover ratio is lower than the country’s average (high changes that’s exactly what you are going to hear) what does the vendor mean?

On one of my recent engagements with a reputable company in Noida, India the staff on the project changed at an amazing rate – while working with 10 member team for about 1 year we saw over 20 people, and only one person stayed on the project from the beginning to the end. No matter what formula I tried apply to that situation it did not seem to align with 18% stated in vendors proposal. And yet every account review my vendor pushed the idea that the turnover ratio was not out of bounds. Ah? ‘Well, Nick:

  • We moved Rajiv and Venkat off the project because they were not performing job well enough;
  • Ramki’s mother got sick and he had to quit to do the right thing;
  • Shushma got married and moved to Hyderabad…

And so on and on and on…

As it turned out my valued partner had a completely different view of the turnover ratio. My guess is that they  calculated the ratio based only on the number people who’d left the company for competitors.

Another trap worth mentioning is an internal transfers.  What difference the company’s average turnover rate makes if your project turnover exceeds it by two or three times?  I’ve seen that numerous times and in a large degree it’s unavoidable. The vendor will move people around to increase their utilization, to appease the loudest customer, and to keep employees motivated.

And one more trap to mention is key resource turnover.   If your team has an average turnover of 20% (you lose and have to retrain 2 people a year on a 10 member team) it might not be so bad if these two are junior QA engineers. What if these two spots both belong to the tech lead on the project?  You find a great TL, he comes to your site for knowledge transfer and after two months go back to India just to resign the next week, two months and countless meetings later another one comes to your office, goes through K-transfer and goes back, and then gets hit by a typhoid fever?

Recognizing that there is much more to turnover than just a percentage sign is a huge step forward.  Dealing with turnover is a much more complex issue and a rather large topic, so I’ll cover it in the next post.

Ready to Outsource?

Organization outsourcing maturity is one of the most important ingredients for an outsourcing initiative. Attempts to force outsourcing to an organization that is not prepared / not ready for it are likely to fail and chances are with a lot of collateral damage. That is true for any outsourcing initiative, not only for offshore; well, it applies to pretty much any organizational change, but my focus is on specifics of offshore though. What does it mean to have your organization ready for offshore outsourcing? Here is a high-level checklist to consider:

  • Solid justification / objective reasons for outsourcing. Jumping into outsourcing following the lemming instinct would end up with pretty much the proverbial result. The industry is full of examples when organizations went after outsourcing just because it was “the best practice” and ended up with massive losses – financial, customer satisfaction, knowledge, etc. Take a look at “reasons for outsourcing” or “my reasons”, go through your own list, and make sure that you have solid objective reasons to even consider offshore. Go through a thorough and very conservative what-if analysis and unless you see a substantial ROI set the idea aside.
  • Sufficient budget. You most likely heard about a hockey-stick or a J-curve – a curve / a pattern of success in business. Look at the budgets you have in-hand with the same perspective. Success requires initial investment and ability to sustain negative cash flow for some time, you need to survive that dip, before you start realizing ROI.
  • Executive commitment / sponsorship. Lack of executive commitment is certain to ruin your offshore strategy. If your executives do not accept the realities of offshore savings, if they are not prepared to wait through a negative stage of the J-curve you are certain run out of budgets prior to getting offshore initiative on the path to success. It’s even worth if your executive team doesn’t buy into offshore idea due to political or personal preferences. The last thing you want to do is spear head an initiative that’s only purpose is to show a bad example.
  • Team understanding, support and commitment. Chances are you can’t do it alone and through out all stages of outsourcing you will need to rely on support of your team. I do not think I need any mountaineering metaphors here. However, strangely enough, I have seen many times when offshore initiatives were driven down the thought of a core team, at the expense of the employee morale and wellbeing, without support and consideration. In my view that results at best in malicious compliance, more frequently in clever or blatant sabotage.
  • Processes and procedures. Often undermined in small organizations immature processes and procedures, in particular in SDLC / project management, are almost certain to result in offshore failure. Large process-savvy organizations are also not immune as the need for the processes is proportional to organization’s size. The most important aspect of the processes is communication channels and their efficiency, inter-departmental handovers, and internal roles and responsibilities definitions.

Add to FacebookAdd to DiggAdd to Del.icio.usAdd to StumbleuponAdd to RedditAdd to BlinklistAdd to Ma.gnoliaAdd to TechnoratiAdd to FurlAdd to Newsvine

Fundamental Laws of Outsourcing

A while ago when answering a question on LinkedIn I suggested applying three well-known principals to managing offshore engagements. I call them Fundamental Laws of Outsourcing a.k.a. FLAWs of Outsourcing (FLOs for short). These laws in my view are as strong as the law of gravity, and as you know if you attempt to ignore it you will at best end up with your face in the dirt…

  • The first FLO is The First Murphy ’s Law: “Nothing is as easy as it looks.”
  • The second FLO is The Second Law of Thermodynamics (Entropy Always Increases)
  • And the Third FLO is the First Law of Military Communications: “If an order could be misinterpreted, it will be”.

That really sounds encouraging you may say. How can you ever consider working with offshore under these laws? Well, back to gravity – it doesn’t seem to prevent us from being ultimately successful in our lives and have fun in the process. So here are just a few basic tips…

1) To deal with FLO # 1consider a few things –

a. Starting with a stellar contract (MSA) which addresses typical offshore traps (quality benchmarks, attrition remediation, knowledge transfer / retention guarantee, etc.).

b. Setting expectations right (your management, your team, and your own). “Right” in this context means as low as possible. For example: there will be no cost savings, the work load will increase, everything that could go wrong will go wrong, things that just can’t go wrong still will, and no matter how low your set the expectations your vendor will surprise you.

c. Setting an “exit water mark”. No matter how long you march down a wrong road you will need stop and go back. In case you made a wrong decision there is a point where you are better off by cutting your losses short.

2) The second law of thermodynamics applies to any organization or project. You might remember these quotes:

The uninspected deteriorates.
– Dwight David Eisenhower

The only things that evolve by themselves in an organization are disorder, friction, and malperformance.
– Peter F. Drucker

In offshore outsourcing the second law of thermodynamic exhibits itself as consistent degradation of quality of services in absence of non-stop energy applied from the on-shore. Consider uninterrupted control from a dedicated resource on your side (make sure it’s someone you trust and who has your company – not the vendor’s – interests in heart). Everything from timesheets to hard core deliverables to be scrutinized and verified. The only way to stay ahead / prevent quality decay is to apply pressure on the vendor even when things are still going (seemingly) well.

3) The FLO # 3 applies to all aspects of communications and in particular to the project/product requirements. Any ambiguity in your documentations, specifications, processes, procedures, and especially verbal instructions will be (innocently) exploited to create maximum damage and cost increase. Consider crystal clear communications and small scope of controlled deliverables. For example, very short project phases, interim builds, etc. Work as many control / feedback points in your process (SDLC) as you can possibly afford.