A Few Words on UAT
This post continues with the topic I started a few months ago – using QA to prevent serious issues with offshore deliverables. In particular I’d like to cover User Acceptance Testing (UAT).
For many software professional UAT has a very clear definition and lucid goals and objectives, yet this understanding at most foundational level varies a lot between different professionals and organizations. In professional services engagements UAT I had pleasure to participate in UAT used to be a final sign off by buyer of the software deliverable. In my new organization UAT has been playing a key role in SDLC acting as a final gate before release to production. In many organizations UAT is interpreted as a smoke test performed by users at each milestone to make sure that the users’ requirements were properly understood.
Whatever the test performed in your organization with a UAT label it is probably an important part of your SDLC and I am not disputing its value. I am also not an abbreviation fanatic demanding that UAT term is only used its original purpose. I think it would be quite important to cover participation of users in the acceptance of deliverables from offshore, and just for the sake of this post let me call those testing activities UAT.
Offshore QA and a Sly Fox
A fairly common model for working with an offshore vendor for SaaS companies is based on black box model – the requirements collected locally go to offshore team and code ready for production comes back, sometimes in a form of binaries. There are variations to that model with the same common thread – the full responsibility for development of the application and its quality assurance belongs to with the vendor.
Can this approach work with an arbitrary software development shop? Absolutely! As a matter of that is the model used by all ISVs that do not employ offshore, so model works for sure. The question is whether offshore components in that model make the difference worth discussing, and unfortunately they do. The fundamental laws of outsourcing (FLO) affect efficiency and reliability of the model to a great extend, often making the model completely unreliable.
There are so many things that can go wrong inside of the proverbial black box turning it more into a Pandora’s Box:
- Communications issues and information loss at every handover
- Ever deteriorating quality of the resources
- Inevitable deterioration of the quality of code
- Growing blind spots in test coverage
- And so on – you can continue this list ad nauseum
Offshore BC & DR
Thinking about Nostradamus predictions for 2012 and all cataclysms that will strike that year? Afraid and developing your bullet proof Business Continuity and Disaster Recovery (BC&DR) plans? Well, if meteors, super-volcanoes, and melting ice caps flatten, burn, and flood most of humanity those BC&DR plans won’t matter much. However, increasingly more powerful floods, hurricanes and earthquakes with enormous toll remind us about vulnerability of even rich nations such as the USA, Canada, or China…
As I pointed out in Force Majeure working with offshore organizations increases risk of substantial losses bring up the importance of having solid BC&DR plans. Of course if you are working with a mature offshore partner they would have their own BC&DR plans. That’s great with one important caveat – will these BC & DR plans work when needed?
It was not long ago when supposedly invincible 365 Main Data Center in SF went lights out for a considerable period of time after a scheduled (!) black out. So there are a number of questions you need to answer:
- Does you offshore vendor has solid BC&DR plans?
- How often are theytested?
- What are the KPIs / metrics associated with these plans?
- Are those metrics sufficient for your business?
- Who audits these plans and activities? (You won’t take the vendor word on it, would you?)
Well probably the first set of questions you should address to yourself or your IT team responsible for BC & DR. Interestingly enough for many, especially small companies the answers from internal team are likely to be much weaker than those from offshore. And as a matter of fact for some offshore is the BC & DR plan.
After you covered two main points you need to check the route between them. There are many aspects to connectivity between offshore and onshore. Things can get lost on the way, connectivity may drop (if major cables are damaged for quite some time as it happen with India couple years ago). But even more important is to make sure that was sent to you is indeed what you expected and that it is what you received. Wrong code pushed to production (happens even to Google) is sometime more serious disaster than interruption in service. That gets into an area that deserves a lot of discussions by itself – QA and in particular acceptance testing. I will cover it in a couple of posts in the future.
One more major aspect to consider – what if the relationship between you and the vendor go sour? In one or another way – you did something wrong, vendor decided to rob you of your IP, etc. There are plenty of sad examples. That kind of disaster is most difficult to deal with. And with them being as unpredictable as the Acts of God you should have solid BC & DR plan for that as well. Starting with solid contractual framework, appropriate and frequent archiving, and so on. Yet you can’t be ready for everything – losing key resources, knowledge transfer cost, etc. Not easy to deal with, sometimes practically impossible. However there is a good news, when it comes to these kind of diseases I know very reliable medicine – Disposable Outsourcing.
Force Majeure
Force Majeure (French for “superior force”), is a common clause in contracts which essentially frees both parties from liability or obligation when an extraordinary event or circumstance beyond the control of the parties, such as a war, strike, riot, crime, or an event described by the legal term “act of God” (e.g., flooding, earthquake, volcano), prevents one or both parties from fulfilling their obligations under the contract.
While working with offshore teams, in particular in countries that are subject to severer environmental conditions such as Bangladesh or Philippines you should never forget about how real Force Majeure is. For my pet project wwhow.com I use support of freelancers from Philippines in SEO and data entry tasks via oDesk service. Here is an email I just received from one of my providers –
Dear Nick,
We had experienced a typhoon Ketsana that hits our area tremendously last Saturday morning since then our power was cut due to heavy rains and floods. Floods were everywhere including in our home. Power was restored last night. In this regard, the two accounts were not able to finish the working hours load for the week.
Rest assured this week, we will continue to post quality deals and able to finish work load. I apologize for this unavoidable circumstances.
–
Kind regards,
Jennifer
Environmental Fears
As I mentioned in At Doorsteps of a New Engagement I have a new vendor to deal with. It is a company that has been working with my team for over two years and thus it’s only new for me. It took about a few days for me to encounter the first set of issues. And that set came from the area so common that it’s worth a post by itself – software environments. Below is an email which I was cc’ed on –
Subject: RE: WCM Publish failed
Ravi,
Please explain to me how the production environment does not match what is in UAT. This is unacceptable and must stop. This is not the first time a production turnover did not match UAT.
We need to review our build, turnover, and documentation procedures. This pattern cannot continue.
Looks familiar? I am sure it is…
If you are in business of delivering software as a service or similar to it the chances you will have the following environments: development, QA, staging, production, disaster recovery. You may also have dedicated environments including Build, UAT, Sand Box(es), Performance Lab, etc. If you work with offshore team the chances are some of those environments are duplicated in the offshore offices.
A number of issues arise as environments proliferate:
At Doorsteps of a New Engagement
I finally started working in my new role, VPE / CTO of PDR Network. It took almost a full year of non-stop activities on two complex M&A projects to get to this point. Medem and its assets moved on to two different organizations and I followed one of those assets. “Asset” in this case meaning product, resources, customers, etc. – basically a business unit. Fortunately, I was able to keep some of my top contributors even though far not everyone.
This economy put a huge strain on teams. I had to let go some people who I immensely respect and enjoy working with. Hopefully the new place will provide an opportunity for some of the guys I lost along the way to rejoin. Well, it remains to be seen; at this point I am facing couple disentanglement and technology merge projects that include working with new offshore partners.
PDR Network was formed by merging two assets acquired by a prominent equity firm: Physicians’ Desk Reference, acquired from Thomson Reuters, and the Health Care Notification Network, acquired from Medem. Medem had its offshore partners and Thomson Reuters many of their own. Naturally Medem partnered with small companies and TR with fairly large ones, even though to my surprise not tier 1.
At this point I do not know practically anything about my new outsourcing partners and the challenges they will bring. It feels like you are standing in front of doors to someone’s house and ringing the door bell. From behind the doors you hear a dog barking. What kind of dog is it? Playful Yorkshire terrier, English Bulldog dripping saliva, huge Newfoundland eager to lick you off your feet, vicious Presa Canario ready to rip you apart? It would be great to know before the door opens.
So let me guess what the new vendor will bring to my plate… At this point your guess is as good as mine, the only thing I know of the vendor that they made it to top 20 Indian IT list. Well, I know a little bit about the history and track record, but not much and only at high level / in general terms… So let me put my expectations in writing and see how the actuals pan out:
- Turnover not less than 30%
- Majority of the resources would not pass through interview by on-shore team
- Poor track record of deliverables (late, over budget, low quality)
- Low quality of code (no comments, inefficient code, etc.)
- Low quality of the processes (incompliance with SDLC in many aspects)
- Very inefficient / over-engineered architecture / designs
- Waterfall SDLC with cushy estimates, slow start and high pressure in the end of each engagement
OK, that’s enough for now. I will keep you posted with what I find out discover over next few months.
Herding Cats
You most likely have seen that commercial.
I think EDS has a great point here, managing software organizations is very much like herding cats. Managing software teams with outsourced distributed components take herding cats to a profoundly higher level – doing it on steroids… or should I say on drugs?
How can one survive the ordeal of getting a product out in multi-sourced distributed environment? Language barriers alone brought the Babel project to its fiasco. And you know that language barrier is often much easier to deal with than with communication barriers or cultural differences.
Well, it’s been done, as a matter of fact many times… on time, within budget and over-delivering on customer’s expectations. Is there a recipe I can offer? Probably not, at least not a panacea that will fit any project or organization. As a matter of fact specific combination of project / team / environment demands its own methodology, approach, techniques. Let me offer just one of those, five C’s of distributed project delivery:
- Casting. That is the first and the most important element – select right tools for the job, find the right providers for the task, put people in positions that match their skills and aspirations.
- Compensation. Make sure that resources on your project are properly compensated for what they do, that applies to everyone from your in-house architects to offshore worker-bees. Considering that working with offshore throws a huge monkey wrench in your ability to control compensation you may need to start early – when establishing your relationships with providers.
- Communications. Put a well thought out communication plan in work and enforce it with the vengeance, at the same time review it and improve with every opportunity. Make sure that you cover all channels and take advantage of multiple media.
- Culture. Cultural differences derailed many projects and engagements. At the same time it is not necessarily as difficult or complex of an issue. Here are just a few things consider – acknowledge / recognize cultural difference, educate the team about them, and keep attention on cultural aspects, in particular when it comes to communications, estimating and delivery patterns.
- Control. Control execution on an ongoing basis in every aspect of the project – schedule, budget, quality. That is a large topic partially covered in the blog already (see for example Peace in Metrics). I will talk more about it in future posts.
Offshore Account Manager
A few days ago an interesting question came over email –
Hi Nick,
Thank you for your valuable inputs and keep them coming as usual at your blog..I have been reading your blogs for sometime now and I enjoy reading them as the info provided are very helpful for me.I have recently taken a new role into account management. My task is to define the scope of account management.
Though I have some specific areas in mind; acct governance, acct communications, acct performance & reporting.. do you have any advise on what should be also included? Might you know if there are any interesting online account management portals/association i can get into?
Your advise will greatly help.
Thanks,
AH
Singapore
Account Manager is somewhat ambiguous position with often very unclear job description. Well, when it comes to job descriptions many companies stay at very high level not providing employees sufficient understanding of the expectations and expecting employees to fill in the blanks. That’s in particular common in management positions (take a look at my post Manager, the Job Description).
As a matter of fact Account Manager (AM) is an exceptionally important role, especially in the Offshore Outsourcing world. There are several definitions / understandings of the role, ranging from “a sales person dedicated to existing accounts” to “a customer advocate”. I think it’s not one or another, it’s the entire spectrum and in that light AM responsibilities include many dimensions, the list below includes the most important three.
Peace in Metrics
Agile techniques and approaches are marching across the software land conquering more and more territories. It was not a blitzkrieg forefathers of agile has dreamed of but it is a successful invasion. Along with true agile aficionados with their well thought out and understood processes the crowds of software anarchists disguised as agile evangelists are capturing organizations by storm. Picking selected items from the agile menu they bask in glory of self-respect enjoying coding / refactoring dance leaving aside schedules, deliverables, and milestones. While religiously following 40-hour work weeks and iteration retrospectives they enjoy weeping sounds of deadlines as they fly by.
While agile made a huge positive impact on software development as any other broad initiative it created some serious problems and could not avoid significant collateral damage. One of the first victims to the new brave world were the software metrics. Indeed, why do you need metrics in a self-monitoring organization with a clear measure of success – the working product? Even Tom DeMarco backed off from some of his ideas, why should not you? … And maybe that’s why it took agile community such a long time to figure out that test-driven development is ridiculously expensive…
Combining agile and offshore is not at all impossible. It is however complex and requires a plenty of prerequisites, including serious agile maturity of the on-shore team. You should think twice before eliminating “overhead” of the waterfall on your offshore projects. And specifically, forgoing the metrics can be detrimental / border line juvenile delinquency.
Of course the topic metrics is also complex and rather controversial. What to measure? How to measure? What to do with the results? Here are a few tips to consider:
Charting a Map to Disposable Outsourcing
Have you heard about PMBOK? In case you did not – the acronym stands for Project Management Body of Knowledge. PMBOK is a very comprehensive document that covers PM processes, procedures, methodologies and techniques promoted by Project Management Institute (PMI), a very well respected organization…
A few months ago I asked two PMs on my team to develop a road map of ensuring that offshoring relationships we have in place are indeed 100% disposable. A very aggressive goal considering a small size of our on-shore organization in particular juxtaposed to the size of our offshore operations. Almost instantly the project was nicknamed OCBOK – offshore contingency body of knowledge. Of course building a real OCBOK would take much more than two part time project managers (even exceptionally good ones) and a couple of offshore teams. So we can only start outlining its structure and put some preliminary content in a few chapters. It would be great to get it to the PMBOK level at least some approximation. To that I need all help I can find – if you have any resources / ideas / suggestions that you think could be applicable please send them my way. Feel free to comment on this post or email me at – krym2000-po@yahoo.com
In meanwhile here are a few interesting observations from writing the first chapter of OCBOK which would be called “You are more dependent than you think” -
Offshore and Code Review
Code review is a rather controversial subject. On one hand many SDLC gurus, tech leads, and architects agree that it could present an incredible value, on the other hand it’s one of the last quality procedures to be exercised in a majority of organizations. There are many reasons code review often takes the back seat, one of the most important ones is morale impact of code review. If let unmanaged code reviews can often cause tension in the team, things quickly become personal and instead of improving quality of code generate turmoil and finger pointing. Another reason of code reviews’ lack of popularity is their complexity in process sense, in particular timing.
I am in general a big proponent of code reviews and when it comes to offshore I found those irreplaceable. Specific implementation of code review depends largely on SDLC and outsourcing model used by the team.
If you fully outsourced your SDLC to an offshore partner you in a large degree will be at mercy of their internal SDLC. If you have an ability to influence your vendor’s processes you may want to request code review as a mandatory step and hope that it will be performed according to the best practices that are generally well known nowadays.
Spam Attack
A few weeks ago we went through upgrade of anti-spam software we use in-house. The Bayesian filter database got corrupted during the upgrade or possibly before it and for some reasons we could not restore it from backups. After much ado we decided to start anew. Software upgrade was quite impressive and new filters reduced the volume of spam dramatically, the majority of the users did not even noticed the little problem we had. It was not the case for me, all knowledge previous version of the software gained about my spam designation was lost and many of my service providers came back.
Today I receive a couple dozens of emails a day which come from valuable IT service providers that somehow got hold of my email address. Apparently they use smart email campaign software that bypasses very solid guards that on daily basis filter out a few hundreds of Canadian pharmacy offerings, lottery winning notifications and love letters from Russian girls. Read more »
Red Flags to Watch Out For
How do you know that a relationship with your vendor is going south? Of course if all your partner’s employees just do not show up for work you know that something is rotten in the state of Denmark. That would be somewhat unusual event and probably irreversible. The goal here is to identify early symptoms of a fatal disease of deteriorating partnership. In techno world we call them red flags.
The earlier you identify developing issues with the relationship the higher the chances that you can nip it in a bud and return to a mutually beneficial rewarding relationship. Small companies typically offer a high level of transparency and short communication channels. That makes identifying issues much easier task. In large companies the issues can develop without knowledge of many stakeholders, that’s why watching the relationship dynamics is especially important for those. Read more »
Offshore Technical Due Diligence
A couple years ago I went through a technical due diligence (TDD) of several relatively small offshore vendors. The vendors were providing product development services for one of my clients, the vendors also supported operations of the SaaS for all of the products. The client had fully outsourced s/w product development and support to those vendors and retained practically no technology resources internally with exception of MIS / SaaS IT support.
The goal of the TDD process was to asses whether the vendors are efficient and can continue performing fairly complex projects involving working with sensitive information. There are a couple important distinctions here:
- The vendors were in large degree focused on the product development for my client and the rest of their business was relatively small.
- The vendors have been performing services for a number of years with very light oversight from the client’s side.
- The quality of work to date has been on a low side yet deemed sufficient for the money.
Laws of Nature
I just added a new post to my new blog Common Sense Management – Laws of Nature and the Need for Management. It touches upon one of my favorite metaphors – application of the Second Law of Thermodynamics to management and offshore outsourcing (see also Fundamental Laws of Outsourcing). The message common to these posts is quite simple – you must stay on a top of your projects, resources and engagements otherwise they would quickly deteriorate.
Having said that I have to admit I should pay more attention to what I preach and follow my own recommendations. Just a few days ago I was going through a small project that seemed to be falling behind. The issues became clear in a few minutes after I walked in the meeting discussing the project status.
The objectives for the project I stated about two months ago were lost and the void was filled in by something substantially more complex (And that illustrates another law – “nature abhors a vacuum”). My offshore team was happy to work on more challenging projects, estimates appeared too high, that required in-depth analysis, more people were getting involved, more issues discovered, that generated substantial amount of R&D that increased the complexity by an order of magnitude… Involved in the meeting there were six people from onsite and almost as many from offshore.
For me, at this point an unattached side observer, it was obvious that the efforts being extended to address the project exceeded the value of it by a great margin. It was not at all clear to people so passionately involved in resolving the problem.
About an hour later it was all over. The vicious circle was broken as the proverbial Gordian Knot. The projects objectives were not only restated but addressed. It required 15 minutes from an engineer who had not been even aware of the discussion. Yet the sunk cost of the project was unrecoverable, exceptionally high and not justifiable by any means. Is there anyone to blame for it except the manager (me) who abdicated the project to those who did not have enough technical depth to cut through the distractions of secondary objectives?
Launching Common Sense Management
I spent a bit of time cleaning up the Site Map and while doing that I noticed that some of the posts have only loose relevance to outsourcing. I put them in a new category – Common Sense Management (CSM). CSM is not a widely used or a popular term. I started using it awhile ago to describe management and leadership style I apply in my day job – running technology teams building software products and services.
CSM is built on top of traditional techniques as well as new methodologies that at some point were considered controversial and now are widely accepted. “Common Sense” in CSM term doesn’t mean “agreed by many” or “widespread knowledge”. CSM stands for application of simple straight-forward solutions to problems that managers face on a daily basis. The best illustration to the term of Common Sense I have ever heard of was a story about “space pen”. Supposedly many years ago NASA spent millions of dollars developing a pen that would write well in the space. Weightless ink presented a serious technical challenge which was successfully addressed by “space pen”… Russian cosmonauts used a Common Sense solution to the problem – pencils…
CSM is a huge topic – it covers multiple aspects of leading and managing people, teams, projects, engagements, and so on. So I decided to start a new blog Common Sense Management – Tips, Tricks and Traps of Technology Leadership. CSM applies to leading all kind of teams and engagements, not only technology, yet with my experience and knowledge limited to that domain I’ll do better staying focused on the technology field.
My first post in the new blog was to some degree a repetition – I wrote about 10 Golden Rules of Bargaining, the post is a bit different from Offshore Negotiations and Rules of Haggling – it is a using Presentation 2.0 style and if you are interested in the topic is probably worth checking out.
I am not sure how frequently I will update my new blog and how it will affect my postings in this blog. Since I only write while commuting the total volume is limited by my abilities and length of BART ride. Well, as they say in France – Vivra verra…
Five Steps to Keeping your Vendor
I started my earlier post Five Steps to Keeping your Business with comparison of an offshoring engagement to a bad marriage: courting, expensive wedding, honeymoon, initial struggles, mundane irritation, aggravated frustration, and bitter divorce. The post touched upon the most important steps vendors should take to keep their business. Now let me cover the buyer’s side. And first let’s discuss a couple reasons for keeping your vendors. Of course if a vendor fails your expectations by a large margin there are not too many reasons for keeping that vendor, if any. If the vendor is perfect the thought of changing them is rather unlikely. (Please do let me know what your vendor is in this case, so far I fail to find one!) The question is relevant to vendors that in general deliver on expectations and sometimes exceed them and yet are in many ways imperfect, sometimes even worse than that. To some degree these reasons for keeping the vendor summarize to “Don’t fix if it’s not broken” – Read more »
Data Entry Gig: Execution Control
In my earlier post Notes from a Data Entry Gig I covered a few areas related to a small data entry project. There were several areas that were left untouched, one of them – execution control, is worth covering by itself.
There are many things you can do to get execution under control on medium to large scope engagements. Project/program management, account reviews, etc. That’s not the methodology to be used on a small gig such as limited scope web research, data entry, SEO, etc.
Considering that discipline and control issues associated with freelancing have to do with a lot of its Cons I was rather skeptical farming out data entry to a few gals in several small cities in Philippines. One would say “it’s a small data entry, what can go wrong”, yet you and I know the possibilities to screw it up are endless. “And then I saw the tool, Now I am a believer.” [almost from Neil Diamond] Read more »
Five Steps to Keeping your Business
Offshore teams delivered a significant portion of the products and services for the companies I worked for. My experience in utilizing offshore whether it is measured in years, number of projects or dollars sent offshore is rather substantial. You would think that by now I should have settled on a couple partners who I use on the majority of my engagements the way most VPEs settle on DB platform, app server, language, etc. Actually that is not the case due to many reasons, the main being different needs call for different partners. There is however another reason worth serious discussion – many vendors lose their clients mainly by not doing good enough of a job of keeping them. Whether you look from the vendor’s or buyer’s side that’s a shame…
It’s a common knowledge that in the service industries the cost of a dollar earned from a new customer is substantially higher than from an existing relationship. Yet for some reason that rule is ignored with unexplainable consistency. In particular I see it common with some of my offshore vendors. All too often a relationship with an offshore vendor goes through typical stages of a bad marriage: courting, expensive wedding, honeymoon, initial struggles, mundane irritation, aggravated frustration, and bitter divorce.
It takes two to tango, and being unbiased marriage counselor I should offer my connubial advice to both sides – buyers and sellers. I will, with this post focused on the vendor’s side:
- Communication is critical element of any engagement and in particular distributed. Even two people who know each other exceptionally well and live under one roof are known to have communication problems, it’s no surprise offshore engagement fall apart due to communication problems. Communication problems have a cumulative nature, meaning that small issues accumulate and result in large scale problems. There is much to be said (and I will do that) about improving communications in offshoring engagements, for now just one critical aspect: You should establish and follow communication process. You should treat communication process as you would treat a manufacturing process. In particular consider no missed steps or other changes to the process unless you expect substantial improvement in efficiency AND the changes are agreed upon by all stakeholders. Read more »
Tools & Links
Bidding Sites and Building Frustration
A couple weeks ago I put an RFP out for a very specific set of SEO activities on one of bidding sites. This SEO project was for my darling app – WWHOW!. Since WWHOW! is based on user generated content it offers serious SEO challenges. Having spent a few months fighting those I knew fairly well what I was looking for and did not make a secret out of my expectations. To no surprise my straightforward SEO request generated a lot of responses primarily from India-based providers. I just finished going through all responses I received to date and it looks like I will have to go through bid-response process again, maybe I have to try a new bidding site, maybe change my request format, content, layout… Frankly, I doubt that changing much on my side will affect the dynamics of the campaign and quality of responses. I might need to change the target development community…
The fact that I received not a single proposal that I could remotely go with was quite irritating. One of the reasons I was annoyed by it is its effect on my “buyer’s reputation”. In some way majority of established bidding sites penalize buyers for not accepting proposals. Some of them will even cut buyers off if they do not meet some criteria, e.g. certain percentage of project acceptance. It appears that they will cut you off independently from the reasons you do not accept the proposals. It happened to me on www.eLance.com a little while ago and since despite multiple attempts I could not reach the customer service I ended up moving to another bidding site.
Read more »
August 10, 2009 Posted by Nick Krym | Making Offshore Decision, News, Articles, Thoughts and Comments | Freelancing, Offshore Risks, Offshore Traps | No Comments Yet