Is Microsoft Lowering the Software Quality Bar for the Industry?

From a paper I wrote in 2006 originally titled “Is Microsoft Ruining Software Quality?”

I have worked as a professional software testing and QA consultant since 1994 and have tested software, as a developer and user, much longer. I scan job postings when looking for my next assignment and to keep a tab on the industry. I noticed that when searching jobs in the Seattle area, a new category of software testing jobs was popping up frequently. This job is called Software Development Engineer in Test or SDET and Microsoft was initially responsible for all these SDET postings.

Microsoft used to have bragging rights when it came to the ratio of testers to developers. For some areas, it was rumored to be 2:1 – unheard of in the non-critical commercial software industry. Perhaps Microsoft is leading the way once again with the hiring of SDET’s. But I also noticed that Microsoft didn’t appear to be hiring any professional software testers.

In 2006, I had a chance to talk with a Microsoft HR person, responsible for interviewing testers. She told me that Microsoft now only hires people with development skills in a current language (C, C++, C# or java) for employee positions. They only hire for SDET positions – all software testing jobs were temporary contract positions filled by agencies. She went on to tell me that SDET’s actually do more programming than developers, so their job was even tougher than the developers’ job. Jokingly, I asked “So SDET’s must earn a lot more than developers, right?” She didn’t answer and changed the subject.

So what does an SDET do? Since SDET’s are developers, they write code – specifically test tools and they also presumably perform testing whenever they haven’t hired temporary testing contractors.

What could be wrong with this? As a test professional, I can see the value of having a developer write custom tools to aid in testing, but by not hiring testers as full time employees and hiring masses of SDET’s is fraught with problems and reveals a lot about Microsoft.

The first problem is that it is a well-known fact that development skills and testing skills are almost mutually exclusive. It is very rare to find a good developer that tests well. It’s hard enough to get devs to even minimally unit test their code, let alone going beyond that. If there really are as many great developer/testers as Microsoft has hired, why not fire all your developers and hire only SDET’s?

Most testers, on the other hand, don’t have development skills, aren’t good developers or do not want to become developers. An obvious questions that always seems to escape the mind of hiring managers and HR people is this: Why would someone accept a testing job if they have good, current development skills, since there are typically 15 dev job openings for every 1 test opening. Test jobs are the first to be cut when times get tough and dev jobs typically pay more than test jobs. Developers also get more respect.

It’s also not productive. Division of labor into specialties is much more effective in both the quantity and quality of work. When people work in the areas that they are good at and enjoy, it is much more effective. Having developers check their own work is like proofreading your own writing – a lot of mistakes are not caught.

The second problem might best by illustrated by a story. Imagine you have won the lottery and you are having your dream house built. You decide to go to the work site to check up on the progress of construction. When you arrive, you find craftsmen, not building your house, but creating a bunch of custom tools. Do you feel good about this? Why do they need to make special tools? Does this mean repairs will be difficult? Oh, and for the actual construction of the house, they will hire day laborers.

In Tools We Trust

When test automation first appeared on the scene, tool vendors were criticized for over promising results. Marketeers made it sound like anyone could use these tools to write tests, run tests and customers would automagically get higher quality software. In 1999, James Bach wrote a paper titled “Test Automation Snake Oil,” which became famous for exposing this kind misrepresentation. But 7 years after Bach’s published paper, it appears that Microsoft has re-bought the test automation myth.

To be sure, test automation can be good, effective and dramatically increase productivity and software quality – but it doesn’t happen automatically. Automation is good for simple tasks, checking detailed calculations, performing tedious tasks and highly repetitive tasks like smoke tests and regression testing. As a tester, I like test automation for a couple of reasons: By freeing up routine testing tasks, it allows me to perform more in-depth exploratory testing.  And writing automated test scripts is also a refreshing break from testing.

By over relying on test automation, many types of errors will be missed, especially errors that users are likely to experience. Much of the test automation is geared towards API-based testing and DB testing which means the GUI is bypassed during testing. Another sign that testing is misunderstood, comes from the failure to recognize that different types of testing reveal different types of errors and that you cannot simply replace one type of testing with another.

But Microsoft does contract out for temporary, manual testing, so perhaps that fills these gaps. That’s possible, but there are more problems lurking here too. These are temporary, and usually short-term contracts, so no product experience built up and carried forward to the next release. Also, many employment agencies pay low wages for these positions. This has a direct consequence on the type of candidate that will accept these offers.

And what message does this send out about how software quality is valued? They company is not willing to hire FTE’s to test. Temps are here today; gone tomorrow – they don’t get to know that application under test intimately. And working as a temp, how well would their suggestions and criticisms be received, and that’s assuming they are asked?

Extreme Makeover – Microsoft Edition

A conversation with a Microsoft employee revealed some key information. He said that Microsoft used to be made up of 20% developers and 80% non-devs. For some reason, Microsoft decided that almost every employee should be a developer (receptionists, janitors and food service, like testers, are temps hired through agencies). Today, Microsoft is 90% developers. To paraphrase Dan Aykroyd from his Saturday Night Live Scottish skit days, “If you aren’t a developer, you’re crap!” If you view Microsoft through this lens, it all becomes clear.

Microsoft is a very dev-centric company. This would be logical, since they are a software development company, but development is much more than programming. So I should say Microsoft is a very programmer-centric company. Development would encompass other areas such as marketing, requirements gathering, design, test, project management in addition to programming.

Why are SDET’s performing testing? Because they want to? No. Is it because they really want to be developers, but for some reason they were not hired as devs? None of the SDET’s that I’ve talked to want to remain in testing. Unlike development, testing has an infinite workload – you can always perform more testing, in addition to documenting test cases, generating and maintaining test data, analyzing requirements, etc.  This has profound implications when it comes to attitudes towards testing and the number of bugs found. With this lack of enthusiasm for testing, it is unlikely that they will do anything beyond the minimum needed. If they want to progress at Microsoft, their time would be better spent learning more programming skills. Microsoft has created a culture that only values developers.

By requiring their “testers” to be developers, we can glean several things from this. First, they probably aren’t very good developers, because if they were, they would be working as developers. They may be developer wannabes that couldn’t make, so they’ve taken a testing job, biding their time until a development job slot opens up. Or they might be testers that do not want a long-term career in testing, so they have learned programming as a way out of testing. Neither case sounds good. All have consequences on software quality.

The fundamental problem that haunts testing, is that companies fail to recognize testing as a skill in and of itself. It is well-known that dev and testing skills require very different skill sets and mindsets. These are not usually found in the same person. Developers focus on creating while testers focus on breaking programs. Apparently, Microsoft either doesn’t know this or think it is common for people to have these two skills or is able to skim the cream of the crop and hire these rare individuals. But then what do they do when a Google comes along and is able to steal talent from them?

Microsoft does have some “staff” test functions for test automation and performance testing for groups that can’t justify hiring these specialists. But here too lies another disconnect: Usually, when writing automated test scripts, you need detailed application knowledge, unless your tests are very simple. Also, many bugs are usually found during the creation of automated test scripts, but these specialists are remote and in my case, did not entered a single bug in the defect tracking system. Maybe it’s because test automation was done so late in the lifecycle, all of the bugs were fixed. Maybe their scripts didn’t exercise the application very well, so no bugs were found. Maybe the automation specialists didn’t know the application’s business rules. Maybe they felt that it wasn’t their job to report bugs, since their task was to automate. I never met or saw people from either staff test group.

Groping in the Dark

Maybe Microsoft is doing the right thing, or something that feels right. Maybe they became so frustrated at trying to hire testers, that they’ve just given up. Maybe there are so few testers that are exceptional, that it’s not worth the effort to recruit them. Maybe previous testers were so poor that they can get the same poor results cheaper by hiring contractors. Attaching programming requirements for full-time employees in testing is much easier and more familiar to Microsoft. Even if a test contractor proves himself valuable, rules are rules and the software giant fails to capitalize on this.

Hiring good testers is not easy. Even experienced testers may not be good due to the common industry practice of moving “duds” into testing, which makes the “testing is for losers” mindset a self-fulfilling prophecy. Then there are different types of testing: GUI, functionality, database, performance, etc. And there are approaches to testing: glass box black box, etc. And each tester needs to be good. Testing tends to be repetitive, but the good tester will always be thinking of news ways to test the same functionality. Thinking is not repetitive. Testing tends to be a high burnout job. Environments can be stressful, especially towards the end of the project, but also usually lacking in resources and being held to a higher standard than analysts and developers are held to in the area of documentation, which doesn’t make sense since requirements and functional specifications are foundational documents for testing.

Another problem that plagues the software testing world is lack of testing curriculum offered in colleges and universities. Anybody can be a “tester.” This means that testers have wildly varying backgrounds and you never know who will be a great tester and who will be a lousy tester. Testers often fall in line with the low self-esteem image that others paint them with. One common response is an emphasis on backend testing and API testing. While these have valid places, testing the application through the GUI, using it as the users would use it, is not in vogue. Besides, it looks so much cooler to be working in an IDE (integrated development environment). Your respect will go up when others see you single-stepping away. Using the actual application puts you on par with the lowly user.

Testing suffers from being defined and run by non-testers. People with little or no test experience (or worse, people who believe they have test experience) always seem to wind up driving the test process. It looks like the same thing has happened at Microsoft.

One might think that people with computer backgrounds would be good testers, but this is not always true. The desire to think, have a strong curiosity and self-motivation are much more important.

Where Have All The Evangelists Gone?

Microsoft created a job position called “Evangelist.” The purpose of the evangelist is to spread the good word of whatever technology or process Microsoft wants others (inside and outside Microsoft) to use.

Does Microsoft have a Test Evangelist? I can’t find one, which I consider a good thing given Microsoft’s current state of testing practices. I doubt any serious, professional tester would ascribe to the testing philosophy Microsoft has adopted. This leaves me wondering who at Microsoft made this decision to contract out all testing and only hire programmers as SDET’s? I also wonder how this conclusion was reached.

The Proof is in the Products

With the heavy emphasis on automation and the use of temporary contractors, what would I expect to experience as a user of Microsoft’s products?

Fewer memory leaks, fewer load/stress bugs, fewer bugs in simple user scenarios (positive test case scenarios) and fewer regression errors (features that a patch breaks or features that do not work on a new version).

On the downside of over reliance on test automation, I would expect to see more errors in negative scenarios, more complex (real-world) user scenarios, error recovery failures (bugs in recovering), GUI bugs and usability issues.

In short, test automation often translates into testers using the program less and certainly not in ways end users would use it (API testing)

The Sincerest form of Flattery

The sad thing is that other software companies look at Microsoft and want to emulate them. After all, Microsoft is the largest software company in the world. These MS-wannabes run the same “Software Developer in Test” job listings.

Ironically, even some of MS’s biggest competitors like Google are following in Microsoft’s footsteps. Fortunately, searching isn’t a critical function, many alternatives to Google exist and you can change search engines instantly. It’s not as easy to change your operating system.

Microsoft is still a world leader in software. If other companies emulate their approach to software testing, this will lead to a collective worsening of software quality in the industry, just when the need for higher quality software is increasing, as computers are placed in the heart of more and more everyday systems.

This entry was posted in Microsoft and tagged , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s