Thursday, 23 February 2012

Today I was 'abused' by a number cruncher

As a tester I try to live by  these two maxims:
  • Any activity which doesn't ultimately lead to finding bugs or stopping bugs from appearing  in the code in the first place is a waste of time - Cem Kaner 
  • I will not fake a test project - James Bach

But, today, I had to fake a test project and I felt like I was abused professionally.

I work in a team which developed an piece of software that is to be embedded inside a commercial electronic device and as a Scrum team we have been developing and doing mainly exploratory testing during the sprints. Now that our software has reached the stage of maturity and being readied for commercial release, more and more people outside our team (but within the company) are demanding to know the quality of our software. 

We duly provided them a report stating the kind of testing we did, the area of functionality covered, a list of all outstanding issues (no Major or Critical ones) and our professional opinion on the quality of the product. Sadly, this report wasn't good enough, since for them, the measure of quality is based solely on game of numbers; how many tests ran, how many passed, how many failed, percentage not run, pass rate, fail rate, etc.

Today, we received an email from an external team who has the power to decide which  software gets to be included in the commercial device demanded that we provide them with these test result figures or else our software will be dropped.

Not wanting to waste many months' worth of our team effort over the test results, I decided to give them what they wanted. I had an old and out of date spreadsheet which has around 85 test cases which checked for the basic features of our software, but had none of the complex test scenarios (that we discovered and used in our exploratory testing) which led us to finding all the critical bugs which we have fixed and retested.

And I spent the rest of the afternoon and early evening dutifully going through all of those out of date, incomplete test cases and marked each one of them as Passed or Failed and at the end we arrived at these magic numbers:

  • Test run: 100%
  • Passed: 95%
  • Failed: 5%

and then I resubmitted our test report with these figures.

And on the way home, I thought about what I did today and I felt used. I felt abused. I felt humiliated. I felt violated. I felt dirty and unclean. No matter how much I washed myself with soap in the shower, I still felt dirty inside and this feeling will stay with me for some time.

Sunday, 5 February 2012

I don't want to find any old bug

Ever since I started out my career as a Software Tester, I have always thought that my job was to help improve the quality of the product, and for many years I believed that to be the case. Likewise, when people hired me during the last 14 or so years to be a tester, they also believed that that was what my job was.

I guess, in that way, everyone's happy; I know what my role was and so did my employers. 

However, lately in these last couple of years, I'm beginning to feel less and less sure of exactly what my job entails. For example, at the beginning I said that I helped improve the software quality. But, do I really? How?

I don't get involved in the capturing user requirements or write user stories, I don't get involved in the design of the product in any way shape or form, I don't write a single line of code. In fact, I don't do a single thing in the creating of a software product. The first time I get to see the product is when it is  either partially or fully formed as a working piece of software.

If I'm not involved in the creative process of a software then how could I have any type of influence on the nature and the quality of the product?

This led me to realise that as a tester, my job is this:- I discover software bugs.

That's it! That's all I do.

It is through the discovery of initial bugs followed by the subsequent elimination and the further lack of discovery of new or old bugs that we somehow arrives at an inference that the software must now be in a solid and stable state because as far as we know, we can't make it fail, and it's good enough to be released to the outside world. 

All I could do, and the only thing I could do, is to point out how bad the software really is, and it's up to the developers and product owners/managers to decide how to take the news that I give them.

In fact, in many cases, we would (have to) release the software (due to marketing, external partners or competitor pressures) knowing full well that there are numerous bugs in the product. And yet, we'd often pad ourselves on the back for a 'job well done' by delivering the product on time and being congratulated by the management for hitting this or that target.

When I'm part of this sad scenario then how I can possibly say to myself and to others that I helped improved the quality of the product when I have spectacularly failed to do so, evident by the fact that the released product has 10s, if not 100s of known bugs.

For me, this is no longer acceptable.

If I could use an analogy, Testers are like Cleaners you'd often see working on the streets at night. Their job is to clean up the mess that people left behind, so the pavements are clean again the next morning. As long as developers are putting in bugs in the software (some obvious and some not so obvious places in the code), we, the testers, will have to keep on finding them and reporting them.

For a long time, I was quite content with that situation. It keeps me in a job of finding bugs and I would even feel elated that I raised 20 bug reports in a single day; going home felling quite good about myself that I discovered so many bugs!

But I don't want to do that any more. I don't want to keep finding bugs in the code. You see, by the time the software arrives on my desk to be tested, it is already too late for me to have any influence on its quality; it will be what is it. The bugs would have been planted and hidden deep within the code and we would then have to find them. Even if we did find them and the developer fixes then, very often it leads to unplanned and unexpected bugs in other parts of the code to raise their heads.

If you're in the same situation as me then what would you rather do? Carry on happily as before or would you find a better way?

I'm not implying that I don't want to find bugs. I do. But I only want to find really interesting bugs. Bugs that require me to be creative and inventive in order to discover them. I don't want to find those obvious bugs, and bugs which keep on recurring in the same area of the code or software. These kinds of bugs should have been quashed during development and not get through to testing to be discovered, because they only add noise and takes away the precious time we have to discover the more serious bugs which may lay deep within the code.

So, the question then arises: should we, as testers, limit ourselves to only finding bugs or should be expand our scope and responsibilities and help developers adapt to new practices that minimises the bugs being created in the first place during development?