Tag Archives: Functional testing

Book Review Pt1: Black-box testing: techniques for functional testing of software and systems

On the 20th anniversary of Black-box testing: techniques for functional testing of software and systems (1995) by Dr. Boris Beizer, the first book I ever read about… actually, the first thing I read about software testing, I review the book to introduce it to the generation of testers that never heard of it.

First, I have to admit that I forgot the premise of the book was to build knowledge about modeling in order to support development of software tools. I generated tests with a product in the late 90’s called TestMaster by Teradyne Software. Yes, I just linked to GeoCities. I also developed my own generator after being inspired by Harry Robinson’s Intelligent Test Automation.

The section about Missing Models explains this book in relation to software testing books from before this book’s existence, focusing mostly on Myers’ The Art of Software Testing. It describes different types of models including logic-based models and language-based models. Thanks for reminding me about how much I have not read.

Right after the preface, BAM! I see the readme.doc chapter. Now I know that I’m dealing with a softare professional, in the greatest sense of the word. Read this, it’s quick. The books is efficiently organized, so it’s to have a good map. The readme explains the structure of the chapters in relation to each other, and the structure of each chapter.

I am already excited about this.

Improving the Value of Automated Testing

I have an idea. I will not get too high on it except that it is intriguing to me. Maybe the idea is not new to other people. The idea came to me while I was thinking about the test automation pyramid (or ice cream cone as automation expert Alister Scott recognized the typical shape). I am fond of the three-layer concept – enough so that I made myself learn how to write unit tests and service-layer tests, in addition to the GUI-based tests that are commonly practiced by software testers.

I am thinking about the shape of the triangle, which part should be how big, etc. Suddenly the problem hits me. The problem is not isn’t the investment in automated tests. It isn’t the maintenance (which I thought maintenance was a big issue). Alright, I am lying. Investment is half of the equation – the return on investment calculation. The problem is the (lack of) focus on the return for the investment.

So much is invested in tests that will find what? Low severity defects that do not halt releases? I do not care about low severity defects until I have sussed the high severity defects. I care even less about them when I am faced with making a substantial investment to find them. Where does that leave me? Focus on the big bugs. Automation for big bugs. Which bugs? The ones we do not want to ship with them in. The ones that require a patch if you miss them. The ones that make you slip release dates if you catch them late.

“I don’t care about low severity defects until

I have sussed the high severity defects.”

I expect to get criticized for this idea. Why? I have no experience on this. I have never tried it. It is an untried idea. What could be less valuable than a hypothesis? I can try it, but this kind of thing would take a while to prove itself out. But I cannot worry about that while I am brainstorming an idea. The idea will be flawed in many people’s minds. Automated tests are really just automated checks. That is not new. People are not particularly suspicious that the results are false positives, they do not like that the test are little positives – that is they test so little compared what a person can do. I believe this concept too.

What I really believe in is the ends, not the means. The means is how I will get it done. The ends is what I want to accomplish. I care about the automation only when it affectedly helps me get the good defects. What are those? In my world, those are the defects that reduce my organization’s ability to meet agreed or assumed service level agreements. I call those Enterprise Readiness defects. Examples include poor performance, poor performance after time (example: caused by a memory leak), poor resilience during or after high loads, problems with failover, and problems with data retention.

How can I accomplish that? By remembering that automation is just a tool in the toolbox.

More on this later…