Monthly Archives: January 2012

Good Practices For Automating Functional Tests


I spend a lot of time talking about the benefits of automating tests – automated checks as explained by Michael Bolton. I call them automated tests out of habit (and clarity to the uninitiated). Part of the responsibility of teaching this subject is teaching people to follow good practices. Almost all of these practices I lifted from somebody else.

Why do I like automated tests? Because I want to reduce the time between when a problem is introduced and when it is reported to the person that introduced the problem. Some time in history, man measured the cost of fixing defects and it was more expensive the longer it went unnoticed. This is a generalization but I take it seriously because my job is to help people make good products. My value is in the reduction of costs and increase in benefit I create for the pay I receive. When I do that more effectively, my value increases. Since I work in the computer software business, I ought to be using technology to reach those ends.

Balance in what you test (Unit, Service, GUI)

I have heard about the automated testing pyramid from several people and read on many blogs such as Michael Cohn’s – I don’t know who originated the idea, but I heard about it from Janet Gregory when she trained my team at HP. If all of the tests for a product are through the graphical user interface, then there’s a lot that isn’t getting tested, the tests will tend to be at a higher cost to maintain because interfaces tend to be change more than the classes and services, and they will be found later in the product cycle as the GUI tests tend to require all of the product layers are built (regardless of the order they are created).

Not Everything Should be Automated

James Bach recently wrote a blog post on the skill required to create scripted tests. The lesson I received was that scripting tests to the point they become checks is high even if people are running the. The investment goes even higher when the interpreter is a computer program because the instruction needs to be so much more precise. The return is lower because the interpreter only sees what it is told to see.

Then where is the value? Automating the activities that cost less to automate than to perform manually. Consider partial automation as a great alternative to the automate/manual question.

  • Inject data to set up the test scenario – possible sql queries or web service calls
  • Verifying the unseen changes – sql queries (new/changed records), parsing logs, or web service calls
  • Navigation to the location of the test – this could be opening web pages, logging in, and going to a certain web page.
  • Capturing screen shots for human eyes to review
  • Notification of environment changes
I recently wrote a blog post on using Interactive Ruby to support manual testing with automation which may help you see the automation and manual thought process marry.
One more thing to consider: do not automate tests that will not be run repeatedly. Do not even script them. Keep notes on what was done just in case you need to do some forensic analysis. Just don’t automate them!

Pass/Fail Criteria

The first mistake I ever made in automated tests was to think I automated a test by creating the automated navigation. Without some criteria to know if the test passed, the “result” is useless.

I like to know which tests failed separately from which did not complete. If the problem is not what you are testing for, it’s an exception. When the test fails, somebody has to figure out if there is a failure in the product, if the product changed without a like change in the tests, or if there is a failure in the tests (presumably, a good practice was not followed). Assuming there is a failure in the product, a defect that will be fixed or not fixed – it is a known problem. In the case of the incomplete test, the problem is unknown because we haven’t actually seen what happens when we get to the end – you probably do not know if the test would pass or not.

Layered Framework

The first thing I got right when I started automating tests was to create layers. In fact, I was so sure that it the only possibility for being successful that I was shocked to hear Dorothy Graham talk about the idea in Lightning Strikes the Keynote at StarWest 2010 (as if it were a new idea). Maybe some automation tools make this separation so difficult that some people don’t do it.

The best way to describe this is to separate the what from the how. The what should be your test cases (like general instructions for a manual tester). The how should be your test framework (classes and methods). The what should be the business logic that needs to be tested (or the workflow, or whatever). The how should be the specific instructions for dealing with the interface (click this, fill in that). The what is your customers actions. The how is the implementation of your interface that supports what the customer wants to do.

The purpose is to simplify maintenance when the product creators (dev team in my case) changes the how. We often do not see that separation in step-by-step manual test scripts (also called checks by the experts). If we change the submit form action from clicking a button to a gesture (such as a Be-Witched nose wiggle), there is one place to change the code so that all tests incorporating that form submission will work.

I have seen the separation in many different ways and levels. In some cases, the framework supported domain specific actions (log in, set field, submit form). In other cases, the framework supported transactions (create account, update profile, purchase subscription).

Run 1 Check per Test

I like the idea of separating each test from the others. Not because I care how many test cases exist but because I want separation of failing and passing results. I do not find value in doing and checking 10 things if they all pass or they all fail because one check failed. This also means to go straight to the tested functionality. Did you want to test the navigation? Do that in another test.

What about an end-to-end test? For example, suppose you want to test creating an order, fulfilling the order, charging a credit card, and notification of completion? If it’s a straight through “complete” use case, then you are testing one thing.

Timing Dependencies

There are two timing-related considerations here. First, sometimes tests must wait. They wait for a web page to load, they wait for JavaScript to be completed. You should not wait by guessing how long is necessary. Many tools come with wait_until type functions such as browser.div(name=>’javascript complete’).wait_until_exists which will delay the script the right amount of time. If you don’t get that in your tool, create loops that check for existence with a timeout.

The second timing consideration is dependence on something happening that we don’t know when it’s going to happen. Suppose I create a situation that will trigger a notification when the notification scheduler runs but I don’t know exactly when it will run. I can either help it along will triggering the notification scheduler manually or … consider not doing that test. Nobody wants the automated tests hung up for 35 minutes.

Do Not Assume the Data Exists

In a previous job that I had, the product under test came with a sample database. I found that it was often used to support manual tests. The problem with the assumption that the sample data will be there to support the test is that other tests could change or remove the data. Manual testers will make the adjustment by creating the data at that point. An automated test will… stop.

When we converting the manual tests into automated tests, one of the first features we added to the framework was to allow us to create the data needed to support the test. In that case, we used web services calls.

The Most Reliable Way to Use Data

A long time ago I worked on a system that had almost no data to support my tests. I would spend an hour creating data through the web interface. I hope that nobody does what I did, not even with a web automation tool. I solved that problem by learning how to import xml files with the data needed to support my tests. Since then, I have used api’s (including web services api’s) and sql scripts to inject data. Use the most reliable way possible.

Clean Up After Your Tests

“A job isn’t finished until you have cleaned up after yourself” said my father. I say the same thing about testing. For the sake of other tests that will run after yours, you may consider cleaning up.


I spend years learning about these practices. Sometimes I learned the hard way, other times I was fortunate enough to learn them from a seasoned professional. I did not want to call this best practices because that would assume I know the context. There are so many situations that I could not know the best practice for each of them. Consider each of them with the help of your team before making a decision. If the others that depend on the test results understand these practices, they can often make implementing the practices much easier.

Additional Acknowledgments

In addition to the ones already in the post, I would like to recognize my co-workers Jean-Philippe Boucharlat and David C. Cooper, whom I worked with at Hewlett-Packard, for sharing their insights to automation best practices.

Update Notes – February 10 2012

I have updated the original post to improve the article from the feedback of the kind readers.

There is an App for That – or Ruby Library

We have heard the ads and promotions for the iPhone. There’s an app for that! Of course there is. Sometimes the app is even good. I am bummed when it is not, but I will keep trying and evaluating them – yes, I have an iPhone.

I realized that this phenomenon also occurs with the Ruby scripting/programming language. I was teaching my co-workers how to use Ruby & Watir, and the benefits of those. I wanted to demonstrate the usefulness of Ruby as a scripting language so I started to show different libraries that could be used. Although it had been around me for years, I finally realized that Ruby covers some major ground thanks to all of the developers and hacks (I use that term in a loving way)  that spent hours, days, weeks, and months extending Ruby to do more.

First, when I read through the Pickaxe book, I learned about many of the installed libraries I can use with Ruby. Sure, we expect to have the string, array, and hash types. But with a simple require ‘net/http’ statement, I can request web pages over the internet – without a browser! I can create my own app server using webrick – I did just that as a target for my squid server test. I can work with file systems, command line parameters, cgi scripts, yaml streams, xml streams, and test it all with test unit.

But wait, that’s not all you get! I learned Ruby to build a test framework. After that, I found myself in a manager position where I could not spend so much time working on automated checking and frameworks. Wanting to keep in practice, I started using Ruby to solve other computer problems for me. I created scripts to help me manager my music library. I automated tedious tasks at home.  For all these, I found websites that could help me identify the best one to use for solving my problem including Ruby Toolbox, RubyGems, and GitHub where many of them are created. On GitHub, you can even fork the project that you kinda like but want to change some stuff. I have seen it done! And you can get it for the low, low price of Free. You will never get that kind of free stuff and you can change it on an iPhone app. And if you order now, you can get an additional Ruby installation, just pay storage fees.

If I had to pick a favorite, it would be Watir (Web Application Testing In Ruby) because that library enticed me to learn Ruby. Because I have learned more libraries and how to use them, I have ventured away by using database client libraries, web service libraries, and faster-performing web libraries.

I would like to know – what is your favorite Ruby library?

Leadership (part 6) – Your Story

As a result of this series, I was able to lead a 1 hour seminar session about leadership at Toastmasters Leadership Institute in San Diego on Saturday where I shared an interactive discussion. I want to keep building on the topic of leadership because it is so interesting. I found the interaction of the comments on this blog to be helpful to me, and even more so with the questions and comments that I received in during the interactive session.

I want your story. I am requesting for you to answer these questions, preferable by video recording, so that I can include them in collage of answers. Please contact me by email using mcnulla at gmail dot com so we can arrange to transfer a copy of your video. Also, see the release form to make sure you agree with it.  Thank You!


For good and valuable consideration, the receipt of which is hereby acknowledged, I hereby consent to the photographing of myself and the recording of my voice and the use of these photographs and/or recordings singularly or in conjunction with other photographs and/or recordings for advertising, publicity, commercial or other business purposes. I understand that the term “photograph” as used herein encompasses both still photographs and motion picture footage.

I further consent to the reproduction and/or authorization by David McNulla to reproduce and use said photographs and recordings of my voice, for use in all domestic and foreign markets. I hereby release David McNulla and organizations that he works with including Toastmasters International, and any of its associated or affiliated organizations, their directors, officers, agents, employees and customers, and appointed advertising agencies, their directors, officers, agents and employees from all claims of every kind on account of such use. I am at least the age of 18 years.

Leadership (part 5) – Getting Others to Participate

Leadership is not an individual sport – others have to play or it won’t happen. I will discuss how you can convince others to join you and participate. I am tailoring this section to Toastmasters because I have been asked to speak on this subject in a couple of weeks, but this kind of leadership applies to so much more than Toastmasters. My great friend Ed, a leader in his church, said to me “You nailed it with leaders being servants. That is the way Jesus modeled true leadership.” This stuff applies all the way around non-profit organizations, families, and sports teams.  Your assignment, if you choose to accept it, is to create more leaders. It’s for you, so you have less dead weight to carry. It’s for them, to have purpose and a better life to live.

“Whoa, Dave,” you might say, “Why does it matter to me if somebody is a poor leader of a Toastmasters club I don’t attend?” It matters because you believe in the organization, otherwise you wouldn’t be part of it. It’s a good organization because of the people – the ones that came before and the ones that come later. It matters because a leader doesn’t want a failure like that on his watch. And I am glad it didn’t take much to convince you, I knew that it wouldn’t.

Let’s get on with the leadership building.

Now the foolish salesman will treat everybody the same. One size fits all. That is a shame because many will miss the point of their leadership opportunity. They need to know how this helps them. The things they need to know depend on where they are at in their journey. Some of the leaders are experienced. Others are just starting. I believe the key is how much or how little they have benefited so far. Somethings must apply to all. Here are some things you can tell them:


  1. Praise their good qualities. Of course, you have to recognize their good qualities first. And tell them. This should start and end every time you give advice.
  2. Remind them about teamwork. If you don’t have time to do it right, you definitely don’t have time to do it alone.
  3. Work the Toastmaster leadership program. You will read about the basic skills to be successful, practice to learn those skills, and fill larger roles to apply those skills at a higher level.

Experienced Toastmasters:

  1. People make the organization happen. A well run organization doesn’t happen automatically like the seasons, flowers, and fruit. You have received that benefit, so pay it forward.
  2. Now you have a chance to create your legacy and greater reputation. You are building your brand. How do people think of that brand?


  1. Others are there to help you. Nearly every situation you encounter, somebody else encountered that or close to that circumstance. Ask for their help, then honor that help by seeing it through.
  2. Leadership is rewarding. Whether you decide to conquer the world, or return to being the servant leader, you will always appreciate your own capability and accomplishments.
  3. You will develop a greater understanding of the organization, and of the skills you need to master your own definition of success.

Let’s go back to the first image that we had in this series – the one about the new leader. The new leader is ready to take on new responsibilities because of learning leadership skills. He is going to be a successful leader. But you are not that leader. You are not the Grasshopper, Kwai Chang Caine. You are the blind master Po. You have taught and encouraged the next generation of leaders. Because the greatest act of a leader is to create a leader that he himself would follow.

Join us at the Test Automation Bazaar

Join us at the Test Automation Bazaar.

There are few products in the world that you can get for free. Fewer that allow you to meet and learn from the product creators for so little money (about 20% the cost of 2 days at StarWest or StarEast). If you are serious about automating tests and test-related tasks, this is a conference for you!

If my employer allows me to go, I guarantee that I will be there.

Update 01/29/2012: I keep asking my boss about attending, but I cannot get a read on what they are thinking. My pessimistic self tends to think that will mean they will not support me in going  😦

Update 02/10/2012: My boss said that my trip is approved. I have my ticket and I am going!

Automation Bazaar Attendees

Leadership (part 4) – How do I become a leader?

In my last post, I said that I would talk about how to become a leader. I conjoin that subject with my experiences here.

Most of my experience in leadership was in leading the peanut gallery. People actually recognized me as the leader of the cut-ups in high school, enough that a guy in my calculus class told me the substitute teacher wouldn’t let go to the bathroom because he knew that I’d make a scene about it. That’s a horrible way to develop leadership skills because it develops the wrong kinds.

Eventually I stopped that kind of leadership (or reduced it enough that it stopped holding me back in my career). I tried to excel in my “individual contributor” position and was eventually rewarded with being in charge of others. I felt inadequate about “leading” so I read books like the ones from Hyrum Smith (Franklin Quest) and Steven Covey (7 Habits) – they eventually merged their companies.

The problem with reading books is that you don’t get any intentional practice in many of the aspects. They will focus on an area or two. I can remember creating a mission statement, listing my roles in life, and objectives to fulfill those roles. I cannot remember reaching any of those goals, but I definitely had a better attitude. Unfortunately, attitude is only a part of being a leader. I needed more skills (the culmination of education and experience).

I learned a lot  from unintentional experience. Given responsibility, you will eventually be put into situations. The problem with unintentional experience is that it rarely tells you if you made good choices or the best choices – that is something even intentional experience falls short in many situations.  In many cases, nobody was mentoring me to suggest key options from which to choose. There was no plan for what to do next.

I also signed up for any day-long courses offered at my company (big companies have lots of courses on-site to choose from). The problem with short courses is the practice is very limited. After it’s over, I could review the booklet but that eventually was packed in a box or drawer.

I had seen people take longer courses and participate in programs, like University of California at San Diego’s Passport to Leadership program.  That seemed to have a big affect on them with self-assessment, practice, and goals. I never joined it because of the financial commitments that it required. But it looked like something I believed would help me.

Then in 2006, in a reason having nothing to do with leadership (except for the self-improvement part), I joined Toastmasters International to help my speaking skills. There was a club starting at my work place. They were willing to pay the small bill. I didn’t have to go anywhere to do it because the meetings were at my office. All I had to do was show up and try to get better at speaking.

Then I got roped into the leadership track. Somebody asked me to be Sergeant at Arms for the coming period. “No big deal,” he said, “just bring stuff to the meeting and set it up.” He was right, it wasn’t hard. Then I was President, started giving some trainings to the club, and found myself on the Leadership Track.  Why was that significant? Because it is the substance behind the style that I was learning from the Communication Track. The significance is that it is one of the best-value educations for becoming a leader that we could ever get. I put that in bold because it is a significant statement that they back up.

How does Toastmasters prepare us to be leaders? The Leadership Track teaches us the skills through training, projects, and plenty of practice situations.

“The Competent Leadership manual features 10 projects, which you complete while serving in various meeting roles and participating in other club activities such as helping with a newsletter or getting involved with a membership building campaign. An evaluator will give you feedback on each project, helping you to improve.”

Toastmasters gives us an opportunity to practice the skills we learn for leadership. When I go to a job interview, I can talk about leading a team of executives. I can discuss the problems we faced and how it helped us. Without that experience, who is going to trust me to lead? What came first, the chicken or the egg? In this case, it’s both.

Toastmasters gives us guidelines on how to be successful. They know what it takes, which is why they have guidelines and predetermined goals (such as how to become a distinguished club). Their system works if you work the system. Is there room for variation? Yes, of course. Clubs are all different and have variations of needs. However, as long as problems are being addressed, you will achieve success.

Everybody in Toastmasters is a volunteer and has to be treated that way. How is that different from real-life leadership? It is not different at all. As a Software QA Manager, I had as many as 24 people reporting to me. All of them volunteered their best effort. As their manager, I couldn’t make them do anything. In addition, most of the people who I needed things from to be successful did not report to me. I needed support from my boss. I needed cooperation from managers of other departments. I needed cooperation from individuals in other departments. Toastmasters prepares people to ask for things, ask again, and communicate what needs to be done very well.

Toastmasters provides lots of help. They have a formal system for training officers, as well as training the district supporting officers. They have a book to explain every necessary job in Toastmasters from the club on up to the highest position, International President. They have a website that can tell what you do not know or forgot. They have a system of sponsors, mentors, and coaches available for whoever wants one. And they have, at least in District 5 that I am in, over 1000 people to talk to about a problem for free advice. Anybody who fails inside Toastmasters probably refused that help. I do not believe that should ever happen.

“Dave, what is the catch?” you ask. The catch is 2 things. First, it is not instantaneous. It takes time to build up the experience and the trust to get the higher levels of experience. I would say that the fastest that anybody could reasonable get through the entire leadership track is 2 years, maybe 3. The second is that it requires you to do it for no cash pay. Toastmasters can afford to let you be in charge because they need you to do it for free. But where else can you get those four key ingredients to become a leader.

I want to hear about your experiences with becoming a leader too. There are a million ways to do it. Please comment with your story and feedback.

In my next post, I will discuss how you can convince others to join you and participate. Leadership is not an individual sport – others have to play or it won’t happen.

Leadership (part 3) – Why do I need to be a leader?

I misread my outline in Part 2. I promised to cover how to become a leader without realizing my next topic was about why we need to be leaders. Obviously you believe in leadership or you wouldn’t be here. But let me tell you some reasons that you need to be a leader. They won’t all apply to you, but nobody gets out of all of them.

Ross Perot’s sucking sound of jobs going overseas has become part of the norm, hitting blue-collar and white-collar jobs alike. In our jobs, we are paid to think, solve problems, and usually work as a team. We have to bring value. We need to differentiate ourselves by being leaders and followers are the right times (only a leader would know which is appropriate at what time).

Leadership is part of our brand. A brand is something we trust, like the usability of Apple, the convenience of, or the on-time deliver of Fed-Ex. Rain Man wanted to fly Qantas Airlines because they were known for not have accidents. He trusted their brand, not because of an advertisement. Our rewards depend on the way people see our brand. We are drawn to good leaders, provided we trust them. There’s trusting people to keep a secret. There’s trusting people to do their job. There’s trusting people to make sure the team is successful.

Nobody wants to be a victim of changes. We want control over our lives and what happens. We have things we want to see happen in our lives, at our jobs, in our worlds. Take it from the most reluctant leader, I just wanted to go for the ride. The problem is that eventually I was unsatisfied with where the bus was taking me. How do we get the outcome if we do not contribute to the outcome? How do we contribute to the outcome without building trust from others? How does another trust us if we haven’t listened, assessed, or shared a vision?

The future will come. What the future will become is not determined yet. Will the world get hotter? Will governments bend too far to the left or the right? What happens when we run out of fossil fuels? Can our children become competent with declining relative investment in schooling them? Nobody is going to solve all these problems, or even one. Teams will work on them just to limit their effect, in fact teams of teams. You have an interest in these things. Will you join one cause to help solve one problem? And as your children watch you will they be inspired to lead in a cause, or will they be inspired by the TV schedule?

In my next post, I will discuss how to become a leader (I really promise, this time I mean it!).