What Everybody Should Know About Buying Test Automation Tools

Vendor Tools Fail for Developing Software

Discussed recently on my Twitter  feed was question about test jobs and the required skills. Somebody pointed out that many ads list selenium or QTP. Why both? The person asking told me at one time he had used both and knew both. I think he asked the question because he knew the products were quite different. Or maybe because the hiring manager did not seem to know about that difference.

I have used many products, vendor, open source, and home-grown for automating tests through the years. A vendor test tool just means that somebody is selling it for money.  In this case, lots of money.

They are all  for the purpose of running tests (typically functional tests). However, they are so different that people tend to join a camp and stick with it.  What’s your preference?

My favorite is open source tools (in general), but I am good with whatever tool gets the job done. Productivity is the most important to me. Maybe that is why open source is my favorite. Let me share some of my analysis on the vendor products. Keep in mind the context of my primary job is testing software that is under development. My analysis is greatly influenced by what I need to do.

Source Control

Vendor tools supply source control systems with their tools. I have seen this with QTP and Test Partner. So you have to install a database server or test management system to connect to (and the test management system stores it’s data in a database like SQL Server or Oracle).

Is that good or bad? When the tools were formed, it was probably good – good because testers weren’t used to fooling with source control systems like svn, git, or perforce; good because crazy people like me could ruin hours, days, or weeks of work with a screwed up search/replace-all action; good because they captured so much data when they recorded scripts. Are they still good to have? At this point,  I say no. They are not needed because people are using the dev source control system anyways – two systems are not better than one. They aren’t needed because they editing/using the tests depend on an active connection – people want a copy on their system and only sync when necessary.


Vendor tools number one strength is the plug-ins they build to tackle automated testing on wicked technologies like Flex. And it’s not new. WinRunner and Astra QuickTest were some of the first products to support testing web technologies. This versatility helps sell the vendor tools.

Unfortunately for them, many of the technologies they sell plugins for will come-and-go. I believe that HTML5 will push those EOL’s a little faster. In the meantime, open source alternatives are available for widely used languages like java, ruby, and perl.

Integrations with other tools

Automated test tools integrate with other products like test management systems, defect tracking systems, and requirements tracking systems. There was a time that vendor-sold tools were the most integrated products. Why? Because they often owned all of the tools. They were able to integrate the products through internal knowledge sharing by teams. The integrations, unfortunately, were often based on technologies like dynamic data exchange (DDE) and object linking & embedding (OLE). That was good once upon a time, as were video cassette recorders.

Today products can integrate through the native language when they are libraries. When they are not, RESTful web services are convenient and reliable. I have seen substantial efforts to update vendor products to use modern integration technologies but they are often handcuffed by their customer-bases’ reliance on current product models.

Visually Oriented/Helpful

The first automated tests I ever wrote were tests that I didn’t write. I used a process that has been around since Microsoft Test (later Visual Test, later Rational Visual Test) called Record & Play. Was that helpful? As a learning experience, it was helpful. The tests created by the recordings were not helpful. The recordings are not helpful for long because they do not follow Good Practices for Automating Functional Tests. Even the tool vendors will concede that the recordings must be edited to minimize maintenance in the future.

Try creating the scripts without the recorder. I haven’t used some of the major vendor tools, but I know with QTP that I wasn’t able to build the script with the recorded objects. The objects are the early friend. They help you get the script created without knowing anything. Eventually, though, you will need to fix all those tests because the software under test has changed. How many tests are depending on that object? Are they multiple objects that look similar? Can you effectively minimize the number of objects during your development iterations as the product under test changes? All these concerns make me want to skip the object repositories. I want to know what is going into my tests, what is coming out, and be able to see what is happening behind the scenes.


Vendor tools sell themselves well with customer support. Actually, they sell the customer support. The revenue from maintenance is usually higher than the revenue from sales. They can do this because the customer needs the fixes and improvements from the version they purchased. There is absolutely no DIY for those tools. The good part is that you do not have to fix it, but that bad side if you can’t.

There are forums for discussing issues in open source tools & libraries. The product authors and other volunteers answer questions about the products to help both experienced and new users, including providing direct access to defect tracking systems. When the cause of issues are found, the plans for release are posted and the user can resolve the issue themselves in their own library code base.

Foot Print

For software, the foot print is the total impact on a computer – space required to install on the hard disk drive, RAM used to run it (both edit and execution modes), licensing connections, integration connections, database connections, and cost to license.

The foot print affects every person that will use the automated test scripts to learn about the product under test. Today’s development methodologies are designed to reduce the gap in time from when a defect is introduced to when it is fixed. Build it regularly and test it regularly using continuous integration systems (e.g. Hudson). Let the failing tests be run on the developers system that checked the defective code in to the repository. Foot print matters.

The vendor tools are bulky and expensive to say the least.  QTP 11 requires a minimum of 1gb storage and 1gb RAM. A use can get by without any connections (to database or to QC) but not a team. I haven’t been able to find the exact cost, but I am seeing ~$6,000 on my Google searches.

Test Complete 8 requires a minimum of 700mb storage and 256mb RAM.  It’s $1,000 for an individual node-lock license and $4,500 for an enterprise floating license (prices in between for mixes of those two).

Telerik Test Studio requires 500mb storage and 500mb RAM. Each of  of those three leading vendor tools require Windows XP, Windows Vista, or Windows 7. Their most popular package (some tools but missing most) is $1,300 while the complete package for one user is $7,334.

Compare those minimum requirements to Selenium test automation tools. Selenium is 31mb (server and java client), runs on Windows, Mac OS X, Linux, and Solaris. Ruby, the core language for the Watir and Watir-Webdriver libraries, supports Windows, Mac OS X, Linux, Solaris, and (I have never heard of) OpenIndiana. Ruby is 275mb (measure for v1.93) with Watir-Webdriver requiring an additional 18b. They are both free, the support is free, and the maintenance is $0.  That’s a huge difference.


For automating tests on software development projects, I believe the vendor tools are inferior to the open source tools. It seems counter-intuitive – why wouldn’t the vendor tools adopt the capabilities of the open source tools?

First, they can’t because of the install-base requires them to remain the same. These products have been selling for up to and even more than a decade. People, departments, companies rely on the tools they have invested so much time into. Regardless of whether they tests are brittle or not, they cannot choose (or allow the vendor to choose) a radical change to how their systems are built. So instead they build new features and modules to sell while the core remains the same.

Second, they do not want to because their target is vast market of uninformed manual testers and managers. I have walked through vendor showcases at StarWest. They have interfaces that are way sexier than VIM. The trials seem so easy. “Come on in here, automate a test – everybody is doing it!” So you do it and it works beautifully. Of course you don’t have to run that test on your system, against your software, or maintain it. Their core target usually knows none of the good automation practices they should follow. But their companies take their advice and pony up the money.

As I mentioned before, the context of my job plays a significant role in my analysis above. If I were choosing a product for testing in an IT department (which is service-oriented), my priorities would change, and the tools I favored may change. I do not dislike vendors. I do not think they mean harm, in fact they provide utility.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s