I have long hoped that San Diego testing community would develop a user group/meetup. My recruiter, Jon Barton of TEKSystems, started a meetup group. I attended a meeting in April. Then I volunteered to speak at a future meeting. That future meeting was last night.
Two things happened to me lately. First, I was trying to find a career tester in the San Diego area that knows at least a little bit about automated testing. It isn’t going well. I’ve reviewed a lot of resumes. all the submitters are career manual testers.
Surely somebody sometime must have wondered if they need to learn more about automation. Elisabeth Hendrickson once asked Do Testers Have to Write Code? They did a survey to figure out what companies were looking for from tester skills. In our case, we aren’t looking for somebody to write the test code, but to write and review the cucumber scenarios. Just the same, even on a light desire level, I was disappointed.
Second, a younger person asked me what he should learn in test automation last week. I had already been contemplating writing this curriculum, so I was resolved to do it. Srini, here it is.
Other people who don’t work with LAMPs, such as .Net environments etc. will probably not appreciate this list. Make your own list on your own blog and put the link in a comment here. I don’t begrudge anybody doing something else. I just don’t want to go there.
I created this curriculum for testers learning test automation. While some addresses how and why, most of the list is about tools that can help create a full solution. Anyway, here is my list in priority order:
An open source tool such as Watir-Webdriver or Selenium/Java – do not mess around with the QTP and TestComplete. The cargo cults that buy those tools will expect “anybody can automate”. With open source tools, you can download your own learning playground and incorporate that with the other products.
Learn how to create page objects. Even if you take advantage of a library like WatirMark or Page-Objects, you will have to do some tailoring yourself. I have been working with Selenium/Java so I am developing my skills on that combination now. Either way, you need to know how to work on that in an efficient way. In fact, you can address a lot of the entries in here just be using Cheezy’s book Cucumbers and Cheese (well worth the $15). I swear that I do not get a dime from it or Cheezy’s work, it’s just such a big benefit for anybody learning that I cannot miss the chance to say how good it is.
Github and Git – there are other good source control tools out there, including subversion. Git is easy to use locally for managing your own practice code. It’s easy go get copies of other people’s public projects onto your own system (how did they do that?). CodeSchool has a free course on git. There is also a nice paper on the differences between git/mercurial and subversion so you can understand the differences.
Ant and Maven if you use Java. Most of what I learned was through osmosis, but being able to shoehorn cucumber into your project is good to know.
Jenkins or Hudson, CruiseControl, or some other open source continuous integration tool. If you ever work at a place that will be introducing automated testing for the first time, this is great to know how to set it up.
Performance testing in JMeter – I think you can find a ruby alternative (BlitzIO or Grinder) but you don’t really need this tool to be in a ruby language. The importance is to learn the different kinds of testing you find under this umbrella (incorrectly) called Performance Testing. The other important skill is creating the right monitors so you can discover where things are bottled up.
Owasp‘s ZAProxy – learn how to capture the http calls between your browser or simulator and the server under test. You will learn a lot there. While you are there, download the GoatWeb project where you can learn about security vulnerabilities through practice.
Monitoring tools (Splunk or Graylog2) – One way to find the errors that are occurring on the system under test is through logging. Those are deleted nearly every time the server is redeployed. You can monitor those logs and server performance much better through a monitoring server.
A true startup is probably not going to hire a newb unless they are cost-control-centered. But if you find you get there are there is no issue tracking, it would be good to know how to set up issue tracking and integrate to your version control and continuous integration server. I’ve tried RedMine and it was fine.
If you see that you think should be on the list that is not there, please add a comment.
Discussed recently on my Twitter feed was question about test jobs and the required skills. Somebody pointed out that many ads list selenium or QTP. Why both? The person asking told me at one time he had used both and knew both. I think he asked the question because he knew the products were quite different. Or maybe because the hiring manager did not seem to know about that difference.
I have used many products, vendor, open source, and home-grown for automating tests through the years. A vendor test tool just means that somebody is selling it for money. In this case, lots of money.
They are all for the purpose of running tests (typically functional tests). However, they are so different that people tend to join a camp and stick with it. What’s your preference?
My favorite is open source tools (in general), but I am good with whatever tool gets the job done. Productivity is the most important to me. Maybe that is why open source is my favorite. Let me share some of my analysis on the vendor products. Keep in mind the context of my primary job is testing software that is under development. My analysis is greatly influenced by what I need to do.
Vendor tools supply source control systems with their tools. I have seen this with QTP and Test Partner. So you have to install a database server or test management system to connect to (and the test management system stores it’s data in a database like SQL Server or Oracle).
Is that good or bad? When the tools were formed, it was probably good – good because testers weren’t used to fooling with source control systems like svn, git, or perforce; good because crazy people like me could ruin hours, days, or weeks of work with a screwed up search/replace-all action; good because they captured so much data when they recorded scripts. Are they still good to have? At this point, I say no. They are not needed because people are using the dev source control system anyways – two systems are not better than one. They aren’t needed because they editing/using the tests depend on an active connection – people want a copy on their system and only sync when necessary.
Vendor tools number one strength is the plug-ins they build to tackle automated testing on wicked technologies like Flex. And it’s not new. WinRunner and Astra QuickTest were some of the first products to support testing web technologies. This versatility helps sell the vendor tools.
Unfortunately for them, many of the technologies they sell plugins for will come-and-go. I believe that HTML5 will push those EOL’s a little faster. In the meantime, open source alternatives are available for widely used languages like java, ruby, and perl.
Integrations with other tools
Automated test tools integrate with other products like test management systems, defect tracking systems, and requirements tracking systems. There was a time that vendor-sold tools were the most integrated products. Why? Because they often owned all of the tools. They were able to integrate the products through internal knowledge sharing by teams. The integrations, unfortunately, were often based on technologies like dynamic data exchange (DDE) and object linking & embedding (OLE). That was good once upon a time, as were video cassette recorders.
Today products can integrate through the native language when they are libraries. When they are not, RESTful web services are convenient and reliable. I have seen substantial efforts to update vendor products to use modern integration technologies but they are often handcuffed by their customer-bases’ reliance on current product models.
The first automated tests I ever wrote were tests that I didn’t write. I used a process that has been around since Microsoft Test (later Visual Test, later Rational Visual Test) called Record & Play. Was that helpful? As a learning experience, it was helpful. The tests created by the recordings were not helpful. The recordings are not helpful for long because they do not follow Good Practices for Automating Functional Tests. Even the tool vendors will concede that the recordings must be edited to minimize maintenance in the future.
Try creating the scripts without the recorder. I haven’t used some of the major vendor tools, but I know with QTP that I wasn’t able to build the script with the recorded objects. The objects are the early friend. They help you get the script created without knowing anything. Eventually, though, you will need to fix all those tests because the software under test has changed. How many tests are depending on that object? Are they multiple objects that look similar? Can you effectively minimize the number of objects during your development iterations as the product under test changes? All these concerns make me want to skip the object repositories. I want to know what is going into my tests, what is coming out, and be able to see what is happening behind the scenes.
Vendor tools sell themselves well with customer support. Actually, they sell the customer support. The revenue from maintenance is usually higher than the revenue from sales. They can do this because the customer needs the fixes and improvements from the version they purchased. There is absolutely no DIY for those tools. The good part is that you do not have to fix it, but that bad side if you can’t.
There are forums for discussing issues in open source tools & libraries. The product authors and other volunteers answer questions about the products to help both experienced and new users, including providing direct access to defect tracking systems. When the cause of issues are found, the plans for release are posted and the user can resolve the issue themselves in their own library code base.
For software, the foot print is the total impact on a computer – space required to install on the hard disk drive, RAM used to run it (both edit and execution modes), licensing connections, integration connections, database connections, and cost to license.
The foot print affects every person that will use the automated test scripts to learn about the product under test. Today’s development methodologies are designed to reduce the gap in time from when a defect is introduced to when it is fixed. Build it regularly and test it regularly using continuous integration systems (e.g. Hudson). Let the failing tests be run on the developers system that checked the defective code in to the repository. Foot print matters.
The vendor tools are bulky and expensive to say the least. QTP 11 requires a minimum of 1gb storage and 1gb RAM. A use can get by without any connections (to database or to QC) but not a team. I haven’t been able to find the exact cost, but I am seeing ~$6,000 on my Google searches.
Test Complete 8 requires a minimum of 700mb storage and 256mb RAM. It’s $1,000 for an individual node-lock license and $4,500 for an enterprise floating license (prices in between for mixes of those two).
Telerik Test Studio requires 500mb storage and 500mb RAM. Each of of those three leading vendor tools require Windows XP, Windows Vista, or Windows 7. Their most popular package (some tools but missing most) is $1,300 while the complete package for one user is $7,334.
Compare those minimum requirements to Selenium test automation tools. Selenium is 31mb (server and java client), runs on Windows, Mac OS X, Linux, and Solaris. Ruby, the core language for the Watir and Watir-Webdriver libraries, supports Windows, Mac OS X, Linux, Solaris, and (I have never heard of) OpenIndiana. Ruby is 275mb (measure for v1.93) with Watir-Webdriver requiring an additional 18b. They are both free, the support is free, and the maintenance is $0. That’s a huge difference.
For automating tests on software development projects, I believe the vendor tools are inferior to the open source tools. It seems counter-intuitive – why wouldn’t the vendor tools adopt the capabilities of the open source tools?
First, they can’t because of the install-base requires them to remain the same. These products have been selling for up to and even more than a decade. People, departments, companies rely on the tools they have invested so much time into. Regardless of whether they tests are brittle or not, they cannot choose (or allow the vendor to choose) a radical change to how their systems are built. So instead they build new features and modules to sell while the core remains the same.
Second, they do not want to because their target is vast market of uninformed manual testers and managers. I have walked through vendor showcases at StarWest. They have interfaces that are way sexier than VIM. The trials seem so easy. “Come on in here, automate a test – everybody is doing it!” So you do it and it works beautifully. Of course you don’t have to run that test on your system, against your software, or maintain it. Their core target usually knows none of the good automation practices they should follow. But their companies take their advice and pony up the money.
As I mentioned before, the context of my job plays a significant role in my analysis above. If I were choosing a product for testing in an IT department (which is service-oriented), my priorities would change, and the tools I favored may change. I do not dislike vendors. I do not think they mean harm, in fact they provide utility.
We have heard the ads and promotions for the iPhone. There’s an app for that! Of course there is. Sometimes the app is even good. I am bummed when it is not, but I will keep trying and evaluating them – yes, I have an iPhone.
I realized that this phenomenon also occurs with the Ruby scripting/programming language. I was teaching my co-workers how to use Ruby & Watir, and the benefits of those. I wanted to demonstrate the usefulness of Ruby as a scripting language so I started to show different libraries that could be used. Although it had been around me for years, I finally realized that Ruby covers some major ground thanks to all of the developers and hacks (I use that term in a loving way) that spent hours, days, weeks, and months extending Ruby to do more.
First, when I read through the Pickaxe book, I learned about many of the installed libraries I can use with Ruby. Sure, we expect to have the string, array, and hash types. But with a simple require ‘net/http’ statement, I can request web pages over the internet – without a browser! I can create my own app server using webrick – I did just that as a target for my squid server test. I can work with file systems, command line parameters, cgi scripts, yaml streams, xml streams, and test it all with test unit.
But wait, that’s not all you get! I learned Ruby to build a test framework. After that, I found myself in a manager position where I could not spend so much time working on automated checking and frameworks. Wanting to keep in practice, I started using Ruby to solve other computer problems for me. I created scripts to help me manager my music library. I automated tedious tasks at home. For all these, I found websites that could help me identify the best one to use for solving my problem including Ruby Toolbox, RubyGems, and GitHub where many of them are created. On GitHub, you can even fork the project that you kinda like but want to change some stuff. I have seen it done! And you can get it for the low, low price of Free. You will never get that kind of free stuff and you can change it on an iPhone app. And if you order now, you can get an additional Ruby installation, just pay storage fees.
If I had to pick a favorite, it would be Watir (Web Application Testing In Ruby) because that library enticed me to learn Ruby. Because I have learned more libraries and how to use them, I have ventured away by using database client libraries, web service libraries, and faster-performing web libraries.
I would like to know – what is your favorite Ruby library?
I want to premise this post with two things. First, I didn’t think of this. I heard Jim Knowlton interviewed on the Watir Podcast. I also attended a web cast with Michael Bolton on using automated testing in a manual form. Both inspired me to try this myself. As some of my software friends are aware, I dig testing with Ruby and Watir (Web Application Testing In Ruby). I didn’t know anything about Ruby or Interactive Ruby (IRB) before working with it, starting in about 2007-2008. But since starting to use it, I found I am more productive by “trying” my code out in the IRB interface before adding it to my functions. Very handy!
My eyes were opened even more when I heard Jim talking about using IRB to facilitate his testing (and his co-workers) by creating libraries that assisted him. I started to envision things I could use it for: pumping data into a system for create conditions for testing business logic; grabbing near-real time information from log files; quickly getting to the ‘state’ in which I want to explore; and grabbing unseen information about the interface that I am looking at such as the state of objects and making them ‘seen’. I remember being so excited that I asked other people to listen to the podcast.
The webcast with Michael was revealing too. I could see with a simple command line IO, inputing numbering and getting numbers back helped me quickly learn about a system that I knew nothing about. He talked strategies to use exploration, as well as tactics that supported the strategy.
Those lessons came to fruition at my new job. I started with needing to test a web service. I have tested SOA web services before. But I had not tested the with Ruby before. I saw the team I was joining using a RestClient. They were modifying the ‘get’ and ‘delete’ calls based on the data set from the ‘create’ call. I asked myself how many times I wanted to copy that data, the answer was three times. So I pulled out the restful web services book that I had opened one or two times in the past, and found a snippet of net/http to call web services. Once I figured out how to add the pieces I needed, in less than a day, I had a class that could perform all three calls and retain the data, allowing me to test the web services api until I was satisfied that it was working according to the design. Running these from IRB allowed me to run manual tests. I kept changing the tests. Even though they were written in QC with instructions on parameters, I kept asking myself “what if I did it this way?” which allowed me to learn and satisfy myself that the API was strong.
I also saw the other team members copying and pasting data from the create api call to use in a lynx (text-based command line browser) for testing business logic. I found all the typing and copying unnecessary and mistake-prone so I ‘printed’ the lynx command out allowing me to copy and paste the entire thing. This allowed me to more quickly get to the business of testing. I used that gained time to add a direct command from Ruby, reducing a step to run quicker, and finally incorporating the Mechanize library with that saved time.
The fruit of that labor was to allow me to test more quickly from the web service call to the browser call. That in itself did not find any defects, but it enabled me to see the need for faster (virtual) browser calls which did find three major defects and a minor defect. In spite of these gains, my manager was concerned about the cost of maintenance of early automation – a lesson I learned from James Bach’s Test Automation Snake Oil that says “Manual testing, on the other hand, is a process that adapts easily to change and can cope with complexity. Humans are able to detect hundreds of problem patterns, in a glance, an instantly distinguish them from harmless anomalies.” I gave her a demonstration to show that my mind was in control of these tests. She felt at ease and asked me to continue.
The final culmination was today when I discovered a problem in a caching algorithm that would not be detected with smaller slower data-flows. I had no intention to test for load. I wanted to test functionality in a system environment and more realistic volumes. At this point, I did create a Ruby “script” but even then it took command line arguments to allow me to modify the test cases as I saw fit. There were no magical red or green colors that filled a test management system with results. Just tests that I ran as I felt were needed to learn if this system was ready to release or not.
I am starting a new job after such a long, wonderful tenure at Hewlett-Packard. After 10 years, if you include my tenure at Peregrine Systems which was purchases by HP, HP gave me walking papers in the nicest way 11 weeks of severance plus vacation pay. And my boss actually got me the lead for this job!
But it’s a break, a change. What’s going to happen? Will I do OK? Will I like the new job? I am a guy that likes small changes on a steady base. So I was not so much of freaking out but ‘concerned’. Let me tell you that those concerns were laid to rest.
Challenge – I am not going to say this place is perfect because it’s not. That is part of the attraction. I have never been satisfied at turning bolts all day. I have to find a way to make it better. If it was heavenly already, the I couldn’t help raise it up. They seem to be caught in a scrumbutt world and I know their issues from experience. Hopefully I can develop enough credibility to have an audience.
Automation! – Yes that’s an exclam. I am NOT a programmer but I like to solve problems and I can solve some test efficiency & capability problems with automation. I was able to build a proof of concept in the first few days using ruby as the language, net/http as the library, and irb as the interface tool. I was running manual tests more efficiently in a hurry, and the developers started using those tools too.
Nice People – I felt that everybody give me criticism in constructive ways, tries to listen to me even when they don’t want to, and answers tons to good and bad questions. They take the time to explain the business logic and why the system works the way it does. The first 20 minutes of the weekly QA meeting is to discuss what happened on the weekend.
Good Coffee & Free Snacks – I would never take a job because of coffee, free cokes, free juice, healthy snacks, and some not so healthy snacks. I still appreciate them being there. The company wants us to be comfortable. I stay away from the sodas.
Treat Contractors like people – I have been a contractor before so I know about the ways they can be treated differently. I am one again (3 month contract before conversion to employee). Despite the difference, I have not been treated any differently. I have been invited to lunch-n-learns, holiday parties, etc. – many things forbidden at other places I have worked at.
Lunch and Learn – the company actively participates in teaching everybody in the company about the products the company makes. Perhaps there are other topics, but that’s what I’ve seen so far. Next week, I will try introducing the QA team to Ruby as a test tool next week in a similar type of event.
Protects People – the product that I work on protects people from their online privacy being invaded. That’s usually a good thing.
Apple – I have not used Apple products (outside my iPhone) since 1989. I now use a Mac and I will seriously consider that for my next home purchase because of the time savings I get (start-up, fewer questions about security, etc.).
HP – As if I couldn’t get away from my beloved former employer, I am using QC, SiteScope, and my baby Service Manager. I am hoping to get a LoadRunner license to help in testing performance in the future.
Supportive – people are giving me some room to run around and try new things… even before I have developed a lot of credibility with them. That kind of support is so valuable to my happiness because I can focus for a while on investigations without worrying about what people will think.
I have no idea how the future will turn out. I just want to be a technical (and spiritual) leader in my new job. I think it’s going to be mutually beneficial to say the least!