Category Archives: Test Stack

Watir Podcast Episode 59

The Watir Podcast is publishing podcasts again. In this week’s episode, Neil Manvar from SauceLabs tells us about the advantages of use SauceLabs, how to get started, and what mistakes can lead to trouble.

Please listen and give feedback on this podcast on SoundCloud (https://soundcloud.com/the-watir-podcast/episode-59-saucelabs), as well as what you would like to hear about on future Watir Podcasts.

You can subscribe to The Watir Podcast feed (http://feeds.soundcloud.com/users/soundcloud:users:248873479/sounds.rss).

Badges? We don’t need no stinking badges!

In the Beginning

I am not a programmer. Well, I am but I never considered myself to be good at it. Most of the places that I worked, I was the only person creating test frameworks. I never received benefit from getting a code review of anything I wrote. I did not pair with anybody on writing any scripts, methods, or page objects. In fact, the first time I showed professionals in test automation my code, they looked embarrassed for me. When they got over the shock, they started giving suggestions that were obvious to them, but I had never learned. Hey, my code worked! Isn’t that what counted?

Learning

Well, since then I got over the big balls of mud that I was creating. I started paying more attention to the grace and simplicity of the code I saw from others. And I learned… some.

Maybe i can use more than one file for my code? Oh, I can let errors throw back to the calling function? Very interesting, I can create tests for my test tools. I can self-document these? I was learning, but I wanted more.

The Next Big Step

Within the past couple years, I have discovered a few things to help me with my programming. First, I found Rubocop (thanks Željko) which helped me to start up the style of my code. I started using standard indention, line lengths, variable names, limit the size of methods, and complexity.

Then I stumbled upon Travis-CI. I can’t remember for sure, but I think that I saw a badge on somebody’s github page. “Hey, I can use that too!” So I added that. I learned that I could run my test suite on several versions of ruby, even on JRuby. I felt that was important because I had written a tool that still used my rest client. I could add my own badge that lets everybody know my library is still good. I want people to feel confident that my builds are good so I added the badge.

Build Status

Then I saw coveralls where I learned that I my tests were missing parts of my code. It was only 10%. But it was coverable, so why not cover it? I improved my coverage to 100%. It did not drop down until… I tried to add support for JRuby9. That’s when I discovered that JRuby9 does not support coveralls yet when my coverage when down. I decided it was too early to support JRuby9. I wanted people to know that the library is tested so I added the badge.

Coverage Status

Then I hit the motherload. I was looking into a library that supported money formatting and discovered RubyMoney. Wow, they had badges for everything.

Quintuple Wow! Most of these tools are free for open-source projects, which is the only status that I would consider. Anybody can add them.

The Benefits

Adding the badges was fun, at least in the beginning. I discovered Reek this week, which is another linter for ruby. It played well with Rubocop so I added it to my project. There were different types of problems that I had to address. To support those changes, I expanded my automated tests too. When I checked in the branch, I could see how it was doing in CI. When I got negative feedback, I reworked the code. When the feedback was good, I knew I could merge the code to the master.

When I check in code changes, whether for refactoring, adding test cases, or increasing the versions of Ruby I support, I learn quickly whether my code was working still. When I learned reek didn’t support JRuby-1.7 because it’s base version of Ruby is 1.9, then I found a way to control that in both the Gemfile and Rakefile. When I tried to support JRuby9, I learned it’s not baked enough for me.

As my friend once stated, “you can never over-communicate.” Those tools are communicating to me, while the Badges are communication to anybody using my libraries.

Do We Need All Those Badges?

I also had to fool around with various configuration files to make sure the linters work working locally and cloudly (yes, I just made that up). Work is cost.

It seems like Gemnasium is going to start charging in 20 days to show my status. Money is cost.

Second thing I learned, while most of the work is in setting them up, maybe I can use some moderation in how many I use. So as long as it helps me, and as long as it’s free, I’ll use them. When I have to pay or they become more work than help, I will drop them.

Test Automation Bazaar

I will be attending the Test Automation Bazaar on January 15-16 2016. There is no agenda, just learning. I look forward to learning and sharing ways to solve problems.

https://testingwithvision.wordpress.com/2015/11/21/announcing-austin-test-automation-bazaar-jan-15-16-2016/

I also like to use these events to help others get involved in open-source software, and am hoping to see other contributors to projects like Selenium and Watir attend. — Bret Pettichord

Test Automation Curriculum

Two things happened to me lately. First, I was trying to find a career tester in the San Diego area that knows at least a little bit about automated testing. It isn’t going well. I’ve reviewed a lot of resumes. all the submitters are career manual testers.

Surely somebody sometime must have wondered if they need to learn more about automation. Elisabeth Hendrickson once asked Do Testers Have to Write Code? They did a survey to figure out what companies were looking for from tester skills. In our case, we aren’t looking for somebody to write the test code, but to write and review the cucumber scenarios. Just the same, even on a light desire level, I was disappointed.

Second, a younger person asked me what he should learn in test automation last week. I had already been contemplating writing this curriculum, so I was resolved to do it. Srini, here it is.

Other people who don’t work with LAMPs, such as .Net environments etc. will probably not appreciate this list. Make your own list on your own blog and put the link in a comment here. I don’t begrudge anybody doing something else. I just don’t want to go there.

I created this curriculum for testers learning test automation. While some addresses how and why, most of the list is about tools that can help create a full solution. Anyway, here is my list in priority order:

  • An open source tool such as Watir-Webdriver or Selenium/Java – do not mess around with the QTP and TestComplete. The cargo cults that buy those tools will expect “anybody can automate”.  With open source tools, you can download your own learning playground and incorporate that with the other products.
    • Learn how to create page objects. Even if you take advantage of a library like WatirMark or Page-Objects, you will have to do some tailoring yourself. I have been working with Selenium/Java so I am developing my skills on that combination now. Either way, you need to know how to work on that in an efficient way. In fact, you can address a lot of the entries in here just be using Cheezy’s book Cucumbers and Cheese (well worth the $15). I swear that I do not get a dime from it or Cheezy’s work, it’s just such a big benefit for anybody learning that I cannot miss the chance to say how good it is.
    • An open source framework such as Cucumber, Cucumber-jvm, or RSpec.
  • Github and Git – there are other good source control tools out there, including subversion. Git is easy to use locally for managing your own practice code. It’s easy go get copies of other people’s public projects onto your own system (how did they do that?). CodeSchool has a free course on git. There is also a nice paper on the differences between git/mercurial and subversion so you can understand the differences.
  • Ant and Maven if you use Java. Most of what I learned was through osmosis, but being able to shoehorn cucumber into your project is good to know.
  • Jenkins or Hudson, CruiseControl, or some other open source continuous integration tool. If you ever work at a place that will be introducing automated testing for the first time, this is great to know how to set it up.
  • Performance testing in JMeter – I think you can find a ruby alternative (BlitzIO or Grinder) but you don’t really need this tool to be in a ruby language. The importance is to learn the different kinds of testing you find under this umbrella (incorrectly) called Performance Testing. The other important skill is creating the right monitors so you can discover where things are bottled up.
  • Owasp‘s ZAProxy – learn how to capture the http calls between your browser or simulator and the server under test. You will learn a lot there. While you are there, download the GoatWeb project where you can learn about security vulnerabilities through practice.
  • Monitoring tools (Splunk or Graylog2) – One way to find the errors that are occurring on the system under test is through logging. Those are deleted nearly every time the server is redeployed. You can monitor those logs and server performance much better through a monitoring server.
  • A true startup is probably not going to hire a newb unless they are cost-control-centered. But if you find you get there are there is no issue tracking, it would be good to know how to set up issue tracking and integrate to your version control and continuous integration server. I’ve tried RedMine and it was fine.

If you see that you think should be on the list that is not there, please add a comment.

Issue & Story Tracking

Everywhere that I’ve ever worked at had a defect tracking system, requirements system, and sometimes merged into one. Some were off the shelf, some were homemade, and others were services. I never felt the need to install one for work, but I did want to see what was involved with doing this, including how to integrate it with other products in the test stack.

I didn’t look around much for an open source solution. I know Bugzilla is commonly used. I also looked at Mantis and Trac. I decided to go with Redmine because it’s a Ruby On Rails app.

Installation

The biggest issue was to identify which ports to use. I already had a Puppies application that I was testing as part of my Cucumber and Cheese book. I added a Graylog2 application, also Ruby On Rails for the web interface (server component is java).

I originally chose to use sqlite3 because it was already installed on my system. I knew that I could use IRB to crack into the database if I needed to. Turned out that I did. I had to create an admin user (default one did not work). I eventually switched to MySQL.

Integrations

I integrated it with Git. I had trouble getting it working until I discovered that I needed to point it to the .git directory of my project directory. By including issue id’s in my checking comments, the issue automatically associated the issue with the checkin. The comment showed up on the repository page and the issue page. It supports diffing, too.
Redmin_GIT

I was able to install a Jenkins Plugin (technically it is for Hudson but works on Jenkins) to integrate with my Jenkins. This allowed me to identify which builds the associated checkins would be built. I also installed a Redmine plugin to Jenkins could also track builds on Jenkins to show Redmine issues associated with the build in aggregate.

It also supported multiple projects, and sub-projects. The wiki and news seemed relevant for keeping track of project information.

There is role based access control. That probably isn’t necessary for an agile team so I ignored this capability. Also non-agile (my opinion) is time tracking and Gantt charts.

Conclusion

I found all the features to be useful. If I were going to build my own stack then I would like using this component.