Exploratory Testing with Interactive Ruby

I want to premise this post with two things. First, I didn’t think of this. I heard Jim Knowlton interviewed on the Watir Podcast. I also attended a web cast with Michael Bolton on using automated testing in a manual form. Both inspired me to try this myself. As some of my software friends are aware, I dig testing with Ruby and Watir (Web Application Testing In Ruby). I didn’t know anything about Ruby or Interactive Ruby (IRB) before working with it, starting in about 2007-2008. But since starting to use it, I found I am more productive by “trying” my code out in the IRB interface before adding it to my functions. Very handy!

My eyes were opened even more when I heard Jim talking about using IRB to facilitate his testing (and his co-workers) by creating libraries that assisted him. I started to envision things I could use it for: pumping data into a system for create conditions for testing business logic; grabbing near-real time information from log files; quickly getting to the ‘state’ in which I want to explore; and grabbing unseen information about the interface that I am looking at such as the state of objects and making them ‘seen’. I remember being so excited that I asked other people to listen to the podcast.

The webcast with Michael was revealing too. I could see with a simple command line IO, inputing numbering and getting numbers back helped me quickly learn about a system that I knew nothing about. He talked strategies to use exploration, as well as tactics that supported the strategy.

Those lessons came to fruition at my new job. I started with needing to test a web service. I have tested SOA web services before. But I had not tested the with Ruby before. I saw the team I was joining using a RestClient. They were modifying the ‘get’ and ‘delete’ calls based on the data set from the ‘create’ call. I asked myself how many times I wanted to copy that data, the answer was three times. So I pulled out the restful web services book that I had opened one or two times in the past, and found a snippet of net/http to call web services. Once I figured out how to add the pieces I needed, in less than a day, I had a class that could perform all three calls and retain the data, allowing me to test the web services api until I was satisfied that it was working according to the design. Running these from IRB allowed me to run manual tests. I kept changing the tests. Even though they were written in QC with instructions on parameters, I kept asking myself “what if I did it this way?” which allowed me to learn and satisfy myself that the API was strong.

I also saw  the other team members copying and pasting data from the create api call to use in a lynx (text-based command line browser) for testing business logic. I found all the typing and copying unnecessary and mistake-prone so I ‘printed’ the lynx command out allowing me to copy and paste the entire thing. This allowed me to more quickly get to the business of testing. I used that gained time to add a direct command from Ruby, reducing a step to run quicker, and finally incorporating the Mechanize library with that saved time.

The fruit of that labor was to allow me to test more quickly from the web service call to the browser call. That in itself did not find any defects, but it enabled me to see the need for faster (virtual) browser calls which did find three major defects and a minor defect. In spite of these gains, my manager was concerned about the cost of maintenance of early automation – a lesson I learned from James Bach’s Test Automation Snake Oil that says “Manual testing, on the other hand, is a process that adapts easily to change and can cope with complexity. Humans are able to detect hundreds of problem patterns, in a glance, an instantly distinguish them from harmless anomalies.”  I gave her a demonstration to show that my mind was in control of these tests. She felt at ease and asked me to continue.

The final culmination was today when I discovered a problem in a caching algorithm that would not be detected with smaller slower data-flows. I had no intention to test for load. I wanted to test functionality in a system environment and more realistic volumes. At this point, I did create a Ruby “script” but even then it took command line arguments to allow me to modify the test cases as I saw fit. There were no magical red or green colors that filled a test management system with results. Just tests that I ran as I felt were needed to learn if this system was ready to release or not.

– Dave McNulla

Advertisements

6 thoughts on “Exploratory Testing with Interactive Ruby

  1. Del Dewar

    Hi Dave,

    Great post. I’m starting to use Ruby/Watir (Webdriver) for some automation at my place of work at the moment. Haven’t used irb as much as I thought I would, mainly because there are some subtle differences in behaviour when the code when run from within irb.

    Have you used any IDEs on your Ruby endeavours? E.g. RubyMine?

    Thanks for sharing
    Del.

  2. dmcnulla Post author

    Del,

    Thank you for the compliment. Right now I am between IDE’s. I used to use Eclipse but I started crashing and I grew impatient. I am working on a Mac now so I would love to use Rubymine but I am too cheap to buy it. I am editing in XCode, which is OK for editing but nothing else. I am running scripts from the command line or from IRB. I have been hearing things about TextMate so I may try that next.

    I am happy to share, as much for how it helps myself as for helping others. Again, I appreciate the encouraging words.

    Dave

  3. Del Dewar

    Dave,

    On an editing note, and the fact you’re a Mac user I have very recently started using BBEdit on my MacBook Pro which I’ve found *extremely* useful for maintaining suites or hierarchies of Ruby scripts. You get the same perks of using something like MacVim (colour/layout-wise) but the much greater ability to switch between scripts, etc. It didn’t cost the earth either which was nice (~ 30 quid)

    Del.

  4. dmcnulla Post author

    John,

    I would except it was a webinar for Hewlett-Packard internal. I am sure that Michael has other webinars on the subject, but I haven’t seen them. Here is an essay he wrote on the general subject: http://t.co/wdBmXfpS

    Dave

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s