I want to premise this post with two things. First, I didn’t think of this. I heard Jim Knowlton interviewed on the Watir Podcast. I also attended a web cast with Michael Bolton on using automated testing in a manual form. Both inspired me to try this myself. As some of my software friends are aware, I dig testing with Ruby and Watir (Web Application Testing In Ruby). I didn’t know anything about Ruby or Interactive Ruby (IRB) before working with it, starting in about 2007-2008. But since starting to use it, I found I am more productive by “trying” my code out in the IRB interface before adding it to my functions. Very handy!
My eyes were opened even more when I heard Jim talking about using IRB to facilitate his testing (and his co-workers) by creating libraries that assisted him. I started to envision things I could use it for: pumping data into a system for create conditions for testing business logic; grabbing near-real time information from log files; quickly getting to the ‘state’ in which I want to explore; and grabbing unseen information about the interface that I am looking at such as the state of objects and making them ‘seen’. I remember being so excited that I asked other people to listen to the podcast.
The webcast with Michael was revealing too. I could see with a simple command line IO, inputing numbering and getting numbers back helped me quickly learn about a system that I knew nothing about. He talked strategies to use exploration, as well as tactics that supported the strategy.
Those lessons came to fruition at my new job. I started with needing to test a web service. I have tested SOA web services before. But I had not tested the with Ruby before. I saw the team I was joining using a RestClient. They were modifying the ‘get’ and ‘delete’ calls based on the data set from the ‘create’ call. I asked myself how many times I wanted to copy that data, the answer was three times. So I pulled out the restful web services book that I had opened one or two times in the past, and found a snippet of net/http to call web services. Once I figured out how to add the pieces I needed, in less than a day, I had a class that could perform all three calls and retain the data, allowing me to test the web services api until I was satisfied that it was working according to the design. Running these from IRB allowed me to run manual tests. I kept changing the tests. Even though they were written in QC with instructions on parameters, I kept asking myself “what if I did it this way?” which allowed me to learn and satisfy myself that the API was strong.
I also saw the other team members copying and pasting data from the create api call to use in a lynx (text-based command line browser) for testing business logic. I found all the typing and copying unnecessary and mistake-prone so I ‘printed’ the lynx command out allowing me to copy and paste the entire thing. This allowed me to more quickly get to the business of testing. I used that gained time to add a direct command from Ruby, reducing a step to run quicker, and finally incorporating the Mechanize library with that saved time.
The fruit of that labor was to allow me to test more quickly from the web service call to the browser call. That in itself did not find any defects, but it enabled me to see the need for faster (virtual) browser calls which did find three major defects and a minor defect. In spite of these gains, my manager was concerned about the cost of maintenance of early automation – a lesson I learned from James Bach’s Test Automation Snake Oil that says “Manual testing, on the other hand, is a process that adapts easily to change and can cope with complexity. Humans are able to detect hundreds of problem patterns, in a glance, an instantly distinguish them from harmless anomalies.” I gave her a demonstration to show that my mind was in control of these tests. She felt at ease and asked me to continue.
The final culmination was today when I discovered a problem in a caching algorithm that would not be detected with smaller slower data-flows. I had no intention to test for load. I wanted to test functionality in a system environment and more realistic volumes. At this point, I did create a Ruby “script” but even then it took command line arguments to allow me to modify the test cases as I saw fit. There were no magical red or green colors that filled a test management system with results. Just tests that I ran as I felt were needed to learn if this system was ready to release or not.
– Dave McNulla