San Diego Quality Meetup 5/4/2016

I have long hoped that San Diego testing community would develop a user group/meetup. My recruiter, Jon Barton of TEKSystems, started a meetup group. I attended a meeting in April. Then I volunteered to speak at a future meeting. That future meeting was last night.

Presentation on Best Practices of Test Automation

Wednesday, May 4, 2016, 5:30 PM

89 Software Engineers in Test Went

Check out this Meetup →

The periscope video:
https://www.periscope.tv/JonBartonTEK/1yoKMmanbQeKQ

The slide deck:
https://drive.google.com/open?id=1oJyGQ2Id9kVPJ7Qlbn79mnuDBbx6MInWy0qkOsuy5Iw

The book I recommend:
https://leanpub.com/cucumber_and_cheese

I am hopeful for the future of software testing in San Diego if they group becomes a place for people to step out of their circle of comfort, to mentor each other, to learn and grow.

Badges? We don’t need no stinking badges!

In the Beginning

I am not a programmer. Well, I am but I never considered myself to be good at it. Most of the places that I worked, I was the only person creating test frameworks. I never received benefit from getting a code review of anything I wrote. I did not pair with anybody on writing any scripts, methods, or page objects. In fact, the first time I showed professionals in test automation my code, they looked embarrassed for me. When they got over the shock, they started giving suggestions that were obvious to them, but I had never learned. Hey, my code worked! Isn’t that what counted?

Learning

Well, since then I got over the big balls of mud that I was creating. I started paying more attention to the grace and simplicity of the code I saw from others. And I learned… some.

Maybe i can use more than one file for my code? Oh, I can let errors throw back to the calling function? Very interesting, I can create tests for my test tools. I can self-document these? I was learning, but I wanted more.

The Next Big Step

Within the past couple years, I have discovered a few things to help me with my programming. First, I found Rubocop (thanks Željko) which helped me to start up the style of my code. I started using standard indention, line lengths, variable names, limit the size of methods, and complexity.

Then I stumbled upon Travis-CI. I can’t remember for sure, but I think that I saw a badge on somebody’s github page. “Hey, I can use that too!” So I added that. I learned that I could run my test suite on several versions of ruby, even on JRuby. I felt that was important because I had written a tool that still used my rest client. I could add my own badge that lets everybody know my library is still good. I want people to feel confident that my builds are good so I added the badge.

Build Status

Then I saw coveralls where I learned that I my tests were missing parts of my code. It was only 10%. But it was coverable, so why not cover it? I improved my coverage to 100%. It did not drop down until… I tried to add support for JRuby9. That’s when I discovered that JRuby9 does not support coveralls yet when my coverage when down. I decided it was too early to support JRuby9. I wanted people to know that the library is tested so I added the badge.

Coverage Status

Then I hit the motherload. I was looking into a library that supported money formatting and discovered RubyMoney. Wow, they had badges for everything.

Quintuple Wow! Most of these tools are free for open-source projects, which is the only status that I would consider. Anybody can add them.

The Benefits

Adding the badges was fun, at least in the beginning. I discovered Reek this week, which is another linter for ruby. It played well with Rubocop so I added it to my project. There were different types of problems that I had to address. To support those changes, I expanded my automated tests too. When I checked in the branch, I could see how it was doing in CI. When I got negative feedback, I reworked the code. When the feedback was good, I knew I could merge the code to the master.

When I check in code changes, whether for refactoring, adding test cases, or increasing the versions of Ruby I support, I learn quickly whether my code was working still. When I learned reek didn’t support JRuby-1.7 because it’s base version of Ruby is 1.9, then I found a way to control that in both the Gemfile and Rakefile. When I tried to support JRuby9, I learned it’s not baked enough for me.

As my friend once stated, “you can never over-communicate.” Those tools are communicating to me, while the Badges are communication to anybody using my libraries.

Do We Need All Those Badges?

I also had to fool around with various configuration files to make sure the linters work working locally and cloudly (yes, I just made that up). Work is cost.

It seems like Gemnasium is going to start charging in 20 days to show my status. Money is cost.

Second thing I learned, while most of the work is in setting them up, maybe I can use some moderation in how many I use. So as long as it helps me, and as long as it’s free, I’ll use them. When I have to pay or they become more work than help, I will drop them.

Update on Book Review

When I was at ResMed, I was reviewing the book Black Box Testing in order to prepare for teaching a class to testers. Because I left ResMed in December, I will not complete the book review right now. Upon re-reading it after two decades, I find that I cherish the things I learned from it. I will continue reading it, but at a slower pace. As I do, I will submit reviews of each chapter at that time.

In the meantime, I will resume posts on investigations, experiments, and lessons that I learn in testing and testing technologies.

Test Automation Bazaar

I will be attending the Test Automation Bazaar on January 15-16 2016. There is no agenda, just learning. I look forward to learning and sharing ways to solve problems.

https://testingwithvision.wordpress.com/2015/11/21/announcing-austin-test-automation-bazaar-jan-15-16-2016/

I also like to use these events to help others get involved in open-source software, and am hoping to see other contributors to projects like Selenium and Watir attend. — Bret Pettichord

Book Review Pt4: Black-box testing: techniques for functional testing of software and systems

Chapter 3: Control Flow Testing

This chapter introduces a basic graph, control flow. I found the tax documents for this exercise.

I read this in two parts. The first part, after the standard definitions section, focused on the relationships of a control-flow model, how they vary, and how to draw them. I liked the use of the truth table to show the relationships in a compound predicate.By the end of this section, I knew (remembered might be a better term) how to use the tactics of this technique. I could differentiate when the nodes would be serial instructions and when they would be selection as an if or case statement..

Dr. Beizer presented a step-by-step instruction set for building and using the control-flow model for testing. The model development seemed to be as if I were modeling a script or coded program. but showed what happened in the methods as well. The steps within the tax instructions gave that feel. The space of each instruction gave us a memory register to store the result. Each instruction did one thing… well, mostly. While I didn’t like the idea of reviewing 18 year old tax instructions, it worked well with this lesson.

The coverage, which is beginning to be developed in the previous chapter, starts to become more relevant. Where is the risk? In some places, the decisions seem somewhat arbitrary; to the whim of the author. Most of the coverage decisions are direct to the risks, and causes of the risks. Even within those, some of the reasons appear to be “old” ways of thinking.

In summary, this chapter gives the tester something they can use right now. It is the first finished feature of model-based testing. It is releasable, usable, and helpful.

Book Review Pt3: Black-box testing: techniques for functional testing of software and systems

Chapter 2: Graphs and Relations

This chapter introduces graphs as the central tool of testing.

Just running through the definitions got me pumped for testing. Every relationship Dr. Beizer described brought about examples in my head from software that I commonly test in my job. Then I saw the Question/Answer of the book. What do you do when you see a graph? Cover it! In 20 years, that has never left me or my test methodology.

I am not an expert on graphing so I did not see anything blatantly left out. I noticed in the undirected links section of definitions that the author didn’t think symmetric relations are not common in testing – I thought that modern software models include load balanced services, as well as multiple threads, which may have those relationships. I have tested those relationships many times.

Regarding the graph types used in testing, I took interest to the those. I felt that the transaction flow model he described doesn’t get enough attention. The finite-state model of menus… modern menus often change based on the current task or context – in the author’s word processing example, one may see a different menu set when editing a table.

While I enjoyed the discussion of transitive/intransitive relationships, I really liked the end. To paraphrase, If specification or programmer is confused about transitivity, then you have a fertile ground for testing.

The examples of the relationship types are helpful. Except the part about equivalence classes. I had to look at 3 web definitions and two YouTube videos before I felt comfortable. Then I reviewed a single partition example and said “oh yeah, I know that.”

The concept of the matrix of relationships between nodes was completely boring, but so much more efficient. If I were modeling with intention to automate coverage of a model, I would use the matrices in some form. But I get why Dr. Beizer says “…the format is cumbersome and error-prone for large matrices.”

The generic testing principles outlined have covered a great deal of my testing lifetime. It’s also the ToC for the rest of this chapter. Each instruction expands to a cookbook for creating a testing tool.

  1. Define the graph
  2. Define the relation
  3. Node Cover
  4. Link Cover
  5. Test All Weights
  6. Design Loop Tests

The instructions are simple, and mostly straight-forward. I didn’t particularly like calling properties as ‘weights’ because I think of weight as a particular type of property. Like the weight might show the likelihood one branch of the graph is used compared to an alternate branch. The author also describes the purpose & significance of each step.

I am not sure the ‘test all weights’ is necessary as a separate step. The properties can be tested within the node cover & link cover. They should be tested there.

In summary, I think this chapter sets up the following chapters by creating vocabulary and tools for them.