People at my church will be working in the community today instead of traditional church services. I will be out there too, working on a food drive to benefit the families of deployed Marines who typically get little in return for the extreme sacrifices. If you see me at Von’s or Stater Bros, come say hi. You can follow my tweets on the hash tag #CommunityService. I will add to this post to describe my experience later.
I got this verbatim from my pastor last night in a church meeting.
Four Different Levels of Change
Mind: Information is the key to change a mind. Make sure they have the data. Facts are more persuasive than opinions, but do not necessarily generate consensus.
Heart: Relationships are the key to spur a change of heart. The focus is on empathetic understanding instead of compelling arguments. An especially difficult hurdle is that emotional reactions are directed at the leader.
Lifestyle: Experiences are the keys to changing lifestyle. Leaders need to give others the opportunity to have the same kind of experiences that they had, which helped bring about their own change.
Culture: Commitment is the key to change in culture. A common mistake is to believe that one has won a commitment when one as one a vote. Cultures change slowly.
I think I got that from an ISO 2001 audit preparation meeting in the mid-90′s during an effort to sell fax machines that we were manufacturing at HP to the EU. I like that so I try to use it.
Do What You Say
I said that I was going to try Improving the Value of Testing. What would be better than security testing? A bunch of things, you might say. But the reality is that security is the highest risk you are facing in your products. The bad guys understand more than you do, and probably more than the people who make the security tools you use already. For me, I do not even understand much about what the tools do, or know the difference between sql injection and cross-site scripting.
Say What You Do
So I am going to venture into this a little by trying to do some security testing with tools that I get from where ever. I will even make some home-grown tools if possible because I like to build and I like control. That would help amp my understanding to a higher level, in my opinion.
My first attempt was to crack open an old book How to Break Web Software by Mike Andrews and James A. Whittaker. Things change a lot in 6 computer years. All the web services are in SOAP – yuck. That’s like getting your mouth washed out. And almost all the tools are for Windows, but I primarily use a Mac. Still, I think I can get some concepts out of this. I try the paso proxy, but it’s not working for me yet.
So I move on to SoapUI. That’s an old friend, but I have never used it for security nor Rest. I spent some time on trying to simply send a request (POST) to my system under test but the SoapUI crashed. And crashed. And crashed. I tried five times before I went to their forum and found an recent unanswered post called Clean Install: Mac OSX beach ball of death. Oh dear.
I spent a lot of time on those without getting anywhere. Edison would have said that I learned some ways that it doesn’t work. I will add more as I have time and additional information!
Tickets for the Test Automation Bazaar in Austin Texas are still available. These guys didn’t write the book on test automation, they created the tools! There is no place to learn more about the latest technologies and good practices in test automation. Contact me before Wednesday for a significant discount.
I have an idea. I will not get too high on it except that it is intriguing to me. Maybe the idea is not new to other people. The idea came to me while I was thinking about the test automation pyramid (or ice cream cone as automation expert Alister Scott recognized the typical shape). I am fond of the three-layer concept – enough so that I made myself learn how to write unit tests and service-layer tests, in addition to the GUI-based tests that are commonly practiced by software testers.
I am thinking about the shape of the triangle, which part should be how big, etc. Suddenly the problem hits me. The problem is not isn’t the investment in automated tests. It isn’t the maintenance (which I thought maintenance was a big issue). Alright, I am lying. Investment is half of the equation – the return on investment calculation. The problem is the (lack of) focus on the return for the investment.
So much is invested in tests that will find what? Low severity defects that do not halt releases? I do not care about low severity defects until I have sussed the high severity defects. I care even less about them when I am faced with making a substantial investment to find them. Where does that leave me? Focus on the big bugs. Automation for big bugs. Which bugs? The ones we do not want to ship with them in. The ones that require a patch if you miss them. The ones that make you slip release dates if you catch them late.
“I don’t care about low severity defects until
I have sussed the high severity defects.”
I expect to get criticized for this idea. Why? I have no experience on this. I have never tried it. It is an untried idea. What could be less valuable than a hypothesis? I can try it, but this kind of thing would take a while to prove itself out. But I cannot worry about that while I am brainstorming an idea. The idea will be flawed in many people’s minds. Automated tests are really just automated checks. That is not new. People are not particularly suspicious that the results are false positives, they do not like that the test are little positives – that is they test so little compared what a person can do. I believe this concept too.
What I really believe in is the ends, not the means. The means is how I will get it done. The ends is what I want to accomplish. I care about the automation only when it affectedly helps me get the good defects. What are those? In my world, those are the defects that reduce my organization’s ability to meet agreed or assumed service level agreements. I call those Enterprise Readiness defects. Examples include poor performance, poor performance after time (example: caused by a memory leak), poor resilience during or after high loads, problems with failover, and problems with data retention.
How can I accomplish that? By remembering that automation is just a tool in the toolbox.
You have to avoid all news outlets to miss the story about Jeremy Lin. I doubt that because a) you are reading the internet right now, and b) you opened this article. The interesting thing about this story is why he is a phenomena. I have heard many reasons for it.
He’s the first American born NBA player of asian descent. Aside from ESPN’s headline and SNL’s parody, does that matter? We’ve had (and have) NBA players from China. I didn’t see that level of excitement for the first Israeli player, the first Iranian player, or the first French point guard for that matter. That’s a small factor.
He’s a Harvard graduate and the NBA typically chooses talent over smart. There have been players from the Ivy League before, in the NFL, NHL, MLB, and even the NBA. Stanford, known for it’s excellent academics, has several graduates in the NBA. That’s not significant.
He’s lifted the NBA Knicks, a storied franchise, from a downward spiral. Anytime the Knicks improve, it’s a story. That’s mostly because the Knicks have been such as bad team over the past 10 or 12 years. However, Carmelo being traded to the Knicks was not that big of a story.
The truth is that he is such an interesting story to most of the people in my world because he is just like you. Here are the ways that he is the same as you.
1. You are Exceptional
Jeremy Lin has shown the world he is exceptional by setting new records points and assists in his first few games in the NBA – more than Magic Johnson. He has helped (at least up until now) turned around the Knicks season. Who knew – apparently not the 30 general managers that didn’t draft him.
You have things that you are better at than most people around you. What are you good at? You know those things you do that make you proud of yourself. There are probably things that you haven’t given yourself credit for being good at doing.
2. Nobody Believed (or Believes) in You
Lin’s high school coach was quoted as saying he thought Jeremy would be a good NCAA Division 3 player. Mike D’Antoni, the Knicks coach, didn’t even know his player’s name for a couple of weeks. He played because D’Antoni did not have a choice. His aging start point guard, Chauncey Billups, was traded to clear cap space. His newly signed point guard, Baron Davis, has been injured the entire season. And the first backup point guard was doing a poor job at running the starting team. Lin getting to start was a desperation move.
You have heard those things too. Maybe your parents said “We just don’t want you to get your hopes up too high.” Maybe a guidance counselor said “I think you should be realistic.” Maybe your co-worker said “but you are just a _[fill in the blank here]_.” Maybe you had times when you didn’t even believe in yourself.
3. You are not Perfect
Jeremy Lin isn’t either. And you don’t mind that he turns the ball over a lot. He can’t rebound like Carmelo Anthony, his Knicks teammate. He is part of a team. He fulfills roles of that team, some better than others. He facilitates the offense of the team. He initiates things. He doesn’t have to be good at all things.
You are not perfect either. I may not know you specifically – I can guarantee that you aren’t perfect. You do, however, have skills. Practice those skills. Shore up your weaknesses. Develop yourself.
4. You Need an Opportunity
Jeremy Lin wasn’t drafted into the NBA. He looked for opportunities. He went to the eventual NBA Champs Dallas Maverick’s mini-camp. He was cut. He played a 10 day contract for the Golden State Warriors as a backup. He played overseas. He played in the NBA’s Developmental League. He did not step back and accept defeat.
You were not hired into the executive fast track of a Fortune 500 company when you graduated college (or maybe you didn’t graduate college). You started at a company that paid you poorly. Maybe that was the only offer you received. You have had to ask for more responsibilities. You have asked for promotions. You have been disappointed too.
My Story
I figured out that Jeremy Lin is like you when I discovered that he is like me. I learned to believe in myself.
I am exceptional. I have a critical mind – that’s part of how I ended up in software testing. I like teaching other people. I like to move to the next level. For most of my career, I have read the latest strategies and techniques in the software testing & quality magazines to see they are recommending what I am already doing.
I had to suffer through doubters to the point that my boss told me that my confidence should by much higher than I projected. I was promoted to be the QA manager for Service Manager (hundreds of millions of US dollars in sales and maintenance per year) because the project manager had a QA manager quit when hiring requisitions were frozen. They believed in me enough to call me “interim manager” until I proved myself in the position.
I am not perfect. I am more excited about what’s next than finishing what is now. I was a lousy public speaker – think Albert Brooks’ character in Broadcast news (sweating and falling apart). I learned to carry myself through Toastmasters. I learned my craft through courses, books, magazines, and reading articles on the internet (like you are doing now).
I found opportunities. I started at HP as a contract hardware tester making $9/hour – pushing paper through the fax machine’s sheet feeder. Years later when I was promoted to be the QA manager, because I asked for that job – and requested a lot of responsibilities in between.
Your Story
The corollary to Jeremy Lin to being like you is that you are like him. His story will be your story if you want it to be. You like Jeremy Lin because you want his story. So make it happen. Start by telling me your story here in the comments below.
Discussed recently on my Twitter feed was question about test jobs and the required skills. Somebody pointed out that many ads list selenium or QTP. Why both? The person asking told me at one time he had used both and knew both. I think he asked the question because he knew the products were quite different. Or maybe because the hiring manager did not seem to know about that difference.
I have used many products, vendor, open source, and home-grown for automating tests through the years. A vendor test tool just means that somebody is selling it for money. In this case, lots of money.
They are all for the purpose of running tests (typically functional tests). However, they are so different that people tend to join a camp and stick with it. What’s your preference?
My favorite is open source tools (in general), but I am good with whatever tool gets the job done. Productivity is the most important to me. Maybe that is why open source is my favorite. Let me share some of my analysis on the vendor products. Keep in mind the context of my primary job is testing software that is under development. My analysis is greatly influenced by what I need to do.
Source Control
Vendor tools supply source control systems with their tools. I have seen this with QTP and Test Partner. So you have to install a database server or test management system to connect to (and the test management system stores it’s data in a database like SQL Server or Oracle).
Is that good or bad? When the tools were formed, it was probably good – good because testers weren’t used to fooling with source control systems like svn, git, or perforce; good because crazy people like me could ruin hours, days, or weeks of work with a screwed up search/replace-all action; good because they captured so much data when they recorded scripts. Are they still good to have? At this point, I say no. They are not needed because people are using the dev source control system anyways – two systems are not better than one. They aren’t needed because they editing/using the tests depend on an active connection – people want a copy on their system and only sync when necessary.
Plug-Ins
Vendor tools number one strength is the plug-ins they build to tackle automated testing on wicked technologies like Flex. And it’s not new. WinRunner and Astra QuickTest were some of the first products to support testing web technologies. This versatility helps sell the vendor tools.
Unfortunately for them, many of the technologies they sell plugins for will come-and-go. I believe that HTML5 will push those EOL’s a little faster. In the meantime, open source alternatives are available for widely used languages like java, ruby, and perl.
Integrations with other tools
Automated test tools integrate with other products like test management systems, defect tracking systems, and requirements tracking systems. There was a time that vendor-sold tools were the most integrated products. Why? Because they often owned all of the tools. They were able to integrate the products through internal knowledge sharing by teams. The integrations, unfortunately, were often based on technologies like dynamic data exchange (DDE) and object linking & embedding (OLE). That was good once upon a time, as were video cassette recorders.
Today products can integrate through the native language when they are libraries. When they are not, RESTful web services are convenient and reliable. I have seen substantial efforts to update vendor products to use modern integration technologies but they are often handcuffed by their customer-bases’ reliance on current product models.
Visually Oriented/Helpful
The first automated tests I ever wrote were tests that I didn’t write. I used a process that has been around since Microsoft Test (later Visual Test, later Rational Visual Test) called Record & Play. Was that helpful? As a learning experience, it was helpful. The tests created by the recordings were not helpful. The recordings are not helpful for long because they do not follow Good Practices for Automating Functional Tests. Even the tool vendors will concede that the recordings must be edited to minimize maintenance in the future.
Try creating the scripts without the recorder. I haven’t used some of the major vendor tools, but I know with QTP that I wasn’t able to build the script with the recorded objects. The objects are the early friend. They help you get the script created without knowing anything. Eventually, though, you will need to fix all those tests because the software under test has changed. How many tests are depending on that object? Are they multiple objects that look similar? Can you effectively minimize the number of objects during your development iterations as the product under test changes? All these concerns make me want to skip the object repositories. I want to know what is going into my tests, what is coming out, and be able to see what is happening behind the scenes.
Support
Vendor tools sell themselves well with customer support. Actually, they sell the customer support. The revenue from maintenance is usually higher than the revenue from sales. They can do this because the customer needs the fixes and improvements from the version they purchased. There is absolutely no DIY for those tools. The good part is that you do not have to fix it, but that bad side if you can’t.
There are forums for discussing issues in open source tools & libraries. The product authors and other volunteers answer questions about the products to help both experienced and new users, including providing direct access to defect tracking systems. When the cause of issues are found, the plans for release are posted and the user can resolve the issue themselves in their own library code base.
Foot Print
For software, the foot print is the total impact on a computer – space required to install on the hard disk drive, RAM used to run it (both edit and execution modes), licensing connections, integration connections, database connections, and cost to license.
The foot print affects every person that will use the automated test scripts to learn about the product under test. Today’s development methodologies are designed to reduce the gap in time from when a defect is introduced to when it is fixed. Build it regularly and test it regularly using continuous integration systems (e.g. Hudson). Let the failing tests be run on the developers system that checked the defective code in to the repository. Foot print matters.
The vendor tools are bulky and expensive to say the least. QTP 11 requires a minimum of 1gb storage and 1gb RAM. A use can get by without any connections (to database or to QC) but not a team. I haven’t been able to find the exact cost, but I am seeing ~$6,000 on my Google searches.
Test Complete 8 requires a minimum of 700mb storage and 256mb RAM. It’s $1,000 for an individual node-lock license and $4,500 for an enterprise floating license (prices in between for mixes of those two).
Telerik Test Studio requires 500mb storage and 500mb RAM. Each of of those three leading vendor tools require Windows XP, Windows Vista, or Windows 7. Their most popular package (some tools but missing most) is $1,300 while the complete package for one user is $7,334.
Compare those minimum requirements to Selenium test automation tools. Selenium is 31mb (server and java client), runs on Windows, Mac OS X, Linux, and Solaris. Ruby, the core language for the Watir and Watir-Webdriver libraries, supports Windows, Mac OS X, Linux, Solaris, and (I have never heard of) OpenIndiana. Ruby is 275mb (measure for v1.93) with Watir-Webdriver requiring an additional 18b. They are both free, the support is free, and the maintenance is $0. That’s a huge difference.
Conclusion
For automating tests on software development projects, I believe the vendor tools are inferior to the open source tools. It seems counter-intuitive – why wouldn’t the vendor tools adopt the capabilities of the open source tools?
First, they can’t because of the install-base requires them to remain the same. These products have been selling for up to and even more than a decade. People, departments, companies rely on the tools they have invested so much time into. Regardless of whether they tests are brittle or not, they cannot choose (or allow the vendor to choose) a radical change to how their systems are built. So instead they build new features and modules to sell while the core remains the same.
Second, they do not want to because their target is vast market of uninformed manual testers and managers. I have walked through vendor showcases at StarWest. They have interfaces that are way sexier than VIM. The trials seem so easy. “Come on in here, automate a test – everybody is doing it!” So you do it and it works beautifully. Of course you don’t have to run that test on your system, against your software, or maintain it. Their core target usually knows none of the good automation practices they should follow. But their companies take their advice and pony up the money.
As I mentioned before, the context of my job plays a significant role in my analysis above. If I were choosing a product for testing in an IT department (which is service-oriented), my priorities would change, and the tools I favored may change. I do not dislike vendors. I do not think they mean harm, in fact they provide utility.
