Our software was able to update the settings on device via wireless module and/or writing to an SD card, which is then inserted into the flow generator. We had a good record for automating test scenarios using cucumber-jvm, but we might not get to include this one.
“But we can’t automate taking the SD card out of the card reader and putting it into the flow-generator! If we can’t automate it from end-to-end, we can’t put it in CI, and there’s no point to investing in it.” How to solve this? We decided to use the gold standard.
We created scenario outlines with combinations of settings in the examples. In the scenario, an authorized person would log in, then navigate to the device settings page, change the settings per the example record, then save them to an SD card. The first time, we would take those files on the SD card, insert them to the device to authenticate that the files “work right” in updating the device with those settings. We could store those files in source control. Then on subsequent runs, when our automated scenarios write new files, they can be compared to the Gold Standard files – the ones we validated already. When this runs in CI (or locally), the scenario fails if there is no match!
And I approve the written settings against the expected file set “<ApprovedFileset>”
Steve Gargan, who wrote the step definitions even did one better. If we were running the scenario locally, not in CI, and designated an approving=true parameter, then should the files not match, we would get asked on-screen whether the originals were right or the new set was right. If we select the new ones, then they would replace the old ones.
Then we created a similar feature for PDF reports. We are testing charts, mostly, so we convert the pdf into images to make gold standards. In addition, we capture the ‘criteria’ for the chart (description of what it should look like based on the data going into it). If the image or the criteria does not match, the scenario fails unless we are in approving=true mode.
I’d like to take credit for this but it was Steve’s implementation, inspired by Llewellyn Falco and Dan Gilkerson at:
http://approvaltests.sourceforge.net/
> Llewellyn Falco
Twitter: @LlewellynFalco
Blog: http://llewellynfalco.blogspot.com
> Dan Gilkerson
Contact info on: http://dangilkerson.com
Software Quality Assurance and Test 