“What is best in life? To crush your enemies, to see them driven before you, and to hear the lamentations of their women.”
-Conan, the Barbarian
As software engineers, we find ourselves fighting a war against an enemy of our own creation. An enemy armed with complexity, obscurity, and redundancy. From a fortified position in legacy code it charges forth using our past compromises as a battle cry. Like the Hydra, software development can work against us: each problem we fix with a cutting change can sometimes spawn more bugs. What is an engineer to do in the face of such odds?
Creating bug-free software can happen in one of two ways. Write no bugs, which given the complexity and breadth of Advantage is impossible, or find the bugs that are created and fix them. To this end, bug detection becomes the most important task in QA, and automated testing. If the bug can be found, then the bug can be killed. Automation is our best tool for this purpose. Through automation we can perform thousands of repetitious tests a minute, and hopefully cover hotspots with enough checks to find bugs before the software can goes live.
A great example of such a hotspot is the Advantage List Frame. List frames display rows of data with column headings. They are meant to mirror database tables, and display raw data in a more user-friendly manner. Users select rows in the list frame, and can perform actions on them. Most people will interact with their Advantage data entirely through list frames, and dialogs launched from them. This makes list frame bugs especially frustrating to users because they are blocked from performing actions in the affected area of Advantage.
To automatically test our list frames, we needed to first find a common point to test. This is where having a shared list frame framework becomes essential. At its core, a list frame will generate a database query to get all the information it needs to display. If something about that request is faulty, then the query fails, and the list frame ends in error without displaying any rows. With no rows to select, a user has very few options to perform actions. Thus the decision was made to provide the list frame framework with the ability to get the most complex query possible for each list frame that uses the framework. Armed with a list of list frames, the testing framework can then step through each of the 1,200 advantage list frames that use the list frame framework, and test each one’s database request. Any malformed requests throw exceptions which are caught and reported. The entire test takes 90 seconds from start to finish.
For many years, in older implementations of Advantage, it was entirely possible for a user to go to a screen of Advantage, hit the search button, and get an error reporting that something was wrong with Advantage’s request for information. These bugs were disruptive and embarrassing. With the move to .NET new possibilities became available to the automated testing framework that nearly eradicated this entire class of bug from live software.
To provide a regular defense, we take the automated List Frame test, and add it to the suite of automated tests we run every hour for every .NET version of Advantage we support. Had we wanted to perform the same validation by hand, it would be prohibitively long. If we allow for the generous estimate of 60 seconds for a human to open a screen, add all of the columns to the display view, and run the search it would take that human 20 hours to verify one version of Advantage. At present there are eight supported versions of Advantage on the .NET platform, and that works out to 160 human hours condensed into a cumulative 12 minutes of automated testing. With build parallelism, we get that down further to 90 seconds. As a result of running this test, Advantage list frame bugs have been almost entirely eliminated in our software.
Frameworks combined with automation are to Advantage what the Code Breakers of Bletchley Park were to the Enigma. With their combined might, and the vigilance of software engineers, we can achieve bug extinction. The dialog box, and data dictionary integrity tests use a similar testing methodology to insure greater stability across Advantage. We continue to look for places to apply this broad testing pattern to augment our 1,500 automated “core-test” suites which are run every hour.
Whenever I see the list frame test fail I smile. Truly, to see your enemy crushed and driven before you is best in life.