As Product Manager, I spend the majority of my days designing new, cool features (e.g., Cider and Call Center) and working with clients to build solutions. This typically involves lots of computer time, and when my day is done, I like to turn to something equally creative, but in the analog world: cooking. My favorite source of recipes is Cook’s Illustrated—I subscribe to both digital and print versions (… sounds like a mixed Advantage agreement to me ). So, what does this have to do with software QA? Well, cooking can serve as an analogy to describe some exciting new features coming with the 2015 release of Advantage due out in January 2015.
While I enjoy cooking, I'm less excited about cooking in someone else’s kitchen. The stove is electric, the salt isn’t in the right place, the pans are too small, the spoons are plastic instead of wooden, etc. I do my best, but the results are not as satisfying as when I’m in my own domain. Wouldn't it be great if I could perform some tests on the unfamiliar kitchen prior to starting?
Sometimes I feel like the Swedish chef from The Muppets!
OK, let’s bring this back to the world of software. With Advantage, we perform automated and manual tests in our QA department before releasing updates. We've been doing that for 35+ years and that will never change. Once we deliver the updates, our clients then apply their testing methodologies to verify the changes. I suspect most of you would rank testing as one of the top five resource bottlenecks in your organizations. The approaches to testing vary widely as well, depending on which philosophies you subscribe to: automated vs. manual, unit vs. integration vs. regression, in-house vs. out-sourced, etc. Although I'm not promising I have the silver bullet to this problem, with the introduction of these features, I am hoping to bump testing to at least #6 on your organization's list.
What features? The ones found in the Advantage Automated Testing Framework.
We apply our testing methodology in our kitchen… I mean, our internal areas. However, the configurations of Advantage at client sites are as numerous as the configurations of all the kitchens in all the countries where we have clients. That is to say, it’s impractical for us to verify that our tests can cover all permutations of configuration and data. This means some issues aren’t discovered until updates are deployed to client installations—basically, until we start cooking in your kitchen. We also know you are continuously making small improvements to your kitchen (regular deployments), and sometimes you may even do a larger renovation (upgrades). Ideally, we want to make sure everything is still in working order after introducing any of these changes.
Let’s imagine the kitchen itself is the code and structure of Advantage. In a new, unfamiliar kitchen, or after a change is introduced to my kitchen, I want to test that everything is in the right place, the right kind, etc. Basically, I want to make sure the kitchen functions in a way that will work for me. Introducing... Core Tests. These are AdvantageCS-created unit code tests which exercise the building blocks of the Advantage logic of your specific configuration. If the tests identify a problem, then we can investigate and resolve any issues. The library of core tests is maintained by AdvantageCS and is constantly expanding as our software grows and evolves—at the time of this article, there are 1,500+ core tests.
If Advantage is the kitchen, orders are the entrees. I want to make sure that if I follow a specific recipe (order inputs), the result (order outputs) is consistent with my previous experiences. For example, if I give the kitchen my chocolate chip cookie recipe, I want to know that the cookies will come out as expected—the size, shape, taste, etc. Introducing... Order Tests. You use our new user-interface approach to create sample orders (or edit the XML documents directly if that’s your thing); either way, you supply the recipe. The tests will initially create a baseline (how my cookies turned out the first time I made them), and subsequent runs will replay the orders and compare the new results to the baseline. This provides you with a regression testing mechanism with a play-back model.
Even if I know the kitchen and my recipes are working, I also want to test the more subjective aspects. Are there expired ingredients in the refrigerator? Did I forget to hand wash the dishwasher unsafe items? Similarly in Advantage, I likely have business-specific data conditions I want to avoid. Introducing... Data Quality Tests. For example, let's assume you use the installment billing with auto-charge option. You want to make sure the installments are going out on time and that the auto-charge ones are hitting customer credit cards according to the schedule. From an operations standpoint, you may have forgotten to run a process, or your selection criteria for when you collect money may be wrong. Quickly identifying unexpected or undesirable data conditions can save businesses time and money while improving customer service. These are the situations that these types of tests are designed to identify.
In summary, there are three types of automated tests we are introducing: Core Tests, Order Tests, and Data Quality Tests. There are configuration options for each of these tests as well: scheduling, active/inactive, grouping tests together, etc. The Automated Testing Framework is designed to reduce the time you spend managing the system, allowing you to focus on building your business.
Pretty cool, huh? Questions? Looking for great recipes? Contact your account manager and he/she will get you connected with the right people.
Bonus recipe time! I love technical recipes where the science drives the technique. Apologies to all grandmas of the world that use “a handful of this and a pinch of that.” This great article breaks down the classic chocolate chip cookie for optimal results: http://sweets.seriouseats.com/2013/12/the-food-lab-the-best-chocolate-chip-cookies.html