Combinatorial Software Test Design – Beyond Pairwise Testing
I put this together to explain combinatorial software test design methods in an accessible manner. I hope you enjoy it and that, if you do, that you’ll consider trying to create test cases for your next testing project (whether you choose our Hexawise test case generator or some other test design tool).
Where I’m Coming From
As those of you know who read my posts, read my articles, and/or have attended my testing conference presentations, I am a passionate proponent of these approaches to software test design that maximize variation from test case to test case and minimize repetition. It’s not much of an exaggeration to say I hardly write or talk publicly about any other software testing-related topics. My own consistent experiences and formal studies indicate that pairwise, orthogonal array-based, and combinatorial test design approaches often lead to a doubling of tester productivity (as measured in defects found per tester hour) as compared to the far more prevalent practice in the software testing industry of selecting and documenting test cases by hand. How is it possible that this approach generates such a dramatic increase in productivity? What is so different between the manually-selected test cases and the pair-wise or combinatorial testing cases? Why isn’t this test design technique far more broadly adopted than it is?
A Common Challenge to Understanding: Complicated, Wonky Explanations
My suspicion is that a significant reason that combinatorial software testing methods are not much more widely adopted is that many of the articles describing it are simply too complex and/or too abstract for many testers to understand and apply. Such articles say things like:
A. Mathematical Model
A pairwise test suite is a t-way interaction test suite where t = 2. A t-way interaction test suite is a mathematical structure, called a covering array.
Definition 1 A covering array, CA(N; t, k, |v|), is an N × k array from a set, v, of values (symbols) such that every N × t subarray contains all tuples of size t (t-tuples) from the |v| values at least once .
The strength of a covering array is t, which defines, for example, 2-way (pairwise) or 3-way interaction test suite. The k columns of this array are called factors, where each factor has |v| values. In general, most software systems do not have the same number of values for each factor. A more general structure can be defined that allows variability of |v|.
Definition 2 A mixed level covering array, MCA (N; t, k, (|v1|,|v2|,…, |vk|)), is an N × k array on |v| values, where
| v |␣ ␣k | vi | , with the following properties: (1) Each i␣1
column i (1 ␣ i ␣ k) contains only elements from a set Si of size |vi|. (2) The rows of each N × t subarray cover all t-tuples of values from the t columns at least once.
– “Construct Pairwise Test Suites Based on the Bak-Sneppen Model of Biological Evolution” World Academy of Science, Engineering and Technology 59 2009 – Jianjun Yuan, Changjun Jiang
If you’re a typical software tester, even one motivated to try new methods to improve your skills, you could be forgiven for not mustering up the enthusiasm to read such articles. The relevancy, the power, and the applicability of combinatorial testing – not to mention that this test design method can often double your software testing efficiency and increase the thoroughness of your software testing – all tend to get lost in the abstract, academic, wonky explanations that are typically used to describe combinatorial testing. Unfortunately for pragmatic, action-oriented software testing practitioners, many of the readily accessible articles on pairwise testing and combinatorial testing tend to be on the wonky end of the spectrum; an exception to that general rule are the good, practitioner-oriented introductory articles available at combinatorialtesting.com.
A Different Approach to Explaining Combinatorial Testing and Pairwise Testing
In the photograph-rich, numbers-light, presentation embedded above, I’ve tried to explain what combinatorial testing is all about without the wonky-ness. The benefits from structured variation and from using combinatorial test design is, in my view, wildly under-appreciated. It has the following extremely important benefits:
- Less repetition from test case to test case
- In the context of discussing testing’s “pesticide paradox” James Bach, I believe, used the analogy that following in someone’s footsteps is a very good way to survive traversing through a mine field but a generally lousy way to find software defects efficiently.
- Maximizing variation from test case to test case, as a general rule, is an absolutely spectacular way to find defects quickly.
- There are thousands, if not trillions of relevant combinations to select from when identifying test cases to execute; computer algorithms will be able to solve the problem of “how can maximum variation be achieved?” far better than human brains can.
- More coverage of combinations of test inputs
- Most of the time, since awareness of pairwise and combinatorial testing methods remain low in the software testing community, combining all possible pairs of values in at least one test case is not even a conscious goal of testers.
- Even if this were a goal of their test design strategy, testers would have a tremendous challenge in trying to achieve such a goal: with hundreds, thousands or tens of thousands of targeted combinations to cover, losing track of a significant number of them and/or forgetting to include them in software tests is virtually a foregone conclusion unless a test case generator is used.
- More thorough coverage leads to more defects being found.
- Efficiency (Testers can “turn the coverage dial” to achieve maximum efficiency with a minimal number of tests)
- The efficiency and effectiveness benefits of pairwise testing have been demonstrated in testing projects every major industry.
- I wanted to prominently include the message that testers using test case generators have the option to dramatically increase the testing thoroughness levels of the tests they generate because it is a topic that often gets ignored in introductions to pairwise testing case studies and introductions
- Thoroughness – (Testers can also “turn the coverage dial” to achieve maximum thoroughness if that is their goal)
- Too often, tester’s view pairwise as a technique that focuses on a very small number of curiously strong tests; that is only part of the story.
- This can lead to the /false/ impression that combinatorial testing methods are inappropriate where high levels of testing thoroughness are required.
- You can create very different sets of tests that are as thorough as possible (given your understanding of what you are testing) no matter whether you have 1 hour to execute tests or one month to test.
Other Recommended Sources of Information on Pairwise and Combinatorial Testing:
- Combinatorial Software Testing (Contains results of a 10-project empirical study)
- Efficient and Effective Software Test Design (Contains some screen shots and worked examples to help make these concepts more concrete)
- Pairwise Testing: a Best Practice that Isn’t (Which has many good cautionary points lest any readers be tempted to embrace these test design methods as a silver bullet cure-all)
- Hexawise (Our test design tool, which includes many explanatory examples and templates of pairwise and combinatorial testing)
Questions or Comments?
If you have questions or comments, please leave a note below. I’d love to hear about people’s experiences using these test design approaches. Thank you.
Same content, different format. It makes wonders! These are the most engaging presentation visuals about pairwise tests that I’ve seen.
It will certainly keep the audience awake curious on what will appear next. It would have been distracting, but the pictures are very related to the content so they’ll still connect.
I recommend you to explain well the asterisk note about 100% coverage, so people will know what to tell their managers when they come back from the conference.
Be warned: I’ll use this presentation in the future when teaching Pairwise. 🙂
Thank you for your kind words!
Yes, the explanation behind the “100% coverage” point is an extremely important caveat. Without the explanation, it could be downright misleading. I know from our conversations that you understand these points well, but to anyone who is reading this I’ll elaborate:
When combinatorial testing methods are used, there is a potential “garbage in, garbage out” problem. When we say there is 100% 2-way coverage in a set of test cases, it means that based on the inputs entered into the test case generator (which might be well-thought out or badly misguided), 100% of all pairs of values will be tested in at least one test case.
Example: If you have listed several browser types, including “IE9” and several user types, including “Platinum Card Holder,” along with a thousand other test inputs, when you click on the “Create Tests” button, there will absolutely, positively, be at least one test case that includes the combination of a Platinum Card Holder using IE9 as a browser. Every single valid combination of 2 test inputs will be tested at least once.
For more thorough combinatorial testing, the same logic applies… When you select “4-way” from the “Create Tests” screen, any four test inputs you include in the “Define Inputs” page will all appear together in at least one test case.
The elephant in the room, as it were, is that testers will routinely forget to include relevant test inputs (because of imperfect information, inexperience, forgetfulness, etc). If a tester forgot to include “IE9” as a browser type, for example, IE9 would not appear in any tests, and no IE9-specific defects would appear.
Additionally, I’d go even further to say that 100% coverage of any System Under Test is a misnomer. We’re not claiming that; Hexawise is simply achieving 100% coverage of all the targeted combinations of identified test inputs in the way described above (not ensuring comprehensive coverage of the System Under Test).
the slides are great. Not dry at all :-).
Agreed, very entertaining presentation.
Sending this to interested parties here.
I really love the idea behind Hexawise,
While how to use it seems quite clear when it comes to positive test cases, I am wondering as to the right approach regarding the so-called negative test cases.
After thinking about it for a while, I currently believe that values which are assumed to give negative result, should first be tested separately, to ensure each of these has failed by itself (while other values in same test case can be generated with regular pair wise).
If we really wish to insist, we can then use more than one negative value at a time, to ensure proper handling when SW encounters such cases – which actually can be handled differently depending on design (fail on 1st only, or reply with all improper values, or etc.)
I wonder what’s your take on this issue.