13 Great Questions To Ask Software Testing Tool Vendors

There are good reasons James Bach is so well known among the testing community and constantly invited to give keynote presentations around the globe at software testing conferences. He’s passionate about testing and educating testers; he’s a gifted, energetic, and entertaining speaker with a great sense of humor; and he takes joy in rattling his saber and attacking well-established institutions and schools of thought that he disagrees with. He doesn’t take kindly to people who make inflated claims of benefits that would materialize “if only you’d perform testing in XYZ way or with ABC tool” given that (a) he can always seem to find exceptions to such claims, (b) he doesn’t shy away from confrontation, and (c) he (rightly, in my view) thinks that such benefits statements tend to discount the importance of critical thinking skills being used by testers and other important context-specific considerations.

Leave it up to James to create a list of 13 questions that would be great to ask the next software testing tool vendor who shows up to pitch his problem-solving product. In his blog post titled “The Essence of Heuristics,” he posed this exact set of questions in a slightly different context, but as a software testing tool vendor myself, they really hit home. They are:

1. Do they teach you how to tell if it’s working?
2. Do they teach you how to tell if it’s going wrong?
3. Do they teach you heuristics for stopping?
4. Do they teach you heuristics for knowing when to apply it?
5. Do they compare it to alternative heuristics?
6. Do they show you why it works?
7. Do they help you understand when it probably works best?
8. Do they help you know how to re-design it, if needed?
9. Do they let you own it?
10. Do they ask you to practice it?
11. Do they tell stories about how it has failed?
12. Do they listen to you when you question or challenge it?
13. Do they praise you for questioning and challenging it?

[Side note: Apparently I wasn’t the only one who thought of Hexawise and pairwise / combinatorial test design approaches when they saw these 13 questions. I was amused that after I drafted this post, I saw Jared Quinert’s / @xflibble’s tweet just now:]

Where do I come down on each of James’ 13 questions with respect to people I talk to about our test design tool, Hexawise, and the types of benefits and the size of benefits it typically delivers? Quite simply, “Yes” to all 13. I enjoy talking about exactly the kinds of questions that James raised in his list. In fact, when I sought out James to ask him questions at a conference in Boston earlier this year, it was because I wanted his perspective on many of the points above, particularly #11: (hearing stories about how James has seen pairwise and combinatorial approaches to test design fail), and #7 (hearing his views on where it works best and where it would be difficult to apply it). I’ll save my specific answers to another post, but I am serious about wanting to share my thoughts on them; time constraints are holding me back today. I gave a speech at the ASQ World Conference on Quality Improvement in St. Louis last week though that addressed many, but not all, of James’ questions.

I’m not your typical software tool vendor. Basically, my natural instincts are all wrong for sales. I agree with the premise that “a fool with a tool is still a fool”; when talking to target clients and/or potential partners, I’m inclined to point out deficiencies, limitations, and various things that could go wrong; I’m more of an introvert than an extrovert, etc. Not exactly the typical characteristics of a successful salesman… Having said that, I believe that we’ve built a very good tool that helps enable dramatic efficiency and thoroughness benefits in many testing situations but our tool, along with the pairwise and combinatorial test design approaches that Hexawise enables both have their limitations. It is primarily by talking to software testers about their positive and negative experiences that our company is able to improve our tool, enhance our training, and provide honest, pragmatic guidance to users about where and how to use our tool (and where and how not to).

Tool vendors who defend their tools (and/or the approaches by which their tools helps users solve problems) as magical, silver bullet solutions are being both foolish and dishonest. Tool vendors who choose not to engage in serious, honest and open discussions with users about the challenges that users have when applying their tools in different situations are being short-sighted. From my own experiences, I can say that talking about the 13 topics raised by James have been invaluable.


4 thoughts on “13 Great Questions To Ask Software Testing Tool Vendors

  1. Hey Justin,

    I can’t take anyone seriously as a colleague until he argues with me. I just don’t understand people who think we can build a strong testing craft without a vigorous process of testing our ideas in open and sincere debate.

    You came to dinner with me in Boston and argued with charm, and conviction. Indeed, you are not the normal tool vendor. And I consider you a colleague in the best sense. Perhaps it’s your legal training, or your upbringing in an intellectual household. Still, I wish more testing experts would put their ideas on the line.



  2. Well, I tried to get a similar discussion of pairwise heuristics going on the Yahoo software-testing group a few years back, but not many too the bait. James has made a few public comments to temper some of the zeal surrounding pairwise testing as the cure to all ills.

    My own experience has been interesting. I understand when pairwise tests might help, but in almost every situation where I thought I found a good candidate test, I instead ended up with one of two outcomes –

    – I found a set of hand-crafted tests based on risk and value which gave me what I felt was more valuable coverage.
    – I went for a high-volume automated test that meant I could test all of the combinations.

    So I see a sweet spot for pairwise approaches in certain kinds of testing, but also that it competes with other test design approaches that aren’t so widely discussed.

    Looking forward to your responses 🙂

  3. Some good points here. For companies who wish to sell it as a silver bullet, you may gain a short term advantage but end up with a long term disadvantage. Overselling the tool means underwhelming the customer. Expectations will be too high and the customer will not put the necessary investment into the tool and you end up with shelfware that will affect your renewal. Worse yet, you may have an unhappy customer who can create untold damage to your reputation. Missing a few deals is worth the opportunity to have happy customers help you generate more easy deals.

  4. Justin,

    In my opinion, a tool vendor needs to juggle with the issue accepting the “context” in which the tool works and where it does not work.

    Competitive business environment dictates sales folks of the tools talk about only those contexts where the tool works (while tactfully avoiding others) or declaring that the tool works universally.

    It is difficult to be a pragmatic tool vendor who is open about contexts where tool does not work.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s