Efficient and Effective Test Design

I enjoyed talking about efficient and effective combination testing strategies (and highlights of a recent empirical study) at yesterday’s TISQA meeting together with Lester Bostic of Blue Cross Blue Shield North Carolina, who shared his team’s experiences of adopting a combinatorial testing approach.  It addresses how tools like Hexawise can help software testers quickly identify the test cases they should execute to find as many defects as possible with as few tests as possible.  I wanted to share it now; once I have more time, I will comment on it and highlight some of the good questions, comments, discussion points, and tester experiences that were raised by the attendees.

The presentation focused on combinatorial testing techniques, such as pairwise testing, orthogonal array-based testing methods, and more thorough combination testing strategies (capable of identifying all defects that could be captured by, say, any possible combination of three or four “things” that you’ve decided to test for (regardless of whether those “things” include features configurations or equivalence class of data or type of user a mix of each).

The middle of the presentation also highlights empirical evidence that shows this method of identifying test cases often has an enormous impact on how quickly software testers are able to identify defects; citing the IEEE Computer article I co-wrote last month on Combinatorial Testing, this approach – on average – led to more than twice as many defects found per tester hour.

The final section of the presentation was delivered by Lester Bostic of Blue Cross Blue Shield and addresses his lessons learned.  Lester used Hexawise to reduce 1,356,136,048,589,996,428,428,909,447,344,392,489,476,985,674,792,960 possible tests (that would have been necessary to achieve comprehensive testing of the application he was testing) to only 220 tests that proved to be extremely effective at identifying defects.  <Side note: No, that absurdly large number with 51 digits after the “1” is not a typo;  it makes me smile every time I see it>.

Comments and questions are welcome.

About these ads

2 thoughts on “Efficient and Effective Test Design

  1. Absolutely fantastic. With the onus on testing teams to deliver more (quality and quantity wise), it is beneficial employing techniques as such and gain productivity, quality and performance levels.

    Very enriching post.

    KNOWLEDGE is POWER.

    Keep it up, Justin Hunter.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s