All-pairs testing

In computer science, all-pairs testing or pairwise testing is a combinatorial method of software testing that, for each pair of input parameters to a system (typically, a software algorithm), tests all possible discrete combinations of those parameters. Using carefully chosen test vectors, this can be done much faster than an exhaustive search of all combinations of all parameters, by "parallelizing" the tests of parameter pairs.

Rationale

Assume that the test function^{[clarification needed]} has $N$ parameters given in a set $\{P_{i}\}=\{P_{1},P_{2},...,P_{N}\}$ . The range of the parameters are given by $R(P_{i})=R_{i}$ . Let's assume that $|R_{i}|=n_{i}$ . We note that the all possible conditions that can be used is an exponentiation, while imagining that the code deals with the conditions taking only two pair at a time, might reduce the number of conditionals.

To demonstrate, suppose there are X,Y,Z parameters. We can use a Predicate_(mathematical_logic) of the form $P(X,Y,Z)$ of order 3, which takes all 3 as input, or rather 3 different order 2 predicates of the form $p(u,v)$ . Clearly $P(X,Y,Z)$ can be written in an equivalent form of $p_{xy}(X,Y),p_{yz}(Y,Z),p_{zx}(Z,X)$ where comma denotes any combination. Thus, if the code is written as conditions taking "pairs" of parameters: then, we notice that the set of choices of ranges $X=\{n_{i}\}$ can be a multiset ^{[clarification needed]}, because there can be multiple parameters having same number of choices.

Let's define $max(S)$ as one of the maximum of the multiset $S$ . Then, the number of pair-wise test cases on this test function would be:-

$T=max(X)\times max(X\setminus max(X))$

Plainly that would mean, if the $n=max(X)$ and $m=max(X\setminus max(X))$ then the number of tests is typically O(nm), where n and m are the number of possibilities for each of the two parameters with the most choices, and it can be quite a lot less than the exhaustive $\prod n_{i}$

For example:

Parameter Name	Value 1	Value 2	Value 3	Value 4
Enabled	True	False	*	*
Choice Type	1	2	3	*
Category	a	b	c	d

In this case the parameters are Enabled with choices range of 2, Choice Type with 3, and Category with 4. That would mean: $X=\{2,3,4\}$ Hence, n = 4, m = 3 and number of tests would be 12. Notice that in this case, an exhaustive test would involve 2 x 3 x 4 = 24 tests, so in this example the number of tests is not reduced by much (see below for another example). The pict tool generated pairwise test cases on the input looks like:-

Enabled	Choice Type	Category
True	3	a
True	1	d
False	1	c
False	2	d
True	2	c
False	2	a
False	1	a
False	3	b
True	2	b
True	3	d
False	3	c
True	1	b

The below table would generate a multiset :

Parameter Name	Value 1	Value 2	Value 3	Value 4
Enabled	True	False	*	*
Choice Type	1	2	3	4
Category	a	b	c	d

In this case the parameters are Enabled with choices range of 2, Choice Type with 4, and Category with 4. That would mean:- $X=\{2,4,4\}$ and it is a multiset. Hence, n = 4, m = 4 and number of tests would be 16, which are shown in the below table:-

Enabled	Choice Type	Category
True	3	b
False	3	d
True	2	a
True	1	c
False	4	b
False	1	a
False	2	b
True	4	d
True	4	a
True	2	d
False	4	c
False	1	b
True	2	c
False	1	d
False	3	a
True	3	c

The reasoning behind all-pairs testing is this: the simplest bugs in a program are generally triggered by a single input parameter. The next simplest category of bugs consists of those dependent on interactions between pairs of parameters, which can be caught with all-pairs testing.^[1] Bugs involving interactions between three or more parameters are progressively less common,^[2] while at the same time being progressively more expensive to find by exhaustive testing, which has as its limit the exhaustive testing of all possible inputs.^[3]

This can be further generalized.^{[citation needed]} The idea is to apply sorting to the set $X=\{n_{i}\}$ so that $P=\{P_{i}\}$ gets ordered too. Let the sorted set be a $N$ tuple :-

$P_{s}=<P_{i}>\;;\;i<j\implies |R(P_{i})|<|R(P_{j})|$

Now we can take the set $X(2)=\{P_{N-1},P_{N-2}\}$ and call it the pairwise testing. Generalizing further we can take the set $X(3)=\{P_{N-1},P_{N-2},P_{N-3}\}$ and call it the 3-wise testing. Eventually, we can say $X(T)=\{P_{N-1},P_{N-2},...,P_{N-T}\}$ T-wise testing.

The N-wise testing then would just be, all possible combinations from the above formula.

One of the main strengths of combinatorial technique is that it enables a significant reduction of the number of test cases without compromising functional coverage.^[4] Many testing methods regard all-pairs testing of a system or subsystem as a reasonable cost-benefit compromise between often computationally infeasible higher-order combinatorial testing methods, and less exhaustive methods which fail to exercise all possible pairs of parameters. For example, consider the case of N=10 binary parameters. An exhaustive set of tests involves $2^{10}$ tests, whereas the all-pair setting would involve just 6.

Notes

^ Black, Rex (2007). Pragmatic Software Testing: Becoming an Effective and Efficient Test Professional. New York: Wiley. p. 240. ISBN 978-0-470-12790-2.
^ D.R. Kuhn, D.R. Wallace, A.J. Gallo, Jr. (June 2004). "Software Fault Interactions and Implications for Software Testing" (PDF). IEEE Trans. on Software Engineering. 30 (6).{{cite journal}}: CS1 maint: multiple names: authors list (link)
^ Practical Combinatorial Testing. SP 800-142 (PDF) (Report). Natl. Inst. of Standards and Technology. 2010.
^ "IEEE 12. Proceedings from the 5th International Conference on Software Testing and Validation (ICST). Software Competence Center Hagenberg. "Test Design: Lessons Learned and Practical Implications".

External links

[1] Black, Rex (2007). Pragmatic Software Testing: Becoming an Effective and Efficient Test Professional. New York: Wiley. p. 240. ISBN 978-0-470-12790-2.

[2] D.R. Kuhn, D.R. Wallace, A.J. Gallo, Jr. (June 2004). "Software Fault Interactions and Implications for Software Testing" (PDF). IEEE Trans. on Software Engineering. 30 (6).{{cite journal}}: CS1 maint: multiple names: authors list (link)

[3] Practical Combinatorial Testing. SP 800-142 (PDF) (Report). Natl. Inst. of Standards and Technology. 2010.

[4] "IEEE 12. Proceedings from the 5th International Conference on Software Testing and Validation (ICST). Software Competence Center Hagenberg. "Test Design: Lessons Learned and Practical Implications".

[1]

[2]

[3]

[4]

Rationale

Notes

See also

External links