As part of your software testing strategy for a new project, how many times have you had to ask, and then try to answer, the question: “How many test cases are enough?” Probably more than you would care to admit. There is no escaping the necessity of this question as well as the necessity of coming to grips with some meaningful answer and its resultant impact on your project. To try to answer this question we need to look at two types of testing involved, structural and functional, and their underlying testing criteria.
Structural (White-Box) Testing
Structural testing, also known as white-box testing, looks at a program’s source code to help create test cases. Structural testing’s main criteria is to test all code. A key conceptual and descriptive tool in planning structural testing is a control flow graph. Control flow graphs allow you to capture the various ways in which a program can execute in terms of its nodes and edges. A node in a control flow graph represents a program statement. An edge represents the ability for a program to flow from its current statement to the statement at the other end of the edge. If an edge is associated with a conditional (ex: IF-THEN-ELSE), it is labeled with the conditional’s value, either TRUE (T) or FALSE (F).
The following example displays pseudo-code for a function called DocumentRequired and a control flow graph that describes the function’s code logic flow. The pseudo-code represents a small piece of a software application/website which my company, Segue Technologies, developed to allow government contractors to evaluate their readiness to undergo a government review of their compliance with Federal Acquisition Regulations. In this example, based on three contract attributes (CeilingValue, BusinesType, Commmercial), function DocumentRequired makes a ‘Y’/’N’ determination if this contract-related document must be maintained by the contractor. The numbered pseudo-code statements are represented in the control flow graph along with the conditionals and flows from one statement to another. Generally, non-executable statements are omitted from the graph.
Control Flow Graph Example
White-Box Testing Criteria
Structural (e.g., white-box) testing strategies deal with how best to construct test cases and test suites to exercise all possible code along the various nodes, edges and paths of a control flow graph. The term code coverage refers to the degree to which the source code in a program, subprogram, or function is tested by a test suite. A test suite which provides high code coverage for a program more thoroughly tests its source code and reduces the chance of the program containing software bugs more than a test suite that provides low code coverage. To measure what fraction of code has been exercised by a test suite, several types of code coverage criteria may be used. Five basic types are:
- Statement Coverage: Every statement is executed at least once
- Edge Coverage: Every edge is traversed at least once
- Condition Coverage: For Boolean logical operators (AND, OR), the individual Boolean sub-expressions are evaluated in every possible combination of TRUE and FALSE
- Relational Coverage: For relational operators (>, <, >=, <=) the equal condition is treated as a separate branch
- Path Coverage: Every possible path is executed at least once
Statement Coverage Testing Example
A statement coverage testing strategy calls for testing each statement in a routine at least once. Pick a test case and plot its path though the control flow graph. Note each statement that is exercised by the test case. Continue to pick test cases until all statements along the control flow graph are covered.
In the statement coverage example below, Test Case 1 (CeilingValue = 650000; BusinessType = ‘F’; Commercial = ‘N’) covers statements 5, 6, 6’, 6’’, 7, and 10 (highlighted in color in the pseudo-code and in the control flow graph). Test Case 2 (CeilingValue = 3000; BusinessType = ‘F’; Commercial = ‘N’) covers statements 5, 6, 9, 10, exercising remaining uncovered statement 9 (also highlighted in color). Statement coverage is achieved with Test Cases 1 and 2.
Edge Coverage Testing Example
An edge coverage testing strategy calls for traversing each edge at least once. Pick a test case and plot its path through the control flow graph. Note each edge that is traversed by the test case. Continue to pick test cases until all edges along the control flow graph are covered. Edge coverage is also referred to as branch coverage. Conditional statements (ex: IF…) are traversed along their TRUE and FALSE edges, without regard to what are referred to as “short-circuit” Boolean operators (e.g., Boolean operators which do not necessarily test each of their operands). For example, in statement 6 in the pseudo code:
=650000ANDBusinessType = ‘F’ AND Commercial = ‘N'”>
The truth value of BusinessType = ‘F’ will only be checked if the truth value of CeilingValue >= 650000 is true. If CeilingValue >= 650000 is false, there is no need to check the truth value of BusinessType = ‘F’ since the entire expression is guaranteed to be false. Likewise, the truth value of Commercial = ‘N’ will only be checked if the truth value of BusinessType = ‘F’ is true.
In the edge coverage example below, Test Case 1 (CeilingValue = 650000; BusinessType = ‘F’; Commercial = ‘N’) covers edges 5-6, 6-6’, 6’-6’’, 6’’-7, and 7-10 (highlighted in color in the pseudo-code and in the control flow graph). Test Case 2 (CeilingValue = 3000; BusinessType = ‘F’; Commercial = ‘N’) covers edges 5-6, 6-9, and 9-10, exercising the remaining required uncovered edge from statement 6: 6-9 (also highlighted in color). Edges 6’-9 and 6’’-9 are associated with short-circuit operators, so they are not required to complete edge coverage.Edge coverage is achieved with Test Cases 1 and 2.
Condition Coverage Testing Example
A condition coverage testing strategy calls for traversing each edge at least once and for complex conditionals (especially short circuit operators) make sure that all possible combinations of TRUE and FALSE are tested. Pick a test case and plot its path through the control flow graph. Note each edge with a conditional that is traversed by the test case. Continue to pick test cases until all edges along the control flow graph are covered.
In the condition coverage example below, Test Case 1 (CeilingValue = 650000; BusinessType = ‘F’; Commercial = ‘N’) covers conditions 5, 6(T), 6’(T), 6’’(T), 7, 10 (highlighted in color in the pseudo-code and in the control flow graph). Test Case 2 (CeilingValue = 3000; BusinessType = ‘F’; Commercial = ‘N’) covers conditions 5, 6(T), 9, 10 (also highlighted in color). Two additional test cases are needed to cover the FALSE conditions for Edges 6’-9 and 6’’-9 to complete condition coverage. Test Case 3 (CeilingValue = 650000; BusinessType = ‘NP’; Commercial = ‘N’) covers conditions 5, 6(T), 6’(F), 9, 10. Test Case 4 (CeilingValue = 650000; BusinessType = ‘F’; Commercial = ‘Y’) covers conditions 5, 6(T), 6’(T), 6’’(F), 9, 10. Edge coverage is now achieved with Test Cases 1, 2, 3 and 4.
Relational Coverage Testing Example
A relational coverage testing strategy provides a stronger form of edge coverage testing in which any relational operator (>, <, >=, <=) has its “equal” condition treated as a separate branch. Therefore, in statement 6 of the pseudo code example, the “IF (CeilingValue >= 650000…” portion of the conditional should be treated as:
IF (CeilingValue > 650000) THEN (TRUE outcome)
ELSE IF (CeilingValue = 650000) THEN (TRUE outcome)
ELSE IF (CeilingValue < 650000) THEN (FALSE outcome)
Relational coverage is thus a stronger form of edge coverage since it is saying that for any conditional you should have at least three test cases: x >y, x<y, x==y. If you combine relational coverage with condition coverage, you will have the strongest form of edge coverage possible.
In the relational coverage example below, existing Test Case 1 (CeilingValue = 650000; BusinessType = ‘F’; Commercial = ‘N’) covers the relation “x = y”, while Test Case 2 (CeilingValue = 3000; BusinessType = ‘F’; Commercial = ‘N’) covers the relation “x < y”. New Test Case 5 (CeilingValue = 700000; BusinessType = ‘F’; Commercial = ‘N’) is required to cover the relation “x > y”. Relational coverage is achieved with Test Cases 1, 2 and 5.
Path Coverage Testing Example
A path coverage testing strategy calls for traversing each path through a program at least once. Unfortunately, full path coverage is usually impractical or impossible. Any routine with a series of n decisions in it can have up to 2**n paths within it. To make matters worse, any loop constructs can result in an infinite number of paths. Additionally, some paths may also be infeasible to traverse/test since there may be no program input value that can cause that particular path to be executed.
One testing strategy you may want to consider is to use the cyclomatic complexity of a module to determine the number of white-box tests that are required to obtain sufficient coverage of the module you want to test. According to this methodology, in almost all cases a module should have at least as many tests as its cyclomatic complexity, and this number should be adequate to exercise all of the relevant paths through that module.
Mathematically, the cyclomatic complexity of a module is defined in terms of the nodes and edges of a control flow graph containing the basic elements of the module. The complexity is defined as:
M = E – N + 2P
where:
M = cyclomatic complexity
E = number of edges on the graph
N = number of nodes on the graph
P = number of connected components (for a single routine, P = 1)
For the control flow graph in this article, the function begins executing at node 5 and exits at node 10. For this graph, there are 7 nodes (5, 6, 6’, 6’’, 7, 9, 10) and 9 edges (5-6, 6-6’, 6’-6’’, 6’’-7, 7-10, 6-9, 6’-9, 6’’-9), so N = 7, E = 9, P = 1, and M = E – N +2P = 4. The function has a cyclomatic complexity of 4.
According to the above methodology, an adequate white-box testing strategy for this function should have at least 4 test cases. The condition coverage testing strategy above, with four test cases which exercise all of the logic path conditions, is consistent with this path coverage testing methodology.
Functional (Black-Box) Testing
Functional testing, also known as black-box testing, tests the functionality of the system without regard to its implementation. We provide inputs, receive outputs, and verify if the outputs are what we would expect based on the supplied inputs. Since this testing typically does not allow us to look inside the system to see how it processes our inputs and compute its outputs, we treat the system as a “black box”.
Functional testing’s main criteria is to test all functions. Test case selection must ensure that the resultant test suite is “complete” with respect to the program’s requirements specification. This requires that all functions are fully tested to include both valid and invalid individual inputs as well as combinations of inputs. To be determined is how many test cases are required to declare the test suite to be “complete”. For each function to be tested, determine “categories” of input that the function should treat equivalently. These categories may be characterized by boundary conditions, as well as both “typical” input and error conditions. Each test suite will need at least one test case for each input category for each function to be tested. After test cases have been created, review them and remove redundant test cases. Finally, prioritize the test cases based on criticality in case project constraints may prevent you from executing the complete test suite.
A functional (black-box) test suite to validate the functionality provided by example FUNCTION DocumentRequired must include test cases to the various combinations of user input values for CeilingValue, BusinessType, and Commercial. The table below illustrates a test suite which will address the functional requirements to be tested. This test suite calls for seven test cases, without including redundant tests, to test the three inputs:
From this example your test suite options may look something like this:
We have touched on only a few of the factors regarding how many test cases in a test suite are necessary to provide an appropriate level of structural (internal) and functional (external) coverage to meet your software project’s quality goals. Other factors may weigh in to shape your strategy, such as available time, budget, and staff. In a related article, we discuss the impact of code quality on test coverage planning. Prudent application of test coverage strategies in your software projects should help you answer the question: “How many test cases are enough?” and help you keep your project costs down while raising the level of quality of the products you deliver to your customers.