“Mirror, mirror on the wall, how much testing to test it all?”
How many times have you asked yourself that question when planning software testing and trying to determine what is ‘enough testing’? Testing, as a part of Quality Control, is an integral part of software project planning and execution at my company, Segue Technologies. A key element of this test planning is creation of suites of tests that exercise (e.g., cover) a maximum number of the software’s functional paths and possible variants. The degree of ‘coverage’ provided by any software test suite will be shaped and constrained by available project resources (budget, schedule, staff) and by the quality of the code to be tested.
Let’s look at how the quality of the code you are planning to test may affect your planning and achievement of your test coverage goals. Before we begin, there are a few things we need to define.
Definitions
Test Coverage: Test coverage is a measure of the amount of software functionality (e.g., number of coverage items) exercised by a test suite. Wherever we can count things and can tell whether or not each of those things has been exercised by some test, then we can measure coverage. Test coverage items come in various flavors: statements, branches, or paths in a program. Test coverage is usually expressed as a percentage:
Number of coverage items exercised
Coverage = ((Number of coverage items exercised)/(Total number of coverage items)) x 100%
A suite of tests that exercises all known coverage items (functional/logic paths, data combinations, and variants) provides 100% coverage. When setting project test coverage goals, you may want to consider these factors and constraints:
- Project quality goals
- Available testing resources, i.e. budget/staff/schedule
- Program size/complexity
- Time required to design/set up/execute tests
- Time required to report/investigate defects
- Amount of rework to address expected/actual defects
Rework Factor: Rework factor, a measure of code quality; represents the fraction of test cases from the current test cycle that can be expected to require rework in the next test cycle (e.g.: the ratio of defects reported per tests performed). Rework factors may be based on current and historical project data or may be pure estimates. A rework factor of 0.10 means that for every 10 tests performed, 1 defect can be expected to be reported. The larger the rework factor, the more rework effort and resources would be required.
Expected Testing: Expected testing represents the maximum number of planned tests which can be performed within the available test cycle time to achieve test coverage goals for a module.
Mathematically, Expected Testing = (Test Cycle Time – Setup Time) / (Test Execution Time).
Expected Defects: Expected Defects represents the estimated number of defects that may need to be addressed in the next test cycle based on the number of tests (planned and unplanned) performed in the current test cycle and the corresponding rework factor.
Tested in Cycle: Tested in Cycle represents the estimated number of planned tests which can be performed within the available test cycle time to attempt to achieve the test coverage goals. This estimate will be impacted by the number of rework factor-based defects injected from previous test cycles which must be retested in the current test cycle.
Mathematically, Tested in Cycle = [(Test Cycle Time – Setup Time) – (Rework Factor)*(Expected Testing)*(Defect investigation time + Retest execution time)]/ (Test Execution Time)
Total Tested: Total Tested represents the total estimated number of planned tests which can be performed across all test cycles to attempt to achieve the test coverage goals. This total will be impacted by amount of rework to address defects encountered during all test cycles.
Setup Time: Setup Time represents the time needed by the tester to be “ready” to test. Based on the information given by the development teams, setup time is defined by module, once per cycle.
Hypothetical Testing Scenario:
The following scenario illustrates the impact of code quality on testing coverage. The scenario considers an application composed of three independent modules (A, B, and C), each developed independently by different developers. Two Test Cycles are considered for each Module. The following parameters apply for this scenario:
- Test cycle time: 90 minutes
- Expected setup time per module: 2 minutes
- Test execution time: 2 minutes per test
- Defect investigation time: 5 minutes per defect
- Retest execution time: 2 minutes per defect
- Expected number of tests to achieve test coverage goals per module: 44 tests
- Rework factors per module (historical):
- Module A: 0.00
- Module B: 0.05
- Module C: 0.1
- Expected defects for each module (based on rework factor):
- Module A: (44)*(0.00) = 0
- Module B: (44)*(0.05) = 2.2 (~2 rounded)
- Module C: (44)*(0.10) = 4.4 (~ 4 rounded)
Test Cycle 1:
For Modules A, B, and C, Expected Testing = (90 – 2)/2 = 44 planned tests per Test Cycle.
Based on the rework factor values for each module, Module A should have no expected defects, while Modules B and C should have 2 and 4 expected defects respectively.
For Test Cycle 1, there would be no defects from prior cycles to retest, so for Modules A, B, and C, Tested in Cycle = [(90 – 2) – 0)]/2 = 44 planned tests.
The table presented below displays the results obtained at the end of the Test Cycle 1.
|
Cycle |
Test Cycle Time (Minutes) |
Expected Testing (# Planned Tests) |
Module |
Expected Defects |
Tested In Cycle (# Planned Tests) |
Total Tested (# Planned Tests) |
|
1 |
90 |
44 |
A |
0 |
44 |
44 |
|
B |
2 |
44 |
44 |
|||
|
C |
4 |
44 |
44 |
At the end of Test Cycle 1, the Test Coverage for each of Modules A, B, and C is 44/44 tests (100%). Based on their respective historical rework factors, testing of Modules A, B, and C should encounter 0, 2, and 4 defects respectively, which must be addressed in Test Cycle 2.
Test Cycle 2:
In Test Cycle 2, expected defects from Test Cycle 1 will need to be investigated and retested, at the expense of the number of planned tests. The rework factor for each module will impact the net number of planned tests which can be executed to achieve test coverage goals.
For Module A, there were no defects from Test Cycle 1 to fix/retest, so the full 90 minute Test Cycle Time is available for new tests and the planned full 44 test suite may be exercised. Tested in Cycle = 44 planned tests.
For Module B, 2 defects from Test Cycle 1 must be investigated and retested, at the expense of available cycle time for planned tests. Tested in Cycle = [(90-2) – (0.05)*(44)*(5+2))]/(2) = 36.3 ~ 36] planned tests.
For Module C, 4 defects from Test Cycle 1 must be investigated and retested, at the expense of available cycle time for planned tests. Tested in Cycle = [(90-2) – (0.10)*(44)*(5+2))]/(2) = 28.6 ~ 29] planned tests.
The table presented below displays the results obtained at the end of Test Cycle 2.
|
Cycle |
Test Cycle Time (Minutes) |
Expected Testing (# Planned Tests) |
Module |
Expected Defects |
Tested In Cycle (# Planned Tests) |
Total Tested (# Planned Tests) |
|
1 |
90 |
44 |
A |
0 |
44 |
44 |
|
B |
4 |
44 |
44 |
|||
|
C |
7 |
44 |
44 |
|||
|
2 |
90 |
44 |
A |
0 |
44 |
88 |
|
B |
4 |
36 |
80 |
|||
|
C |
7 |
29 |
73 |
At the end of Test Cycle 2, effective Test Coverage is 88/88 tests (100%) for Module A, but has dropped to 80/88 tests (91%) for Module B and 73/88 tests (83%) for Module C.
If additional Test Cycles are planned, these expected defects can be expected to further reduce the estimated number of planned tests for Modules B and C, which can be completed within the available Test Cycle Time. For example, at the end of a planned Test Cycle 3, you should expect an effective Test Coverage of 132/132 tests (100%) for Module A, 116/132 tests (88%) for Module B, and 102/132 tests (77%) for Module C.
To address code quality impacts on planned testing, you may want to consider one or more of the following options: increase available Test Cycle Time, reduce root causes of defects and rework, and reduce planned Test Coverage goals.
Software project test planning at Segue considers a spectrum of factors, constraints, and tradeoffs to provide its clients with software solutions of the highest caliber. The impacts of code quality are a key part of our planning matrix, and they should be part of yours as well.