Verification Test Planning

System Verification represents a significant effort associated with the development of any medical device. Consideration of test planning early in the process can significantly lower the overall effort.   This post addresses how to develop a plan for verification, specifically

• Testing at the right level (system or subsystem)
• Minimizing testing overlap
• Ensuring full requirements coverage

Testing at the Right Level

Testing at the right level requires careful analysis of requirements flow down. Requirements that are fulfilled by a single subsystem should be tested at the subsystem level. As a first step in test planning, carefully go through the system requirements and flow down and identify the testing level for the requirement.

Minimize the Overlap

During the test planning, avoid the common mistake of designing tests around large functional actions. Large functional test involve too many requirements and likely have too much overlap. Simple requirements failures, or the inevitable bug-fix to address an issue, lead to repeating large numbers of tests. Targeted small tests provide much less overlap.

Ensuring Optimal Coverage

In designing tests , structured test planning prevents the development of costly test methods with minimal effectiveness.  With this in mind, a tool such as factorial design. common with design of experiments, effectively minimizes the scope of testing .  

Leveraging factorial test design initially considers the requirements as factors of the overall test design.  During requirements review, considering how requirements interact significantly reduce the size of the factorial designs.  For example, a full factorial design of 3 requirements with 2 levels , referred to as a 2³ design requires 8 distinct tests with the 3 factors at the high or low level.  If one of the factors does not interact with the other factors, test scope requires just a 2² design of 4 tests. 

The following steps should be followed to establish the scope of the verification for performance requirements:

• Identify which requirements represent an input to the subsystem, and which requirement represents an output from the subsystem
• Review requirements and establish groupings of requirements identified.  Grouping should focus on reviewing the design to identify the requirements that interact
• Establish the factorial design for each of the groupings

For functional testing, develop the testing scope based upon the level of requirement interaction. This process organizes the functional requirements into groups with dependency and uses this as the basis for the test development.

An Example – Factorial Design

As an example, assume that a subsystem consists of a circuit board with the following requirements:

• The digital signal output shall accurately digitize the input signals over the range of 0-200 millivolts
• The Power supplied to the board shall be 12 volts +/- 0.5 volts
• The board shall weigh 0.6 Kg +/- 0.1 Kg

Clearly, the weight and power input would not be related, but the power and output signal could be related.   Signal output has been identified as subsystem performance requirement that will require a test method to be developed.   Using the test method developed to evaluate the signal output should have the following range of tests.

Factorial Test Design - Setting Up Test Runs

Test Run	Power	Input Signal
1	11.5 volts	0 millivolts
2	11.5 volts	200 millivolts
3	12.5 volts	0 millivolts
4	12.5 volts	200 millivolts