From arrizza.org wikiJump to navigation Jump to search
|Previous ⇦ Main Page||Main Page ⇫ Up||Algorithms ⇨ Next|
- 1 Requirements
- 1.1 Types of Requirements
- 1.2 Format of Requirements
- 1.2.1 Conditions & Asserts
- 1.2.2 Fundamental Rules
- 1.2.3 Indeterminacy
- 1.2.4 Implied Conditions
- 1.2.5 Compound Conditions
- 1.2.6 Compound Asserts
- 1.2.7 Multiple Asserts
- 1.2.8 Testability
- 1.3 Test Protocols
- 1.4 Test Result
- 1.5 Using Requirements
This may seem like an odd topic. Requirements are easy, you just write what you want the system to do and that's it. But there is a bit of an art and a bit of a science to it.
Requirements exist so we can state what we want the system to do and then compare what the system actually does and verify those match. Requirements clarify the most important behaviors of a system. There will be many behaviors of a system that are not important enough to write down. But there are those behaviors that are important and so we commit to the extra effort of writing them down.
The "compare" part means that requirements exist so that testing can be performed efficiently and correctly. A requirement focuses the testing effort and clarifies the intent of the test(s). A test team may or may not test other aspects of the system, but they will test the behavior specified by the requirements.
Having a common, simple structure that each requirement follows helps ensure they are more legible and are easier for readers to understand. A simple structure will also help testers adequately prepare for and test the requirement.
Types of Requirements
There are three broad types of Requirements:
- User (not covered here)
User Requirements are usually specified before designing and implementing the system. They provide a set of constraints from a User's perspective. Examples:
- The system shall weigh no more than 2kg
- The system shall operate on battery for at least 8 hours
- The system shall be compliant with common practices used in a majority of hospitals in the USA and Europe
- All User Interface instructions shall use a maximum of Grade 8 level language
User Requirements are verified using a different type of test methodologies and will not be covered here.
Prescriptive Requirements are usually specified before designing and implementing the system. They provide a set of constraints that the design and implementation needs to comply with.
These requirements specify what the system should behave like from a high level point of view. They are sometimes called "Product Requirements".
They also answer other basic, high level questions:
- Are there external standards to which the system should comply with? Examples:
- The system shall comply with IEC-60601-2
- What are the system level parameters? Examples:
- The pump shall be able to provide 0.1ml/hr to 1000 ml/hr delivery rates for all therapies.
- There shall be 2 types of therapies: Basic and Intermittent
- The pump shall communicate to the Server via 802.11abgn and a cellular modem
Most User Requirements can be traced to Prescriptive Requirements, i.e. it should be possible to confirm that a User Requirement is being met using a set of one or more Prescriptive Requirements. Some simply "bleed through" into Prescriptive Requirements. Example:
- The battery capacity shall be sufficient to provide at least 8 hours of operation
Note that there will be Prescriptive Requirements that may not tie back to a User Requirement. E.g. The system shall comply with IEC-60601-2
Prescriptive Requirements rarely change since they describe what the final product's overall characteristics are. As the design and implementation is completed, there may be changes to clarify them or to mark them as for future development.
Descriptive Requirements are usually specified after design is complete. They are sometimes called "Subsystem Requirements"
A system can be designed and implemented in many ways. Each design has a set of dependencies on sub-components or external systems. These dependencies need to be specified so that their interactions are clear and tested.
Format of Requirements
Conditions & Asserts
All requirements have the form:
If C then A
where C is a Condition and A is an Assert.
A Condition specifies the state of the system within which the Assert is allowed to be evaluated.
The Assert specifies the system's behavior when the Condition is in effect. An Assert has the word "shall" in it to indicate the Assert phrase must be met if the system is behaving correctly.
A Condition C may or may not be true, and an Assert A may or may not be true. These rules describe what the possible combinations mean:
- If C is true and A is true then the requirement has been met. The system is behaving correctly.
- If C is true and A is false then the requirement is not met. The system has a defect.
- If C is false then the requirement does not apply and so it is indeterminate.
A requirement can be indeterminate. This can occur when, for example:
- the requirement applies elsewhere but not here
- the requirement applies at some other time, but not now
The Assert clause may still be true (or false), but we can't use the result. An example:
if on the ABC Screen, the Next button shall be displayed
is indeterminate if the current screen is the XYZ Screen i.e. it's not the ABC screen. There may, in fact, be a Next button displayed, but it doesn't matter. We are not on the ABC screen and so we have to throw out that result.
It may be indeterminate if, during the test execution, the system should have been on the ABC screen, but it is not. That is, when the tester was preparing to test, they incorrectly ended up on some other screen. Perhaps there is a defect in the system that prevents them from going to the correct screen. Or some system behavior can prevent them from testing the behavior at this point in time. In other words, they ran the preparatory test steps correctly, but the system has timing related behavior that prevents the correct screen from showing up now.
In any case, if a requirement is indeterminate, the automated test framework should exit the Test Case, or for manual testing, the test should be stopped. The rationale is that if a Condition is false:
- subsequent Conditions are likely to be false (whether rightly or wrongly)
- likely to lead to further indeterminate behavior
- may cause false positives
An Assert A is never implied. On the other hand, a Condition C may be implied.
Each requirement has a context. That context may be obvious or implied, e.g.
- The system is on
- The device under test is SystemABC
- The system has a color GUI
Specified in an indirect way. For example, the document section of the requirement may imply the Condition, e.g.
Section 1. ABC Screen Section REQ-123 The Next button shall be displayed
In this case, Requirement REQ-123 only applies if you are testing the ABC Screen.
A Condition may be universal:
REQ-234 The background image shall be blue
Requirement REQ-234 applies no matter where in the system you are testing. Note, that there may still be implied Conditions even in this case, e.g. the system is on or the application has been started.
The Condition C may be compound, e.g.
if C1 and C2 then A
In this case both Conditions (C1 and C2) must be true for the Assert A to be evaluable. If either C1 or C2 is false, the requirement is indeterminate.
Typically the word "and" in a Condition is OK. However, the word "or" indicates that there are two (or more) requirements being specified, they just happen to have the same Assert. e.g.
SRS123 if C1 or C2 then A
It is usually not good practice to do this. Test Cases are tied to specific requirements. When the requirement is "C1 or C2", a question is raised as to which Condition is that Test Case is using, is it "C1" or is it "C2"? To remove this potential confusion, it is better to split the requirement into two:
SRS123 if C1 then A SRS124 if C2 then A
This can be cumbersome at the very least and problematic as well. The Assert in both requirements need to change together, e.g. if the Assert in A SRS123 is changed, then SRS124 must change as well.
The Assert A may be compound, e.g.
if C then * A1 * A2 * A3
If Condition C is true then all of A1, A2 and A3 must all evaluate to true for the requirement to be met. Conversely, if any of A1, A2 or A3 is false, then the requirement is not met.
Using "and" in the Assert is OK and it's OK for the "and" to be implied via list like the example above.
However, using the word "or" is problematic. For example:
If C then A1 or A2
Which option "A1" or "A2" should the test should confirm? The tester could check for both and confirm if either scenario occurs when the system is set up with Condition "C". But it does imply that the system is non-deterministic which is rare. In that case, the system is randomly choosing which path to take. If that is the true expected behavior of the system, then having the "or" does make sense.
The more likely case is the Condition "C" is missing a clause that differentiates between the two scenarios. The requirement should be split into two or more requirements such that the "or" is not needed. For example:
If C and D then A1 If C and not D then A2
Note it is possible for the additional Condition to be embedded in language inside the Assert. For example:
REQ-500 On the Main Screen, if the User presses "Enter", the system shall navigate to the Therapy Screen if they've logged in already or to the Login Screen if they haven't.
The system is deterministic, and the requirement does capture the behaviors correctly. For clarity (and tracing purposes), it should be split into two requirements:
REQ-500 On the Main Screen, if the User presses "Enter" and they've logged in already, the system shall navigate to the Therapy Screen REQ-501 On the Main Screen, if the User presses "Enter" and they haven't logging in already, the system shall navigate to the Login Screen.
It is possible for the Assert to become very complex and may apply to multiple requirements. One way to do this is to use a document section to specify the Assert, for example:
Section 1 Main Screens SRS-100 On the Main Screen, there shall be a navigation bar that complies with all requirements in the Navigation Bar Section SRS-101 On the Therapy Screen, there shall be a navigation bar that complies with all requirements in the Navigation Bar Section <snip> Section 2 Navigation Bar SRS-300.1 The navigation bar shall indicate the current screen's unique name SRS-300.2 The navigation bar shall indicate which step in the current process the User is performing SRS-300.3 The navigation bar shall be at the top of the screen
If this is done, there are complications in the testing. For example, the navigation bar has to be tested in each screen that it appears.
If a navigation bar requirement fails, then that "calling" requirement also has to fail, e.g. say SRS-300.2 failed for the Therapy Screen. Then the Test Report needs to indicate that SRS-300.2 failed and that SRS-101 failed as well. But note that SRS-300.2 passed on the Main Screen (SRS-100).
Multiple Asserts are evaluated instantaneously. In other words the order that the Asserts are evaluated is irrelevant.
For example the following are all equivalent to each other:
1) If C1 then A1; A2; A3 2) If C1 then A3; A1; A2 3) If C1 then A2; A3; A1 etc.
This also spans requirements, e.g. If C1 and C2 are both true, then the following are equivalent:
1) If C1 then A1; If C2 then A2 2) If C2 then A2; If C1 then A1 3) If C1 & C2 then A1; A2 4) If C1 & C2 then A2; A1 etc.
A Requirement is considered Testable if all of the following are met:
- There is an appropriate list of set up instructions that can make the Condition(s) true. If there are Compound Conditions, then the setup instructions can make all of the Condition(s) true at once.
- When the Condition is true, the Assert can be determined to be either true or false
- The Assert is either true or false, and cannot be both true and false at the same time.
#1 All Conditions Can be True
It must be possible, somehow, to make the Condition or Conditions all true. If not, the requirement can't ever be tested and therefore is invalid. Invalid requirements must be removed from the requirements document.
Here are some (contrived) examples:
REQ-100 On the Foo Screen, the label1 shall display "bar" !! The Foo Screen does not exist in this application REQ-101 If the value of field1 is 0 and the value of field1 is 21, the label2 shall be "hello" !! It is not possible for the Compound Condition to ever be true REQ-102 If the sensor value is greater than 200, label3 shall be "error" !! The sensor in use can not measure values above 199. REQ-103 If the Bar function is enabled, hide the label1 field REQ-104 On the Foo Screen if the Bar function is enabled, the label1 shall display "bar" !! The Foo Screen exists, the Bar function is enabled, but the label1 field can not be seen at that time
There may need to be a reasonableness standard applied to the requirement as well. It may be "possible" to make the conditions true, but that scenario is extremely rare or it is short-lived. Or there may be a set of instructions that make the conditions true, but they take the tester a very long time to run, the requirement may still be untestable from a practical point of view.
#2 Assert must be determinable
There must be a method to determine if the Assert is true or false. Here is a (contrived) example.
REQ-200 When the hard drive is powered off, the hard drive shall contain the file "some_file.txt" !! Can't check the contents of the hard drive unless it is powered on
In this example, since the hard drive can't be checked while it is powered off, the existence of the file can't be determined. The Requirement is therefore untestable.
Note there may be extreme methods available for a tester to use. For example, the hard-drive platter could be tested independently using a secondary sensor and therefore the file existence could, in fact, be determined. And so, again, there may need to be a reasonableness standard applied to the requirement.
#3 Assert must be either true or false, not both
The tester must be able to make a clear determination the system is meeting the Assert or not.
Some (contrived again) examples:
REQ-300 The Startup screen shall display for longer than 2 seconds and less than 1 second !! The assert is mutually inconsistent REQ-301 The Next button shall be blue. !! The Next button button is in fact blue, but there are several kinds of blue buttons in the system.
A Tester may need specific expertise to determine if the Assert is being met. That expertise is usually gained through training. In this case, there may be a prerequisite in the Test Case indicating that the Tester has finished the training and provides evidence of the same in the Test Result.
A Test Case may require multiple runs to be done and a statistical method applied to determine if the Assert is being met. Eg.
Req-400 The therapy shall deliver 1L of fluid in 120 seconds +/-5 seconds at most and +/- 3 seconds at least 70% of deliveries.
The Test Case would run the test, say 10 times, and measure the time it takes to deliver 1L. For all deliveries, the time to deliver must be 115 - 125 seconds, and at least 7 times it would be 117 to 123 seconds to pass the test case.
A Test Protocol is a set of instructions for testing the requirements.
A Test Protocol is usually divided into a set of Test Cases. Each Test Case performs tests for a small number of requirements (usually one or two).
The Test Case instructions will prepare the system such that the Conditions for the requirement are all true. Once the Conditions are true, the Assert is evaluated to produce a Test Result. An Assert produces one and only one Test Result.
If there are a sufficient number of Test Cases with common setup instructions or from a organizational point of view (e.g. "test the Basic Therapy"), they may be grouped together into a Test Suite. A Test Suite's result is an aggregate of it's individual Test Cases e.g. "Basic Therapy is Passing".
A Test Result has two values: Pass or Fail.
- If the Assert is true, then the Test Result is Pass
- if the Assert is false, then the Test Result is Fail
Note for logistical convenience, a Test Result may have other values. For example:
- todo: indicates there is more work to be done, e.g. used during test development
- deviation : indicates this is a known failure, e.g. the system will ship with the deviation
- skip: indicates this is a temporary skip of the Assert, e.g. the Assert is known to fail and will be fixed at some time later
In a Waterfall software methodology, Requirements are tested when they are complete, when the design is done and the implementation is complete. There are specific times in the development flow where a review is done of the Requirements, Test Protocols, etc.
In an Agile methodology this is done in an ongoing process. For example, to implement a Feature:
- Prescriptive Requirements are reviewed to ensure the feature is outlined correctly there. If not, additional requirements are added, modified or deleted
- an informal design session is held to determine how the Feature will behave and how it fits in to the existing system.
- Existing Descriptive Requirements are reviewed to determine if any need to change because of the new Feature.
- New requirements are added for the Feature
- The developers, testers, user representatives and other interested parties review the changes to ensure they are correct.
- Design documents are similarly updated and development is started. If any questions arise, the requirements are reviewed for updates as necessary.
- Test Protocol and Test Cases are modified or updated to test to the new Requirements. If any questions arise, the requirements are reviewed for updates as necessary.
- The development team delivers an initial version of the software to the test team.
- The test team runs their initial version of the Test Cases.
- This is repeated until all the Tests pass and the following need no further changes:
- Test Protocols
- Test Cases
- the software