I am changing gears a little in this blog and am discussing a recent paper and presentation (and soon to be book) that was made known to me via a LinkedIn discussion. The topic deals with common testing problems.
Donald Firesmith from the Software Engineering Institute (SEI) at Carnegie Mellon University has put together a comprehensive report on common testing problems. You can read the full text in his two-part blog: “Common Testing Problems: Pitfalls to Prevent and Mitigate” at:
In his paper and in his blog, Donald makes the following statement: “A widely cited study for the National Institute of Standards & Technology (NIST) reports that inadequate testing methods and tools annually cost the U.S. economy between $22.2 and $59.5 billion, with roughly half of these costs borne by software developers in the form of extra testing and half by software users in the form of failure avoidance and mitigation efforts. The same study notes that between 25 and 90 percent of software development budgets are often spent on testing. “
I was especially pleased to read about how “requirements” fit into the common problems Donald found. Note: This topic is of special interest to me since I have seen many projects fail because of poor planning for (1) the verification and systems validation phases of projects and (2) the activities required to address those defects that are attributed to requirements.
Donald included defective requirements as one of the categories of common testing problems: “Requirements-related testing problems are related to the requirements that should be driving testing. Often, the requirements are ambiguous, missing, incomplete, incorrect, or unstable. Lower-level requirements may be improperly derived from their higher-level sources. Likewise, verification methods may be unspecified and the tracing between requirements and tests may be lacking.”
Donald lists the following requirement defects as pitfalls to testing:
– Ambiguous Requirements
– Obsolete Requirements
– Missing Requirements
– Incomplete Requirements
– Incorrect Requirements
– Requirements Churn
– Improperly Derived Requirements
– Verification Methods Not Specified
– Lack of Requirements Trace
For each of these, Donald goes into more detail, including: defect description, potential applicability, characteristic symptoms , potential negative consequences, potential causes, and recommendations for mitigating these defects.
We address all of these defects and their mitigation in our training.
Donald’s work is excellent and is sorely needed. Response to his paper and presentation was so great that Donald has turned the paper into a book: Common Testing Pitfalls: Ways to Avoid Them and Lessen Their Impact, to be published by Addison Wesley, as part of SEI’s series in Software Engineering, later this year.
Comments to this blog are welcome. Also, if you are interested, send me an email at firstname.lastname@example.org and I will send you a copy of my paper and presentation on “Thinking Ahead to Verification and Validation”.
If you have any other topics you would like addressed in our blog, feel free to let us know via our “Ask the Experts” page and we will do our best to provide a timely response.Tags: Carnegie Mellon University, common testing problems, requirement defects, Software Engineering Institute, validation, Verification
By Ken Edwards October 10, 2013 - 1:39 pm
Lou, would you be able to point to the LinkedIn discussion group you found this in? I’m interested in these topics and would like to take a look. Thanks.
By Lou Wheatcraft October 10, 2013 - 8:18 pm
The LinkedIn groups I participate that frequently have discussions on SE and Requirements include:
– MBSE, Model Based Systems Engineering
– Requirements Engineering
– Requirements Management and Analysis
– Systems Engineering
By Jim Nelson October 17, 2013 - 2:43 pm
Great article and relation to requirements. So many design people don’t realize the effects of poorly defined requirements. We (leaders and managers) need to find a way to quantify the costs and also promote the benefits better to help executives and design tacticians value the up-front time and effort spent on requirements development. Many professionals lose sight of rationale, levels, allocation and organization to streamline the verification process.
Thanks again for the awareness!
By Don Firesmith January 6, 2014 - 4:07 pm
By the way, the book is now out and available via Amazon. Pretty inexpensive too, about $36. The book addresses 92 testing pitfalls in 14 categories. Since six weeks ago when I had to baseline the book, I have been having people provide additional input. I have now added an additional 15 pitfalls and one new category (executable models). This testing of models (including requirements models) is naturally quite relevant to requirements engineering. Why wait until code exists to begin testing? With executable requirements models, you can stimulate them with test inputs under test preconditions, observe the resulting behavior and test postconditions and compare the results with a test oracle (say a customer/user representative and an application domain subject matter expert)and determine if the requirements themselves have defects.