Dunchurch Management Services

Test Design and Analysis

A 3-Day Course

Getting the Right Results from a Test Requires Effective Test Design

Systems are growing more complex and are developed at high stakes.  With unprecedented complexity, effective test engineering plays an essential role in development.

This three-day course is designed for military and commercial program managers, systems engineers, test project managers, test engineers, and test analysts. Although the course is technical in nature, the analytical and design methodology is taught from a layman's viewpoint. The focus of the course is giving individuals practical insights into how to acquire and use data to make sound management and technical decisions in support of a development program. Numerous examples and test design or analysis "traps or pitfalls" will be highlighted in class. Many design methods and analytic tools will be introduced. During this extensive study, the students (in groups) will participate in a detailed practical exercise designed to demonstrate the application of testing tools and methods for system evaluation.

Test design and analysis from a systems viewpoint, with sufficient detail to provide information for test engineers.

By the end of the course students should be able to:

• Design a cost effective test for a system
• Conduct specification and comparative tests
• Choose appropriate test and evaluation designs
• Determine if system effectiveness is influenced by outside factors
• Collect and analyze data, evaluate the results, and report conclusions and recommendations
• Determine if test results indicate any statistical or user/military significance

Topics Covered in the Course

Testing and Evaluation - Basic concepts for test and evaluation. 

• Why do we test?
• Verification and validation concepts
• Common T&E objectives
• Types of Test and Evaluation (DT&E, OT&E, concurrent T&E)

The T&E Context - View, model and process as a way to observe T&E.  Context and relationships between T&E and systems engineering.  Phases of T&E.

• Test and Evaluation context
• Systems Engineering context
• Evolutionary acquisition in DoD within the context of DOD-5000
• T&E support to acquisition, design reviews, configuration management
• The Test and Evaluation Master Plan (TEMP)

Needs and Requirements - Identifying the need for a test.  The requirements envelope, and how the edge of the envelope defines testing.  Understanding the design structure.

• Stakeholders, system, boundaries, motivation for a test.
• Requirements and their effect on test design.
• Design structure and how it affects the test

Issues, Criteria and Measures - Identifying the issues for a test.  Measures of effectiveness and measures of performance. Evaluation planning techniques.  Other sources of data.

• Identifying and formulating the issues.
• The Requirements Verification Matrix
• Developing evaluation criteria:  Measures of Effectiveness (MOE), Measures of Performance (MOP)
• Analysis to plan the test:  Operational analysis, Design/engineering analysis, Matrix analysis, Dendritic analysis
• Practical and efficient evaluation
• Modeling and simulation for test planning

Designing Evaluations & Tests - Specifics of the methods to design a test.  Analytic methods.  Sequences, input/output, statistical design, sequence testing.

• Sequence units under test - relationships of different units
• Input/output analysis - where test variable come from, choosing what to measure, types of variables.
• Review of statistics and probability distributions
• Statistical design of tests - basic types of statistical techniques, choosing the techniques, variability, assumptions and pitfalls.
• Sequencing test events - the low level tactics of planning the test procedure

Conducting Tests - Preparation for a test.  Writing the report first to get the analysis methods in place.  How to work with failure.

• Test preparation
• Forms of the test report
• Evaluating the test design
• Determining when failure occurs.

Evaluation - Comparing results to the criteria. Performance analysis.  Analyzing problems.  Types of evaluation reports.

• Analyzing test results.
• Comparing results to the criteria.
• Test results and their indications of performance
• Types of test problems and how to solve them.
• Test failure analysis - analytic techniques to find fault.
• Test program documents.
• Pressed Funnels Case Study - How evaluation shows the path ahead.

Special Types and Best Practices of T&E - Survey of special techniques and best practices. 

• Special types:
• Software testing, Design for testability, Combined testing, Evolutionary development, Human factors, Reliability testing, Environmental issues, Safety, Live fire testing, Interoperability
• Best Practices
• The Nine Best Practices of T&E

Summary - Review of the important points of the course.  Interactive discussion of participant experiences that add to the material. 

Need to know more? Contact Dunchurch about the Test Design and Analysis.