Create a VV&A methodology for DARPA for the Conflict Modeling, Planning and Outcome Experimentation (COMPOEX) Program. COMPOEX is a large political, military, economic, social, information, and infrastructure (PMESII) simulation.
The figure below illustrates the complexity of COMPOEX.
Each PMESII area is represented by a set of models, divided by level of resolution. In general, the models communicate with each other by posting variable results to the backplane at the end of each time step and reading variable values at the beginning of the next time step.
From a V&V perspective, each model must be addressed and the nature of the assumptions built into the communications links must be addressed as part of the V&V of the entire system. The fact of multiple simulation paradigms used within the system (systems dynamics models, SOAR technology, and possible other agent-based modeling systems) complicates the process.
However, the most challenging aspect from a VV&A perspective is the proposed use of different sets of models, depending on the situation, with models being custom built or modified immediately prior to use.
The team was composed under the Evidence Based Research (EBR) corporate umbrella, as follows.
Stuart Starr, Project Lead
Dean Hartley, Hartley Consulting
Al Sciarretta, CNS Technologies
Identified and defined five critical elements
VV&A Support to Risk Mitigation
VV&A to Support Compressed & Hyper-Compressed Situations
The figure below illustrates the elements of the modeling and V&V processes. Each of the model building and using processes involves a risk. The V&V processes permit the identification and correction of errors, reducing risk.
The V&V processes generally identify problems that cannot be simply corrected. However, these problems can often be mitigated, further reducing risk. Because no V&V of a major model will be complete, a portion of the decision concerning the use of the model will involve some level of trust, indicated in the figure below as "Initial Confidence." Finally, after all corrections and mitigations have been carried out, there will remain some level of risk. The accreditation decision is based on the comparison of the residual risk and the collective confidence built by the V&V concerning the use to which the model will be put. If the accreditation decision is positive, then the residual risk still exists, but is now termed "accepted risk."
Testing is divided into developmental testing, triggered testing, and periodic testing and is followed by accreditation, as defined in the following figure. The actual tests are of varying types and are adequately described in the V&V literature.
|All tools go through developmental testing
Certain events trigger additional testing
Periodically supplemental tests are performed to increase the understanding of the toolset
The various types of testing and the accreditation processes occur at specific times during the lifecycle of the model, as shown in the following figure.
The parts of the VV&A methodology can be illustrated in a flow diagram, as shown below.
Validation metrics for conceptual models (CM) are as follows:
The basic measures are
The results are shown over time in a table to place in proper context
Various pivot tables show the breakdown of
Description of unresolved problems
Validation metrics for coded models and the ensemble are divided into nine components
PMESII and DIME components are measured based on
Connections among the models are measured
User Issues are measured for each model and for the ensemble
The values are entered into a spreadsheet, which calculates the values. To enhance the transparency of those tests, the results are linked a set of “spider” diagrams to the spreadsheet (see figures below). The “spider” diagrams support visualization of multiple dimensions in a single chart and support an overview and segmentation by each individual model. The metrics and their diagrams provide the deep insight into the model’s strengths and weaknesses that emerge from the VV&A process . The metrics and their diagrams also support progress measurement as problems are fixed and are useful in conveying V&V status to the user.
Risk mitigation strategies are based on the CM validation level and the various sources of risk. The two figures below show the levels of risk and the appopriate mitigations for each combination.
DARPA proposed that COMPOEX should be modular and that the modules might be modified for each particular use. Naturally, this poses validation problems as the modified model does not necessarily partake of the validity of the base model. The VV&A procedures described above are referred to as the "Baseline VV&A." Normal, developmental modifications are covered under the baseline procedures; however, if the time between modification and use is on the order of a month (Compressed) or a week (Hyper-Compressed), special procedures are required. Several approaches and potential methods were explored, as shown in the figure below.
One promising method consists of exploring the PMESII space to determine a small number of input sets that can be used to test any modified model. This "Exploratory Design" is illustrated in the following figure.
A second method for testing the validity of a modified model, "Logic Tracing," is illustrated in the following figure. Time sequenced data are captured during the model run and displayed as graphs, organized by precursors to each given variable. This permits investigation of unexpected behaviors.
The third promising method involves the comparison of a high level model (HLM), constructed independently of COMPOEX. COMPOEX and the HLM are each run from some prior date up to the current date. Their outputs and the changes in the real situation are compared. If the results are sufficiently close, then COMPOEX can be used with some confidence to extrapolate the current situation. If the results are not in agreement, an attempt can be made to calibrate COMPOEX to the actual changes in the real situation. If successful, this calibrated version can be used with some confidence. If the calibration is unsuccessful, there may be an error in the modified model.
(Approved for Public Release, Distribution Unlimited)
For more on the details of tool requirements see Analytical Tools for OOTW.
For more on the ISSM (the prototype for the HLM) see Interim Semi-static Stability Model (ISSM).
For details on the VV&A tool that was developed to implement these results, see DIME/PMESII VV&A Tool.
A version of this work was also included as a chapter in a book.
If you arrived here using a keyword shortcut, you may use your browser's "back" key to return to the keyword distribution page.
Return to Hartley's Projects Page