8  Analysis

Important

This version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs.

The draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised.

8.1 Introduction and overview

The analysis stage is where planned analysis is undertaken and assured, and progress and relevance are monitored. During this stage, the design may be amended to account for changing circumstances, emerging information or unexpected difficulties or limitations encountered. This stage also includes maintaining appropriate and traceable records of the analysis and assurance activities conducted, changes, decisions and assumptions made. In some cases, changes or limitations encountered may necessitate a return to either the design or scoping stage.

8.1.1 The Analyst’s responsibilities during the analysis stage

  • The Analyst shall follow the assurance plan, and conduct the specified verification and validation. They shall provide traceable documentation of the assurance they have undertaken. They shall respond to recommendations from the Assurer and act on them as appropriate.

  • The Analyst shall produce documentation of the data and methods used. The Analyst shall ensure these are sufficient for the Assurer to understand the approach.

  • The Analyst shall document any changes to the analytical plan in a proportionate manner.

  • The Analyst shall maintain appropriate contact with Commissioner and Assurer. This provides and opportunity for them to advise on whether the analysis is still meeting the Commissioner’s needs or whether there are any new requirements.

8.1.2 The Assurer’s responsibilities during the analysis stage

The Assurer shall review the assurance completed by the Analyst, carry out any further validation and verification they may see as appropriate, and report errors and areas for improvement to the Analyst. The Assurer may then need to re-review the analytical work completed, as required.

The Assurer may be required to provide feedback on changes to the analytical plan, and consider whether they are qualified to provide rigorous assurance on the revised methodology.

8.1.3 The Commissioner’s responsibilities during the analysis stage

  • The Commissioner should be available to provide input and clarifications to the Analyst.

  • The Commissioner’s should review any changes in design or methodology that the Analyst brings to their attention.

8.1.4 The Analyst’s responsibilities during the analysis stage

  • The Analyst shall follow the conduct the verification and validation activities that were designed as part of the analytical plan in the design stage. They shall provide traceable documentation of the assurance they have undertaken. They shall respond to recommendations from the Assurer and act on them as appropriate.

  • When the analysis includes coding, the Analyst shall proportionately follow best practice for code development.

  • The Analyst shall produce documentation of the data and methods used. The Analyst shall ensure these are sufficient for the Assurer to understand the approach.

  • The Analyst shall document any changes to the analytical plan in a proportionate manner.

  • The Analyst shall maintain appropriate contact with Commissioner and Assurer. This provides and opportunity for them to advise on whether the analysis is still meeting the Commissioner’s needs or whether there are any new requirements.

8.1.5 The Assurer’s responsibilities during the analysis stage

  • The Assurer shall review the assurance completed by the Analyst, carry out any further validation and verification they may see as appropriate, and report errors and areas for improvement to the Analyst. The Assurer may then need to re-review the analytical work completed, as required.

  • When the analysis includes coding, the Assurer shall review that the work proportionately adheres to coding best practice.

  • The Assurer may be required to provide feedback on changes to the analytical plan, and consider whether they are qualified to provide rigorous assurance on the revised methodology.

8.1.6 The Approver’s responsibilities during the analysis stage

The Approver should be aware of the progress of the analysis and ensure that they are available for approving the work at the delivery stage.

8.2 Assurance activities in the analysis stage

8.2.1 Verification and validation

Verification that the implemented methodology meets the intended plan should be incorporated as part the analysis. Whitener and Balci (1989) review verification techniques in relation to simulation modelling, but these techniques extend to analysis more broadly. These include:

  • Informal analysis: techniques that rely on human reasoning and subjectivity.
  • Static analysis: tests that the implementation of the analysis before it is run. For example, checking that code adheres to code conventions, structural analysis of the code by examining graphs of control and data flows, .
  • Dynamic analysis: tests the behaviour of the system or code to find errors that arise during execution. This includes unit testing, integration testing and stress testing
  • Symbolic analysis: particularly relevant to modelling and tests the transformation of symbolic proxies of model inputs into outputs during the execution of a model. Includes path tracing and cause-effect testing (see Whitener and Balci (1989) )
  • Constraint analysis: particularly relevant to modelling and tests the implementation of constraints during model execution. This includes checking the assertions of the model and boundary analysis.
  • Formal analysis: tests logical correctness through formal verification such as logic or mathematical proofs

Validation refers to testing whether the product meets the requirements of users. Hence, it is important to involve the users in the process. Methods for validation include quantification and judgment of acceptable sensitivity, specificity, accuracy, precision and reproducibility.

Validation of models includes testing the validity of the conceptual model, and testing the operational validity of any computerized model. Techniques that may be useful in validation of models are reviewed by Sargent (2011).

The Analyst has primary responsibility for conducting verification and validation. The Assurer is responsible for reviewing the verification and validation that is carried out by the Analyst, and for conducting or recommending additional verification and validation as required. The Assurer may refer to the specification document to assure that the analysis meets the specification.

8.2.2 Data validity and data considerations

Testing data validity (i.e. that data meet the specification for which they are used) is a key part of analysis. Procedures for assuring data validity include testing for internal consistency, screening for data characteristics (outliers, trends, expected distributions etc), and assuring robust data management practices (e.g. automating data creation and data sourcing).

It is rare to have the perfect dataset for an analytical commission. Reasons for this include:

  • The data is not available in the time frame required for the ideal analysis;
  • The data definition does not perfectly align with the commission;
  • There are data or coverage gaps;
  • The data may be experimental or there are other reasons why it is not ‘mature’.

Often, no data is available that are directly and precisely relevant to the parameter and conditions of interest. In such cases, it is often possible to use surrogate data. This is measurements of another parameter, or of the parameter of interest under different conditions, that is related to the parameter and conditions of interest. This implies an extrapolation between parameters, or between conditions for the same parameter. The use of surrogate data introduces further uncertainty, additional to that associated with the data itself. It may be possible to quantify this additional uncertainty using expert knowledge of the relationship between the surrogate and the parameter of interest.

The impact of using a proxy dataset should be explored and, if the uncertainty associated with the dataset has a large impact on the analysis, its appropriateness should be revisited. This exploration, and the decision to use a particular dataset or input, should be recorded for the benefit of the Assurer.

8.2.3 Assurance of code

The Duck Book provides detailed guidance on developing and assurance for delivering quality code. This includes guidance on structuring code, producing documentation, using version control, data management, testing, peer review, and automation.

8.3 Documention in the analysis stage

The Analyst should:

  • Maintain appropriate records of the work;
  • Fully document any code following agreed standards;
  • Log the data, assumptions and inputs used in the analysis, and decisions made (see documentation);
  • Record the verification and validation that has been undertaken, documenting any activities that are outstanding and noting what remedial action has been taken and its impact on the analysis;
  • Produce user and technical documentation).

8.4 Treatment of uncertainty in the analysis stage

While the Scoping and Design stages identified and described risks and uncertainties, the Analysis stage aims to assess and quantify the impact of uncertainty on the analytical outcome and their contribution to the range and likelihoods of possible outcomes. The Uncertainty Toolkit for Analysts reviews methods of quantifying uncertainty. The verification and validation by the Analyst and Assurer should assure the appropriate treatment of uncertainty.

8.5 Black box models and the analysis stage

Bblack box models such as AI and ML models are not as transparent as traditionally coded models. This adds challenge to the assurance of these models as compared to other forms of analysis. Assurance activities during the Analysis stage include performance testing and formal verification. See the Introduction to AI Assurance for further details.

8.6 Multi-use models and the analysis stage

In multi-use models, analysis and edits may be carried out on individual elements of the model at differing times. This calls for mechanisms for assuring that the changes integrate into the larger model as expected, for example, through the use of test-suites.