9  Delivery, communication and sign-off

Important

This version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs.

The draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised.

9.1 Introduction and summary

The successful delivery of analysis to its Commissioner marks its transition from being a product under development to one that is fit and ready to be used to inform decision making in your organisation and possibly inform the public.

This chapter provides information on the processes around assurance of communication of analysis and delivery of analytical output.

9.1.1 The Analyst’s responsibilities during delivery, communication and sign-off

  • The Analyst shall follow organisational governance procedures for delivery and sign-off, including, where appropriate, updating the business-critical analysis register, and making the analysis publicly available.
  • The Analyst should ensure that communication meets audience requirements, including accessibility.
  • The Analyst may be required to communicate the assurance state to the Approver, if not done directly by the Assurer.
  • The Analyst should be prepared to respond to challenge from the Approver. For example, this challenge might arise through project or programme boards.

9.1.2 The Assurer’s responsibilities during delivery, communication and sign-off

The Assurer shall communicate the assurance state to the Approver. This includes confirmation that the work has been appropriately scoped, executed, validated, verified, documented, and provides adequate handling of uncertainty. This communication may go via the Analyst.

9.1.3 The Commissioner’s responsibilities during delivery, communication and sign-off

The Commissioner shall use the analysis as specified at the start of the analytical cycle, applying any limitations to its use as described by the Analyst.

9.1.4 The Approver’s responsibilities during delivery, communication and sign-off

  • The Approver shall review the assurance evidence that has been provided to them.
  • The Approver should provide sufficient challenge to the analysts to gain assurance that the analysis is fit for purpose.
  • When they are satisfied with the validity and robustness of the analysis, the Approver should provide the Analyst with evidence that the analysis outputs have been properly reviewed and formally approved.
  • The Approver shall follow organisation governance procedures for sign-off.

9.2 Assurance activities in the delivery, communication and sign-off stage

9.3 Delivery

When delivering a piece of analysis, the Analyst and/or Assurer should communicate its assurance state to the Approver and provide evidence that the analysis and associated outputs have undergone proportionate quality assurance and to demonstrate that the analysis is ready for delivery, for example:

  • It uses suitable data and assumptions;
  • It meets the purpose of its commission;
  • It has been carried out correctly and to its agreed specification;
  • It has a risk assessment and statement against the programme risk register;
  • If meets analytical standards, such as those around coding standards and documentation;
  • It adheres to any professional codes of practice (e.g. The Code of Practice for Statistics
  • Where appropriate the analysis is accompanied by a completed assurance statement.

Though not strictly assurance, the analyst should also consider areas such as security ratings, retention policies, intellectual property, ethics and related concerns.

The Approver should scrutinise the evidence delivered and approve the work if the analysis meets specified organisational criteria. The Approver should then feedback the outcome of any approval activities to the analyst so that the analysis can be updated if required.

The exact nature of any scrutiny made by the Approver should be proportionate to the impact of the analysis, the governance process of their programme/ organisation, and follow the principles of proportionality described in Chapter 3 of this document.

To ensure that the analysis is used as intended, the Commissioner should use the analysis as specified at the start of the analytical cycle, applying any limitations to its use as described by the Analyst.

9.4 Communication

The effective and transparent communication is essential to enable analysis to be adopted and trusted by the Commissioner and onward users. Depending on its final use and likelihood of publication, any analysis may be communicated to a wide audience including:

The form of communication should be tailored to the audience. The communication should be quality assured in a proportionate manner to ensure an accurate reflection of the analytical results.

The Analysis Function’s Making Analytical Publications Accessible Toolkit gives guidance to help ensure that any that websites, tools, and technologies produced from analysis are designed and developed so that people with disabilities can use them. More specifically, people can: perceive, understand, navigate, and interact with the web.

If publishing the outcome of any analysis is required, the analyst should follow departmental and statutory guidance.Some examples are given below:

9.4.1 Sign-off

The exact nature of the approval process may vary depending on:

  • The impact of the analysis;
  • The approval process of the organisation, and
  • The nature of the programme, project or board approving the analysis.

The formality of the sign-off process should be governed by organisational procedures, and be proportionate to the analysis.

The Approver should provide the Analyst with evidence that the analysis outputs have been properly reviewed and formally approved. For example, through the notes of a project/programme board where the decision to approve the analysis was made or similar.

9.5 Documentation

When the Analyst and Assurer are satisfied that the analysis is ready to hand over to the Commissioner, they should ensure that any associated documentation supporting the analysis is ready and has also undergone quality assurance. Supporting documentation might include:

  • Specification and design documentation
  • Logs of data, assumptions and decisions including their source, ownership, reliability and any sensitivity analysis carried out;
  • User and technical documentation
  • Advice on uncertainty and its impact on the outputs of the analysis;
  • A description of the limits of the analysis and what it can and cannot be used for;
  • Any materials for presenting the analysis to the Commissioner, for example slide decks or reports;
  • A record of the analysis including methods used, dependencies, process maps, change and version control logs and error reporting;
  • The code-base, when it has been agreed to publish the analysis openly
  • The test plan and results of the tests made against that plan;
  • A statement of assurance;
  • A statement that ethical concerns have been addressed, especially for the application of black-box models;

9.6 Treatment of uncertainty in the delivery, communication and sign-off stage

Government has produced a range of guidance to support analysts in presenting and communicating uncertainty in analysis. This includes:

Each provides valuable advice on how to estimate and present uncertainty when describing the limitations of use of a piece of analysis.

9.7 Black-box models and the delivery, communication and sign-off stage

The Approver is responsible for signing-off that all risks and ethical considerations around the use of black-box models have been addressed. The aspects to be considered are detailed in the Introduction to AI assurance.

9.8 Multi-use models and the delivery, communication and sign-off stage

There is a greater risk that multi-use models may be used for purposes outside the intended scope. This places a greater onus on the Analyst to clearly communicate to all users the limitations and intended use. The Analyst may consider testing communication with different user groups to ensure that the analytical outputs are used as intended.

9.9 Analytical transparency

Enabling the public to understand and scrutinise analysis promotes public confidence in decisions. This includes providing the the public with information on models used for business-critical decisions and making analysis open. Further guidance on transparency can be found here.

9.9.1 Business-critical analysis register

This section applies to publishing lists of business critical analysis (BCA), including models.

  1. Departments and Arm’s Length Bodies1 (ALBs) should publish a list of BCA in use within their organisations at least annually.
  2. Each department and ALB should decide what is defined as business critical based on the extent to which they influence significant financial and funding decisions; are necessary to the achievement of a departmental business plan, or where an error could lead to serious financial, legal or reputational damage.
  3. Departments and ALBs should align their definitions and thresholds of business criticality with their own risk framework respectively. The thresholds should be agreed by the Director of Analysis or equivalent.
  4. ALB’s are responsible for publishing their own BCA list, unless agreed otherwise with the department. The ALB’s Accounting Officer is accountable for ensuring publication and the sponsor department’s AO oversees this.
  5. The BCA lists should include all business-critical analysis unless there is an internally documented reason that the analysis should be excluded, agreed with the Director of Analysis (or equivalent) and the agreement documented.
  6. Justification for not publishing a model in the list may include exemptions under the Freedom of Information (FOI) Act 2000 where relevant, for example, including, but not limited to: National Security, policy under development or prejudicing commercial interests.
  7. In addition to these exemptions, there may be further reasons where the risk of negative consequence is deemed to outweigh the potential benefits resulting from publication of the model. One example is where population behaviour may change in response to awareness of a model or modelling.
  8. For clarity, the name of model/analysis and what it is used for should be included, alongside links to published material.
  9. To ensure the list is accessible, content and structure should follow guidance for writing plainly

9.9.2 Open publishing of analysis

To facilitate public scrutiny, departments may choose to make analysis/models (e.g. source code or spreadsheets) and details which may include data, assumptions, methodology and outputs open to the public. Open publishing source code and other elements of analysis allows others to reuse and build on the work (https://www.gov.uk/service-manual/service-standard/point-12-make-new-source-code-open). Practical guidance to open coding can be found here.

Publication of analysis should also draw on the guidance for BCA lists related to accessibility and justification for when publication may not be appropriate. For analysis that is extremely complex in nature, it may be more appropriate to publish summary information instead, to aid the accessibility.

  1. ALBs include executive agencies, non-departmental public bodies and non-ministerial departments, please see Cabinet Office guidance on Classification of Public Bodies↩︎