9  Delivery, communication and sign-off

Important

This version of the AQuA Book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs.

The draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised.

The successful delivery of analysis to the commissioner marks its transition from being a product under development to one that is fit and ready to be used to inform decision-making in your organisation and, possibly, inform the public.

This chapter provides information on the assurance of communication of analysis and delivery of analytical output.

9.1 Roles and responsibilities in delivery, communication and sign-off

9.1.1 The commissioner’s responsibilities

The commissioner should:

• confirm that the analysis is likely to meet their needs
• use the analysis as specified
• understand and apply any limitations to its use

9.1.2 The analyst’s responsibilities

The analyst shall:

• follow organisational governance procedures for delivery and sign-off, including, where appropriate, updating the business-critical analysis register and making the analysis publicly available
• describe any limitations to using the analysis

The analyst should:

• ensure that communication meets audience requirements such as accessibility
• be prepared to respond to challenge from the approver or scrutiny from project or programme boards

The analyst should communicate the assurance state to the approver if this is not done directly by the assurer.

9.1.3 The assurer’s responsibilities

The assurer should communicate the assurance state to the approver. This includes confirmation that the work has been appropriately scoped, executed, validated, verified, documented and that it provides adequate handling of uncertainty. This communication may be undertaken by the analyst.

9.1.4 The approver’s responsibilities

The approver shall:

• review the assurance evidence that has been provided to them
• be confident that the analysis meets the design requirements, is of sufficient quality and is adequately and proportionately documented
• follow organisation governance procedures for sign-off, including updating of the business-critical analysis register, where appropriate

The approver should:

• provide sufficient challenge to the analysts to gain assurance that the analysis is fit for purpose
• provide the analyst with evidence that the analysis outputs have been properly reviewed and formally approved when they are satisfied with the validity and robustness of the analysis

9.2 Assurance activities

9.2.1 Delivery

When delivering a piece of analysis, the analyst (or assurer), should communicate its assurance state to the approver and provide evidence that the analysis and associated outputs have undergone proportionate quality assurance. They should also demonstrate that the analysis is ready for delivery. This may include confirming that the analysis:

• uses suitable data and assumptions
  
• has provisions for regular review
• meets the purpose of its commission
  
• has been carried out correctly and to its agreed specification
  
• has a risk assessment and statement against the programme risk register
• meets analytical standards, such as those around coding standards and documentation
• adheres to relevant professional codes of practice (for example, The Code of Practice for Statistics
• is accompanied by a completed assurance statement, where appropriate

Though not strictly assurance, the analyst should also consider areas such as:

• security ratings
• retention policies
• intellectual property
• ethics and related concerns

The approver should scrutinise the evidence delivered and approve the work if the analysis meets the required standard. The approver should then feedback the outcome of any approval activities to the analyst so that the analysis can be updated if required.

The exact nature of any scrutiny made by the approver should be proportionate to the effect the analysis is likely to have and the governance process of their programme or organisation. It should follow the principles of proportionality.

To ensure that the analysis is used as intended, the commissioner should use the analysis as specified at the start of the analytical cycle, applying any limitations to its use as described by the analyst.

9.2.2 Communication

Effective and transparent communication is essential to ensure analysis is adopted and trusted by the commissioner and onward users. Depending on its final use and likelihood of publication, any analysis may be communicated to a wide audience including:

• commissioners and users of the analysis
• external scrutineers such as the Public Accounts Committee, the National Audit Office, internal and external auditors
• the public, through publications and Freedom of Information Act requests
  
• academic experts, possibly through a departmental Areas of Research Interest document
• national and international governmental partners

The form of communication should be tailored to the audience. The communication should be quality assured in a proportionate manner to ensure an accurate reflection of the analytical results.

The Analysis Function’s Making Analytical Publications Accessible Toolkit gives guidance to help ensure that any that websites, tools, and technologies produced from analysis are designed and developed so that they meet accessibility guidelines.

If the outcome of any analysis needs to be published the analyst should follow departmental and statutory guidance. There are different guidelines depending on the work that is being published. These are:

• The regulatory guidance for publishing official statistics and national statistics
  
• The Government Social Research Publication protocol
  
• any recommendations from the Evaluation Task Force for evaluations

9.2.3 Sign-off

The exact nature of the approval process may vary depending on the:

• effect the analysis is likely to have
  
• approval process of the organisation
  
• nature of the programme, project or board approving the analysis

The formality of the sign-off process should be governed by organisational procedures and be proportionate to the analysis.

The approver should provide the analyst with evidence that the analysis outputs have been properly reviewed and formally approved. For example, through the notes of a project or programme board where the decision to approve the analysis was made.

9.3 Documentation

When the analyst and assurer are satisfied that the analysis is ready to hand over to the commissioner, they should ensure that any associated documentation supporting the analysis is ready and has also undergone quality assurance. Supporting documentation may include:

• specification and design documentation
• logs of data, assumptions and decisions including their source, ownership, reliability and any details of any sensitivity analysis carried out
• user and technical documentation
• advice on uncertainty and its affect on the outputs of the analysis
• a description of the limits of the analysis and what it can and cannot be used for
  
• any materials for presenting the analysis to the commissioner (for example, slide decks or reports)
• a record of the analysis including methods used, dependencies, process maps, change and version control logs and error reporting
  
• the code-base, when it has been agreed to publish the analysis openly
• the test plan and results of the tests made against that plan
  
• a statement of assurance
  
• a statement confirming that ethical concerns have been addressed, especially in cases that include the application of black-box models

9.4 Treatment of uncertainty

Government has produced a range of guidance to support analysts in presenting and communicating uncertainty in analysis, providing valuable advice on how to estimate and present uncertainty when describing the limitations of use of a piece of analysis. This includes:

• The Office for Statistical Regulation’s Approaches to presenting uncertainty in the statistical system
• The Uncertainty Toolkit for Analysts
  
• The Government Analysis Function guidance note on Communicating quality, uncertainty and change

9.5 Black-box models and the delivery, communication and sign-off stage

The approver is responsible for signing-off that all risks and ethical considerations¹ around the use of black-box models have been addressed.

This may include:

• formal consultation and approval by an ethics committee or similar depending on internal governance structures
• provisions for regular review, including whether on-going peer review is required to ensure the latest guidance and assurance methodology is taken into account
• communicating the “health” of the model at regular intervals to the commissioner.

Regularly reviewing model health

Model effectiveness can change over time. For example, does the model continue to behave as expected or has there been data drift where the model’s performance decreases? This can happen when a model is trained on historical data, but then uses current data when it is being used in production. Such a model may become less effective because it is no longer conditioned on the current state of the data.

You can read more about this in the Introduction to AI assurance.

9.6 Multi-use models

There is a greater risk that multi-use models may be used for purposes outside the intended scope. This means it is important that the analyst very clearly communicates to all users the limitations and intended use. The analyst may consider testing communication with different user groups to ensure that the analytical outputs are used as intended.

9.7 Analytical transparency

Supporting and encouraging the public to understand and scrutinise analysis promotes public confidence in decisions. This includes providing the public with information on models used for business-critical decisions, making analysis open and ensuring transparency.

The UK Statistics Authority has published guidance on how the principles of Trustworthiness, Quality and Value from the Code of Practice for Statistics can be applied in the design, development and use of models.

9.7.1 Business critical models register

Departments and Arm’s Length Bodies² (ALBs) should publish a list of business critical models (BCM) in use within their organisations at least annually. The list should meet accessibility guidelines.

Each Department and ALB should decide what is defined as business critical based on the extent to which they influence significant financial and funding decisions, are necessary to the achievement of a departmental business plan, or where an error could lead to serious financial, legal or reputational damage.

The definitions and thresholds of business criticality should be aligned with their organisation’s own risk framework. The thresholds should be agreed by the director of analysis or equivalent.

ALB’s are responsible for publishing their own BCM list, unless agreed otherwise with the department. The ALB’s accounting officer is accountable for ensuring publication and the sponsor department’s accounting officer oversees this.

The BCM list should include all Business critical models unless there is an internally documented reason for the analysis to be excluded. This should be agreed with the director of analysis (or equivalent) and that agreement should be documented.

Justification for not publishing a model in the list may include, but is not limited to:

• exemptions under the Freedom of Information (FOI) Act 2000
• national security
• policy under development
• commercial interests

In addition to these exemptions there may be further reasons where the risk of a negative consequence is deemed to outweigh the potential benefits resulting from publication of the model. For example, where population behaviour may change in response to awareness of a model or modelling.

For clarity, the name of the analysis or model and what it is used for should be included alongside links to any published material.

9.7.2 Open publishing of analysis

To facilitate public scrutiny departments may choose to make the analysis or model (for example, source code or spreadsheets) and any relevant data, assumptions, methodology and outputs open to the public. Open publishing source code and other parts of the analysis allows others to reuse and build on the work.

You can read more about making source code open and reusable.

The guidance for publishing Business Critical Analysis lists should be applied to the publication of analysis as in some cases it might not be appropriate to publish the work. For example, if the analysis is extremely complex it may be more appropriate to publish summary information to make the analysis more accessible.

---

1. See principles in Artificial Intelligence Playbook for the UK Government

2. ALBs include executive agencies, non-departmental public bodies and non-ministerial departments, you can read more in the [Cabinet Office guidance on Classification of Public Bodies(https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/519571/Classification-of-Public_Bodies-Guidance-for-Departments.pdf)