1  Introduction

Important

This version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs.

The draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised.

The AQuA book provides guidance on producing quality analysis for government. It aims to support well informed decision making to deliver better outcomes and improve the lives of citizens.

The AQuA Book has made a significant contribution to the cultural change in assurance practices in government. It is about the process for assuring analytical evidence in all forms. It sets out the core framework for assuring all forms of analytical evidence.

The last version of the Analytical Quality Assurance (AQuA) Handbook was published in 2015, following Sir Nicholas Macpherson’s Review of quality assurance of government models. Since then, assurance has become part of the fabric of good practice for developing evidence to support policy development, delivery and operational excellence.

The world of analysis has developed since we published the first edition of the AQuA Book.

Increasingly in our data driven world, insights from analysis underpin almost all policies and support operational excellence. This underlines the continuing importance of assuring our evidence. In parallel our working practices have developed. The dominant analytical tools when we wrote the last edition were spreadsheets and proprietary software. Since then we have broadened the range of methods to include open-source software, machine learning and artificial intelligence.

Users of the AQuA book have pointed out some things we did not cover in the first edition and areas where guidance was unclear or insufficient. In this edition we have added guidance on:

• multi-use models - large models used for many purposes with many stakeholders;
  
• assuring “black box”¹ analysis, including artificial intelligence;
  
• development, maintenance and continuous review;
  
• working with third parties such as contractors and academic groups and,
  
• publishing models.

We provide improved guidance on what a proportionate approach to assurance means and have made the whole guide applicable to all types of analysis.

The AQuA Book describes what you need to do but not how to do it, although it does contain many worked examples. Large organisations will have their own processes and practices covering “the how”.

For those of you who do not work in places with bespoke guidance you will find a collection of helpful resources in chapter 10.

The AQuA Book is a key supporting guide for the Analysis Function Standard. This Standard refers extensively to the AQuA book and notes that “detailed guidance on the analytical cycle and management of analysis, included in the Aqua Book should be followed.”

It is also referred to by the Green Book, the Magenta Book and other Functional Standards, such as the Finance Function Standards.

1.1 Who is the AQuA Book for?

In this edition we have tried to make our guidance relevant to anyone who commissions, uses, undertakes or assures analysis. It is about the whole process of producing analysis that is fit for purpose and not just about the checks after the analysis has been completed.

We would like to see producers and users of analysis from all backgrounds using this book, especially those producing analysis, evidence and research to support decision making in government. Our intended audience includes:

• Users of analysis – helping you to get the most out of your commission;
• Those who carry out analysis such as members of the government analytical professions, including:
  • operational researchers, statisticians and economists;
  • geographers;
  • finance professionals making spending forecasts;
  • actuaries;
  • social researchers carrying out qualitative research;
  • data scientists developing advanced analytics;
  • and anyone else carrying out analysis.
• Senior leaders with an interest in analytical assurance.

1.2 Why should I pay attention to this guidance?

Here are a few reasons why.

• Your analytical insights will be used for major decisions and operations. You need to do your best to get them right, thus minimising the risk of being complicit in causing operational, business or reputational damage;
• Trust is hard to obtain but easy to lose. A simple error that could have been prevented by assurance could lead to your and your team’s work being doubted;
• Prevention is better than cure. Analysis is more likely to be right first time when you consider quality from the start. Having appropriate quality assurance in place helps to manage mistakes, handle changes to requirements and ensure appropriate re-use;
• Delivering quality analysis provides the confidence that is needed for transparency and public openness;
• Assurance is required for audit purposes²; and
  
• Professional pride in your work.

1.3 How to use this book

The first four chapters of this book cover definitions and overarching themes, whilst the second half of the book goes into more detail on the analytical life cycle. This can be pictured as follows:

Each chapter in the second half of the book is structured as follows:

• Introduction and overview
• Roles and responsibilities
• Assurance activities
• Documentation
• Uncertainty
• Black box models
• Multi-use models
• Any other elements specific to the stage of the life cycle

This guidance uses the following terms to indicate whether recommendations are mandatory or advisory.

The terms are:

• ‘shall’ denotes a requirement, a mandatory element, which applies in all circumstances, at all times
• ‘should’ denotes a recommendation, an advisory element, to be met on a ‘comply or explain’ basis
• ‘may’ denotes approval
• ‘might’ denotes a possibility
• ‘can’ denotes both capability and possibility
• is/are is used for a description

These are the same terms as those in the UK Government Functional Standards.

Acknowledgements

The AQuA Book is the work of many authors from across the Government Analysis Function. The original version of the book was compiled by the Quality Assurance Working Group set up after Sir Nicholas Macpherson’s review of modelling in government. This revised edition of the book was produced by a task and finish group drawn from across the Analysis Function community. We would like to thank everybody who has given of their time and expertise to produce the revised edition.

---

1. Black box: system which can be viewed in terms of its inputs and outputs (or transfer characteristics), without any knowledge of its internal workings.

2. Managing Public Money, Annex 4.2 Use of models