2 Definitions and concepts
This version of the AQuA Book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs.
The draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised.
This chapter sets out definitions and concepts that are used throughout the rest of the book.
Analysis
Analysis is the collection, manipulation and interpretation of information and data for use in decision-making. Analysis can vary widely between situations and many different types of analysis may be used to form the evidence base that supports the decision-making process.
Examples of types of analysis that are frequently encountered in government are:
- actuarial
- data science
- economic
- financial
- geographical
- operational research
- scientific, technical and engineering research
- statistical
- social research
Assurance
Analytical assurance is the process and set of practices that ensure analysis is fit for purpose.
Assurance activities
Assurance activities are any actions carried out in order to validate and verify analysis.
This may include:
- analyst testing
- peer review
- reconciliation of results to independent sources
Artificial Intelligence
Artificial Intelligence (AI) attempts to simulate human intelligence using techniques and methods such as machine learning, natural language processing, robotics, and generative AI. AI aims to perform tasks that typically require human intelligence, such as problem-solving, -making, and language understanding. AI models are usually considered black box models. They can be pre-trained models or custom built.
Black box models
Black box models’ internal workings are not visible, easily understood, or succinctly explained. These models take input and produce output without providing clarity about the process used to arrive at the output. This also includes proprietary models with protected intellectual property. Artificial Intelligence models (including Machine Learning) can often be considered a type of black box model. Other types of black box model may arise in future.
Business critical analysis
Business critical analysis refers to analysis that: * has a significant influence over finance and funding or departmental operation(s) * is necessary to the achievement of a departmental business plan or * is analysis where an error could have a significant reputational, economic or legal implications.
Business critical models
Business critical models are models to support or provide Business critical analysis.
Documentation
Specification documentation
Specification documentation records the initial engagements with the commissioner. It describes the question, the context and any boundaries of the analysis. The specifications provide a definition of the scope of the project and a mechanism for agreeing project constraints (for example, deadlines and available resources) and define what level of assurance is required by the commissioner.
Design documentation
Design documents describe the analytical plan, including the methodology, inputs and software that will be used. They also contain details of the planned verification and validation of the analysis. They provide a basis for the analytical assurer to verify whether the analysis meets the specified requirements.
You can read more about design documentation in the Design chapter.
Assumptions log
The assumptions log is a register of assumptions, whether provided by the commissioner or gathered from the analysis, that have been risk assessed and signed off by an appropriate governance group or stakeholder. Assumption logs should:
- describe each assumption
- quantify its effect and reliability
- set out when it was made
- explain why it was made
- explain who made the assumption and who signed it off
Decisions log
The decisions log is a register of decisions, whether provided by the commissioner or derived from the analysis. Decisions logs should:
- describe each decision
- set out when it was made
- explain why it was made
- explain who made the decision and who signed it off
Data log
A register of data provided by the commissioner or derived from the analysis that has been risk assessed and signed-off by an appropriate governance group or stakeholder.
User and technical documentation
All analysis shall have user documentation, even if the only user is the analyst leading the analysis. This documentation should include:
- a summary of the analysis including the context to the question being asked
- what analytical methods were considered
- what analysis was planned and why
- what challenges were encountered and how they were overcome
- what verification and validation steps were performed
Where relevant, the analyst may include a model map that describes data flows and transformations.
For analysis that is likely to be revisited or updated in the future, more comprehensive documentation should be provided to assist a future analyst. It may also be helpful to include guidance on what should be considered or updated.
Assurance statement
A brief description of the analytical assurance that have been performed to assure the analysis. The statement should refer to known limitations and conditions associated with the analysis.
The Department for Energy Security and Net Zero (DESNZ) and Department for Business and Trade (DBT) have published a range of quality assurance tools and guidance to help people with the quality assurance of analytical models. Modelling Quality Assurance tools and guidance are used across the departments to ensure analysis meets the standards set out in the AQuA Book and provide assurance to users of the analysis that proportionate quality assurance has been completed.
Machine Learning
Machine Learning uses algorithms to learn from patterns in data without needing to programme explicit business rules. Some models are white box models and others are considered black box models. Machine Learning is a subset of Artificial Intelligence.
Model
A model is a tool used to study, or understand, a part of the real world. Often, they use quantative methods and theories drawn from statistics, economics, or mathematics together with assumptions, to turn input data into numerical estimates. Models can be:
- descriptive - what is?
- predictive - what might be?
- prescriptive - what should be done if?
Maturity Model
A maturity model is a structured way to assess an organization’s capabilities within a specific area, like quality assurance.
Multi-use models
Multi-use models are used by more than one user or group of users for related but different purposes. These are often complex and large.
A Steering Group may be created to oversee the analysis of these models. This Steering Group would be chaired by the senior officer in charge of the area that maintains the model and consist of senior representatives of each major user area. The members of the Steering Group may have decision-making responsibilities in their area of work.
Quality analysis
Quality analysis is fit for the purpose it was commissioned to meet. It should be:
- accurate
- appropriately assured
- evidenced
- proportionate to its effect
- adequately communicated
- documented
- accepted by its commissioners
Roles and responsibilities
The AQuA Book defines the following roles:
- commissioner
- analyst
- assurer
- approver
You can read more in the Roles and Responsibilities section.
Third party
Any individual or group of individuals that are not commissioners or the analysts. For example, they may be working for a different government department, a different function or an outside company, person or group besides the two primarily involved in a situation, especially a dispute.
Uncertainty
Uncertainties are things that are not known, are in a state of doubt or are things whose effect is difficult to know. They have the potential to have major consequences for a project, programme or piece of analysis meeting its objectives. The NAO have published a good practice guide on managing uncertainty
There are different types of uncertainty. A common classification divides uncertainty into known knowns, known unknowns and unknown unknowns. The type of uncertainty will influence the analytical approach and assurance activities required.
The Uncertainty Toolkit for Analysts in Government is a tool produced by a cross-government group to help assessing and communicating uncertainty.
Validation
Validation ensures the analysis meets the needs of its intended users and the intended use environment.
You can read more in Verification and validation for the AQuA Book by Paul Glover.
Verification
Verification ensures the analysis meets it specified design requirements.
You can read more in Verification and Validation for the AQuA Book by Paul Glover.
Version control
It is important to ensure that the latest version of the analysis is being used and any changes made can be easily seen and quality assured by the analytical assurer. There are tools and templates that can be used to record any updates and checks made during a project. They can help to provide a log of the changes that have been made including why and when they were made, and who made them.