How to use ONS Quality Questions
This guidance is an ALPHA draft. It is in development and we are still working to ensure that it meets user needs.
Please get in touch with feedback to support the guidance by creating a GitHub Issue or emailing us.
Structure of the ONS Quality Questions
There are 61 quality questions in total. They cover all the stages of the analytical cycle.
Answering 61 questions must seem quite daunting at first! The questions are designed to help you as you work your way through the analytical process, rather than to be answered all at once. Creating a log of answers as you move through the stages of the analysis workflow will help you to check that your work meets analytical standards, follows good practice and (if relevant) complies with the Code of Practice for Statistics.
Answering the questions will also help you make sure that everybody working on the analysis has a clear understanding of how and why it works as it does, and to support your users when writing your outputs. Moreover, most of the answers will help you to produce the critical documents that mitigate risk like an assumptions log, decisions log, issues log, and technical guides for your team and your users.
The AQuA book divides the analytical cycle into four stages:
- Scoping
- Design
- Conducting and checking analysis
- Delivery
You can find the questions in the ONS Quality Questions section of the website. The questions have tabs which relate to three themes:
Theme1: ONS Quality Questions and why they matter
Alongside each question we explain why it matters and explain the potential risks and benefits around it. We try to address the very sensible question of “why should I care about this?”.
Theme 2: The questions and the Code of Practice for Statistics
Each question is linked with the Code of Practice for Statistics pillars of Trustworthiness, Quality and Value. We explain the importance and relevance of each question in light of the three pillars so teams can better understand and apply these principles through out the project life cycle.
Even if your work does not directly feed into the production of official statistics, compliance with the principles and practices of the Code is a good way to strengthen the resilience of your work, increase transparency and clarity and reduce risk.Theme 3: Linking the questions to AQuA roles
Quality questions are also linked to which responsible role from the AQuA book would usually answer them. The idea is to highlight the clear line of accountability set out in the AQuA book in an easy-to-understand manner. We want to make it easier for teams to decide how the four key assurance roles of Commissioner, Approver, Analyst and Assurer are covered in their own workflows.
The AQuA book sets out four roles that cover different areas of assurance responsibility. Taken together, they provide a comprehensive set of assurance for an analytical project. The roles are:
- Commissioner (may be known as customer)
- Analyst
- Assurer (may be known as the analytical assurer or assuring analyst)
- Approver (may be known as senior analyst or senior responsible officer)
Let’s look at the roles more detail.
Responsibilities of the Commissioner
The commissioner is focused on making sure the analysis meets the required user needs.
Ensures that the context around the work is understood, so quality assurance is appropriate and proportionate. Ensures that there is enough time and resource for required assurance, and accounts for risk. Must understand strengths and limitations including uncertainty, so results are interpreted correctly. Delivers QUALITY OF OUTCOME — “The analysis meets user needs and we understand its limitations.” Requests the analysis and sets out their requirements. Agrees that what the analyst is going to do will satisfy the need. Accepts the analysis and assurance as fit for purpose.
Responsibilities of the Approver
The Approver is accountable for the analytical workflow throughout its lifecycle. Usually a senior member of the analytical team, they work closely with (or manage) the analyst team.
Signs off all important decisions made about the analysis to ensure that it is fit-for-purpose prior to use. Scrutinises the work of the analyst and the assurer. Confirms (if necessary) to the analyst, assurer, and commissioner that the work has been appropriately assured. There is no minimum grade for the SRO role, but they should have the expertise, resources and accountability to ensure that the analysis is well-designed, complies with relevant standards, works as intended and is fit for purpose. Delivers QUALITY OF CONTENT — “The analysis is well-designed and uses the right tools and methods to meet user needs.” — alongside the analyst team.
Responsibilities of the Analyst
The analyst team usually works to the SRO. Analysts are responsible for setting up, running, checking and reporting on the analysis.
Assist the Commissioner and SRO in framing the question to ensure the right analysis is done. Manage external specialists. Design, build, document, and run analyses. Carries out their own assurance. Acts on findings from the assurer. Can be a group of analysts, in which case the lead analyst is responsible. Delivers QUALITY OF CONTENT — “The analysis is well-designed and uses the right tools and methods to meet user needs.” — alongside the SRO.
Responsibilities of the Assurer
The assurer is there to make sure high-level assurance takes place.
Assurance takes place throughout the lifecycle — from design through to use. Reviews the assurance completed by the analyst. Carries out any further validation and verification they may see as appropriate. Reports errors and areas for improvement to the analyst. Undertakes repeated reviews as required. Confirms to the approver that the work has been appropriately scoped, executed, validated, verified, and documented. Must be independent from the analyst. Can be a group of assurers, in which case the leader of the group is responsible. Delivers QUALITY OF PROCESS — “The analysis does what it’s supposed to do — and we can prove it.”
You can read more about the four roles in the AQuA Book and Verification and Validation for the AQuA Book.
Answering the ONS Quality Questions
We have made templates for you to download and use to record your answers to the questions, as well as key project information. There is an Excel template and an editable HTML template.
Save the template in your project repository or on Sharepoint so that changes can be tracked through version control. You can update it as your analytical work progresses.
We encourage the commissioner, analyst team and assurer to contribute response to all the questions that are relevant to their responsibilities. To keep the level of assurance proportionate to your analysis it might be decided that only a subset of questions need be covered.
The commissioner and lead analyst should agree the level of assurance required and identify which questions are most appropriate together with any possible additional ones.
The Excel template has mapped the questions to various categories including Analysis Function’s Quality Questions and Red Flags and the OSR’s Statistical Thinking – both these publications promote their questions as a core set that anyone doing analysis should consider. To identify these questions it is only necessary to filter on the appropriate columns in the worksheet.
It is also possible to filter questions relevant for each stage of the Generic Statistical Business Process Model. Many questions are relevant for more than GSBPM stage and only need be answered once!
We aim to add further categories to subset questions best suitable for production, research and adhoc projects. These would only be suggestive but could provide a start.
For audit purposes, remember to sign and date your responses.
When analysis is due to complete the lead analyst or their designated representative should provide sign-off for the whole template if satisfied that responses provide sufficient assurance of the analysis.