4 Quality assurance culture
This version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs.
The draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised.
Creating and maintaining a strong culture of quality assurance is a key element of ensuring analysis is high quality and robust.
This chapter particularly addresses senior leaders and describes their role in developing a strong culture of quality assurance. It outlines processes and approaches to support and embed quality assurance that senior leaders should consider for their teams. However, everyone has a role to play in creating a strong quality assurance culture. As such, the approaches outlined are useful for everyone to understand, regardless of their role in a given team.
For purposes of this chapter, culture is defined as the shared ways of working, beliefs and habits of an organisation.
A strong culture of quality assurance means that quality assurance is understood, expected, and valued across all those involved in the analytical process, including analysts, managers, senior leaders and stakeholders.
Culture enables the risk management described elsewhere in this document.
4.1 Leadership
A quality assurance culture starts with senior leaders. They are accountable for the quality of analysis carried out in their departments. A key element of discharging this accountability is to clearly set out the priority of quality within their teams and create processes for embedding quality assurance.
The guidance in Annex 4.2 of Managing Public Money assigns accountability for ensuring appropriate assurance processes are in place to the Accounting Officer. In practice the Accounting Officer may assign the responsibility to a senior leader reporting to the senior management board. The Accounting Officer may collect information on the state of assurance processes and include this in their annual report. The Department of Energy Strategy and Net Zero reports this annually - page 91 of DESNZ Annual report.
Senior leaders should ensure there is clear messaging and standards on quality assurance, through guidance, training, and regular updates. Senior leaders can demonstrate the importance of quality assurance through long term initiatives, like setting up and embedding quality assurance processes within the team and creating roles and teams to support quality assurance. Senior leaders should also regularly talk about quality with their teams and highlight quality successes.
HM Revenue and Customs have developed a Quality Champions network, made up of analysts from across the department. The network discuss quality assurance initiatives, quality issues and how they were resolved and shares wider best practice.
As part of a strong quality assurance culture, senior leaders should empower all those in the analytical process to identify any risks to the quality of the work, ensure people at any level can raise any quality concerns, and be able to discuss and constructively challenge each other if they feel those standards are not being met. To underpin this, senior leaders should ensure teams have a common understanding of the quality standards required for their work.
Creating transparency at all levels can help embed a culture of quality assurance. This includes peer review, open source of code (where possible), external publication of models or methods and publication of the register of Business Critical Models.
Senior leaders should also develop processes so teams can report when things go wrong, be open and honest when issues occur, carry out reviews to understand the failures in the assurance process and share the lessons learnt across the analytical community.
When the Department for Education made an error producing the schools national funding formula allocations for 2024-25, they ran a detailed internal review to understand what went wrong and why it was not detected by the QA process. The department also commissioned and published an external, independent review to assess the error and put forward recommendations. The independent review praised the team for its open learning culture and a culture of taking responsibility for mistakes.
4.2 Capacity and capability
Senior leaders should create the conditions in which quality assurance processes can operate effectively, by ensuring staff have time built-in for quality assurance and documentation, can draw on expertise and experience, and have the access to the tools and data they need.
4.2.1 Capacity
There is a risk that work, and time pressures mean teams cut corners on quality or mistakes are made as analysis is rushed. Senior leaders can mitigate this through strong prioritisation, supporting teams to push back on lower value work or by making tough choices on team wide priorities. Through this prioritisation, senior leaders can emphasise the importance of quality. Team leaders can support quality assurance, by ensuring it is considered throughout the lifecycle of a project and not simply considered at the end, and making time for adequate quality assurance even when timescales are tight.
If time constraints mean insufficient assurance has taken place, senior leaders should ensure this is explicitly acknowledged and reported. This should be reported via an assurance statement that sets out the known limitations and conditions associated with the analysis.
Where analysis requires peer review, this should be carried out by independent, skilled and competent individuals or groups. However, it can be difficult to identify available experts who are able to provide a review. There are several approaches to support and embed independent review, such as setting up specific teams to review and audit a sample of analytical projects, developing assurance networks of analysts who can provide reviews when needed, partnering with another Department, or procuring an an independant review from an independant source such as the Government Actuary’s Department (GAD) (https://www.gov.uk/government/publications/gad-services/government-actuarys-department-services), an academic institution or contractor. Team leaders can support this by making time for analysts to carry out peer reviews and ensuring analysts are clear that supporting peer reviews is part of their role.
HMRC has a small analytical team which independently reviews analysis from across the department, including a sample HMRC’s business-critical models. The reviews provide assurance for high profile analysis and support the sharing of best practice.
4.2.2 Capability
There is a risk that errors occur because of a lack of skills or experience. Senior leaders can identify common skills gaps, creating training or mentoring to help fill gaps in analysts’ knowledge. Processes to support knowledge sharing, innovation and dissemination of best practice will all help develop capability. Rolling out training on departmental assurance processes can also mitigate this risk.
There are various cross-government resources designed to provide support and guidance for commissioners and users of analysis. For example, the Analysis Function’s Advice for policy professionals using statistics and analysis aims to help policy professionals to work effectively with analysts and analysis. It introduces some important statistical ideas and concepts to help policy professionals ask the right questions when working with statistical evidence.
The Department for Energy Security and Net Zero are building assurance capacity with a programme of quality assurance Colleges. The Colleges run regular virtual sessions, open to colleagues across Government and Partner Organisations.
These sessions include an interactive activity looking at a purposefully sabotaged model, used to introduce and familiarise colleagues with the quality assurance logbook and the departmental system of actively monitoring models through these logbooks.
College participants also join the Modelling Integrity Network, as potential Assuring Analysts where these cannot be found by Lead Analysts in their own policy areas.
4.3 Quality assurance champions
In an organisations with large analytical community, it is good practice for the senior leaders to appoint a quality assurance champion. This person can share best practice in implementation of quality assurancem and provide advice on issues such as proportionality and communication of assurance.
4.4 Tools
There is a risk that technology or analytical tools are out of date meaning analysts cannot follow best practice or must spend time fixing processing issues instead of focussing on quality. Senior leaders can support teams by making funding available for new tools or improving existing tools, working with digital/IT teams to escalate issues and gathering cross team issues.
4.5 Data
There is a risk that data quality and data understanding cause quality issues with analysis. Senior leaders can escalate data quality issues with wider data teams, and champion and oversee changes to improve data quality.
4.6 Governance and control
Governance supports a strong quality assurance culture, by overseeing the management and assurance of analysis. The Analysis Function Standard sets out the requirements for a governance framework for analysis. Each organisation should have a defined and established approach to assurance, which should be applied proportionately to the risk and value of the activity and integrated with the organisation’s overall assurance framework.
Project level governance can provide oversight over a particular model or work area, allowing the Approver to ensure the analysis is fit for purpose. For example, formally agreeing assumptions will reduce the need for reworking the analysis providing more time for assurance. Projects governance can also fit within wider programme level governance.
Analytical governance boards for new, high-profile or complex pieces of analysis, can allow senior analytical leaders and experts to provide oversight and challenge of analysis and ensure best practice is followed. These groups are multi-disciplinary and can cover a range of analytical approaches based on their expertise and experience. This can help ensure that innovations and new approaches are disseminated across teams, and standards are applied equally across similar work.
4.7 Transparency
Transparency at all levels can help embed a culture of quality assurance. For example peer review, sharing lessons learnt, and making analysis open (where possible), all contribute to an open culture of high quality work.
4.7.1 Externally commissioned analysis
When commissioning analysis externally, either qualitative or quantitative research, or the production of models, then the accountability for the quality sits with the commissioning department. It is the commissioning analyst’s role to ensure that quality standards are clear and being met and documented to the commissioning departments requirements. Further information can be found in the commissioning chapter.
For Arm’s Length Bodies (ALBs), the commissioner of analysis is accountable for ensuring the requirements set out in the AQuA Book are met.
The guidance mentioned above also applies to Arms Length Bodies. They may set out in a framework document how their Accounting Officer will demonstrate compliance with Annex 4.2 of Managing Public Money and the Analysis Function Standard
User groups for types of analysis can also provide assurance by setting standards and ensuring projects follow topic specific or specialised guidance, such as coding best practice.
When working with third parties, the commissioner shall ensure it is clear which role or roles in which stages of the analytical lifecycle that third party is responsible for. See Roles and Responsibilities for details on the roles.
For example, the third party may only undertake the analyst role in the analysis phase or they may undertake the analyst, assurer and approver roles in all stages of the lifecycle.