3 Proportionality
This version of the AQuA Book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs.
The draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised.
All analysis shall be assured.
The assurer and the analyst shall be independent. The degree of separation depends on many factors including the importance of the output, and the size and complexity of the analysis. This does not mean that the analyst should not undertake assurance, rather that there shall also be some formal independent assurance.
The assurance should be proportionate to the potential effect it will have and the size and complexity of the analysis. The level of assurance should be guided by a structured assessment of the business risks.
3.1 Factors for determining appropriate assurance
While there is a need to be confident in the analysis, it is not necessary to spend months assuring simple analysis that will have a minor influence on a decision. The level of analysis should be appropriate (proportionate) to the analysis.
Table 3-1 provides a list of factors that should be considered when determining what level of assurance is appropriate.
Factor | Comments |
---|---|
Business criticality | Different issues will affect how business critical the analysis is. For example, its financial, legal, operational, political and reputational effects. |
Relevance of the analysis to the decision-making process | When analysis forms only one component of a broad evidence base, less assurance is required for that specific analysis than if the decision is heavily dependent on the analysis alone. Significant assurance is still likely to be required for the whole evidence base. |
Type and complexity of analysis | Highly complex analysis requires more effort to assure. The nature of that analysis may also require the engagement of appropriate subject matter experts. |
Novelty of approach | A previously untried method requires more assurance. Confidence will grow as the technique is repeatedly tested. |
Reusing or repurposing existing work | Reusing work may require validation and verification to confirm that original approach. For example, confirming the original method and assumptions data are still appropriate for the new requirement. |
Level of precision required in outputs | Lower precision analysis often uses simplified assumptions, models and data. The assurance approach is the same but will take less time than more precise analysis. |
Amount of resource available for the analysis and assurance | The value for money of any additional assurance must be balanced alongside the benefits and risk appetite that exists. Analysis that is used for many purposes (for example, population projections) may require greater levels of quality assurance than might be suggested by any of the individual decisions they support. |
Longevity of the analysis | Ongoing analysis will require robust change control and regular review. |
Effects on the public | Analysis which will have a significant effect on the public may require more assurance. |
Repeat runs for the same analysis | Assurance should concentrate on version control and the assurance of data and parameters for each run. |
You can read more on the Data Quality Hub’s Quality Questions and Red Flags.
Figure 3-1 shows some assurance techniques that might be considered for different levels of analysis complexity and business risk. The need for more assurance interventions increases with the complexity of the analysis and the business risk associated with it.
The interventions in figure 3-1 must not be viewed in isolation. The interventions should build on each other. For example, some complex and risky analysis that would benefit from an external review should also use interventions closer to the axes, such as version control and analyst-led testing.
The total elimination of risk will never be achieved, so a balance needs to be found that reduces the overall business risk to an acceptable level. The diagram indicates a few practical assurance techniques. The analayst should consider consider and implement other techniques where necessary.
3.2 Structured assessment of business risk and complexity
A structured approach should be taken to determine what assurance is needed when reviewing business risks. Business risk should be viewed as the combination of the potential effect of analytical errors and the likelihood of errors occurring. In situations where the risk is high, it is more important to reduce likelihood of errors than the level of the effect.
This can be visualised by considering the situation as a risk matrix (table 3-2). The effect the analysis will have is usually beyond the control of the analyst to change, so there will be few options to lessen the effect of a risk. However, there will usually be treatments (or mitigations) involving additional assurance measures that will allow the assessed business risk to become less likely to occur.
Likelihood of errors occurring | |||||||
---|---|---|---|---|---|---|---|
Effect from errors occurring | Highly unlikely | Unlikely | Realistic possibility | Likely or probably | Highly likely | ||
Critical | Medium | Medium | High | High | High | ||
Severe | Low | Medium | Medium | High | High | ||
Major | Low | Medium | Medium | Medium | High | ||
Moderate | Very Low | Low | Medium | Medium | Medium | ||
Minor | Very Low | Very Low | Low | Low | Medium |
Table 3-3 shows appropriate responses to a risk assessment. Where business risk is high, appropriate mitigations must be considered to reduce the probability of errors occurring. This will depend on the mitigations already in place and on the complexity of the analysis (figure 3-1).
For a situation where simple analysis is being employed, a review by an appropriate expert may be sufficient as the additional mitigation. However, for complex analysis that is already employing a wide range of internal assurance measures, options such as external peer review may be necessary.
In cases with significant time and resource constraints, it may not be possible to do as much assurance as desirable. In these situations, addressing the areas of greatest risk should take priority. You should inform the commissioner about how you have addressed the risks and any remaining risks together with appropriate caveats.
Assessed risk | Mitigations to consider |
---|---|
High | High risk should not be tolerated. New assurance measures must be considered to treat the likelihood of errors occurring. If treatment isn’t an option, consideration must be given to terminating or transferring the risk. If it remains necessary to tolerate the risk the commissioner should to fully understand this risk. |
Medium | Medium risk should not be tolerated without the agreement of the commissioner. New assurance measures should be put in place to treat the likelihood of errors occurring. Continue with planned or existing mitigations. |
Low | Low risk can be tolerated. Continue with existing or planned mitigations. New treatments may also be considered. |
Very Low | Very low risk can be tolerated. Continue with existing or planned mitigations. |
You can read more on risk management in the Orange Book which covers risk management principles and risk control frameworks.
3.3 Externally commissioned work
Proportionate assurance of externally commissioned work is just as important as for internally produced analysis. The commissioner should be fully informed of the business risk associated with the work. This should be provided by an appropriate mix of documented risk assessments provided as part of the work and by joint risk assessments planned throughout the life of the project. For commissioned work the options for mitigation will be similar to those for internal analysis.
The difference will be in ensuring the assessment of risks and the applied mitigations are fully understood by the commissioner.
3.4 Black-box models and business risk
Increasingly analysis may be underpinned by Artificial intelligence (AI) or other forms of black-box models. With these models the need to understand business risk remains and the same structured approach to assessing business risk should be taken. The challenges in providing this assessment will be in ensuring the transparency of the analysis, availability of a suitable mix of experts and developing understanding of what mitigations are possible.
To note the Generative AI Framework for HMG has assurance highlighted in “Principle 10”.