Organizations that test products and certify them themselves face requirements of two different kinds. The adoption of ISO 9001 [1] is voluntary and defines a general quality management system. The requirements that apply specifically to accredited testing laboratories and to product certification bodies are ISO/IEC 17025 [2] and ISO/IEC 17065 [3]; they set the conditions for accreditation, including the treatment of risks and, for certification bodies, of impartiality. Each standard asks for risk-based thinking, the general risk management process is described in ISO 31000 [4], but the risks differ: business risks, risks to the validity of results and risks to impartiality. A single register therefore has to rank risks of different nature on one scale. The ICH Q9(R1) guideline [5] is a sector-specific document on quality risk management in the pharmaceutical industry; it does not govern testing and certification organizations in general. It is cited only because it states explicitly that subjectivity in risk assessment must be managed and minimized, a concern that applies equally to the scoring of risks in other sectors.

In practice, a risk is still often scored as the product of ordinal probability and consequence scores. Such products have coarse resolution and make different risks look equal [6]. Weighting by risk type and process criticality moves part of the subjectivity from scores to weights [7]. In a recent robustness study of such a weighted index on a register of 19 risks [8], the author found that the ranking was stable under moderate disagreement about weights (mean Spearman correlation 0.95 for ±20% weight deviation, against 0.45 for random weights), but was affected more by errors in the scores: when 20% of scores were shifted by one point, the correlation fell to 0.88. The study concluded that effort should be directed to the quality of the scores, for example, calibrated scales, independent assessors and data from the nonconformity log. The data came from one organization.

This paper builds on that conclusion. Its question is narrow: where could AI support the production of better scores and earlier signals, under what conditions, and how should such support be validated before it is relied upon? The paper reports no AI experiment. It adds one computation, a simulation on the author’s register that shows how accurate score support would have to be to change the ranking noticeably, followed by a validation protocol. The functions proposed are untested.

The baseline is the weighted index R = P × C × Wt × Wp [7], where P and C are probability and consequence scores, Wt is the weight of the risk type and Wp the weight of the criticality of the process in which the risk arises. The register contains laboratory technical risks, impartiality risks of the certification body and management risks. The index separates risks that the unweighted product P × C scores equally and, in the cited study, classified nine of 19 risks as high [8]. In the integrated system, the register is connected to corrective actions and to the KPIs of the system [17]. The baseline is deliberately kept unchanged here; AI would act on its inputs, not on its structure.

Figure 1 shows the proposed arrangement. In all three functions, the output is advisory: a human assessor decides the final scores and actions, and the AI output is validated and logged.

Figure 1. Proposed advisory use of AI in risk prioritization (conceptual; not implemented).

Function A:evidence for probability and consequence scores. Probability and consequence scores are currently assigned by experts. A model trained on the organization’s nonconformity and audit records could show, for each risk, how often related events occurred and how serious they were, so that the assessor works with evidence and a documented range rather than an impression. This addresses directly the instability shown in [8]. It requires that records be coded to the same risk categories as the register; otherwise, the evidence cannot be linked to risks.

Function B:early-warning signals from KPI trends. The KPI system of the integrated management system includes indicators that are calculated from process data (for example metrological traceability, report issue time and share of decisions made on time). Anomaly detection on these time series, similar in idea to patient-based real-time quality control [10], could raise a signal before a quarterly or annual review. The signal would prompt a reassessment of the related risk; it would not change a score automatically.

Function C:mapping of changed requirements to risks. The register of the studied organization contains a risk of nonconformity with accreditation requirements caused by changes in the normative base. Natural language processing could compare new or revised standards and regulations with the organization’s documents and propose which processes and risks are affected, which is the kind of document and compliance support described in [9]. A person would check every proposed link.

Functions not proposed. AI is not proposed for making certification decisions or for judging impartiality of persons. These decisions require competent persons who can be held accountable, and the risk of misjudgment from a model that cannot explain itself [16] would itself be a threat to the impartiality that the integrated system must protect [17].

Four conditions follow from the literature and from the standards cited above. First, data: the barriers named in [9] apply, and records must be complete, coded to risk categories and kept separately for the laboratory and the certification body, because a model trained on pooled data could carry information across the boundary between the two activities. Second, validation: an AI function used in the system is an information system whose fitness has to be demonstrated before use [2]. Third, risk management of the AI function itself, using the approach of [13] and [14], including logging of inputs and outputs and a defined fallback to the manual procedure. Fourth, explainability: an assessor must be able to see why the model suggests a value [16].

Design. The question can be partly answered without any AI model by asking how much the ranking improves if the share of wrong scores falls. The register of 19 risks of the earlier study [8] was taken as the reference. Following the scenario of score errors in that study, each probability score P and each consequence score C was shifted down by one point with probability q and up by one point with probability q (within the range 1 - 5), so that a nominal share 2q of the scores is shifted in a run. Scores were limited to the range 1 - 5: a shift that would leave the range was not applied, so the score stayed unchanged. The realized share of altered scores is therefore lower than 2q (Table 1); the values of q in this paper are nominal shift probabilities, not realized error rates. For each q, 20,000 runs with a fixed random seed (7) were made with the weights unchanged. In each run, the index R was recalculated and compared with the reference ranking by the Spearman rank correlation, the probability that the same four risks stay the highest, and the share of risks that change their level (low, medium, high, critical), with the cutoffs low R < 3, medium 3 ≤ R < 9, high 9 ≤ R ≤ 36 and critical R > 36. Ties were handled as follows: the Spearman correlation uses average ranks, and the four leading risks are selected by a stable sort, so that equal values keep the register order.

Table 1. Effect of the share of wrong scores on the ranking (register of 19 risks, 20,000 runs per row, seed 7).

Check. The simulation reproduces the earlier results: mean rank correlation 0.88 at q = 0.10 and 0.74 at q = 0.25 [8].

Reproducibility. The 19-risk register (P, C, Wt, Wp; Table A1), the cutoffs, the tie rules, the seed and the complete Python script (Appendix B; Python 3, NumPy 2.5.3, SciPy 1.18.1) are given in the appendices, so the results of Table 1 can be regenerated independently. With seed 7 the script reproduces every value of Table 1; the last column of Table 1 gives the realized share of altered scores (the mean over the 20,000 runs).

Results. Halving the error rate from q = 0.10 to q = 0.05 raised the mean correlation from 0.88 to 0.94, the retention of the four leading risks from 37% to 62%, and cut the share of risks that change level from 12.7% to 6.5%. In terms of realized altered scores, these two rows correspond to 17.6% and 8.8% of all scores. Reaching a mean correlation of 0.97 needs q around 0.025. The gain is therefore not small, but it requires score errors to be reduced by half or more, not by a few percent. The last two rows of Table 1 give a second result: improving only the eight laboratory risks and improving only the eleven impartiality and management risks give the same mean correlation (0.91), but the second keeps the four leading risks much more often (54% against 43%), because three of the four leading risks in the register are impartiality or management risks.

Interpretation and limits. The result is a requirement, not a performance: it says how accurate AI-supported scores would have to be, and it does not say that any AI model can reach this accuracy. The errors were modelled as independent random shifts of one point around the register scores, which were treated as correct; real errors may be systematic, correlated between risks, or larger than one point, and the register scores themselves are expert judgments. The second result suggests a practical problem: data-driven evidence is most likely to exist for frequent laboratory events, while the leading risks of the register are impartiality and management risks for which records are probably scarce. If this is true, Function A would help most where it is least likely to have data, and the benefit of AI support in this register would be smaller than the first rows of Table 1 suggest. This is an inference from the structure of the register and has to be tested on real records.

The author’s previous study [8] used rank correlation with a baseline ranking and the share of simulation runs in which a risk stays at the high level. The same measures can be used to validate AI support. The protocol is retrospective and uses the organization’s register and its records of events (nonconformities, complaints, impartiality incidents, audit findings and results of proficiency testing). Its steps are: 1) fix a scoring date and build the baseline ranking from expert scores alone, using only information available on that date (Section 7.1); 2) produce a second ranking in which the expert scores are replaced, or supplemented, by the AI evidence of Function A, again using only information available on that date; 3) build the reference ranking from the events recorded after the scoring date (Section 7.2) and compare both rankings with it, not with each other; 4) report the measures of Section 7.4.

The paper has important limitations. No AI method was built and no AI result is reported; the simulation of Section 6 evaluates the ranking, not a model, and rests on assumed random errors and on the register scores taken as correct. The baseline model has been tested in one organization with a register of 19 risks, which is small for statistical learning, and the number of past events available for training is probably small as well. The literature reviewed comes from other fields, construction and clinical laboratories, and its transfer to testing and certification organizations is an assumption. The validation protocol has not been run, and the suggestion that records are scarce for the leading risks has not been checked on real data. The reference ranking built from later events is affected by preventive actions that suppress events (Section 7.3), and this effect can be reduced but not removed. The values of q are nominal shift probabilities; the realized share of altered scores is lower because of the score limits.

The author’s previous robustness study suggests that the weakest point of the weighted risk index is the quality of expert scores. Three advisory AI functions are proposed to address it: evidence for scores, early-warning from KPI trends, and mapping of changed requirements to risks, all with a human assessor retaining the decision. The conditions for use are data quality, validation, risk management of the AI function and explainability. A simulation on the register shows that halving the nominal share of shifted scores would raise the rank correlation from 0.88 to 0.94 and the retention of the four leading risks from 37% to 62%, and that the benefit depends on which risks are improved. A retrospective validation protocol with pre-specified criteria is proposed. The next step is to collect coded records and to run the protocol on them; until that is done, the proposals remain hypotheses.

Table A1 lists the 19 risks of the register of [8] with their scores and weights. The index is R = P × C × Wt × Wp; risk levels: low R < 3, medium 3 ≤ R < 9, high 9 ≤ R ≤ 36, critical R > 36. Risks 1 - 8 are laboratory risks, 9 - 15 impartiality risks and 16 - 19 management risks.

Table A1. Baseline register (19 risks).

The script below (file ica_sim_reproduce.py) uses NumPy and SciPy and the seed 7 for every row of Table 1. Run as “python ica_sim_reproduce.py”.

"""Reproduction script for Table 1 (Section 6): effect of score errors on the ranking

of the 19-risk register. Python 3, numpy, scipy. Fixed seed 7 for every row."""

import numpy as np

from scipy.stats import spearmanr

# Register: P, C, Wt, Wp for risks 1-19 (1-8 laboratory, 9-15 impartiality, 16-19 management)

REG = [(2,3,1.5,2.0),(2,3,1.2,1.5),(2,2,1.7,1.0),(1,3,1.5,1.5),(2,2,1.7,1.0),(1,3,1.5,1.5),

(1,2,1.5,0.8),(1,3,1.2,1.5),(2,3,2.0,2.0),(2,3,2.0,2.0),(1,3,2.0,2.0),(1,3,2.0,2.0),

(2,2,2.0,1.0),(1,2,2.0,1.5),(1,3,2.0,1.5),(1,3,1.2,1.0),(2,3,1.0,1.5),(2,3,1.7,1.5),

(2,2,1.0,1.0)]

P, C, Wt, Wp = np.array(REG, float).T

n = len(P)

BASE = P * C * Wt * Wp # reference index R = P*C*Wt*Wp

def level(R): # 0 low (<3), 1 medium [3,9), 2 high [9,36], 3 critical (>36)

return np.where(R < 3, 0, np.where(R < 9, 1, np.where(R <= 36, 2, 3)))

def top4(R): # ties broken by register order (stable sort)

return set(np.argsort(-R, kind="stable")[:4])

BASE_LEVEL, BASE_TOP4 = level(BASE), top4(BASE)

def run(q, N=20000, seed=7):

qv = np.asarray(q, float) * np.ones(n) # per-risk shift probability (one direction)

rng = np.random.default_rng(seed)

rho, keep, chg, altered = [], [], [], []

for _ in range(N):

def shift(s):

u = rng.random(n); d = np.zeros(n)

d[u < qv] = -1 # down with probability q

d[(u >= qv) & (u < 2 * qv)] = 1 # up with probability q

return np.clip(s + d, 1, 5) # scores limited to 1-5

p, c = shift(P), shift(C)

R = p * c * Wt * Wp

rho.append(spearmanr(BASE, R).correlation) # ties get average ranks

keep.append(top4(R) == BASE_TOP4)

chg.append((level(R) != BASE_LEVEL).mean())

altered.append(((p != P).sum() + (c != C).sum()) / (2 * n))

rho = np.array(rho)

return (round(rho.mean(), 2), round(float(np.percentile(rho, 5)), 2),

round(100 * np.mean(keep)), round(100 * np.mean(chg), 1), round(100 * np.mean(altered), 1))

if __name__ == "__main__":

print("baseline R:", np.round(BASE, 1))

print("baseline top-4 (0-based index):", sorted(BASE_TOP4), "levels:", BASE_LEVEL.tolist())

print("q, mean rho, 5th pct rho, top-4 kept %, level change %, realized altered scores %")

for q in (0.25, 0.15, 0.10, 0.075, 0.05, 0.025, 0.0125):

print(q, run(q))

lab = [i < 8 for i in range(n)]

print("lab q=0.05, others 0.10:", run([0.05 if l else 0.10 for l in lab]))

print("lab q=0.025, others 0.10:", run([0.025 if l else 0.10 for l in lab]))

print("non-lab q=0.05, lab 0.10:", run([0.10 if l else 0.05 for l in lab]))