Success likelihood index method

Success Likelihood Index Method (SLIM) is a technique used in the field of Human reliability Assessment (HRA), for the purposes of evaluating the probability of a human error occurring throughout the completion of a specific task. From such analyses measures can then be taken to reduce the likelihood of errors occurring within a system and therefore lead to an improvement in the overall levels of safety. There exist three primary reasons for conducting an HRA; error identification, error quantification and error reduction. As there exist a number of techniques used for such purposes, they can be split into one of two classifications; first generation techniques and second generation techniques. First generation techniques work on the basis of the simple dichotomy of ‘fits/doesn’t fit’ in the matching of the error situation in context with related error identification and quantification and second generation techniques are more theory based in their assessment and quantification of errors. ‘HRA techniques have been utilised in a range of industries including healthcare, engineering, nuclear, transportation and business sector; each technique has varying uses within different disciplines.

SLIM is a decision-analytic approach to HRA which uses expert judgement to quantify Performance Shaping Factors (PSFs); factors concerning the individuals, environment or task, which have the potential to either positively or negatively affect performance e.g. available task time. Such factors are used to derive a Success Likelihood Index (SLI), a form of preference index, which is calibrated against existing data to derive a final Human Error Probability (HEP). The PSF's which require to be considered are chosen by experts and are namely those factors which are regarded as most significant in relation to the context in question.

The technique consists of two modules: MAUD (multi-attribute utility decomposition) which scales the relative success likelihood in performing a range of tasks, given the PSFs probable to affect human performance; and SARAH (Systematic Approach to the Reliability Assessment of Humans) which calibrates these success scores with tasks with known HEP values, to provide an overall figure.

Background

SLIM was developed by Embrey et al. [1] for use within the US nuclear industry. By use of this method, relative success likelihoods are established for a range of tasks, and then calibrated using a logarithmic transformation.

SLIM methodology

The SLIM methodology breaks down into ten steps of which steps 1-7 are involved in SLIM-MAUD and 8-10 are SLIM-SARAH.

Definition of situations and subsets
Upon selection of a relevant panel of experts who will carry out the assessment, these individuals are provided with as fully detailed a task description as possible with regards to the individual designated to perform each task and further factors which are likely to influence the success of each of these. An in depth description is a critical aspect of the procedure in order to ensure that all members of the assessing group share a common understanding of the given task. This may be further advanced through a group discussion prior to the commencement of the panel session to ascertain of consensus. Following this discussion, the tasks under consideration are then classified into a number of groupings depending upon the homogeneity of the PSF's that have an effect on each. Subsets are thus defined by those tasks which have in common specific PSFs and also by their weighting within a certain sub-group; this weighting is only an approximation at this stage of the process.
Elicitation of PSFs
Random sets of 3 tasks are presented to experts from which they are required to compare one against the other two and subsequently identify an aspect in which the highlighted task differs from the remaining two; this dissimilarity should be a characteristic which affects the probability of successful task completion. The experts are then asked to highlight the low and high end-points of the identified PSF i.e. the optimality of the PSF in the context of the given task. For example, the PSF may be Time Pressure and therefore the end points of the scale would perhaps be “High level of pressure” to “Low level of pressure”. Other possible PSFs may be stress levels, task complexity or degree of teamwork required. The aim of this stage is to identify those PSF's which are most prevalent in affecting the tasks as opposed to eliciting all the possible influencing factors.
Rating the tasks on the PSFs
The endpoints of each individual PSF, as identified by the expert, are then assigned the values 1 and 9 on a linear scale. Using this scale, the expert is required to assign to each task a rating, between the two end points, which accurately reflects, using their judgement, the conditions occurring in the task in question. It is optimal to consider each factor in turn so that the judgements made are independent from the influence of other factors which otherwise may affect opinion.
Ideal point elicitation and scaling calculations
The “ideal” rating for each PSF is then selected on the scale constructed. The ideal is the point at which the PSF least degrades performance – for instance both low and high time pressure may contribute to increasing the chance of failure. The MAUD software then rescales all other ratings made on the scale in terms of their distance from this ideal point, with the closest being assigned as a 1 and the furthest from this point as a 0. This is done for all PSF's until the experts are agreed that the list of PSF's is exhausted and that all the scale positions identified are correctly positioned.
Independence checks
Using the figures which represent the relative importance of each task and their rating on the relevant scale, these are multiplied to produce a Success Likelihood Index (SLI) figure for each task. To improve the validity of the process it is necessary to confirm that each of the scales in use are independent to ensure no overlap or double counting in the overall calculation of the index.

To help carry out this validation task, MAUD software checks for correlations between the experts’ scoring on the different scales; if the scale ratings indicate a high correlation, the experts are consulted to reveal whether they agree in their meanings of the ratings on the two scales which are showing similarities. If this situation occurs, the experts are asked to define a new scale which will be a combination of the meaning of the two individually correlated scales. If the correlation is not significant then the scales are treated as independent; in this case, the concerned facilitator is required to make an informed decision as to whether or not the PSFs showing similarities are actually similar and should therefore ensure that a strong justification is explainable for the final decision.
Weighting procedure
This stage of the process concentrates on eliciting the emphasis required to be weighted to each of the PSFs in terms of the influence on the success of a task. This is done by enquiring, with the experts, the likelihood of success between pairs of tasks while considering two previously identified PSFs. By noting where the experts’ opinion is changed, the weighting of the effect of each PSF on the task success can thus be inferred. To enhance the accuracy of the outcome, this stage should be carried out in an iterative manner.
Calculation of the SLI
The Success Likelihood Index for each task is deduced using the following formula:
${\mbox{SLI}}_{j}=\sum _{i}R_{ij}W_{i}$
Where
- SLI_j is the SLI for task j
- W_i is the importance weight for the ith PSF
- R_ij is the scaled rating of task j on the ith PSF
- x represents the number of PSFs considered.
These SLIs are estimates of the probability with which different types of error may occur.
Conversion of SLIs to probabilities
The SLIs previously calculated require to be transformed to HEPs as they are only relative measures of the likelihood of success of each of the considered tasks.

The relationship
$\log P=a\,{\mbox{SLI}}+b$

is assumed to exist between SLIs and HEPs. P is the probability of success and a and b are constants; a and b are calculated from the SLIs of two tasks where the HEP has already been established.
Uncertainty bound analysis
Uncertainty bounds can be estimated using expert judgement methods such as Absolute probability judgement (APJ).
Use of SLIM-SARAH for cost-effectiveness analyses
As SLIM evaluates HEPs as a function of the PSFs, considered to be the major drivers in human reliability, it is possible to perform sensitivity analysis by modifying the scores of the PSFs. By considering the PSFs which may be altered, the degree to which they can be changed and the importance of the PSFs, it is possible to conduct a cost-benefit analysis to determine how worthwhile suggested improvements may be i.e. what-if analysis, the optimal means by which the calculated HEPs can be reduced.

Worked example

The following example provides a good illustration of how the SLIM methodology is used in practice in the field of HRA.

Context

In this context an operator is responsible for the task of de-coupling a filling hose from a chemical road tanker. There exists the possibility that the operator may forget to close a valve located upstream of the filling hose, which is a crucial part of the procedure; if overlooked, this could result in adverse consequences, of greater effect to the operator in control. The primary human error of concern in this situation is ‘failure to close V0204 prior to decoupling filling hose’. The decoupling operation required to be conducted is a fairly easy task to carry out and does not require to be completed in conjunction with any further tasks; therefore is failure occurs it will have a catastrophic impact as opposed to displaying effects in a gradual manner.

Required inputs

This technique also requires an ‘expert panel’ to carry out the HRA; the panel would be made up of for example two operators possessing approximately 10 years experience of the system, a human factors analyst and a reliability analyst who has knowledge of the system and possesses a degree of experience of operation. The panel of experts is requested to determine a set of PSFs which are applicable to the task in question within the context of the wider system; of these, the experts are then required to propose those PSFs, of the identified, which are the most important in the circumstances of the scenario. For this example, it is assumed that the panel put forth 5 main PSFs for consideration, which are believed to have the greatest effect on human performance of the task: training, procedures, feedback, perceived risk and time pressure.

Method

PSF rating

Considering the situation within the context of the task under assessment, the panel are asked to provide further possible human errors which may occur that have the potential of affecting performance e.g. mis-setting or ignoring an alarm. For each of these, the experts are required to establish the degree to which each is either optimal or sub-optimal for the task under assessment, working on a scale from 1 to 9, with the latter being the optimal rating. For the 3 human errors which have been identified, the ratings decided for each are provided below:

Errors	Training	Procedures	Feedback	Perceived Risk	Time
VO2O4 open	6	5	2	9	6
Alarm mis-set	5	3	2	7	4
Alarm ignored	4	5	7	7	2

PSF weighting

Were each of the identified human errors of equal importance, it would then be possible to obtain the summation of each row of ratings and come to the conclusion that the row with the lowest total rating- in this case it would be alarm mis-set- was the most probable to occur. In this context, as is most often the case, the experts are in agreement that the PSFs given above are not of equal weighting. Perceived risk and feedback are deemed to be of greatest importance, twice as much as training and procedures, which these two are considered to be one and a half times more important than the factor of time. The time factor is of considered of minimal importance in this context as the task is routine and is therefore not limited by time.

The importance of each factor can be observed through the allocated weighting, as provided below. Note that they have been normalised to sum to unity.

PSF	Importance
Perceived Risk	0.3
Feedback	0.3
Training	0.15
Procedures	0.15
Time	0.10
Sum	1.0

Using the figures for the scaled weighting of the PSFs and the weighting of their importance, it is now possible to calculate the Success Likelihood Index (SLI) for the task under assessment.

Weighting	PSF's	V0204	Alarm Mis-set	Alarm Ignored
0.3	Feedback	0.60	0.60	2.10
0.3	Perceived Risk	2.70	2.10	2.10
0.15	Training	0.90	0.75	0.60
0.15	Procedures	0.75	0.45	0.75
0.10	Time	0.60	0.40	0.20
	SLI (total)	5.55	4.30	5.75

From the results of the calculations, as the SLI for ‘alarm mis-set’ is the lowest, this suggests that this is the most probable error to occur throughout the completion of the task.

However these SLI figures are not yet in the form of probabilities; they are only indications as to the likelihood by which the various errors may occur. The SLIs determine the order in which the errors are most probable to occur; they do not delineate the absolute probabilities of the PSFs. To convert the SLIs to HEPs, the SLI figures require to first be standardised; this can be done using the following formulation.

\log _{10}{\mbox{HEP}}=a\,{\mbox{SLI}}+b

Result

If the two tasks for which the HEPs are known are incorporated in the task set which is undergoing quantification then the equation parameters can be determined by using the method of simultaneous equations; using the result of this the unknown HEP values can thus be quantified. In the example provided, were two additional tasks to be assessed e.g. A and B, which had HEP values of 0.5 and 10 -4 respectively and SLIs respectively of 4.00 and 6.00, respectively, then the formulation would be:

\log {\mbox{HEP}}=-1.85\,{\mbox{SLI}}+7.1

The final HEP values would thus be determined as

V0204 = 0.0007

Alarm mis-set = 0.14

Alarm ignored = 0.0003

References

[1] EMBREY, D.E., Humphreys, P.C., rRosa, E.A., Kirwan, B. & Rea, K., SLIM-MAUD: An approach to assessing human error probabilities using structured expert judgement. NUREG/CR-3518. 1984, US Nuclear Regulatory Commission: Washington DC.

[2] Humphreys, P. (1995) Human Reliability Assessor's Guide. Human Factors in Reliability Group.

[3] Kirwan, B. (1994). A Practical Guide to Human Reliability Assessment. CPC Press.

[4] Corlett, E.N., & Wilson, J.R. (1995). Evaluation of Human Work: A Practical Ergonomics Methodology. Taylor & Francis.