Zoekresultaten

By accident or design

Critical review of safety performance metrics

Various tools for safety performance measurement have been introduced in order to fulfil the need for safety monitoring in organisations, which is tightly related to their overall performance and achievement of their business goals. Such tools include accident rates, benchmarking, safety culture and climate assessments, cost-effectiveness studies, etc. The current work reviews the most representative methods for safety performance evaluation that have been suggested and applied by a variety of organisations, safety authorities and agencies. This paper discusses several viewpoints of the applicability, feasibility and appropriateness of such tools, based on the viewpoints of managers and safety experts involved in a relevant research that was conducted in a large aviation organisation. The extensive literature cited, the discussion topics, along with the conclusions and recommendations derived, might be considered by any organisation that seeks a realistic safety performance assessment and establishment of effective measurement tools.

How does aviation industry measure safety performance?

In this paper we present a review of existing aviation safety metrics and we lay the foundation for our four-years research project entitled “Measuring Safety in Aviation – Developing Metrics for Safety Management Systems”. We reviewed state-of-the-art literature, relevant standards and regulations, and industry practice. We identified that the long-established view on safety as absence of losses has limited the measurement of safety performance to indicators of adverse events (e.g., accident and incident rates). However, taking into account the sparsity of incidents and accidents compared to the amount of aviation operations, and the recent shift from compliance to performance based approach to safety management, the exclusive use of outcomes metrics does not suffice to further improve safety and establish a proactive monitoring of safety performance. Although the academia and aviation industry have recognized the need to use activity indicators for evaluating how safety management processes perform, and various process metrics have been developed, those have not yet become part of safety performance assessment. This is partly attributed to the lack of empirical evidence about the relation between safety proxies and safety outcomes, and the diversity of safety models used to depict safety management processes (i.e. root-cause, epidemiological or systemic models). This, in turn, has resulted to the development of many safety process metrics, which, however, have not been thoroughly tested against the quality criteria referred in literature, such as validity, reliability and practicality.

The concept towards a Standard Safety Model (STASAM v.0)

In safety science and practice, there have been various safety models, each of them reflecting a particular approach to safety management and accident causality. The large variety of models suggested in literature and applied in practice serve the communication of diverse perspectives towards safety and the need to consider contextual factors, but it does not allow the establishment of a common language within and across organisations and industry sectors. Considering the potential benefits of talking a lingua franca when it comes to safety and inspired by the Standard Model used in particle physics and recent suggestions from relevant studies, we thought of exploring the possibility to introduce a Standard Safety Model (STASAM). As a first step, we focused on four representative safety and accident models widely used, discussed and debated: the Swiss Cheese Model, AcciMap, Functional Resonance Analysis Method (FRAM) and Systems-Theoretic Accident Model and Processes (STAMP). We reviewed literature which compares the particular models, and we listed the strengths and weaknesses of each as a means to set the grounds for the STASAM. The combinations of these models with a focus to host their advantages and avoiding their disadvantages led to a three-level STASAM. The concept STASAM was used in two random incident investigation reports to assess its applicability and visualisation against the original models. The results of the application along with the STASAM concept were reviewed by three safety professionals and three safety researchers. The comments received were in the positive direction and indicated the potential of establishing an inclusive and commonly accepted safety/accident model. The next research phase will be the additional review of the STASAM and its pilot application to a variety of safety events and systems as a means to test its reliability and strengthen its validity.

Do experts agree when assessing risks?

Risk matrices have been widely used in the industry under the notion that risk is a product of likelihood by severity of the hazard or safety case under consideration. When reliable raw data are not available to feed mathematical models, experts are asked to state their estimations. This paper presents two studies conducted in a large European airline and partially regarded the weighting of 14 experienced pilots’ judgment though software, and the calculation of agreement amongst 10 accident investigators when asked to assess the worst outcome, most credible outcome and risk level for 12 real events. According to the results, only 4 out of the 14 pilots could be reliably used as experts, and low to moderate agreement amongst the accident investigators was observed.

Effectiveness of risk controls as indicator of safety performance

The objective of the study described in this paper is to define safety metrics that are based on the effectiveness of risk controls. Service providers define and implement such risk controls in order to prevent hazards developing into an accident. The background of this research is a specific need of the aviation industry where small and medium-sized enterprises lack large amounts of safety-related data to measure and demonstrate their safety performance proactively. The research department of the Aviation Academy has initiated a 4-year study, which will test the possibility to develop new safety indicators that will be able to represent safety levels proactively without the benefit of large data sets. As part of the development of alternative safety metrics, safety performance indicators were defined that are based on the effectiveness of risk controls. ICAO (2013) defines a risk control as “a defence with specific mitigation actions, preventive controls or recovery measures put in place to prevent the realization of a hazard or its escalation into an undesirable consequence”. Examples of risk controls are procedures, education and training, a piece of equipment etc. It is crucial for service providers to determine whether the introduced risk controls are indeed effective in reducing the targeted risk. ICAO (2013) describes the effectiveness of risk control as "the extent to which the risk control reduces or eliminates the safety risks”, but does not provide guidance on how to measure the effectiveness of risk control. In this study, a generic metrics for the effectiveness of risk controls based on their effectiveness was developed. The definition of the indicators allows, for each risk control, derivation of specific indicators based on the generic metrics. The suitability of the metrics will subsequently be tested in pilot studies within the aviation industry.