Measuring an aviation system's safety performance

PhD candidate and Safety Manager at NATO's Airlift Management Programme, Ilias Panagopoulos, has collaborated with Professor Chris Atkin and Dr Ivan Sikora, senior academics in the Department of Mechanical Engineering & Aeronautics at City, University of London, for his doctoral research in aviation safety performance.

Ilias presented his methodology for measuring performance and research results at the 19th EURO Working Group on Transportation Meeting (EWGT2016) on 5th to 7th September 2016, Istanbul, Turkey, and at the 1st International Cross-industry Safety Conference (ICSC 2016), organised by the Amsterdam University of Applied Sciences (Aviation Academy) in the Netherlands from 3rd to 4th November 2016.

Annex 19 of the International Civil Aviation Organization (ICAO) says: 'safety is the state in which risks associated with aviation activities are reduced and controlled to an acceptable level'.

New harmonised approach

Safety is a system quality stemming from a legal and regulatory framework which stipulates strict and high- performance targets as well as a number of activities which must be performed by air operators. Annex 19 goes on to say that aviation service provider (i.e. airlines, airports, aircraft maintenance organisations, air training organisations and air traffic services) shall, as a minimum:

• Establish a Safety Management System (SMS)
• Provide continuing monitoring and regular assessment of safety performance
• Ensure remedial action to maintain agreed performance
• Aim at a continuous performance improvement

At the European level, the European Aviation Safety Agency (EASA), in parallel with management system requirements, outlined its new harmonised approach for establishing a Performance-Based Environment by introducing a clear set of indicators and targets against which the oversight performance of civil aviation authorities is assessed.

In the aviation industry there is nevertheless a level of uncertainty about the extent existing methodologies for measuring performance are suitable for those operators who have achieved excellent safety records (i.e. zero accidents or serious incidents) and in-control processes, and as such the need to look for further improvements.

In addition, within the aviation industry, the measurement process regarding a set of pre-defined indicators for measuring an aviation system's safety performance has not yet been introduced or standardised.

Besides, the development and measurement of proper Safety Performance Indicators (SPIs) or metrics is not straightforward and the operational experience for measuring the effectiveness of SMS is very limited, since there are many questions yet to be answered on measuring safety performance.

Continuous improvement methodology

Consequently, the main challenge remaining is how to control and maintain performance within agreed safety specification limits and how to develop an objective methodology that will proactively investigate and measure system performance variability from target.

As a consequence, this study further investigated the following key research questions:

• What methodology could proactively measure system safety performance and improve the safety performance measurement process?
• Could a conceptual framework assist the continuous improvement of the safety performance measuring process?

So as to address the key research questions, the research presents a conceptual framework that will improve the safety performance measurement process and the aviation system safety performance.

In this framework, the Safety-Performance Indicator Lean Sigma (Safety-PILS) model has been embedded within Define-Measure-Analyse-Improve and Control (DMAIC) continuous improvement process. This integration results in a continuous improvement methodology that measures system safety performance and reduces the safety process variability. In addition, the study provides an implementation guide on how organisations could use this framework to design and develop a proactive, performance-based methodology for measuring Acceptable Levels of Safety Performance (ALoSP) at sigma (σ) level, a statistical measurement unit.

In Phase I of the safety measurement process, the Safety-PILS model provides guidance on how organisations could design, implement and use a proactive, performance-based measurement tool for assessing and measuring ALoSP. Also, Safety-PILS model assists operators to comprehend and design their safety system in accordance with the agreed Safety Performance Indicators (SPIs), targets and specification limits.

Nevertheless, the Safety-PILS model provides a holistic view on how organisations could set leading performance indicators and monitor metrics on the top of identified root-causes that affect safety performance or how to set lagging indicators and feedback metrics on the top of safety outcomes (e.g. number of occurrences).

Moreover, the core advantage of the Safety-PILS model is that applies the Central Limit Theorem and since it repeatable uses a large size of data and means, the distribution of the sample means will finally approach a normal distribution. Accordingly, the next step for the operator is to follow at Phase II the DMAIC process for continuously improving the overall system's safety performance measurement process.

Through DMAIC process shown in the Figure below, the operator could apply Lean Six Sigma methodology for measuring both the performance of each established indicator and system safety performance variability at sigma level from core safety objectives.

The research study introduces an integrated, empirical-tested conceptual framework that may satisfy the requirements of aviation authorities for establishing a performance-based approach in aviation safety.

Furthermore, the study identified and filled the gap existing in the literature and proposed a practical implementation guide and tools for measuring aviation system safety performance.

Finally, the study revealed that the application of Lean Six Sigma methodology can enhance the safety measuring process.

To this end, the proposed guide is a new way of thinking for designing a safety case aims to achieve desired outcomes within agreed specifications limits.