@phdthesis{14045,
  author = {Sunil Nair},
  title = {Characterization of Safety Evidence for Assessment and Certification of Critical Systems},
  abstract = {Safety assurance and certification are amongst the most expensive and timeconsuming activities in the development of safety-critical systems. Deeming a system to be safe involves gathering convincing evidence to argue the safe operation of the system, usually according to the requirements of some safety standard. To handle large collections of safety evidence effectively, practitioners need knowledge regarding (1) the different types of evidence, (2) what characterizes the different evidence types, (3) how to effectively structure the evidence along with its argumentation to show fulfilment of system safety claims, and (4) how to assess the confidence in the evidence presented. Failing to clearly understand the above evidence needs while assessing the system safety can result in major problems. First, the system supplier may fail to record critical details during system development that the certifier may require later on. Building the missing evidence after-the-fact can be both expensive and laborious. Second, the certifier might find it hard to develop sufficient confidence in the system undergoing certification if there is no common understanding of the evidence needs (between the system supplier and the certifier) a priori. An agreed common knowledge regarding evidence requirements might help make certification and assurance more credible. Third, for a real life large-scale complex system, it is highly important to demonstrate sufficient traceability among the thousands of development and verification artefacts that might be used as safety evidence. If the evidence is not structured properly, its sheer volume and complexity can jeopardize the clarity of the safety arguments, in turn make safety assessment difficult. Finally, it is important to be able to determine how definitive the evidence used to support a particular claim is. Though safety standards mandate sufficient and credible evidence to show compliance, they are often vague on what sufficient and credible means in a particular context, often intentionally and for the sake of being general.This thesis is aimed at understanding and characterizing the safety evidence used for certification and assessment of safety-critical system. The thesis analyses the current evidence management practices and proposes means for future improvements. The contribution provided in this thesis have been developed in the scope of OPENCOSS, a large-scale European research project whose goal is to devise a common certification framework for the automotive, avionics, and railway domains.The thesis defines a taxonomy of evidence types towards enhancing the knowledge and understanding regarding what safety evidence is. The taxonomy was built by analysing and synthesizing the existing knowledge in the academic literature about safety evidence. This was achieved by means of a Systematic Literature Review, based on 218 peer-reviewed studies between 1990 and 2012. The taxonomy was then validated by 52 practitioners from 11 application domains through a survey aimed at analysing the state of the practice on safety evidence management. The benefits of such a taxonomy are three-fold: (1) the taxonomy can provide a common terminology for communication among the system suppliers and safety assessors about evidence requirements during the certification process, helping in reducing certification risks and costs by avoiding terminological mismatches; (2) the taxonomy is a useful reference to new researchers, helping them better get acquainted with the area, and; (3) the taxonomy can be a helpful tool for practitioners to gain a clearer understanding of what information may be relevant for demonstration of compliance with safety standards. Other contributions related to the survey of literature and practice also include (1) a detailed analysis of the various techniques for evidence structuring and evidence assessment proposed in the literature and used in practice, (2) an overview of existing challenges for provision of safety evidence, and (3) a comparison of the knowledge gathered from the state of the art and the state of the practice, allowing us to find potential gaps.Another important research area addressed in this thesis relates to safety evidence traceability. The results of the literature review and the survey with practitioners showed that safety evidence traceability is topic that has been little addressed in literature and is a major challenge in practice. Towards addressing the gap in literature, the thesis analyses the current state of the art in traceability by looking at the topics that have been studied, the challenges that have been addressed, the contributions that have been made, and the type of artefacts traced. This analysis helped in identifying gaps in past work and determining research needs. To address the challenges faced in practice, based on the analysis and the knowledge gathered previously, the thesis identifies the information that characterises safety evidence traceability. The thesis presents the set of traces that is regarded as necessary for safety evidence and proposes SafeTIM, a traceability information model for safety evidence. SafeTIM provides the set of fundamental concepts and relationships for enacting evidence traceability in real industrial settings. We have validated the model with documentation from three different industrial case studies. Other contributions include the definition of the motivations for evidence traceability and of potential challenges relating to evidence traceability.The final research area of the thesis is themed towards improving expert judgement in safety evidence assessment. The results of the survey with practitioners indicated that expert judgement is one of the most common means to assess safety evidence. In spite of its high importance, we know little about how safety experts assess evidence, how they gain confidence in evidence and what information do they consider when assessing the evidence. To address these issues, the thesis studied and presents results of the current practice of evidence assessment. With the help of in-depth interviews and focus group meeting with safety experts, we identified that the safety evidence assessment process varied substantially from expert to expert and that safety assessments were frequently based on subjective evaluations. More importantly, we identified a set of generic factors that influence the expert\’s decision on the acceptance of the safety evidence. To further improve expert judgement in safety evidence assessment context, the thesis proposes a novel approach for assessing the confidence in safety evidence. The proposed approach automatically builds secondary confidence arguments that detail the various reasons for having confidence in the evidence while accounting for uncertainty in the judgement using Evidential Reasoning. The approach decomposes the abstract notion of confidence in evidence into sub-factors such as confidence on the process, the personnel, tools, etc. and performs low-level assessment of each of these factors. The lower-level confidence values are then automatically propagated up to tree and present final assessment on confidence. As part of the approach, the thesis also proposes a confidence argument pattern that is represented in Goal Structuring Notation. The proposed argument pattern incorporates various factors that influence expert\´s confidence in a piece safety evidence and build a clear argument structure to support the use of the evidence. The thesis also develops a prototype tool named EviCA (Evidence Confidence Assessor) that supports the application of the proposed approach.},
  year = {2015},
  journal = {University of Oslo},
  month = {03/2015},
  publisher = {Akademika Publishing, University of Oslo},
}