October 30, 2013

A new study by Dr. Jennifer Taylor, an associate professor in the Department of Environmental and Occupational Health at the Drexel University School of Public Health, and her team found that with advances in computerized information retrieval techniques, large-scale near-miss narrative text analysis is possible. The study, which was detailed in a recent article in Accident Analysis & Prevention, is the first to successfully apply machine learning algorithms to assign mechanism of injury codes to near-miss narratives reported to the National Fire Fighter Near Miss Reporting System (NFFNMRS). Previous research using machine learning only examined definitive injury narratives. With the computer algorithm, the researchers were able to create two new variables that previously did not exist in the near-miss dataset – (1) did an injury happen and; (2) what type of injury occurred or would have occurred and what was the cause of that injury. Dr. Taylor and her team believe that this research can be applied to other industries that use injury reports, such as mining, nuclear power and construction, among others. Additionally, the ability to use computers to identify near-miss and injury events allows researchers to better understand these reports and more importantly, to develop primary prevention activities.

To conduct the narrative text analysis, a computer algorithm using Fuzzy and Naïve Bayesian models was used to predict the precipitating and the proximal mechanism for injury or near-miss events reported to the National Fire Fighter Near Miss Reporting System (NFFNMRS). The injury codes used to evaluate these data were modified versions of mechanism of injury codes from the International Classification of Disease 9 Clinical Modification Manual (ICD-9-CM). In order to accurately predict what type of injury code would represent the incident discussed in the narrative, the computer program had to be trained by the research team. To train the algorithm, researchers manually assigned an injury code to 764 fire events. The training was an incremental process starting with 100 narratives.  From the initial training set of 100 to the final set of 764, the algorithm’s sensitivity improved 31% to 74% for the Fuzzy model and 35% to 68% for the Naïve model. The sensitivity was continuing to improve at the final interval indicating that if more narratives were available the results would improve further.

In the absence of new narratives additional modifications to the word set such as paired words, word sequences, morphs and drop word lists improved the hit rate. Preliminary analysis using paired words and 3-word sequences in the Fuzzy model generated a sensitivity of 82% and 85% respectively. Such minor modifications could significantly improve the predictive capability of the algorithm thereby reducing the amount of narratives that would need to be manually reviewed. Dr. Taylor’s research team is continuing to research these enhancements.

This study is the first to successfully apply machine learning algorithms to assign mechanism of injury codes to near-miss narratives. Previously, computer algorithms were only used to look at injury narratives. Near miss events reveal hazards that could have resulted in injury, but did not, calling attention to opportunities for safety interventions at an earlier point in the process than can be identified by data describing injuries and their consequences. Many near misses are potentially fatal events, and the narratives contained in these reports are opportunities to examine vital causal information that would not have been available had the reporter died. Near miss events happen more frequently than injuries hence their study affords a more precise picture of failure points in systems.  However, as detailed in the article and in an excerpt of a near-miss narrative below, working with near-miss narratives in which an injury could have happened but did not presents unique challenges:

On arrival, there was fire showing on the second floor "B" side of a two and one half story wood-frame residential structure. We had been operating a two and one half inch attack line for approximately ten minutes. As Division Two Commander, I felt at that time that we were beginning to lose progress. On orders of the Incident Commander, orders were given to immediately evacuate the second floor. As Division Two Commander, all crews were evacuated, excluding myself and two other crew members in a final effort, despite the Incident Commanders orders.  Upon evacuating, after the Incident Commander's second order to evacuate, we observed the second floor flashover and collapse.

First, there are multiple outcomes that could have occurred (burns, struck by falling roof) and determining which outcome is most likely is a time consuming process requiring expert opinion.  Second, determining what would have been the proximal mechanism of a near-miss is challenging because an injury did not occur and a fire scene is often chaotic and the chain of events complex. Third, on average near-miss narratives are 216 words long whereas national survey injury narratives usually average around 11 words. The presence of more words makes it harder to train the algorithm. Finally, the instructions to individuals completing the Near-Miss Forms are vague – asking the individual to “describe the event”. In an attempt to set the scene, many narratives, such as the one above, contain information that is not directly related to the chain of events that contributed to the injury or near-miss.

The ability to use computers to identify near-miss and injury events and then classify them by cause offers researchers a powerful new tool to better understand injuries. Before this study, there was not an efficient way to determine if an injury occurred and what the cause of the injury or near-miss event was. Now researchers will be able to expediently identify near-miss narratives relevant to the research they are conducting, which means firefighters can benefit from the researcher sooner.

Dr. Taylor has been trained in the field of injury prevention and control and uses its principles to address safety issues related to the unique tasks of the fire service. Dr. Taylor received her Ph.D. from the Johns Hopkins Bloomberg School of Public Health, Department of Health Policy and Management. She obtained her Master’s of Public Health degree in Health Services from the Boston University School of Public Health.  Dr. Taylor is currently an Associate Professor in the Department of Environmental and Occupational Health at Drexel University School of Public Health.