Royal Society Publishing

Bayesian assessment of overtriage and undertriage at a level I trauma centre

Paul B DiDomenico, Jan B Pietzsch, M. Elisabeth Paté-Cornell

Abstract

We analysed the trauma triage system at a specific level I trauma centre to assess rates of over- and undertriage and to support recommendations for system improvements. The triage process is designed to estimate the severity of patient injury and allocate resources accordingly, with potential errors of overestimation (overtriage) consuming excess resources and underestimation (undertriage) potentially leading to medical errors.

We first modelled the overall trauma system using risk analysis methods to understand interdependencies among the actions of the participants. We interviewed six experienced trauma surgeons to obtain their expert opinion of the over- and undertriage rates occurring in the trauma centre. We then assessed actual over- and undertriage rates in a random sample of 86 trauma cases collected over a six-week period at the same centre. We employed Bayesian analysis to quantitatively combine the data with the prior probabilities derived from expert opinion in order to obtain posterior distributions. The results were estimates of overtriage and undertriage in 16.1 and 4.9% of patients, respectively.

This Bayesian approach, which provides a quantitative assessment of the error rates using both case data and expert opinion, provides a rational means of obtaining a best estimate of the system's performance. The overall approach that we describe in this paper can be employed more widely to analyse complex health care delivery systems, with the objective of reduced errors, patient risk and excess costs.

Keywords:

1. Introduction and background

Numerous studies during the 1970s and 1980s (Cales & Trunkey 1985) demonstrated high rates of preventable deaths among trauma patients who were transported from the scene of injury to the nearest hospital. These figures were judged to be unacceptably high and provided the motivation for the creation of today's trauma systems. Since the late 1970s, regional trauma systems have been established in many areas throughout the United States. Trauma systems serve to direct seriously injured patients to specific facilities on local, regional and statewide bases. They link hospitals, pre-hospital care providers and other emergency medical services, as well as health care and public safety agencies. Designated trauma centres that have the essential resources and medical staff to care for major trauma cases serve as the hubs of these systems. The benefits of this approach are well documented in the literature; for example O'Keefe et al. (1999) described the positive effect of cumulative experience in designated trauma centres in decreasing mortality.

(a) Pre-hospital trauma triage

An integral part of any trauma system is pre-hospital trauma triage that attempts to differentiate critically injured patients from those with minor injuries, and to transport the critically injured to designated trauma centres that have the personnel and facilities best able to care for the patient. The triage process is designed to make a prediction of the appropriate level of care that a patient will require, given limited information as to the full extent and true nature of the patient's injuries (Knopp et al. 1988). This inherent uncertainty is compounded by the time critical nature of the process, as well as the potentially significant costs of mistakes, resulting in significant patient risk.

Attempts at devising prediction rules to categorize trauma patients at the pre-hospital level date back to the early 1970s, when many such rules were developed to identify major trauma by empirical criteria. In the 1980s, more analytical approaches were used to develop triage indexes using regression analysis of large inpatient databases (Champion 1986). The American College of Surgeons Committee on Trauma (1991) later defined and recommended a set of triage criteria. These ‘field categorization of trauma patients’ criteria were categorized according to physiological, anatomical and mechanism of injury (MOI) parameters. Although using these criteria permits effective identification of the injured patients who will benefit from trauma centre care, they are still relatively non-specific. Owing to their subjective nature, MOI criteria, in particular, have been studied with regard to their usage for activating a resource-intensive trauma response (Hoff et al. 1995). Paramedic judgement has also been advocated as a triage criterion that is equal or superior to other less subjective tools in its ability to identify major trauma victims (Emerman et al. 1991).

Pre-hospital triage criteria are not only used to determine whether a patient needs to be transported to a trauma centre, but are also responsible for initiating a cascade of pre-hospital and hospital events that affect patient care and resource use. For any set of decision-making criteria, it is possible that the system will make errors, with false positives, or overtriage, resulting in excessive resources (personnel, equipment) devoted to a patient unnecessarily, and false negatives, or undertriage, resulting in inadequate treatment and potential medical errors (Esposito et al. 1995). Overtriage results in overestimation and undertriage in underestimation of the patient's true risk state. While undertriage is a patient quality-of-care issue, Hoff et al. (1995) showed that overtriage is more of an economic problem in that it (i) stresses the human, material and economic resources of designated trauma centres, (ii) increases the demands of pre-hospital personnel, and (iii) financially penalizes the non-designated hospitals, which may lead to animosity and resistance against regionalization.

Generally, there are no benchmark rates of over- and undertriage. The goal is minimization of each type of triage error within resource constraints, striking an acceptable balance in individual systems based on local factors and individual judgements. On the one hand, the development of criteria that are insufficiently sensitive results in too many victims not being transported to a trauma centre, and can possibly lead to an increase in trauma mortality. On the other hand, the use of criteria that lack specificity results in too many patients transported to the trauma centre, with potential overuse of resources. This trade-off between over- and undertriage has been recognized in the literature. The American College of Surgeons Committee on Trauma has suggested that a 50% rate of overtriage might be necessary to maintain an acceptable undertriage rate (Hoff et al. 1995). While overtriage is relatively easy to assess, assessing undertriage is a challenge that has not been clearly solved yet. This is a difficult problem in particular since data for patients who are not treated in a trauma centre are not collected and thus cannot be analysed. In addition, the trade-off between the two types of errors involves a value judgement. For example, Champion (1986) stated that less than 1% of trauma deaths should occur in non-trauma hospitals and that an undertriage rate of 5% is an acceptable error rate trade-off when minimizing overtriage.

(b) In-hospital trauma alerts

Closely related to field triage is the process of trauma team activation within the trauma centres themselves. Soon after the introduction of trauma systems, it was recognized that different levels of response were needed to use hospital resources efficiently. Early experience revealed that very few patients who were brought to the trauma centre solely based on MOI criteria had severe or life-threatening injuries requiring the response of a full trauma team. Consequently, to avoid unnecessary disruption of the activities of key hospital personnel, differential trauma team activation systems were introduced in most trauma centres (Shatney & Sensaki 1994). Trauma alerts were then usually classified as ‘major’ or ‘minor’, with a corresponding full or modified trauma team response. The sensitivity and specificity of a differential alert system depend on the information from the field, since the call for a major or minor trauma response is usually made before the patient arrives at the centre. The quality of pre-hospital triage and communication between the field and the trauma centre thus directly affect patient classification and trauma centre response.

The studied trauma centre uses a three-level alert system for trauma triage. This system employs many of the recommendations of the American College of Surgeons Committee on Trauma. In summary, the three-level system consists of the following designations based on the combination of physiological and anatomical criteria with the MOI.

  1. Major trauma. Severe trauma (such as penetrating injuries, multiple fractures or active bleeding, as well as severe burns), physiological compromise in the face of significant anatomical findings or high risk of vital sign deterioration.

  2. Minor trauma. Significant MOI (such as automobile collision with a pedestrian, rollover motor vehicle accident with unrestrained occupant or fall from a height of greater than 15 ft/4.6 m). Anatomical findings may be present but there is no evidence of significant physiological deterioration.

  3. Mechanism only. Less severe MOI (such as fall from a height of less than 15 ft/4.6 m or a restrained occupant in a motor vehicle collision). No obvious evidence of significant anatomical injury other than extremity fractures distal to the knees/elbows, superficial laceration, abrasions or contusions. No evidence of physiological compromise.

This approach is similar to that used at other trauma centres that may use a three-level system with designations such as ‘911’ for major trauma, ‘922’ for minor trauma and ‘933’ for mechanism-only cases. However, it should be noted that trauma alert systems are not necessarily uniform among all level I trauma centres in the United States and this may limit the generalizability of any conclusions resulting from this study.

2. Material and methods

We sought to apply methods of engineering risk analysis to this setting. These methods that have been traditionally applied to high hazard industries, such as nuclear power generation, aerospace systems and chemical plants, can be brought to bear on problems relating to the delivery of medical care. These methods rely on systems analysis and probability theory (mostly Bayesian). Early applications of engineering risk analysis methods to the medical field were performed, for example, by Paté-Cornell et al. (1996, 1997) and Paté-Cornell (1999) towards the goal of reducing accidents in anaesthesiology, and the authors showed this interdisciplinary approach to be of value. The assessment of a trauma triage system lends itself particularly well to this analysis. It is quite comparable to warning systems in engineering systems that involve similar trade-offs between the probabilities of errors of both types and have been studied in considerable detail (Paté-Cornell 1986).

(a) System analysis

The objective of the system analysis in this case was to examine the studied trauma centre's current implementation through the systems perspective, with a particular emphasis on the interactions between the pre-hospital care personnel and the emergency department. It specifically addressed prevailing issues in patient triage that the system has to face in its day-to-day operations. The enquiry encompassed the entire process flow, beginning with the initiating event, continuing through the pre-hospital response and the hospital arrival, and ending with the actual medical care administered and its eventual effect on the patient's outcome.

The first step was to create an influence diagram model of how patients are triaged, both in terms of assessment by local paramedics and response of the emergency department in terms of preparation for patient reception and appropriate care. Influence diagrams were first introduced in decision and risk analysis in the late 1970s and have proved to be a very useful modelling approach (Howard & Matheson 1984). An influence diagram incorporates key decisions and relevant uncertainties in a system, as well as their dependencies, and links decisions and uncertainties to a value of interest.

We created our model based on a series of expert interviews, patient case records and historical performance data. Once the model was finalized, an analysis of dependencies within the model provided insight into the over- and undertriage problems. This kind of analysis in anaesthesiology previously showed that certain recommendations to improve patient safety (such as periodic recertification of personnel and training on simulators) would be effective in reducing patient risk, whereas others (such as drug testing of personnel) would be much less effective in improving patient outcome and would impose high costs (Paté-Cornell et al. 1996). Therefore, modelling the sources of patient risk provided a context for the final recommendations.

(b) Sample of trauma cases

With the initial understanding of the overall trauma system, the next step was to go into the emergency department and collect data to assess the incidence rates of over- and undertriage. The studied trauma centre treats approximately 1500 trauma cases per year. By designing a survey instrument, a cohort of 86 trauma cases was randomly sampled during a continuous six-week period in the early autumn. The cases observed during this period were considered to be representative for other periods during the year as well. We assumed that the rates are stationary. Extraordinary events (e.g. mass casualties) were not considered, as the focus of this study was on day-to-day trauma triage. Each case survey involved collection of information about the MOI and paramedic report, anatomical and physiological findings, the initial trauma alert level and the findings from the primary and secondary surveys in the emergency department. Once the patient had been evaluated, the senior resident or attending physician was asked to assess in retrospect what initial alert level would have been appropriate, given the actual level of care the patient required.

The 86 patient cases provided statistical point estimates of the over- and undertriage rates. Part of the difficulty in estimating these rates is that one is attempting to measure the occurrence rate of relatively rare events from a small sample, especially in the case of undertriage. Usually, the frequentist statistical approach is employed to obtain a single number within a specified confidence interval, based on empirical sampling from the given population. However, this approach does not take into account any information outside of the empirically observed sample. Furthermore, it can be limiting in that collecting sufficient data to characterize rare events can be costly. In order to obtain a full statistical distribution of over- and undertriage using all available information about the underlying parameters of interest, a Bayesian approach was employed. The Bayesian methods address the fundamental issue of how empirical evidence should change one's beliefs about the value of a parameter of interest (Resnic et al. 2004), and provide a rational means of combining expert opinion with observational data.

(c) Survey of trauma surgeons

Without disclosing any information about the gathered case data, six experienced trauma surgeons were interviewed to obtain their estimates of the overall over- and undertriage distributions at this trauma centre, based on their own work experience. They were asked for estimates of the mean rates of over- and undertriage incidence rates as well as the 25th and 75th percentiles of the related distributions. Knowledge of the 25th and 75th percentiles of the distribution allows its spread (i.e. standard deviation) to be elicited independently of the mean. It was assumed that the surgeons were able to specify the individual parameters with equal precision.

(d) Bayesian analysis

The Bayesian approach builds fundamentally on Bayes' theorem, which itself builds on the fundamental axioms of probability theory. It relates the probability of a hypothesis (θ) conditional on a given body of data (x) to the probability of the data given the hypothesis. In the notation used in equation (2.1) below (cf. Resnic et al. 2004), f(θ|x) denotes the conditional probability of a parameter θ, given the evidence that x is true (the posterior distribution), f(θ) is the prior distribution, f(x|θ) is the likelihood function and f(x) denotes the probability of the evidence observed:Embedded Image(2.1)

In the example discussed here, the prior distribution describes the experts' opinion about the rate of overtriage (or undertriage) before observing any of the collected data. The Bayesian updating process according to equation (2.1) facilitates the formal combination of the a priori beliefs with the collected evidence, and yields the posterior distribution, which is then used as the estimate of over- and undertriage.

Owing to its distinct advantages over other possible distributions, the standard beta distribution was chosen for this updating process. The beta distribution can relatively easily be fitted to the percentile data obtained from the experts, and has positive probability density only in the interval between 0 and 1 (0 and 100%), reflecting the range of possible values for over- and undertriage rates. Furthermore, the beta distribution is the natural conjugate to the Bernoulli distribution, which makes it well suited for the process of updating the aggregated prior distribution with the gathered case data. This implies that if the prior distribution of the error rate is characterized by a beta distribution, the posterior distribution is also beta distributed.

Equation (2.2) shows the probability density function of the standard beta distribution, where x reflects the incidence rate and α and β are the shape parameters of the distribution:Embedded Image(2.2)where Γ(γ) represents the gamma function Embedded Image.

The shape parameters α and β allow us to fit the distribution to the data elicited from the experts. Equation (2.3) describes the mean and equation (2.4) the variance of the standard beta distribution:Embedded Image(2.3)Embedded Image(2.4)

For each of the six interviewed trauma surgeons, beta distributions were fitted to the mean and percentile information given, using a least-squares approach. A heuristic search was employed to obtain a reasonable fit, with the mean of the distribution required to match the physicians' assessment, and the distances for the least-squares method defined as the distances between the physicians' 25th and 75th percentiles and the respective values of the beta distribution. The obtained six beta distributions were then aggregated to a combined beta distribution using a linear weighting approach (Stone 1961) with equal weight of 1/6 assigned to each of the six physicians. This joint prior distribution then constituted the prior distribution that was employed for the Bayesian updating process.

Having obtained this combined prior distribution, we then performed a Bayesian update using the survey data collected from the 86-case sample, in order to obtain the best estimate of the over- and undertriage rates. Using this approach, we obtained a robust ‘best estimate’ of the system's current performance, combining wide differences of opinion with limited actual evidence.

3. Results

Table 1 shows the results of the 86-case data sample. As shown in the table, there were 13 cases of overtriage (above the diagonal) and 2 cases of undertriage (below the diagonal). The 13 overtriage cases consisted of 5 cases of major alerts when a minor alert was appropriate and 8 cases of minor alerts when a mechanism-only alert was appropriate. Notably, most (71%) of the trauma cases fall into the minor alert category. This phenomenon that a disproportionate majority of trauma cases falls into one category resulted in a recommendation to consider modifying the triage criteria in order to split the minor alert category into two categories to enable more accurate triage of patients.

View this table:
Table 1

Random sample of trauma cases over a six-week period.

Calculating the conditional probabilities of each type of triage error provides insight into the kinds of errors most likely to occur. The probability of initial alert given the final evaluation is known is shown in table 2, and the probability of the final evaluation for the given initial alert is shown in table 3. Some interesting observations can be derived from these calculations: 20% of all true major traumas are initially undertriaged as minor trauma (table 2), and 38% of all major trauma alerts are initially overtriaged, since they are later shown to be minor trauma after evaluation (table 3). The values in the diagonal may be thought of as the diagnostic ‘sensitivity’ of the triage process (table 2) and ‘positive predictive value’ (table 3).

View this table:
Table 2

Conditional probabilities, p(initial alert) given post-evaluation assessment of the correct triage level. (Values in the diagonal may be thought of as the ‘sensitivity’ of the triage process.)

View this table:
Table 3

Conditional probabilities, p(post-evaluation) given initial alert triage level. (Values in the diagonal may be thought of as the ‘positive predictive value’ of the triage process.)

We found vast differences in perception among the trauma surgeons, ranging from 15 to 40% for the mean rate of overtriage and 3 to 16% for undertriage (table 4). These differences were reflected accordingly in the individual prior distributions fitted to the surgeons' expert opinions. The aggregated prior distribution obtained through linear weighting was a beta distribution with parameters α=2 and β=5.05 for overtriage and α=2 and β=22.3 for undertriage, yielding (from equation (2.3)) means of 28.4% for overtriage and 8.2% for undertriage (table 5).

View this table:
Table 4

Physician judgement (interview) data from six experienced trauma surgeons surveyed. (OT, overtriage; UT, undertriage; MA, major alert; MI, minor alert; MO, mechanism only; NR, no response.)

View this table:
Table 5

Mean estimates of over- and undertriage.

Using Bayesian updating to incorporate the case data with physician judgement, we obtained the posterior distributions to show that overtriage occurs at a mean rate of 16.1% while undertriage occurs at a mean rate of 4.9% (table 5, with distributions shown in figures 1 and 2). These rates can be extrapolated to represent approximately 242 cases of overtriage and 74 cases of undertriage on an annual basis. The parameters of the posterior distributions are α=4 and β=106.3 for undertriage and α=15 and β=78.05 for overtriage.

Figure 1

Overtriage posterior distribution (dotted line, prior distribution; solid line, posterior distribution).

Figure 2

Undertriage posterior distribution (dotted line, prior distribution; solid line, posterior distribution).

4. Discussion

We find that over- and undertriage occur at not-insignificant rates. Although the overall rates are low, it should be noted that the rates were previously unknown and the subject of speculation, as shown in table 5. Based on the posterior rates, more informed judgements can be made and specific actions can be taken. Many of the intuitions of the experienced trauma surgeons were confirmed: that overtriage does occur; that it occurs more often than undertriage; and that the current system can be improved. Note, however, that undertriage is rare, and that it occurs much less often than overtriage. These phenomena are the results of a value judgement about the trade-offs involved and may be considered characteristics of a well-designed triage system.

A traditional warning system is usually constrained by a trade-off between false-positive and false-negative alerts. We believe that trauma triage is similarly constrained, i.e. reducing overtriage will tend to increase the rate of undertriage. Since undertriage is currently a rare event and overtriage exceeds undertriage, any changes to the system should seek to maintain the low undertriage rate while reducing overtriage. The system currently acts in a conservative manner by absorbing excess cost for the sake of patient safety. This implies that the risk of error is currently shifted away from the patient. This characteristic of the triage system is desirable (Hoff et al. 1995) and should be maintained while the overtriage rate is driven down.

This study did not attempt to quantify the actual cost of overtriage. It is assumed that these costs are primarily borne by the hospital in the form of

  1. marginal financial costs of equipment and personnel time devoted to a patient unnecessarily,

  2. opportunity costs of diverting resources away from the sickest patients in the hospital, i.e. some patient care may be delayed, resulting in deterioration of the patient's condition and creating additional costs, and

  3. ‘burnout’ (or fatigue) and complacency on the part of trauma team members who respond to overtriage cases unnecessarily.

Since it is desirable to reduce these costs while maintaining a low undertriage rate, potential improvements to the current system should be considered. Although the trauma triage process is a complex decision-making system, the major processes of the system may be categorized generally as paramedic policy, communication and trauma centre policy. With the understanding gained from the initial system analysis, we correlated policy and managerial choices to particular outcomes and effects. These findings, along with recommendations to address the system shortfalls, are summarized below.

  1. A few key sources of uncertainty introduce the bulk of errors and inefficiencies. Of particular note are the MOI trauma triage guidelines that attempt to correlate the cause of an impulse injury to the expected condition of the patient. These guidelines, although effective, contribute greatly to the high level of overtriage within the studied region. A targeted statistical study could be undertaken to improve the criteria (especially MOI) currently employed.

  2. The current system could benefit from greater standardization. Each of the three major trauma centres in the local region employs differing standards and methods to determine the level of resources that will be activated to treat the incoming patient. At the studied hospital, we noted uncertainty among personnel as to the definitions of the three triage alert levels. Training on the current system in use could provide immediate benefit while a larger region-wide review of triage criteria is conducted.

  3. Inefficiencies in information flow introduce decision errors. The critical information leak exists at the pre-hospital–hospital interface, where paramedics attempt to convey their best assessment of the patient's condition to the trauma centre's charge nurse. This information is critical in determining the trauma centre's level of response. Yet, communications need improvement. The information transfer process could benefit from instituting common reporting protocols, technical improvement and better collaboration among stakeholders in the process.

From the mathematical perspective, this study has shown the advantages of the Bayesian approach for parameter estimation in situations where existing knowledge can be incorporated. In a frequentist framework, it would not have been possible to incorporate any of the available expert opinion residing with the physicians. Using a Bayesian approach, the obtained probability distribution can be monitored for convergence, and the results are available at any given point to inform decisions. This can be particularly helpful when rare events are monitored, or when collection of evidence is costly or time consuming. It should be noted that the Bayesian posterior distribution, in most cases, eventually converges to the classical (frequentist) maximum-likelihood estimate. This means in turn that subjective elements of the assessment in the Bayesian framework are in most cases eventually eliminated if the sample size of the collected dataset is large enough. For the purpose of comparison and to gauge the impact of the subjective (expert opinion) element in the analysis, it can be helpful to conduct the updating process in parallel for the experts' prior distribution and for an uninformative prior distribution (e.g. a uniform distribution).

A potential limitation of our study alluded to earlier is the fact that while the studied triage system is based on the nationally recognized approach to trauma triage, it should be noted that we analysed data from only one centre. This may limit the generalizability of our conclusions.

5. Conclusions

Over- and undertriage errors do occur at a significant rate, even when highly trained experts are involved in the triage process. Triage errors are the result of complex system interactions, and they can be understood by performing a system analysis using engineering risk analysis methods. An empirical Bayesian approach can be beneficial to permit the incorporation of expert judgement about the process, and to obtain a more robust estimate about a system's performance when gathering data is difficult or costly. The Bayesian approach presented here was judged reasonably practical in terms of expended data collection, and provided a reproducible means of measuring over- and undertriage incidence rates based on a relatively small empirical sample.

The findings of the study suggest that causal factors of over- and undertriage can be addressed by revisiting the triage criteria currently in use and by improving communications and collaboration between the paramedic and the physician communities. While we did not assign a value judgement to the absolute rates of over- and undertriage, the fact that overtriage occurs about three times more often than undertriage indicates a degree of conservatism built into the current system. This means that the current system tends to err on the side of caution, absorbing some excess costs for the sake of patient safety. What we recommend is to keep the patient risk at a low level while reducing the probability, and the costs, of overtriage. Finally, we conclude that an increased collaboration between the engineering risk analysis and the medical communities can provide a better understanding of the sources of patient risk in complex systems such as the triage process, enabling improvements in patient safety and a potential reduction in the rate of medical errors.

Acknowledgments

We would like to thank Dr Adam Seiver (Department of Management Science and Engineering, Stanford University, CA) and the members of the MS&E 250B project class of 2001 for their support on this project.

Footnotes

  • One contribution of 13 to a Theme Issue ‘Mathematical and statistical methods for diagnoses and therapies’.

References

View Abstract