DEPARTMENTS
PRESIDENT'S MESSAGE
JOHN MULHAUSEN, PhD, CIH, CSP, FAIHA, retired in 2018 from 3M where he worked for 31 years in a variety of global health and safety risk management roles, most recently as director of corporate safety and industrial hygiene. Send feedback to The Synergist.
Faulty Judgment
BY JOHN MULHAUSEN, AIHA PRESIDENT
Our profession, like many others, values experience—and with good reason. Our work requires such a broad range of skills that most of us take years to become fully competent. Some workplaces are so complex that extensive experience is needed to propose the right interventions.
We often frame our deference to experience in terms of “professional judgment.” In situations where little data exists on which to base decisions about worker protection, professional judgment is our trump card—it’s the reason why we’re making the decision and not someone whose experience is different or lesser than ours.
But there’s a major issue with our reliance on professional judgment: studies have shown that we aren’t always accurate when characterizing exposures. In fact, some studies have found that qualitative exposure judgments made by OEHS experts are not significantly better than judgments based on random chance and, in at least one study, worse than those made by novices.
Worse still, some studies show that our exposure judgments tend to underestimate exposures. If we’re going to be wrong, we would much rather overestimate exposures, which would lead us to recommend overprotective health and safety measures. When we underestimate exposures, we put workers’ health at risk.
EXPERTS, NOVICES, AND RANDOM CHANCE Three of the studies in question are listed in the Resources section and are worth reviewing in full. Each used the AIHA exposure assessment strategy—as described in A Strategy for Assessing and Managing Occupational Exposures—to examine the accuracy of exposure judgments for OEHS professionals with varying levels of experience.
The AIHA strategy calls for the OEHS professional to assign exposures for a given similar exposure group (SEG) to one of four exposure control categories: ECC 1 (less than 10 percent of the OEL), ECC 2 (10–50 percent of the OEL), ECC 3 (50–100 percent of the OEL), or ECC 4 (greater than the OEL). According to the strategy, the OEHS professional should determine in which ECC the 95th percentile of the exposure distribution is most likely to be found for a given SEG.
Two of the studies examined the accuracy of both qualitative and quantitative (based on monitoring data) exposure judgments. Participants in these studies were trained on how to estimate the 95th percentile. Prior to training, accuracy was poor and not much better than that achieved by assigning an ECC randomly. After training, the accuracy of quantitative judgments increased significantly. Not surprisingly, as there was no data with which to apply the statistical training, there was no change in the accuracy of qualitative judgments. See below for a graph of results from one study that compares pre-training judgment accuracy to random chance judgments.
The third study, published in the March 2016 issue of the Journal of Occupational and Environmental Hygiene, examined the accuracy of judgments before and after the use of an algorithmic checklist tool. Judgments without the use of the tool were not significantly different from random chance and improved significantly with use of the tool.
Of interest is the observation in this study that novices performed as well as or better than experienced professionals. This may suggest that the professional experience that serves us so well in some aspects of our jobs may actually work against us when we need to make qualitative exposure judgments.
Perhaps as we gain experience, we become overconfident in our abilities or incorrectly calibrate through inaccurate data interpretation because we do not consistently use statistical tools. By contrast, a less experienced individual is more likely to adopt a more systematic approach.
Since the effectiveness of our exposure risk management efforts relies on accurate exposure judgments, our weakness in this area is alarming. We can take solace from the fact that research in other professions has drawn similar conclusions. Medical practitioners, for example, are among the professionals who need to synthesize a great deal of information and make decisions based on incomplete data. Research has found that their judgments, too, are often wrong when they rely solely on their professional judgment.
The proper use of algorithmic and statistical tools that ensure a systematic approach was shown to significantly improve the accuracy of exposure judgments.
TOOLS FOR BETTER JUDGMENT There is a silver lining, however. In each of these studies, the proper use of algorithmic and statistical tools that ensure a systematic approach was shown to significantly improve the accuracy of exposure judgments. Simple rules of thumb that OEHS professionals can use to determine the correct ECC were helpful. One such rule is to assign an SEG to ECC 4 if there are fewer than six exposure measurements and one measurement exceeds the OEL.
These algorithms and other tools for characterizing exposures are available on the AIHA website: •The IH/OEHS Exposure Scenario Tool, or IHEST, guides the assessor by collecting information about workplace scenarios and agents. It provides cues for estimating important determinants of exposure such as generation and ventilation rates and prompts the user to specify the type of engineering controls. •The Basic Exposure Assessment and Sampling Spreadsheet is an Excel-based template for entering air and noise sampling data. •IHMOD 2.0 is an Excel-based mathematical modeling spreadsheet that can be used in deterministic mode (using point value parameters) or in Monte Carlo simulation mode, with choices of distributions of parameter values. •The Qualitative Exposure Assessment Checklist requires only four pieces of information: the OEL, the vapor pressure, the observed workplace control measures, and the required level of workplace control. It can be applied in just a few minutes using readily available information and is significantly more accurate than subjective, intuitive judgments. •IHSkinPerm is an Excel application for estimating dermal absorption. •IHSTAT is an Excel application that calculates various exposure statistics, performs goodness of fit tests, and graphs exposure data.
For more information about how these tools fit into an effective exposure risk management program, view the free AIHA webinar “Top 10 Imperatives for the AIHA Exposure Risk Management Process.”
These tools are similar to those that have improved judgment in other professions. For example, doctors rely on the Apgar test to make a quick determination of the health of newborn infants. The Apgar test, which uses just five inputs to predict the need for medical intervention, is credited with substantially improving health outcomes for newborns. The airline industry has also found that pilots’ use of checklists and feedback loops have led to significant improvements in flight safety.
INCREASING ACCURACY Given the critical importance of exposure judgment aids, one would expect the IH tools and algorithms on AIHA.org to be mainstays of current industrial hygiene practice. Yet anecdotal evidence suggests that a distressingly small percentage of AIHA members use these tools despite their free availability.
Some professionals may be reluctant to rely on tools that appear, at first glance, to oversimplify a complex process. Perhaps using these tools runs counter to the idealized notion of professional judgment that many of us hold dear. But as research makes clear, our reverence for professional judgment is sometimes misplaced. The sooner we accept this truth, the better off we—and the workers we’re charged with protecting—will be.
Qualitative judgment results for accuracy of 93 pre-training judgments for all industrial hygienists compared to random chance judgments. Adapted from “Effect of Training on Exposure Judgment Accuracy of Industrial Hygienists,” Figure 5, Journal of Occupational and Environmental Hygiene, April 2012.
Tap on the graph to open a larger version in your browser.
RESOURCES
AIHA: A Strategy for Assessing and Managing Occupational Exposures, Chapter 6, “Approaches to Improving Professional Judgment Accuracy,” 4th ed. (2015).
AIHA: “Top 10 Imperatives for the AIHA Exposure Risk Management Process” (webinar, 2021).
Annals of Occupational Hygiene: “Occupational Exposure Decisions: Can Limited Data Interpretation Training Help Improve Accuracy?” (April 2009).
Journal of Occupational and Environmental Hygiene: “Effect of Training on Exposure Judgment Accuracy of Industrial Hygienists” (April 2012).
Journal of Occupational and Environmental Hygiene: “Using Checklists and Algorithms to Improve Qualitative Exposure Judgment Accuracy” (March 2016).
The Synergist: “Judgment Day: How Accurate Are Industrial Hygienists’ Qualitative Exposure Assessments?” (January 2014).