DEPARTMENTS
ETHICS
ALAN LEIBOWITZ, CIH, CSP, FAIHA, is the president of EHS Systems Solutions LLC, vice chair of the Joint Industrial Hygiene Ethics Education Committee, a past chair of the Board for Global EHS Credentialing, and a past Board member of AIHA.
Send feedback to The Synergist.
Performance vs. Performative: Make Metrics Matter
BY ALAN LEIBOWITZ
Editor’s note: The case study in this article is fictitious and is intended to highlight ethical issues in the practice of industrial hygiene. Any resemblance to real people or organizations is coincidental. The opinions expressed in this article are those of the author and do not necessarily reflect the opinions of AIHA, The Synergist, the JIHEEC, or its members.
Please share your thoughts on this article by emailing the Synergist team or submitting them through the form at the bottom of this page. Responses may be printed in a future issue as space permits.
The second edition of AIHA’s Industrial Hygiene Performance Metrics manual provides guidance regarding the use and misuse of metrics. It includes many examples of metrics that can be used to measure the performance of specific elements of an OEHS program, including biological monitoring, Total Worker Health, sustainability, and program management. When adapted to local programs, these tools can provide real insight into program needs and potential corrective actions. Organizations can use performance metrics to understand their operations and make data-driven decisions for improvement.
OEHS metrics are most useful when they work in harmony with individual or organizational goals. They are a key element of effective management systems that focus on continual improvement of organizational programs and processes. This activity most often concentrates on internal efforts.
VANITY METRICS AND COGNITIVE BIASES
While some sources report that Peter Drucker wrote “what gets measured gets managed” in his 1954 book The Practice of Management, it appears that he has been misquoted. While the quote may be inaccurate, the concept is worthy of exploration. It is true in a broad sense that you need to measure the current state of a process to obtain the information required to select appropriate system adjustments. However, numbers alone are not sufficient to make OEHS decisions. They must be evaluated in the context of knowledge of the organization and professional judgment.
The metrics selection process needs to ensure that limited available resources are applied to measuring actionable areas of operations. If change is not likely to be required in a given process, there is limited value in measuring its performance. However, it is common for OEHS programs to report on areas where conditions are under control and performance exceeds goals and industry comparisons. While confirmation that a process is performing within acceptable parameters is initially important and may require ongoing monitoring, reports that identify program gaps are more likely to provide useful information for management.
When a measure is included in program reporting to generate a positive impression of program performance without addressing a control need or real concern, it can be termed a “vanity” or “performative” metric. These non-value-added metrics are often used for promotional reasons to either obfuscate real issues or imply performance that exceeds reality. Most OEHS programs have many elements that, if viewed in isolation, appear to exceed performance expectations. A facility might report a low accident rate without specifying that they omitted accidents that did not meet their definition of reportability. A press release might report results that indicate chemical use has been drastically reduced without noting that the reduction was based on sales dollars, a deceptive comparison given increases in unit prices.
Inappropriate use of denominators with limited relationship to operational performance can mislead an audience. Examples are often financial in nature but could also involve elements such as work hours that include time without potential for exposure to occupational risk or using data sets that selectively exclude elements that do not support the desired result. All these choices have the potential to violate section II(A)(1) of the BGC Code of Ethics (PDF), which requires that a CIH deliver competent services with objective and independent professional judgment in decision-making, and section II(A)(5), which requires truthful and accurate representations to the public in advertising, public statements, or representations.
Numbers alone are not sufficient to make OEHS decisions. They must be evaluated in the context of knowledge of the organization and professional judgment.
Sometimes this deceptive reporting is the result of individual biases that affect how data is collected and interpreted. Every individual is biased, often unintentionally, in some form or fashion. Many types of cognitive bias are of OEHS interest, including the following:
Anchoring bias: focusing on one or a small number of data points when making decisions. An example is deciding that seasonal allergies caused a rash of eye irritations as had been identified in the past despite reports of new chemical irritants.
Apophenia: perceiving relationships between unrelated things. Mistaking correlation for causation is an example of apophenia.
Availability heuristic: assigning greater weight to what is already known or easiest to remember, such as assuming an employee’s sunburn was due to misuse of provided sunscreen without exploring potential photosensitizers.
Cognitive dissonance: the mental burden of reconciling contradictory data. In other words, people find it difficult to act contrary to their beliefs for an extended period without adjusting those beliefs. This can lead, for example, to avoidance of emergency planning. However, it can also be used for positive benefit in an OEHS program. For example, when eye protection is consistently required, wearing it will become a habit over time even if there is strong initial resistance.
Confirmation bias: assigning greater weight to information that supports one’s preconceptions. This bias is related to “false priors” where stereotyping is a concern. An example of confirmation bias is only testing known contaminants in the workplace without surveying all potential sources.
Egocentric bias: valuing your own perspective and knowledge above all others. For example, OEHS professionals sometimes allow their education and training to lead them to inappropriately dismiss line workers’ knowledge in analyzing an OEHS issue. It is also related to the Dunning-Kruger effect, where unskilled individuals overestimate their knowledge and abilities. This is a particular OEHS concern with the rise of AI and internet-based pseudo-expertise.
Extension neglect: assigning inappropriate relevance to analysis of small sample sizes. This issue is particularly relevant to OEHS, where small sample sizes are common. Statistical methods can offer better analysis of this data, but the result will never absolutely reflect reality.
Framing effect: drawing different conclusions from the same data based on how it is presented. A common example in OEHS is presenting data in a narrow band (that is, not starting at zero) or on a different scale (for example, log-normal) to imply the greatest variability.
Logical fallacy: making interpretations not supported by facts, such as assuming that the numerical result of a metric is all that is required to assess a real-world concern.
Other potential biases include objectivity bias, an individual’s belief that they are more objective or less biased than others. Many biases can be minimized by using diverse teams to develop and review metrics programs. In this context, diversity includes not only cultural and ethnic elements but also age, experience, education, and occupation.
To reduce the potential for misuse, metrics programs should be developed in conjunction with well-established goals and reporting parameters determined prior to the outcomes being calculated. However, the reporting parameters should be flexible enough to allow analysis and communication of unexpected results. Using commonly available algorithms for calculations enables comparison of the results to similar operations and avoids selection of only positive results. More broadly, data transparency is key to eliminating misuse. If all appropriate data is available to interested stakeholders, they can both validate the reported information and perform their own, potentially more useful, calculations.
CASE STUDY: INJURY-RATE DILEMMA
Hector is a long-standing employee of Overseas Inc. (OI), a services business with employees stationed around the globe. He is a corporate manager responsible for the industrial hygiene and safety program for the global workforce. OI has just become aware of the opportunity to bid on a very lucrative contract that has a significant EHS component. One of the requirements is that bidders must have an injury rate less than one to even submit a proposal.
Hector is concerned that, while OI has few on-the-job accidents, a high number of off-the-job injuries have occurred in OI-supplied housing. The nature of their work requires that the employees are covered 24 hours a day by the OI safety and health program, and some of their contracts require reporting on that basis. Historically, OI’s public reporting has touted a low injury rate based on traditional eight-hour workdays.
For discussion: What ethical issues does Hector face in deciding how to complete his portion of the proposal? Should previous reporting have included some sort of caveat regarding the varying coverage? How should this dilemma be communicated to management?
NicoElNino/Getty Images
JIHEEC: Promoting Ethical Practice
The Joint Industrial Hygiene Ethics Education Committee (JIHEEC) promotes awareness and understanding of the enforceable code of ethics (PDF) published by the Board for Global EHS Credentialing (BGC) as well as the ethical principles of AIHA and ACGIH.
JIHEEC includes representatives from all three organizations. JIHEEC is not an enforcement body or resolution board. It serves the profession by bringing attention to ethical dilemmas and challenges potentially encountered by industrial hygienists and other OEHS professionals.