A holistic evaluation of sexual health disease investigation: case study of the 2022 Mpox outbreak in Santa Clara County, California

Background

Disease investigation (DI) services play an integral part in preventing and controlling infectious diseases within local health departments. These services, which can include identifying new cases, eliciting contacts, and facilitating treatment, can positively improve health outcomes. Despite this significance, there is limited research aiming to evaluate health department DI services, particularly in the realm of sexually transmitted infections (STIs). In 2023, the California Department of Public Health presented an evaluation of syphilis DI using a composite measure that incorporated several individual process outcomes to account for the various steps disease investigators undertake while completing an investigation. Examples of these process outcomes included the number of cases referred to syphilis partner services and the number of partners treated through partner services. Their findings revealed the potential of each investigation to yield multiple outcomes to better represent the impact of a multi-step investigation process. This holistic approach to evaluation may similarly illuminate the impact of DI services for other STIs, especially those that have emerged and become more prevalent in recent years and have therefore required focused public health efforts to understand, monitor, and control the spread of disease.

Mpox is a viral zoonotic disease that emerged as a growing public health concern in 2022 in the United States,¹ where top case counts were recorded, especially in the state of California.² Accurate diagnosis of Mpox poses challenges because of the similarity of symptoms to other STIs such as syphilis and herpes virus.³ The complexities of Mpox underscore the importance of meticulous and effective DIs to ensure immediate treatment initiation and ongoing monitoring for those infected. To our knowledge, there are no studies that have evaluated the impact of DI to prevent or control the spread of Mpox. Therefore, this evaluation aims to understand the range of outcomes achieved through DI services using the 2022 Mpox outbreak as a case example.

Methods

Study measures

A composite measure that included multiple individual Mpox DI processes was constructed. A composite measure is an aggregate evaluation metric that combines multiple individual indicators into a single, more comprehensive measure for a holistic evaluation of any particular intervention.^4–6 This approach to evaluation can be useful to analyze performance and can further be used to understand the full impact of resources that are designated to the intervention that is being evaluated.^4–6 The Mpox composite measure comprised the following process indicators: (1) treatment completion, (2) monitoring completion, (3) partial vaccine dose completion, (4) full vaccination series completion, and (5) STI and HIV testing. These measures were purposefully selected given their key and integral roles in completing overall case and contact investigations for local Mpox DI. These measures and their definitions are described below.

Treatment completion

This indicator was defined as the successful completion of a 2-week duration of tecovirimat. Tecovirimat was generally prescribed for severe Mpox cases.

Monitoring completion

All Mpox case and contact investigations required symptom monitoring for a period of 21 days or an attested status of self-monitoring for those investigations determined to be low risk. An investigation was considered to have been completed if the status of the case or contact was designated as either ‘completed isolation’ or ‘complete, no monitoring needed’. An investigation outcome of ‘complete, no monitoring needed’ was recommended for contacts who were classified as low risk and did not need active monitoring by a member of the DI team.

Partial vaccine dose completion

This indicator was defined as contacts having received one dose of the Jynneos vaccine.

Full vaccination series completion

This indicator was defined as contacts having received two doses of the Jynneos vaccine.

STI and HIV testing

If the case or contact successfully completed testing for either STI and HIV, where chlamydia, gonorrhea, and syphilis were the recommended STI tests, they were considered to have received STI and HIV testing.

Each of these individual measures was included in the construction of the overall composite score and select measures, as outlined in Fig. 1, were included in the composite score calculation for both case and contact investigations.

Fig. 1.

Individual process indicators by type of investigation.

Results

A total of 214 case and 272 contact investigations were conducted, which led to 924 outcomes achieved: 402 and 522 outcomes for cases and contacts, respectively. The overall Mpox composite score was 0.53, which indicated that each investigation led to 1.90 outcomes achieved. For case investigations, 0.53 investigations led to one achieved outcome, meaning there were 1.88 outcomes per investigation. For contact investigations, 0.52 investigations led to one achieved outcome, meaning there were 1.92 achieved outcomes per investigation.

This same approach was further applied to contextualize each process outcome’s contribution to the overall composite score. Of the total achieved outcomes, 44% represented monitoring completion, and 22% represented partial and full-dose vaccine completions (Table 2). Among the 99 cases eligible for treatment, 2.16 case investigations yielded one treatment completion; 1.19 case and contact investigations yielded one monitoring completion; 2.21 contact investigations yielded one partial vaccine completion; 3.53 contact investigations yielded one full vaccination series completion; and 2.25 case and contact investigations yielded one STI or HIV test. Table 2 provides a breakdown of each achieved outcome by individual process indicators.

Discussion

The monitoring completion process indicator, which represented 44% of the achieved measured outcomes, coupled with treatment completion, which represented 11% of total achieved outcomes, showed that more than half (55%) of all achieved outcomes represented a successful end result. Perhaps this representation would be even higher if outcome data from cases and contacts that became lost to follow-up could be included. The percentage of contact investigations yielding partial or full vaccination completion was 39%, which shows that although a high proportion of contacts were amenable to being vaccinated, a greater proportion did not obtain a vaccine through DI services; reasons for this may include that there were individuals who were already vaccinated at the time of intake or were not considered medium or high risk, they had a low risk perception for Mpox, or they were hesitant to get vaccinated for other reasons.⁷ Even though STI and HIV testing was highly recommended for anyone who may be at greater risk for acquiring these infections,³ only 23% of all achieved outcomes represented completion of this measure. Reasons for this could be attributed to known barriers to testing^8,9 or patients testing at a location not tracked within the clinic system that was evaluated.

There are no other studies, to our knowledge, that report evaluation results pertaining to Mpox DI outcomes. Historically, success of DI services for STIs such as Mpox has been measured solely by assessing treatment completion status, implying that 100% of all weight is placed on this process indicator without accounting for other key process outcomes that may be achieved during DI. Although this is the case for most other STIs being investigated, treatment was not always recommended for all cases of Mpox, so it became important to include monitoring completion status as an outcome measure when constructing the composite measure. However, even if the traditional approach to evaluation was applied in this case study, with treatment completion and monitoring completion being the only outcome variables, it can be seen from Table 2 that these two variables would only represent 55% of the total outcomes achieved, and the remaining 45% would not have been captured in the evaluation.

In recent years, composite measures have more widely been used to measure healthcare quality and performance.^4–6 In our study, we demonstrated how applying this evaluation approach to construct a composite measure for one disease area can be done through purposive selection of individual measures, which can easily be modified when evaluating DI in other disease areas. In contrast, the use of single or individual measures for evaluation typically assesses each individual measure independently, which can provide a more granular assessment of outcomes achieved in an intervention. As such, though individual measures offer insight into a single indicator, they cannot be used to provide a holistic and overall understanding of several indicators. These single outcomes, if combined, can show the big picture of the effectiveness of an intervention with the ability to gain granular, individual indicator–level detail as needed. Thus, incorporating a composite measure for a process evaluation, such as the one seen within this case study, can help to better align the focus of DI processes to ensure the completion of all aspects of DI in comparison to only focusing on treatment completion which, from the patient perspective, may not necessarily provide insight into the extent to which DI helped prevent an individual from acquiring the STI again in the near or long-term future.

Limitations include the limited ability to select additional individual process indicators for inclusion in the overall composite measure because of unavailable or incomplete data. This presents a key opportunity area to strengthen data collection systems through future quality improvement efforts in order to inform a better understanding of intervention impact. Other indicators that could have been included are HIV PrEP or PEP initiation and linkage to HIV treatment and care. Also, as this evaluation was conducted in the setting of an emergency response, worker burden can serve as a limitation in fully completing each process step. Studies have shown that in the emergency response setting, epidemiologic responders face challenges such as lack of role clarity, limited knowledge sharing, and lack of communication.¹⁰ This limitation also illustrates the importance of applying this evaluation approach in non-emergency settings to better understand and improve routine DI operations.

Conclusion

The Mpox DI team achieved several positive outcomes during the local emergency response. The composite measure served as an integrative tool that assessed different aspects of Mpox DI. By measuring trends within individual outcomes, public health departments can identify the strengths and weakness of their DI processes to shape program funding and protocols, and to improve DI processes. Health departments are encouraged to establish clear outcome definitions using a composite measure to evaluate the impact of their DI services.