Synoptic reporting accuracy for computed tomography pulmonary arteriography among patients suspected of pulmonary embolism

Abstract

Background

Structured reporting is an efficient and replicable method of presenting diagnostic results that eliminates variability inherent in narrative descriptive reporting and may improve clinical decisions. Synoptic element reporting can generate discrete coded data that then may inform clinical decision support and trigger downstream actions in computerized electronic health records.

Objective

Limited evidence exists for use of synoptic reporting for computed tomography pulmonary arteriography (CTPA) among patients suspected of pulmonary embolism. We reported the accuracy of synoptic reporting for the outcome of pulmonary embolism among patients who presented to an integrated health care system with CTPA performed for suspected pulmonary embolism.

Methods

Structured radiology reports with embedded synoptic elements were implemented for all CTPA examinations on March 1, 2018. Four hundred CTPA reports between January 4, 2019 and July 30, 2020 (200 reports each for which synoptic reporting recorded the presence or absence of pulmonary embolism [PE]) were selected at random. One non‐diagnostic study was excluded from analysis. We then assessed the accuracy of synoptic reporting compared with the gold standard of manual chart review.

Results

Synoptic reporting and manual review agreed in 99.2% of patients undergoing CTPA for suspected PE, agreed on the presence of PE in 196 of 199 (98.5%) cases, the absence of PE in 200 of 200 (100%) cases with a sensitivity of 87.6% (76.1–96.1) a specificity of 99.9% (99.7%–100%), a positive predictive value of 99.5% (98.1–100), and a negative predictive value of 98% (95.7%–99.5%).

Conclusion

The overall rate of agreement was 99.2%, but we observed an unacceptable false‐negative rate for clinical reliance on synoptic element reporting in isolation from dictated reports.

INTRODUCTION Background

A Radiology reports are commonly structured with a narrative summary of observations followed by a section of interpretation and summative remarks.¹ This form of reporting allows for variability in the reporting of structure and content, introduces risks which include inadvertent omission, an inadequate emphasis on clinically relevant findings, and misunderstanding of report contents by clinicians.^2–4 These risks are amplified in high-volume clinical settings, like emergency departments (ED) or urgent care clinics. Among surgical reports, non-standardized narrative reporting includes non-essential information up to 80% of the time and may fail to include clinically essential information up to 70% of the time.⁵

Importance

Structured reporting's benefits include standardized ordering of observed findings⁶ improved efficiency for the individual interpreting the report,^2,7 a reduction in the probability that the radiologist might inadvertently omit a key descriptor, and improved completeness in reporting findings observed.⁸ A structured report assures that critical information about both the procedure and the patient history are included with efficiency and accuracy,⁹ acknowledging that limitations exist in the number of variables that humans can process.¹⁰

Synoptic reporting permits data analytics to provide clinical decision-making using codable data generated from the radiology report. Synoptic reporting can generate discrete data elements that may be embedded in clinical decision support (CDS) systems to facilitate autonomous function. Discrete coded data elements also facilitate quality improvement, auditing, coding, and other activities.

Adoption of synoptic reporting in radiology^11,12 has included its use in computed tomography pulmonary arteriography (CTPA) reports among patients suspected of pulmonary embolism.^13,14 However, description of the quality of synoptic reporting for CTPA to identify pulmonary embolism (PE) represents an unmet need.^15–17 PE is the third most common cause for cardiovascular death,¹⁵ accounting for 60,000–100,000 deaths in the United States annually.¹⁶ Delayed treatment of acute PE can lead to poor clinical outcomes, and pre-emptive anticoagulation for patients in whom a delay in diagnostic testing is anticipated has been suggested.^18,19 Synoptic element reports can generate discrete coded data that can be used to trigger clinical alerts, improve the time to PE diagnosis, and perhaps decrease delay in initiating treatment. However, to assure that synoptic reporting can accurately report PE diagnosis in the emergency department the test characteristics must be determined.

Goal of this investigation

The goal of this investigation was to describe the performance of synoptic reporting compared to traditional reporting methods for the ascertainment of PE when CTPA is performed in the ED among patients suspected of PE.

METHODS

This retrospective observational cohort study was conducted in Intermountain Healthcare, a vertically integrated health care system in Utah and Idaho with more than 550,000 ED visits annually that employs more than 10,000 nurses, 2400 physicians and advanced-practice clinicians, and another 3800 affiliated physicians and advanced-practice clinicians. The system's 24 hospitals include a teaching and referral hospital, 3 regional referral hospitals, 9 urban and suburban community hospitals, an orthopedic specialty hospital, 8 rural community/critical access hospitals, a pediatric specialty hospital, and a virtual tele-hospital, providing care for patients physically residing in one of the other hospitals or alternate care venue. The study was approved by the institutional review board that waived the requirement of informed consent. No funding was provided.

The Bottom Line

Structured synoptic data reporting offers many appealing features for efficiently communicating findings of emergency department radiologic tests. In this series of 200 chest computed tomography pulmonary arteriography scans, agreement between synoptic and manual reporting was high (92%), but a false-negative report occurred in 3 cases. Despite its promise, implementation of synoptic reporting must be done with caution.

Study design and setting

Structured reporting of radiologist reports with embedded synoptic elements was adopted as a clinical standard of care at all 23 hospitals on March 1, 2018.

Selection of participants

An electronic random number generator was used to randomly select 200 patient encounters that occurred between January 4, 2019 and July 30, 2020 for which synoptic reporting identified PE, and 200 patient encounters for which synoptic reporting identified the absence of PE. We chose 200 encounters for each group because of practical considerations given the resources available. Furthermore, confidence intervals (CIs) on proportions based on 200 charts are estimated to have at most a margin of error of 7 percentage points (when the true percentage was far from 100% and precision is of less interest). With true percentages near 100% the CIs have margins of error near 1 percentage point or lower.

Synoptic reporting protocol

Radiologists were required to select one or more of the following discreet coded data fields on every CTPA report: (1) acute PE present, (2) chronic PE present, (3) PE absent, (4) limited, but no PE identified at the segmental level or above; or (5) non-diagnostic. If the radiologist identified acute PE as present, then requisite fields included selecting the anatomic indicators of PE being present in the (1) main, (2) segmental lobar or central, or (3) only subsegmental vasculature, to indicate the most proximal thrombosis present. If PE was present, the radiologist was required to report the right ventricle to left ventricle ratio (RV:LV) ratio as ≥1.0 or <1.0 with a free text field to voluntarily enter the integers. To facilitate efficiency for the radiologist, the synoptic report field for PE was defaulted to “PE absent.” Radiologists also recorded free verse dictation including a clinical impression as standard of care for all patients.

Measurements

Manual chart review was performed by 2 investigators (I.A.W. and S.C.W.) with extensive research experience conducting manual chart review of CTPA and venous studies that has led to former publications^20,21 in reporting accuracy of computer decision support to assess for PE. Agreement of the 2 reviewers was necessary for each event, and in the setting of a discrepant interpretation a third investigator (J.R.B.) was designated to adjudicate any disagreement. A κ coefficient of agreement among the reviewers was calculated.

Outcomes

The primary outcome was agreement between the synoptic element report and the radiologist-free verse dictated report upon manual chart review. The PE outcome was binary for each encounter and was reported as either present or absent. Studies that were identified as “Limited, but no PE identified at the segmental level or above” were analyzed as “PE absent” and included for analysis. Studies in which the synoptic report indicated the CTPA was determined to be non-diagnostic by the interpreting radiologist were not included for analysis. The secondary outcome was agreement in anatomic location of PE between the synoptic report and the radiologist-free verse dictated report for patients in which PE was present.

Analysis

To calculate the negative and positive predictive values of synoptic reporting we ascertained the prevalence of PE from our health care system among imaging ordered for suspected PE which was found to be 8.7%, as we formerly described.²⁰ We combined prevalence information with the results of the chart reviews and a full Bayesian analysis was used with a simplex of the 4 population proportions (proportions true and false-positives and negatives) as the parameters of interest (Dirichlet [1,1,1,1] prior) and binomial likelihoods based on the observed prevalence and results of the chart reviews. Parameter estimates and 95% credible intervals were computed from the posterior distributions of the transformed parameters. Calculations were done using the Stan program²² and R Stan package for R.²³

RESULTS

Synoptic reporting matched manual chart review in 200/200 (100%) studies in which synoptic reporting indicated PE absent, and in 196/199 (98.5%) studies in which synoptic report indicated the presence of PE, yielding an overall agreement of 99.2% (396/399). Agreement between the 2 reviewers of the 400 reviewed reports for the presence and absence of PE yielded a κ = 1.0. Therefore, when compared with manual chart review for the outcome of thrombosis synoptic reporting had a sensitivity of 87.6% (95% CI, 76.1–96.1), a specificity of 99.9% (95% CI, 99.7–100), a positive predictive value of 99.5% (95% CI, 98.1–100), and a negative predictive value of 98.0% (95% CI, 95.7–99.5). In studies with PE present the location of the most proximal thrombus reported by synoptic reporting matched manual chart review for 200/200 (100%) of the studies (Table 1). The radiologist dictated narrative report for 3 studies in which manual chart review was discordant with the synoptic element report did accurately capture the PE diagnosis and these texts are reported in Table 2.

TABLE 1 Classification of pulmonary embolism by vasculature and reported outcome

PE	No. of scans	Rate of agreement (%)
Positive
Chronic	24	24/24 (100)
Main	17	17/17 (100)
Segmental	121	121/121 (100)
Subsegmental	38	38/38 (100)
Total	200	200/200 (100)
Negative*
Limited	10	10/10 (100)
No PE	188	185/188 (98.4)
Total	199	196/199 (98.5)

Abbreviations PE, pulmonary embolism.

*Limited quality of 1 study disallowed interpretation.

TABLE 2 Discordant synoptic element report events

Synoptic element reporting	Narrative summary impression	Error type
FINDINGS: “No evidence of pulmonary embolism.”	IMPRESSION: “Significant pulmonary emboli involving upper and lower lobes bilaterally and extending as far proximally as the proximal left pulmonary artery.”	False–negative
FINDINGS: “No evidence of pulmonary embolism.”	IMPRESSION: “Pulmonary embolism is confirmed.”	False–negative
FINDINGS: “No evidence of pulmonary embolism.”	IMPRESSION: “Extensive pulmonary embolus seen in the lungs right greater than left.”	False–negative

LIMITATIONS

We chose the radiologist-free text dictation as the gold standard for PE outcomes in our study. Although this is the clinical gold standard at most institutions, overread of every CPTA by an expert panel was not thought to be feasible and would have made our study less generalizable. Potential disadvantages of synoptic reporting include reports of clerical errors associated with data elements in certain transcription settings,²⁴ which were also true in our study. Human readers may easily overlook terms such as “no” and “not” so phrasing of reports must be carefully constructed.²⁵ Synoptic element reporting accuracy is dependent on programming and validation to assure a process that reduces systematic errors.

DISCUSSION

Synoptic reporting could enhance clarity, accuracy, safety, and satisfaction of radiology reporting. However, synoptic element reporting is not well validated and is currently rarely assimilated in radiology workflows. To our knowledge, this is the first study reporting the accuracy of synoptic reporting for CTPA among patients who present to the ED who are suspected of PE. We found that synoptic reporting, when compared with manual chart review, yielded an agreement of 99.2% with a sensitivity of 87.6% (95% CI, 76.1–96.1), a specificity of 99.9% (95% CI, 99.7–100).

Certainty must exist that synoptic reporting accurately captures and reports the findings of the radiologist. Furthermore, although a burgeoning body of evidence exists for the value of synoptic reporting aiding in the domains of speed, satisfaction, and confidence by the consumer, recent research²⁶ suggests that quality evidence is lacking regarding accuracy. This observation makes our manuscript even more important.

Our results are encouraging that synoptic element reporting could be implemented for pulmonary embolism resulting, yet we observed 3 instances where, in the same report, a discordant result was reported. In all 3 cases when the synoptic element reported “PE absent” the narrative portion of the radiology report communicated that PE was present (Table 2). Our study was not designed to determine the reasons for discrepancy. As part of a workflow accommodation with the introduction of mandatory synoptic reporting, it was elected to default the field for PE to “PE absent,” with the intention of introducing efficiency for the radiologist, given that the absence of PE is approximately 10 times more likely for any given study. We hypothesized that this discordant reporting would be resolved if the defaulted result of “PE absent” in the synoptic coded data cell be removed and the radiologist be required to select the correct synoptic element for each study. No other errors in the performance of synoptic element reporting accuracy were observed, and because of feedback associated with this study the default setting is no longer embedded in radiology workflow.

Our experience serves as a cautionary tale regarding the balance of enhancing efficiency and workflow with assuring that processes for efficiency do not inadvertently introduce systematic errors, contradictory statements, or confusion in radiology reporting. Future studies could focus on the effects of eliminating default settings, optimal timing of synoptic reporting, before or after radiologist free verse dictation, or use of synoptic reporting embedded within decision support tools.

Our synoptic reports were highly accurate for the anatomic location of PE present. Benefits associated with this knowledge may include patient triage to optimize the setting in which care would be given. For example, if the most proximal aspect of a clot burden is isolated in the subsegmental circulation and certain parameters are met, then concomitant messaging could include refraining from anticoagulation as some guidelines recommend.¹⁸ In the circumstance of large volume central PE being identified, the ordering physician could be alerted to consider activating a pulmonary embolism response team (PERT) that could expedite optimal care beginning in the ED. This information might inform long-term follow-up including maintaining a heightened vigilance for the development of pulmonary arterial hypertension.²⁷ Therefore, a simple, reliable, automated, and accurate mechanism to search for coded data field elements that report the anatomic distribution of PE has the potential to improve ongoing care of patients.

The use of discrete coded data elements could significantly improve the reporting of outcomes for CTPA. Most health care systems report PE incidence and outcomes based on the International Classification of Diseases codes that have been demonstrated to be inaccurate,²⁸ however, synoptic reporting may improve the accuracy of these reports. Synoptic report data may also be surfaced in real time to inform clinical decision support embedded into the electronic health record and speed decision making for treating clinicians. This is impactful in the reporting of CTPA given that delay in initiating anticoagulation may lead to increased PE mortality²⁹ and because anticoagulation is rarely initiated expeditiously as guidelines recommend.¹⁹ Synoptic report data could be used to alert clinicians to the presence of PE and reduce treatment delay. Additional benefits to clinical research projects and to evaluation of technical factors impacting diagnostic accuracy (eg, percentages of limited or non-diagnostic scans) also are made possible by synoptic reporting.

We found that synoptic reporting for the outcome of CTPA among ED patients suspected of PE in an integrated health care system had high correlation with radiologist dictated reports. Given our findings, we advise against defaulted data fields that may lead to error or confusion. Additional study is needed to validate these findings prospectively and refine optimization of synoptic reporting in radiology.

AUTHOR CONTRIBUTIONS

The authors declare that they had full access to the data in this study and they take complete responsibility for the integrity of the data and the accuracy of the data analysis. JRB takes full responsibility for the integrity of the manuscript in it's entirety. Concept and design: JRB, IAW, SCW, SMS, KEC, BHG. Analysis, acquisition, or interpretation of data: JRB, IAW, SCW, JFL, GLS, PJ. Drafting of the manuscript: IAW, JRB, SCW. Critical revision of the manuscript: IAW, SCW, SMS, JFL, KEC, BHG, GLS, PJ, JRB.

CONFLICT OF INTERESTS

The authors declare no conflicts of interest.