Evaluation of Sensitivity and Specificity of Acuity 360 Telemedicine Vision Screening System

Abstract

Purpose: Our aim is to evaluate the sensitivity and specificity of Acuity 360 telemedicine system, as compared to in-person clinic examination, in identifying clinically significant eye disease. Acuity 360 is a combination of commercially available ocular imaging devices used together to provide a comprehensive evaluation of the structures and diseases of the eye.

Methods: Observational cross-sectional study of consecutively examined patients where 19 remote examiners analyzed 80 patients using Acuity 360 images. The examiners’ diagnoses were compared to the diagnosis obtained from in-person clinic examination of all the patients. The sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), false-negative rate, and inter-rater agreement were calculated.

Results: Compared to in-person clinic examination, the sensitivity, specificity, PPV, NPV, and false-negative rate for the optical coherence tomography (OCT) of optic nerve were 95.8%, 98.5%, 86.8%, 99.6%, 0.39% and for the combined retina examination (OCT of the macula and widefield fundus photography) were 93.4%, 88.5%, 80.2%, 96.4%, 2.19%, respectively. The median inter-rater agreement the OCT of optic nerve and the combined retina examination were each 95%.

Conclusion: The Acuity 360 telemedicine system has a low false-negative rate and is highly sensitive and specific when compared to an in-person clinic examination. It can determine the necessity for specialist referral and triage the patients that require urgent treatment. High inter-rater agreement shows that it is effective with minimal variability in analyzing the Acuity 360 images by remote examiners.

Introduction

Several studies have established the validity and feasibility of tele-ophthalmology screening in identifying eye disease states such as diabetic retinopathy,^1–3 glaucoma,^2,4,5 retinopathy of prematurity,² age-related macular degeneration,^2,3 choroidal nevi and iris nevi.⁶ Tele-ophthalmology can support the health caregivers in effectively managing and preventing diabetes and its complications.⁷ Tele-ophthalmology has also shown high patient satisfaction levels and acceptance because of the convenience, no need for dilation of the pupils, reduced examination time, and reduced traveling time and cost.^8,9 We believe that newer and improved diagnostic imaging modalities compared to routine telehealth screenings can increase the practicality and accessibility of tele-ophthalmology services.

After multiple consultations with physician focus groups, we have identified a group of diagnostic tests, which, we believe, can accurately diagnose specific eye diseases/conditions. We have combined them into the Acuity 360 telemedicine vision screening system.

The diagnostic tests included in Acuity 360 telemedicine system are: Optical coherence tomography (OCT, Heidelberg Spectralis™) of the optic nerve, macula (including OCT angiography) and anterior segment, slit lamp video, and widefield retinal imaging (Optos™). These are in addition to checking visual acuity, intraocular pressure, color vision, and stereovision. There is no risk or radiation exposure to the patient while performing Acuity 360 imaging and the entire process takes around 30 minutes. All of the Acuity 360 tests are non-invasive and do not require pupillary dilation or physical contact with the patient’s eye. It has been documented that non-mydriatic testing can accurately detect retinal pathology with high sensitivity and specificity.^10,11

The collective evaluation of this group of tests in a tele-screening modality would be appropriate and comprehensive in screening the patients for clinically significant eye diseases that include diabetic retinopathy, hypertensive retinopathy, macular hole, macular degeneration, macular pucker, retinal holes or detachments, choroidal or iris nevi, shallow anterior chamber angles, and chronic open angle glaucoma.¹²

The primary objective of the study is to evaluate the sensitivity and specificity of Acuity 360 telemedicine examination as compared to in-person clinic examination in determining any clinically significant eye diseases and the necessity for specialist referral. The secondary objective is to assess the inter-rater agreement and inter-rate reliability among the clinicians to determine the variability in analyzing the Acuity 360 images.

Methods

We conducted an observational cross-sectional study of consecutively examined patients. The above-described panel of tests constituting the Acuity 360 system (Acuity 360™) were used, between July 2019 and November 2019 at four telemedicine offices in Southern California operated by the Acuity Eye Group, to acquire the eye images of patients with/without eye diseases. The patient history, visual acuity, intraocular pressure (by iCare™ IC100), color vision (monocular Ishihara),¹³ slit lamp photos and OCT of the anterior segment, and five of the imaging components of the telemedicine examination (widefield retinal imaging and OCT of the cornea, angles, macula, and optic nerve) conducted by ophthalmic technicians were sent to remote examiners. All examiners were blind to the interpretation of other examiners and to the in-person clinic examination of all patients.

Examiners reviewed the images, evaluated the abnormal/normal state of eye structures, and diagnosed the presence/absence of certain eye diseases. Guidelines were provided to the examiners on what constituted clinically significant/insignificant diseases and how to give the ratings (Appendix A).

During the analysis, examiners chose either of the three options: 1) continued yearly screening in telemedicine clinic (no clinically significant disease), 2) refer for an in-person examination with an optometrist or ophthalmologist (clinically significant disease present), 3) poor image quality. A fourth option “Refer for Humphrey visual field (HVF)” was also given for the OCT of the optic nerve. If HVF turns out normal, examiner recommended continued yearly screening and if it was abnormal, examiner recommended in-person examination. HVF option was added as sometimes OCT of the optic nerve alone cannot differentiate between glaucoma suspects and patients with early glaucoma.

Out of a total of 19 examiners included for evaluation, eight were optometrists (42.1%), five were general ophthalmologists (26.3%), four were glaucoma specialists (21.1%), and two were retina specialists (10.5%).

Similarly, for the widefield retinal imaging, fourth option was “Artifact/peripheral lesion of unknown significance”. Sometimes small shadows/image artifacts are present in an Optos photograph for which some examiners may have chosen to continue yearly screening in the telemedicine clinic and others may have referred for an in-person examination depending on the examiner’s comfort level. This fourth option was meant to separate the observations of image artifacts from those having true disease.

All patients included in the study had an in-person clinic examination within six months either before or after their telemedicine exam. The remote examiners’ determination of presence/absence of clinically significant disease were compared to the patient’s in-person examination to assess sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and false-negative rate.

The responses for widefield retinal imaging and OCT of macula were grouped into a “Combined retina examination” for purposes of statistical analysis. This was done specifically because OCT of the macula or widefield retinal imaging in isolation cannot give a full picture of clinically significant retinal disease, but together they can be used as an assessment that approaches (or surpasses) in-person fundus examination at the slit lamp. The other imaging modalities, OCT of optic nerve and OCT of the angles, were analyzed individually.

Raw data was captured, and statistical values were analyzed in Microsoft Excel. Contingency tables were prepared for the imaging modalities and the sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), false-negative rate, and inter-rater agreement were calculated. All the examiners’ responses stating “poor image quality” and “artifact/peripheral lesion of unknown significance” were excluded from statistical analysis. If any image was reported as “poor image quality” by >50% of the examiners, then that image was excluded completely from the analysis.

Data Summary and Technical Validation

Out of 121 consecutive patients, 41 patients did not have in-person clinical examination within six months either before or after their telemedicine exam. Therefore, the remaining 80 patients were included for study analysis. There were 22 examiners. Three of them analyzed <20 patients and their responses were excluded from analysis. Two of the remaining 19 examiners analyzed 60/80 patients and their responses were included only for those 60 patients. The remaining 17 examiners analyzed all the 80 patients. This gave a total of 1480 unique remote examinations (17 examiners x 80 patients + 2 examiners x 60 patients).

For the combined retina examination, 20 (1.3%) remote examinations were excluded due to “poor image quality” responses leaving a total of 1460 examinations. For OCT of the optic nerve, a significantly higher number of 446 (30.1%) examinations had to be excluded due to “poor image quality” responses leaving 1034 for analysis. For OCT of the angles, 184 (12.4%) remote examinations were excluded due to “poor image quality” responses leaving 1296 examinations for analysis.

Out of 80 patients analyzed, 46 patients (57.5%) did not have clinically significant eye disease and 34 patients (42.5%) did have clinically significant eye disease, according to the in-person clinical examination.

Cornea data was collected but not analyzed as there was no clinically significant corneal pathology present in this group as determined by both the in-person examination and the remote examination.

Compared to in-person clinic examination, the sensitivity, specificity, PPV, NPV, and false-negative rate obtained for the OCT of optic nerve and for the combined retina examination are shown in Table 1 and Table 2, respectively.

Table 1 Contingency Table and Statistical Values for the Optical Coherence Tomography of the Optic Nerve

Table 2 Contingency Table and Statistical Values for the Combined Retina Examination

The OCT of angles showed a specificity of 86.7%. The sensitivity, PPV, NPV, and false-negative rate could not be calculated for OCT of angles as all the instances of in-person clinic examinations were normal in this test.

Figure 1 shows the box and whisker plots of the inter-rater agreement and Table 3 shows the percentage of inter-rater agreement for the OCT of angles, the OCT of optic nerve and the combined retina examination.

Table 3 Inter-Rater Agreement for the Imaging Modalities Used in Acuity 360 Vision Screening System

Figure 1 Box and whisker plots of the inter-rater agreement of the imaging modalities used in Acuity 360 vision screening system. OCT: optical coherence tomography. The top and bottom of the boxes are the 75th and 25th percentiles of the data. There are no top whiskers as third quartile is equal to the maximum data value in all the three imaging modalities. The line drawn through the middle of the box is the median (50th percentile). The box contains the middle 50% of the data. The length of the box is the inter-quartile range. Bottom whiskers indicate the minimum data values. Round dots are the outliers.

Discussion

The main objective of the study was to evaluate the sensitivity and specificity of Acuity 360 telemedicine screening system, as compared to in-person clinic examination, in determining any clinically significant eye diseases and the necessity for specialist referral. Our results showed that the OCT of the optic nerve and the combined retina examination, used in the Acuity 360 system, have very low false-negative rates of 0.39% and 2.19%, respectively. Hence, very few patients with clinically significant disease would be missed by using these imaging modalities in isolation. Compared to in-person clinic examination, the sensitivity and specificity of the OCT of optic nerve (Table 1) were 95.8% and 98.5%, respectively. For the combined retina examination (Table 2) the corresponding values were 93.4% and 88.5%, respectively.

In-line with other studies, which reported high level of clinical accuracy in diagnosing eye diseases,^{4,8,9,14–18} Acuity 360 telemedicine system showed very low false-negative rates and high sensitivity and specificity as compared to in-person clinic examination in its ability to diagnose clinically significant eye diseases. These results suggest that the Acuity 360 system can provide valuable tele-ophthalmology services by identifying patients that require specialist consultation and treatment for the identified disease. Thus, it can triage the patients that require urgent treatment and reduce the burden of clinic visits for both patients and clinicians.

Yearly screening with the Acuity 360 system may also help to identify eye diseases at an early stage, which helps in improving the clinical outcomes.³ Moreover, tele-ophthalmology has been shown to increase the patient participation in screening¹⁸ probably due to the convenience, reduced examination time, lack of dilation, and reduced traveling cost and time.⁹ Our study also showed that the Acuity 360 system can be easily operated by ophthalmic technicians and has the potential to be used in remote/rural clinics or dangerous environments such as prisons, where clinicians are not easily accessible.

For the screening of diabetic retinopathy,^19,20 retinopathy of prematurity,¹⁹ and glaucoma,^19,20 teleophthalmology has been shown to be more cost-effective than clinic examination particularly if the disease is more prevalent among the patients screened.²⁰ This indicates that the Acuity 360 system can be cost-effective if the rate of screening is increased by regular monitoring and population screening. Community-based screening programs for high-risk individuals have the potential to identify eye diseases and refer patients to appropriate specialists.²¹ Screening older patients and fully utilizing the equipment are some other factors, which will optimize the cost-effectiveness of teleophthalmology.²⁰ Non-mydriatic and larger population screening programs are cost-effective and also decrease the clinic workload.²²

Inter-rater agreement helps to determine the variability in analyzing the Acuity 360 images by different examiners. The median inter-rater agreements for the combined retina examination, the OCT of optic nerve, and the OCT of angles were 95%, 95% and 100%, respectively (Table 3 and Figure 1). These imaging modalities showed highly similar responses from the examiners with less variability in analyzing the Acuity 360 images.

A major limitation of this study was exclusion of a high number (30.1%) of OCT optic nerve examinations due to remote examiners deeming the images to be of poor quality. This highlighted within our own practice the need for better technician training in image acquisition for this particular modality. In practice, these patients would have been brought back for re-imaging or in-person clinic examination.

Limitations including differentiating between types of cataracts (nuclear sclerotic versus posterior subcapsular) as well as patients who had advanced dry eyes who made some of the imaging not as clear but also did not show up on anterior segment imaging. We could not analyze the corneal data as no significant corneal pathology was present in the studied group. Similarly, none of the patients in the study group showed any anterior segment pathology with respect to OCT of the angles. Hence, the sensitivity and false-negative rate of the OCT of angles could not be performed, whereas it showed a specificity of 86.7%.

The current study validates the combination of comprehensive OCT imaging of anterior and posterior segments of the eye along with widefield retinal fundus photography as a viable system for screening of clinically significant eye disease in a telemedicine setting. Most importantly it has a very low false-negative rate, implying very little clinically significant pathology would be missed by the remote examination. Future considerations for study will be a more robust validation with remote and in-person examinations occurring on the same day and with a larger cohort that includes disease findings on OCT of the cornea and angles.

Abbreviations

OCT, optical coherence tomography; PPV, positive predictive value; NPV, negative predictive value; HVF, Humphrey visual field; COVID-19, coronavirus disease-19.

Ethical Approval

This case report was conducted in accordance with the Declaration of Helsinki. The collection and evaluation of all protected patient health information was performed in a US Health Insurance Portability and Accountability Act-compliant manner. Informed consent was obtained from the study participants. The full name of the ethics committee that reviewed this study was Advarra IRB ([email protected] or 877-992-4724). Protocol number A360.1. Reference number Pro00036632.

Statement of Informed Consent

Informed consent was obtained from the study participants. This report does not contain any personal information that could lead to the identification of the patient.

Acknowledgments

The authors acknowledge the support provided by Mounika Bodala of Hazlo Consultancy Pvt Ltd, Delhi, India, in preparing the first draft of the article and revising it with intellectual inputs from the authors. She also formatted the tables and figures included in the manuscript. Tom Chang, Jeffrey Bloom and Nicholas Binder, although no conflicts of interest, do work at Acuity Eye Group, who conducted this Acuity 360 trial. Acuity 360 was developed by a for-profit, private group of clinicians in ophthalmic care. Attached is the link to the website: https://www.acuityeyegroup.com/our-services/acuity-360. While the technology was developed by a for-profit group, the intent of this technology and manuscript is to demonstrate the utility of this approach, particularly in locations where the physical presences of physicians is either difficult or impossible.

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. All authors attest that they meet the current ICMJE criteria for authorship.

Disclosure

The Authors all declare that there is no conflict of interest with respect to the research, authorship, and/or publication of this article.

References

1. Afshar AR, Oldenburg CE, Stewart JM. A novel hybrid fixed and mobile ultra-widefield imaging program for diabetic teleretinopathy screening. Ophthalmol Retina. 2019;3:576–579. doi:10.1016/j.oret.2019.03.007

2. Rathi S, Tsui E, Mehta N, et al. The current state of teleophthalmology in the United States. Ophthalmology. 2017;124:1729. doi:10.1016/j.ophtha.2017.05.026

3. Vaziri K, Moshfeghi DM, Moshfeghi AA. Feasibility of telemedicine in detecting diabetic retinopathy and age-related macular degeneration. Semin Ophthalmol. 2015;30:81–95. doi:10.3109/08820538.2013.825727

4. Chandrasekaran S, Kass W, Thangamathesvaran L, et al. Tele-glaucoma vs clinical evaluation: the New Jersey Health Foundation Prospective Clinical Study. J Telemed Telecare. 2019:1357633X19845273.

5. Giorgis AT, Alemu AM, Arora S, et al. Results From the first teleglaucoma pilot project in Addis Ababa, Ethiopia. J Glaucoma. 2019;28:701–707. doi:10.1097/IJG.0000000000001271

6. Lapere S, Weis E. Tele-ophthalmology for the monitoring of choroidal and iris nevi: a pilot study. Can J Ophthalmol. 2018;53:471–473. doi:10.1016/j.jcjo.2017.11.021

7. DeBuc DC. The role of retinal imaging and portable screening devices in tele-ophthalmology applications for diabetic retinopathy management. Curr Diab Rep. 2016;16:132. doi:10.1007/s11892-016-0827-2

8. Mohammadpour M, Heidari Z, Mirghorbani M, et al. Smartphones, tele-ophthalmology, and VISION 2020. Int J Ophthalmol. 2017;10:1909–1918. doi:10.18240/ijo.2017.12.19

9. Sreelatha OK, Ramesh SV. Teleophthalmology: improving patient outcomes? Clin Ophthalmol. 2016;10:285–295. doi:10.2147/OPTH.S80487

10. Prathiba V, Rajalakshmi R, Arulmalar S, et al. Accuracy of the smartphone-based nonmydriatic retinal camera in the detection of sight-threatening diabetic retinopathy. Indian J Ophthalmol. 2020;68(Suppl 1):S42–S46. PMID: 31937728; PMCID: PMC7001191. doi:10.4103/ijo.IJO_1937_19

11. Kornberg DL, Yannuzzi NA, Klufas MA, D’Amico DJ, Orlin A, Kiss S. Ultra-widefield imaging of posterior segment pathology in the setting of the Boston keratoprosthesis. Retina. 2016;36(6):1101–1110. PMID: 26536101. doi:10.1097/IAE.0000000000000833

12. Conway MP, Forristal MT, Treacy MP, Duignan ES. Investigating the role of optometrists in teleophthalmology and the implications of increasing access to advanced imaging techniques and digital referral: a systematic search and review. Telemed J E Health. 2021;27(9):974–981. PMID: 33275866. doi:10.1089/tmj.2020.0284

13. Gündogan NÜ, Durmazlar N, Gümüş K, et al. Projected color slides as a method for mass screening test for color vision deficiency (a Preliminary Study). Int J Neurosci. 2005;115:1105–1117. doi:10.1080/00207450590914365

14. Nanji K, Kherani IN, Damji KF, et al. The muranga teleophthalmology study: a comparison of virtual (Teleretina) assessment with in-person clinical examination to diagnose diabetic retinopathy and age-related macular degeneration in Kenya. Middle East Afr J Ophthalmol. 2020;27:91. doi:10.4103/meajo.MEAJO_144_19

15. Duchin KS, Asefzadeh B, Poulaki V, et al. Teleretinal imaging for detection of referable macular degeneration. Optom Vis Sci. 2015;92:714–718. doi:10.1097/OPX.0000000000000598

16. Shi L, Wu H, Dong J, et al. Telemedicine for detecting diabetic retinopathy: a systematic review and meta-analysis. Br J Ophthalmol. 2015;99:823–831. doi:10.1136/bjophthalmol-2014-305631

17. Tozer K, Woodward MA, Newman-Casey PA. Telemedicine and diabetic retinopathy: review of published screening programs. J Endocrinol Diabetes. 2015;2. doi:10.15226/2374-6890/2/4/00131

18. Kawaguchi A, Sharafeldin N, Sundaram A, et al. Tele-ophthalmology for age-related macular degeneration and diabetic retinopathy screening: a systematic review and meta-analysis. Telemed J E Health. 2018;24:301. doi:10.1089/tmj.2017.0100

19. Au A, Gupta O. The economics of telemedicine for vitreoretinal diseases. Curr Opin Ophthalmol. 2011;22:194–198. doi:10.1097/ICU.0b013e3283459508

20. Sharafeldin N, Kawaguchi A, Sundaram A, et al. Review of economic evaluations of teleophthalmology as a screening strategy for chronic eye disease in adults. Br J Ophthalmol. 2018;102:1485–1491. doi:10.1136/bjophthalmol-2017-311452

21. Hark LA, Adeghate J, Katz LJ, et al. Philadelphia telemedicine glaucoma detection and follow-up study: cataract classifications following eye screening. Telemed J E Health. 2019;26:992–1000. doi:10.1089/tmj.2019.0170

22. Ullah W, Pathan SK, Panchal A, et al. Cost-effectiveness and diagnostic accuracy of telemedicine in macular disease and diabetic retinopathy: a systematic review and meta-analysis. Medicine. 2020;99:e20306. doi:10.1097/MD.0000000000020306