Interobserver Reliability of Endoscopic Ultrasonography: Literature Review

Abstract

Endoscopic ultrasonography (EUS) has been applied to the diagnosis of various digestive disorders. Although it has been widely accepted and its diagnostic value is high, the dependence of EUS diagnosis on image interpretation done by the endosonographer has persisted as an important difficulty. Consequently, high interobserver reliability (IOR) in EUS diagnosis is important to demonstrate the reliability of EUS diagnosis. We reviewed the literature on the IOR of EUS diagnosis for various diseases such as chronic pancreatitis, pancreatic solid/cystic mass, lymphadenopathy, and gastrointestinal and subepithelial lesions. The IOR of EUS diagnosis differs depending on the disease; moreover, EUS findings with high IOR and those with IOR that was not necessarily high were used as diagnostic criteria. Therefore, to further increase the value of EUS diagnosis, EUS diagnostic criteria with high diagnostic characteristics based on EUS findings with high IOR must be established.

1. Introduction

Endoscopic ultrasonography (EUS) was first reported in 1980 [1,2]. Since then, EUS has been applied to the diagnosis and treatment of pancreatic diseases [3,4,5,6,7]. Currently, EUS is used to diagnose diseases of various organs, including the pancreas. Generally, EUS is regarded as an effective modality because of its excellent image resolution, which allows the detection of small lesions and accurate assessment of staging in addition to the possibility of tissue sampling. Consequently, numerous reports of the diagnostic ability of EUS exist. Despite its benefits, EUS is not always perfect; limitations that hinder its accuracy are reportedly attributed to the following: (i) EUS imaging and diagnosis depend on the skill of the physicians, i.e., inter-operator variation; (ii) endoscopic ultrasound-guided fine needle aspiration (EUS-FNA) is often necessary for differential diagnosis because distinguishing between neoplastic and non-neoplastic tissue is difficult with EUS imaging; and (iii) conditions such as biliary/pancreatic duct stenting and chronic pancreatitis can interfere with the detection and staging of small lesions [8,9,10,11,12,13]. Particularly, inter-operator variation presents major difficulties. Consequently, high inter-observer reliability (IOR) is important to improve the reliability of imaging by EUS.

Currently, numerous diagnostic criteria based on EUS for various diseases are available. In these diagnostic criteria, a high IOR of EUS diagnostic findings is crucially important to support its diagnostic value. Some studies of IOR of EUS diagnosis have been conducted for various gastrointestinal diseases. This review presents a summary of the literature regarding the IOR of EUS diagnosis and explains some of the “wonders and problems” of current EUS diagnosis. For this study, IOR is expressed as kappa (K) values, which are defined as follows: <0, no agreement; 0–0.20, slight; 0.21–0.40, fair; 0.41–0.60, moderate; 0.61–0.80, high; and 0.81–1.00, almost perfect [14]. The literature from 1966 through 2019 was searched in PubMed using the keywords EUS and IOR.

2. Chronic Pancreatitis

EUS has been employed for the diagnosis of patients with chronic pancreatitis (CP) or suspected CP for over 20 years. Results of an earlier study suggested that CP was detectable by EUS even when findings of noninvasive imaging studies, such as computed tomography and magnetic resonance imaging, were negative [15]. Lees et al. demonstrated the usefulness of EUS morphology and histology after pancreatic resection in post-pancreatic resection specimens [16]. The results showed that CP was diagnosed histologically in six of the seven patients with CP findings on EUS. Other reports have described good correlation between EUS findings of CP and histological findings of CP, except in elderly patients [17,18]. Albashir et al. reported that the diagnostic performance of EUS based on histological findings was 84% for sensitivity and 100% for specificity [19].

Because of the well-known utility of EUS in the diagnosis of CP, this has been based on standard scoring of CP, defined by the International Working Group as standard criteria (SC) since these guidelines were published in 1998 [20] (Supplementary Table S1). The nine findings adopted were the following: hyperechoic foci, hyperechoic strands, lobularity, calcifications/stones, cysts, main pancreatic duct dilation, main pancreatic duct irregularity, hyperechoic main pancreatic duct (MPD) walls, and visible side branches. Subsequently, new diagnostic criteria for CP, i.e., the Rosemont Criteria (RC), were adopted in 2009 [21] (Supplementary Table S2).

The IOR of each finding has been reported in numerous studies (Table 1). The first report on IOR was by Wallence et al. in 2001 [22]. The K value of duct dilatation was 0.61 (high). That of lobularity was 0.51 (moderate), whereas that of the other features was <0.4 (slight and fair). Stevens et al. reported a similar result. Only the IOR of duct dilation was high (K = 0.61) [23]. Del Pozo et al. reported that the K values of lobularity, cysts, and MPD dilation were large (0.75, 0.66, and 075, respectively), whereas the IOR was only fair in lobularity with honeycombing (K = 0.31) [24]. Gardner et al. reported that the IOR is almost perfect in cysts (K = 1.00) and high in strands (K = 0.62) [25]; the other findings had low concordance rates: moderate in duct dilation, hyperechoic MPD wall, and lobularity (K = 0.53, 0.53, and 0.44, respectively) and fair in hyperechoic foci (K = 0.39). Lieb et al. reported that the IOR of stones and duct dilation were high (K = 0.78 and 0.77, respectively), whereas those of hyperechoic foci, strands, and side branch dilation were slight (K = 0.19, 0.07, and 0.11, respectively) [26]. A T study by Topazian et al. found low IOR in all cases: the IOR of cysts, which was the highest value (K = 0.48), was fair; those of the other findings were slight [27].

Final diagnosis of CP by EUS based on SC and RC has also been reported. The assessment of IOR of SC has been studied previously. It was found to be moderate at best. Wallence et al., Stevens et al., Del Pozo et al., and Kalmin et al. reported moderate IORs of the final diagnoses (K = 0.45, 0.54, 0.53, 0.50, respectively) [22,23,24,28]. The IOR of the final diagnosis in the study reported by Lieb et al. was fair (K = 0.39) [26].

Furthermore, several studies evaluated the IOR of the final diagnosis for both SC and RC. Del Pozo et al. reported that the IOR of the final diagnosis was moderate for both criteria (K = 0.53 for SC and K = 0.46 for RC) [24]. The study reported by Stevens et al. revealed the IOR of the final diagnosis as moderate for SC (K = 0.54) and high for RC (K = 0.65). However, the difference was not statistically significant (p = 0.12) [23]. Kalmin et al. reported that the IOR of the final diagnosis was moderate for SC (K = 0.50) and fair for RC (K = 0.27) [28]. These results suggest that the IOR did not seem to improve with the RC compared with the SC.

The examiners participating in most of the studies were experts. However, some studies have investigated differences in IOR according to years of experience with EUS. Wallence et al. reported the IOR of the final diagnosis as moderate for fellows and experts (K = 0.42 and 0.46, respectively) [22]. Stevens et al. reported no significant difference in IOR based on years of experience, irrespective of the criteria used (i.e., SC or RC) [16].

Although the IOR of the respective findings for CP varied, several reports have described that the IOR of the final diagnosis produced using both SC and RC is moderate. Moreover, the findings suggest that the IOR might not depend on the years of experience with EUS. 3. Solid Pancreatic Lesion

Imaging of a solid pancreatic lesion (SPL) is fundamentally important for treatment. SPLs include pancreatic adenocarcinoma, inflammatory mass in the context of CP, pancreatic neuroendocrine tumor, autoimmune pancreatitis, and other tumors. Among them, pancreatic adenocarcinoma has a particularly poor prognosis. For that reason, early detection is desirable. EUS plays a major role as a diagnostic modality: even small lesions of 5 mm can be visualized reliably by EUS. Moreover, contrast-enhanced harmonic endoscopic ultrasonography (CH-EUS) is an extremely useful test in the differentiation of SPL [29,30]. CH-EUS can assess tissue perfusion in real time without Doppler-related artifacts [30,31,32]. A recent study demonstrated that the microvascular pattern of SPL based on CH-EUS correlates closely with histological features [32,33]. Gong et al. reported that the sensitivity of CH-EUS for the differential diagnosis of pancreatic adenocarcinoma is 94%; the specificity was 89% [34].

Based on EUS, pancreatic adenocarcinomas are unevenly margined and well-defined. Small tumors present with a homogeneous hypoechoic appearance. However, as the tumor enlarges, a heterogeneous hyperechoic region develops in the center. Pancreatic duct dilation caudal to the mass and dilated surrounding branches accompanied by storage cysts are important indirect findings. CH-EUS often shows hypo-enhancement in the arteriovenous phase because of a desmoplastic reaction and an internal heterogeneous contrast pattern. Moreover, EUS images of neuroendocrine tumors (NETs) show a hypoechoic appearance with smooth edges, well-defined borders, and uniform interiors. In CH-EUS, it is of hyper-vascular-type. Inflammatory masses in the context of CP present an overall hypoechoic appearance. The boundary of the pancreatic parenchyma is somewhat obscured. They also often present with duct-penetrating signs.

Several studies have examined IOR of the diagnosis of SPL using CH-EUS (Table 2). Some studies have examined differences in IOR by years of experience with EUS. Kitano et al. reported that the combination of CH-EUS and EUS-FNA increased the sensitivity for the diagnosis of pancreatic adenocarcinoma to 100% [35]. After they evaluated the IOR of CH-EUS in 277 SPLs conducted by two experienced endo-sonographers, they reported a K value of 0.94 (almost perfect). Fusaroli et al. reported that the IOR of contrast uptake was moderate for all endo-sonographers (K = 0.56). Similar results were reported for experienced endo-sonographers (K = 0.56), and for non-experienced endo-sonographers (K = 0.55) [36]. Gincul et al. evaluated the IOR of CH-EUS by experienced endo-sonographers and non-experienced endo-sonographers. The overall IOR (K = 0.66) and the IOR of experienced (K = 0.65) and non-experienced (K = 0.76) endo-sonographers were high [37]. In this study, CH-EUS and EUS-FNA had high accuracy, with sensitivity of 95% and 96%, of 95% and 93% (p < 0.05). Bunganič et al. reported that the IOR of pancreatic adenocarcinoma diagnosis by EUS and CH-EUS was moderate (K = 0.45 and 0.60, respectively) [38]. The sensitivity and specificity of EUS and CH-EUS for pancreatic adenocarcinoma diagnosis were, respectively, 83.1% and 62.5% and 94.5% and 61.7%.

Palazzo et al. examined the IORs of the diagnoses of 81 cases of pancreatic NET (P-NET) [39]. The sensitivity and specificity of CH-EUS for the P-NET diagnosis were 96% and 82%, respectively. The IOR was high (K = 0.66) for all endo-sonographers, almost perfect (K = 0.83) for experienced endo-sonographers, and almost perfect (K = 0.82) for junior endo-sonographers.

Soares et al. evaluated the diagnostic ability of EUS-elastography (EUS-E) for SPLs [40]. Eleven endo-sonographers were divided into four groups: group A (long experience in EUS and EUS-E), group B (short experience in EUS and EUS-E), group C (long experience in EUS but no experience in EUS-E), and group D (no experience with EUS or EUS-E). The overall IOR was moderate (K = 0.42). The IOR of group A (K = 0.80) was significantly higher than that of groups B (K = 0.54), C (K = 0.54), and D (K = 0.28).

The IOR of SPL diagnosis by EUS was high. However, it might vary depending on the years of experience of endo-sonographers. 4. Pancreatic Cystic Lesion

Pancreatic cystic lesions include intraductal papillary mucinous neoplasm (IPMN), mucinous cystic neoplasm, and serous cystic neoplasm; all have a malignant potential [41,42,43]. EUS can provide morphological details of pancreatic cysts. However, a benign or malignant cyst is sometimes difficult to diagnose [44,45]. Earlier studies showed the diagnostic accuracy of EUS for pancreatic cystic lesions as 82–96% [46,47,48].

The first report of the IOR of pancreatic cystic lesion diagnosis was presented by Ahmad et al. in 2003 [49] (Table 3). Thirty-one cases with histologically classified pancreatic cystic lesions were reviewed independently by eight expert endo-sonographers. The IOR of diagnosing a benign or malignant cyst was fair (K = 0.24). Moreover, the IORs of the respective diagnostic finding were as follows: presence or absence of solid component, K = 0.43 (moderate); presence or absence of abnormal pancreatic duct, K = 0.29 (fair); debris, K = 0.21 (fair); septations, K = 0.30 (fair); presence or absence of margins, K = 0.01 (slight); and presence or absence of abnormal pancreatic parenchyma, K = 0.01 (slight). Fusaroli examined the IOR of pancreatic cystic lesion diagnosis by CH-EUS and found the IOR of contrast uptake to be moderate (K = 0.58) [36]. Gonzalez et al. examined the IOR of branch duct IPMN diagnosis by EUS and magnetic resonance cholangiopancreatography [50]. The IOR was moderate for PD = 5–9 mm (K = 0.45) and for abrupt change in PD (K = 0.529), fair for wall thickening (K = 0.259), and slight for non-enhanced mural nodule (K < 0) and lymphadenopathy (K < 0). In high-risk stigmata, the IOR was reported as slight for enhanced solid component (K = 0.12) and PD > 10 mm (K < 0). Jong et al. examined the IOR in three groups based on EUS experience [51]. The IOR of nodules was high in the expert group (K = 0.65) and fair in semi-expert and novice groups (K = 0.32 and 0.37, respectively). The IOR of the presence of solid components was moderate in the expert group (K = 0.52) and significantly higher than that in the other two groups (semi-experts, K = 0.09; and novices, K = 0.03).

Kim et al. reported the IOR of cyst communication with the duct as high (K = 0.69) and that of differentiating malignant from benign cysts as almost perfect (K = 0.92). The IOR of pancreatic cystic lesions is expected to be elevated with CH-EUS. Nonetheless, the challenge remains of eliminating the difference in IOR by years of experience with EUS. 5. Other Diseases 5.1. Lymphadenopathy

Differentiating benign from malignant lymphadenopathy is extremely important: EUS is appropriate for such evaluation [52,53,54]. Catalano et al. established the following EUS features as predictive of malignancy (in increasing order of importance): hypoechoic structures, sharply demarcated borders, rounded contours, and size >10 mm [52]. EUS-FNA is an efficient method for diagnosing lymphadenopathy, with recent reports describing sensitivity and specificity of 74–98% and 100%, respectively [55,56,57]. Although EUS-FNA has a high diagnostic value in lymphadenopathy, it is important to use only EUS to predict benign or malignant lesions.

Takasaki et al. evaluated the IOR of EUS features and proposed EUS diagnostic norms for lymphadenopathy based on the IOR [58]. The IOR was moderate for shape (K = 0.44) and fair for echogenicity, homogeneity, border, and hilum of the lymph node (K = 0.33, 0.34, 0.22, and 0.22, respectively). Moreover, they suggested the establishment of new EUS diagnostic criteria using shape, long axis (>20 mm), and short axis (>10 mm). Melo et al. studied the IOR of EUS diagnostic imaging for lymphadenopathy [59] and reported fair IOR for shape (K = 0.35) and moderate IOR for echogenicity and borders (K = 0.46 and 0.43, respectively). The overall IOR was high (K = 0.65).

Xia et al. evaluated the IOR of CH-EUS features for intra-abdominal lesions of undetermined origin. Results showed that about 80% of the eligible cases were in lymphadenopathy and that the IOR was almost perfect (K = 0.95). Furthermore, they reported that the sensitivity, specificity, positive predictive value, negative predictive value, and accuracy with which CH-EUS differentiated malignant from benign lesions were 96.3, 100, 100, 94.1 and 97.6% [60].

Several studies of lymphadenopathy using EUS-E have been conducted. Based on the general EUS-E patterns, lymph nodes were classified as benign (e.g., green color) or malignant (e.g., blue color) and scored in the study of Giovannini et al. [61]. Janssen et al. reported that the accuracy range among the examiners was 82–88% for benign lymph nodes and 85–87% for malignant ones. The IOR was almost perfect (K = 0.84) [62]. Based on results of a study by Larsen et al., the IOR of distinguishing benign or malignant lymphadenopathy was reported as moderate (K = 0.59) [63] (Table 4).

The IOR of lymphadenopathy is expected to be elevated through the use of CH-EUS and EUS-E. Nevertheless, further development of diagnostic criteria to predict benign or malignant lesions is warranted. 5.2. Gallbladder Wall Thickening

Gallbladder (GB) wall thickening might be caused by a broad spectrum of pathologies, including GB carcinoma, chronic cholecystitis, and adeno-myomatosis. Differentiation between GB carcinoma, adeno-myomatosis, and GB inflammatory diseases presenting as wall thickening is an important clinical issue because misinterpretation of GB wall thickening might result in unnecessarily extended surgery in patients with GB inflammatory diseases or delayed treatment in patients with GB carcinoma. Because it allows observation of the laminar structure of the GB wall, EUS is useful for the diagnosis of GB thickening [64,65]. Recently, some studies showed that CH-EUS might be useful for the diagnosis of GB disease [66,67]. Imazu et al. evaluated the IOR of diagnosing GB wall thickening using EUS and CH-EUS [68]. Univariate analysis results indicate that GB wall thickening of >12 mm and disruption of the GB wall layer structure on EUS and inhomogeneous distribution pattern of contrast agent on CH-EUS are significantly associated with malignant GB wall thickening with a high odds ratio. The addition of contrast enhancement to conventional EUS was extremely useful for differential diagnosis of GB wall thickening. Overall sensitivity and specificity for diagnosing malignant GB wall thickening of EUS and CH-EUS were 83.3 vs. 89.6, 65 vs. 98%, respectively. The IOR of the differential diagnosis of malignant or nonmalignant GB wall thickening was moderate with EUS (K = 0.51) and high with CH-EUS (K = 0.77) (Table 4).

The IOR of GB wall thickening diagnosis by CH-EUS is high; nevertheless, examination of more cases is necessary to further evaluate the usefulness of the method. 5.3. Subepithelial Lesion

EUS is also useful for diagnosing subepithelial lesions (SELs). The adjunctive application of CH-EUS can augment the diagnostic work-up of SEL. Recently, sensitivity, specificity, and accuracy rates of 100%, 63%, and 83%, respectively, were reported for the prediction of malignancy of gastrointestinal SEL by CH-EUS [69]. Gress et al. demonstrated that the IOR of SEL was almost perfect for cystic lesions (K = 0.80) and extrinsic compressions (K = 0.94), high for lipoma (K = 0.65), moderate for leiomyoma and vascular lesions (K = 0.53 and 0.54, respectively), and fair for other submucosal lesions (K = 0.34). Overall agreement among observers was strong (K = 0.63) [70]. Furthermore, significant association was found between the total years of experience with EUS and the number of correct answers (p = 0.01). Moreover, Fusaroli et al. [36] determined the IOR of the parameters (i.e., uptake, pattern, washout) examined in SEL of three types by CH-EUS. The IOR was large for uptake (K = 0.638), slight for pattern (K = 0.183), and fair for washout (K = 0.394) (Table 4). The IOR of SEL is high and independent of years of experience, suggesting that EUS is useful for diagnosing SEL.

5.4. Intrapancreatic Accessory Spleen

Accessory spleen (AS) might be encountered as an intrapancreatic lesion on EUS. It can present as a diagnostic dilemma because it might resemble other pancreatic pathologies, such as pancreatic adenocarcinoma and neuroendocrine tumors on imaging. Intrapancreatic AS (IPAS) is usually an incidental finding that requires no treatment [71,72]. Nevertheless, IPAS must be distinguished from other pancreatic neoplasms to avoid unnecessary interventions. Rodriguez et al. showed that an AS tends to be an isoechoic or hypoechoic mass with well-defined, smooth borders on EUS [73]. Kim et al. reported that the IOR of determining whether or not the pancreatic lesion is IPAS is fair (K = 0.37) [74] (Table 4). They also reported that the sensitivity and specificity for IPAS were greater than 70%.

Few studies have examined the IOR of AS; nonetheless, it has been suggested that combining EUS with other modalities might increase the diagnostic capabilities for AS. 5.5. Mucosa-Associated Lymphoid Tissue (MALT) Lymphoma

Several studies have demonstrated the reliability of EUS for staging MALT lymphoma before and after therapy [75,76,77,78]. Particularly, as reported earlier, EUS has sensitivity of 89%, specificity of 97%, and overall accuracy of 95% for depth of invasion [79]. Fusaroli et al. reported the IOR of EUS staging of gastric MALT lymphoma before and after treatment [80]. The overall IOR of T-stage was fair before and after treatment (K = 0.38 and 0.37, respectively). That of N-stage was large before treatment and fair after treatment (K = 0.63 and 0.34, respectively) (Table 4). Although the IOR of EUS staging of gastric MALT lymphoma varies, EUS for gastric MALT lymphoma might determine the treatment effects that were not observable using macroscopic morphology.

5.6. Gastrointestinal Malignancy

EUS is used for staging of gastrointestinal malignancies. Yanai et al. reported the IOR of staging of submucosal invasion of superficial esophageal cancer (SEC) [81]. The IOR was fair (K = 0.46). It was suggested that EUS is useful for the evaluation of lymph node metastasis and might be useful in predicting the prognosis of SEC. Burtin et al. evaluated the IOR of staging of rectal cancer [82]. The IOR of uT1 tumors was fair (K = 0.40), uT2 tumors was slight (K = 0.20), and uT3 tumors was moderate (K = 0.58). In rectal cancer, EUS was also excellent for the evaluation of lymph node metastasis (K = 0.65). Roubein et al. was evaluated similarly for rectal cancer but stated only that the evaluation of lymph node metastases was a high IOR (K = 0.42) [83] (Table 4).

Reportedly, the IOR for staging of gastrointestinal malignancies cancers varies widely among observers. Further study of this point is required [84].

6. Future Perspectives

In recent years, as the development of artificial intelligence (AI) technology has progressed, it has been applied to the field of diagnosis by EUS [85,86,87]. Kuwahara et al. reported that the malignancy of IPMN can be diagnosed using AI with accuracy of 94%, which is higher than those obtained using conventional EUS features (40–60%) and endo-sonographer’s diagnosis (56%) [88]. Advances in AI are expected to be important for improving IOR-independent imaging capabilities in the field of EUS imaging.

7. Conclusions EUS imaging has become a widely used approach for diagnosis of numerous diseases because of its high resolution and noninvasive nature. In addition, several EUS diagnostic criteria are extremely useful for diagnosing various diseases. However, the IOR of EUS diagnosis differs depending on the disease. Not only EUS findings with high IOR but also those with not necessarily high IOR are used as diagnostic criteria. Therefore, to further increase the value of EUS diagnosis, EUS diagnostic criteria with high diagnostic characteristics, based on EUS findings with high IOR, must be established.

EUS Findings	Topazian (2007) n = 20	Wallence (2001) n = 33	Stevens (2010) n = 50	Lieb (2011) n = 16	Gardner (2011) n = 44	Kalmin (2011) n = 36	Del Poza (2011) n = 69
Hyperechoic foci	0.12	0.29	0.21	0.19	0.39	NC	0.48
Strands	0.14	0.31	0.29	0.07	0.62	NC	0.55
Lobularity	0.23	0.51	0.16	0.53	0.44	NC	0.75
Cysts	0.48	0.32	0.35	NC	1	NC	0.66
Stones	0.03	0.38	0.36	0.78	NC	NC	NC
Duct dilation	NC	0.61	0.61	0.77	0.53	NC	0.75
Duct irregularity	0.2	0.29	0.5	0.60	NC	NC	NC
Hyperechoic MPD margin	0.2	0.36	0.33	0.34	0.53	NC	NC
Dilated side branch	0.09	0.18	0.46	0.11	NC	NC	NC
Rosemont classification	NC	NC	0.65	NC	NC	0.27	0.46

NC; not calculated.

Author	Year	Disease	Degree of Experience	K-Value
Kitano et al. (n = 277)	2012	Adenocarcinoma	Experienced	0.56
Fusaroli et al. (n = 40)	2012	SPL	Overall Experienced Non-experienced	0.56 0.56 0.55
Gincul et al. (n = 100)	2014	SPL	Overall Experienced Non-experienced	0.66 0.76 0.65
Bunganic et al. (n = 116)	2018	SPL	Overall	0.60
Palazzo et al. (n = 81)	2018	P-NET	Overall Experienced Non-experienced	0.66 0.82 0.83

SPL, solid pancreatic lesion; P-NET, pancreatic neuroendocrine tumor.

EUS Findings	Ahmad (2003) n = 31	Fusaroli (2012) n = 40	Jong (2011) By Expert n = 22	Jong (2011) By Semi-Expert n = 22	Jong (2011) By Novice n = 22
Contrast-enhanced	(-)	(+)	(−)	(−)	(−)
Solid component	0.43	NC	NC	NC	NC
Irregular pancreatic duct	0.29	NC	NC	NC	NC
Septation	0.30	NC	NC	NC	NC
Irregular margins	0.01	NC	0.65	0.32	0.37
Abnormal pancreatic parenchyma	0.01	NC	NC	NC	NC
Final diagnosis	0.24	0.58	0.43	0.09	0.30

NC, not calculated.

Disease	Year	Author	n	Modality	K-Value
Lymphadenopathy	2019	Takasaki et al.	68	EUS	0.65
	2010	Xia et al.	43	CH-EUS	0.95
	2007	Jassen et al.	50	EUS-E	0.84
	2011	Larsen et al.	61	EUS-E	0.59
Gallbladder Wall Thickening	2014	Imazu et al.	36	EUS	0.51
	-	-	-	CH-EUS	0.77
Subepithelial Lesion	2001	Gress et al.	20	CH-EUS	0.63
	2012	Fusaroli et al.	40	CH-EUS	uptake 0.64, pattern 0.18 washout 0.39
Intrapancreatic Accessory	2019	Kim et al.	12	EUS	0.37
MALT Lymphoma	2002	Fusaroli et al.	54	EUS	T-stage before and after treatment = 0.38 and 0.37 N-stage before treatment and fair after treatment = 0.63 and 0.34
Staging of superficial esophageal cancer	2003	Yanai et al.	31	EUS	0.31
Staging of rectal cancer	1997	Burtin et al.	27	EUS	T2 0.20 T3 0.58

MALT, Mucosa-associated Lymphoid Tissue.

Supplementary Materials

The following are available online at https://www.mdpi.com/2075-4418/10/11/953/s1: Table S1: Endoscopic ultrasound criteria for the diagnosis of chronic pancreatitis, Table S2: Classification of patients based on endoscopic criteria.

Author Contributions

A.Y. (Akira Yamamiya) and A.I. designed this manuscript, collected and analyzed the data, and drafted the manuscript; A.I. checked the manuscript and approved the final version; K.K., Y.K., K.N., T.M., N.I., A.Y. (Akane Yamabe), K.H., and K.T. collected the literature and created the tables; M.I. and K.G. supervised this manuscript. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Conflicts of Interest

The authors declare no conflict of interest

References

1. DiMango, E.P.; Buxton, J.L.; Regan, P.T.; Hattery, R.R.; Wilson, D.A.; Suarez, J.R.; Green, P.S. Ultrasonic endoscope. Lancet 1980, 315, 629-631.

2. Strohm, W.D.; Phillip, J.; Hagenmüller, F.; Classen, M. Ultrasonic tomography by means of an ultrasonic fiberendoscope. Endoscopy 1980, 12, 214-244.

3. Hisanaga, K.; Hisanaga, A.; Nagata, K.; Ichie, Y. High speed rotating scanner for transgastric sonography. AJR Am. J. Roentgenol. 1980, 135, 627-639.

4. DiMagno, E.P.; Regan, P.T.; Clain, J.E.; James, E.M.; Buxton, J.L. Human endoscopic ultrasonography. Gastroenterology 1982, 83, 824-829.

5. Khalid, A.; Nodit, L.; Zahid, M.; Bauer, K.; Brody, D.; Finkelstein, S.D.; McGrath, K.M. Endoscopic ultrasound fine needle aspirate DNA analysis to differentiate malignant and benign pancreatic masses. Am. J. Gastroenterol. 2006, 101, 2493-2500.

6. Gardner, T.B.; Levy, M.J. EUS diagnosis of chronic pancreatitis. Gastrointest. Endosc 2010, 71, 1280-1289.

7. Levy, M.J.; Wiersema, M.J. EUS-guided celiac plexus neurolysis and celiac plexus block. Gastrointest. Endosc. 2003, 57, 923-930.

8. Fusaroli, P.; Manta, R.; Fedeli, P.; Maltoni, S.; Grillo, A.; Giovannini, E.; Bucchi, L.; Caletti, G. The influence of endoscopic biliary stents on the accuracy of endoscopic ultrasound for pancreatic head cancer staging. Endoscopy 2007, 39, 813-817.

9. Bhutani, M.S.; Gress, F.G.; Giovannini, M.; Erickson, R.A.; Catalano, M.F.; Chak, A.; Deprez, P.H.; Faigel, D.O.; Nguyen, C.C. The No Endosonographic Detection of Tumor (NEST) study: A case series of pancreatic cancers missed on endoscopic ultrasonography. Endoscopy 2004, 36, 385-389.

10. Fritscher-Ravens, A.; Brand, L.; Knöfel, W.T.; Bobrowski, C.; Topalidis, T.; Thonke, F.; de Werth, A.; Soehendra, N. Comparison of endoscopic ultrasound-guided fine needle aspiration for focal pancreatic lesions in patients with normal parenchyma and chronic pancreatitis. Am. J. Gastroenterol. 2002, 97, 2768-2775.

11. Varadarajulu, S.; Tamhane, A.; Eloubeidi, M.A. Yield of EUS-guided FNA of pancreatic masses in the presence or the absence of chronic pancreatitis. Gastrointest. Endosc. 2005, 62, 728-736.

12. Krishna, N.B.; Mehra, M.; Reddy, A.V.; Agarwal, B. EUS/EUS-FNA for suspected pancreatic cancer: Influence of chronic pancreatitis and clinical presentation with or without obstructive jaundice on performance characteristics. Gastrointest. Endosc. 2009, 70, 70-79.

13. Eloubeidi, M.A.; Varadarajulu, S.; Desai, S.; Wilcox, C.M. Value of repeat endoscopic ultrasound-guided fine needle aspiration for suspected pancreatic cancer. J. Gastroenterol. Hepatol. 2008, 23, 567-570.

14. Landis, J.; Koch, G.G. The measurement of observer agreement for categorical data. Biometrics 1977, 33, 159-174.

15. Chong, A.K.; Hawes, R.H.; Hoffman, B.J.; Adams, D.B.; Lewin, D.N.; Romagnuolo, J. Diagnostic performance of EUS for chronic pancreatitis: A comparison with histopathology. Gastrointest. Endosc. 2007, 65, 808-814.

16. Lees, W.R. Endoscopic ultrasonography of chronic pancreatitis and pancreatic pseudocysts. Scand. J. Gastroenterol. 1986, 123, 130-134.

17. Bhutani, M.S.; Arantes, V.N.; Verma, D.; Moezzi, J.; Suryaprasad, S.; Kapadia, A.S.; Gopalswamy, N. Histopathologic correlation of endoscopic ultrasound findings of chronic pancreatitis in human autopsies. Pancreas 2009, 38, 820-824.

18. Varadarajulu, S.; Eltoum, I.; Tamhane, A.; Eloubeidi, M.A. Histopathologic correlates of noncalcific chronic pancreatitis by EUS: A prospective tissue characterization study. Gastrointest. Endosc. 2007, 66, 501-509.

19. Albashir, S.; Bronner, M.P.; Parsi, M.A.; Walsh, R.M.; Stevens, T. Endoscopic ultrasound, secretin endoscopic pancreatic function test, and histology: Correlation in chronic pancreatitis. Am. J. Gastroenterol. 2010, 105, 2498-2503.

20. The International Working Group for Minimal Standard Terminology in Gastrointestinal Endosonography. Minimal standard terminology in gastrointestinal endosonography. Dig. Endosc. 1998, 10, 159-184.

21. Catalano, M.F.; Sahai, A.; Levy, M.; Romagnuolo, J.; Wiersema, M.; Brugge, W.; Freeman, M.; Yamao, K.; Canto, M.; Hernandez, L.V. EUS-based criteria for the diagnosis of chronic pancreatitis: The Rosemont classification. Gastrointestndosc 2009, 69, 1251-1261.

22. Wallace, M.B.; Hawes, R.H.; Durkalski, V.; Chak, A.; Mallery, S.; Catalano, M.F.; Wiersema, M.J.; Bhutani, M.S.; Ciaccia, D.; Kochman, M.L.; et al. The reliability of EUS for the diagnosis of chronic pancreatitis: Interobserver agreement among experienced endosonographers. Gastrointest. Endosc. 2001, 53, 294-299.

23. Stevens, T.; Lopez, R.; Adler, D.G.; Al-Haddad, M.A.; Conway, J.; Dewitt, J.M.; Forsmark, C.E.; Kahaleh, M.; Lee, L.S.; Levy, M.J.; et al. Multicenter comparison of the interobserver agreement of standard EUS scoring and Rosemont classification scoring for diagnosis of chronic pancreatitis. Gastrointest. Endosc. 2010, 71, 519-526.

24. Del Pozo, D.; Poves, E.; Tabernero, S.; Beceiro, I.; Moral, I.; Villafruela, M.; Sanz, C.; Borrego, G. Conventional versus Rosemont endoscopic ultrasound criteria for chronic pancreatitis: Interobserver agreement in same day back-to-back procedures. Pancreatology 2012, 12, 284-287.

25. Gardner, T.B.; Gordon, S.R. Interobserver agreement for pancreatic endoscopic ultrasonography determined by same day back-to-back examinations. J. Clin. Gastroenterol. 2011, 45, 542-545.

26. Lieb, J.G.; Palma, D.T.; Garvan, C.W.; Leblanc, J.K.; Romagnuolo, J.; Farrell, J.J.; Savides, T.J.; Eloubeidi, M.A.; Draganov, P.V.; Forsmark, C.E.; et al. Intraobserver agreement among endosonographers for endoscopic ultrasound features of chronic pancreatitis a blinded multicenter study. Pancreas 2011, 40, 177-180.

27. Topazian, M.; Enders, F.; Kimmey, M.; Brand, R.; Chak, A.; Clain, J.; Cunningham, J.; Eloubeidi, M.; Gerdes, H.; Gress, F.; et al. Interobserver agreement for EUS findings in familial pancreatic-cancer kindreds. Gastrointest. Endosc. 2007, 66, 62-67.

28. Kalmin, B.; Hoffman, B.; Hawes, R.; Romagnuolo, J. Conventional versus Rosemont endoscopic ultrasound criteria for chronic pancreatitis: Comparing interobserver reliability and intertest agreement. Can. J. Gastroenterol. 2011, 25, 261-264.

29. Kitano, M.; Kudo, M.; Maekawa, K.; Suetomi, Y.; Sakamoto, H.; Fukuta, N.; Nakaoka, R.; Kawasaki, T. Dynamic imaging of pancreatic diseases by contrast enhanced coded phase inversion harmonic ultrasonography. Gut 2004, 53, 854-859.

30. Fusaroli, P.; Saftoiu, A.; Mancino, M.G.; Caletti, G.; Eloubeidi, M.A. Techniques of image enhancement in EUS (with videos). Gastrointest. Endosc. 2011, 74, 645-655.

31. Kersting, S.; Konopke, R.; Kersting, F.; Volk, A.; Distler, M.; Bergert, H.; Saeger, H.D.; Grützmann, R.; Bunk, A. Quantitative perfusion analysis of transabdominal contrast-enhanced ultrasonography of pancreatic masses and carcinomas. Gastroenterology 2009, 137, 1903-1911.

32. Napoléon, B.; Alvarez-Sanchez, M.V.; Gincoul, R.; Pujol, B.; Lefort, C.; Lepilliez, V.; Labadie, M.; Souquet, J.C.; Queneau, P.E.; Scoazec, J.Y.; et al. Contrast-enhanced harmonic endoscopic ultrasound in solid lesions of the pancreas: Results of a pilot study. Endoscopy 2010, 42, 564-570.

33. Fusaroli, P.; Kypraios, D.; Caletti, G.; Eloubeidi, M.A. Pancreatico-biliary endoscopic ultrasound: A systematic review of the levels of evidence, performance and outcomes. World J. Gastroenterol. 2012, 18, 4243-4256.

34. Gong, T.T.; Hu, D.M.; Zhu, Q. Contrast-enhanced EUS for differential diagnosis of pancreatic mass lesions: A meta-analysis. Gastrointest. Endosc. 2012, 76, 301-309.

35. Kitano, M.; Kudo, M.; Yamao, K.; Takagi, T.; Sakamoto, H.; Komaki, T.; Kamata, K.; Imai, H.; Chiba, Y.; Okada, M.; et al. Characterization of small solid tumors in the pancreas: The value of contrast-enhanced harmonic endoscopic ultrasonography. Am. J. Gastroenterol. 2012, 107, 303-310.

36. Fusaroli, P.; Kypraios, D.; Mancino, M.G.; Spada, A.; Benini, M.C.; Bianchi, M.; Bocus, P.; De Angelis, C.; De Luca, L.; Fabbri, C.; et al. Interobserver agreement in contrast harmonic endoscopic ultrasound. J. Gastroenterol. Hepatol. 2012, 27, 1063-1069.

37. Gincul, R.; Palazzo, M.; Pujol, B.; Tubach, F.; Palazzo, L.; Lefort, C.; Fumex, F.; Lombard, A.; Ribeiro, D.; Fabre, M.; et al. Contrast-harmonic endoscopic ultrasound for the diagnosis of pancreatic adenocarcinoma: A prospective multicenter trial. Endoscopy 2014, 46, 373-379.

38. Bunganič, B.; Laclav, M.; Dvořáková, T.; Bradáč, O.; Traboulsi, E.; Suchánek, S.; Frič, P.; Zavoral, M. Accuracy of EUS and CEH EUS for the diagnosis of pancreatic tumours. Scand. J. Gastroenterol. 2018, 53, 1411-1417.

39. Palazzo, M.; Napoléon, B.; Gincul, R.; Pioche, M.; Pujol, B.; Lefort, C.; Fumex, F.; Hautefeuille, V.; Fabre, M.; Cros, J.; et al. Contrast harmonic EUS for the prediction of pancreatic neuroendocrine tumor aggressiveness (with videos). Gastrointest. Endosc. 2018, 87, 1481-1488.

40. Soares, J.B.; Garcia, J.; Goncalves, B.; Lindkvist, B.; Noia, J.; Bastos, P.; Caetano, A.C.L.; Ferreira, A.; Pimentel-Nunes, P.; Lopes, L.; et al. Interobserver agreement of EUS elastography in the evaluation of solid pancreatic lesions. Endosc. Int. Open 2015, 4, 244-249.

41. Fernandez-del, C.C.; Targarona, J.; Thayer, S.P.; Rattner, D.W.; Brugge, W.R.; Warshaw, A.L. Incidental pancreatic cysts: Clinicopathologic characteristics and comparison with symptomatic patients. Arch. Surg. 2003, 138, 427-430.

42. Kiely, J.M.; Nakeeb, A.; Komorowski, R.A.; Wilson, S.D.; Pitt, H.A. Cystic pancreatic neoplasms: Enucleate or resect? J. Gastrointest. Surg. 2003, 7, 890-897.

43. Talamini, M.A.; Pitt, H.A.; Hruban, R.H.; Boitnott, J.K.; Coleman, J.; Cameron, J.L. Spectrum of cystic tumors of the pancreas. Am. J. Surg. 1992, 163, 117-123.

44. Ahmad, N.A.; Kochman, M.L.; Lewis, J.D.; Ginsberg, G.G. Can EUS alone differentiate between malignant and benign cystic lesions of the pancreas? Am. J. Gastroenterol. 2001, 96, 3295-3300.

45. Hernandez, L.V.; Bhutani, M.S. Endoscopic ultrasound and pancreatic cysts: A sticky situation! Am. J. Gastroenterol. 2001, 96, 3229-3230.

46. Koito, K.; Namieno, T.; Nagakawa, T.; Shyonai, T.; Hirokawa, N.; Morita, K. Solitary cystic tumor of the pancreas: EUS-pathologic correlation. Gastrointest. Endosc. 1997, 45, 268-276.

47. Muller, M.; Meyenberger, C.; Bertschinger, P.; Schaer, R.; Marincek, B. Pancreatic tumors: Evaluation with endoscopic US, CT, and MR imaging. Radiology 1994, 190, 745-751.

48. Sahani, D.V.; Kadavigere, R.; Saokar, A.; Fernandez-del Castillo, C.; Brugge, W.R.; Hahn, P.F. Cystic pancreatic lesions: A simple imaging-based classification system for guiding management. Radiographics 2005, 25, 1471-1484.

49. Ahmad, N.A.; Kochman, M.L.; Brensinger, C.; Brugge, W.R.; Faigel, D.O.; Gress, F.G.; Kimmey, M.B.; Nickl, N.J.; Savides, T.J.; Wallace, M.B.; et al. Interobserver agreement among endosonographers for the diagnosis of neoplastic versus non-neoplastic pancreatic cystic lesions. Gastrointest. Endosc. 2003, 58, 59-64.

50. Gonzalez, L.U.; Keane, M.G.; Pereira, S.P.; Iglesias-García, J.; Dominguez-Muñoz, E.; Lariño-Noia, J. Agreement among magnetic resonance imaging magnetic resonance cholangiopancreatography (MRI-MRCP) and endoscopic ultrasound (EUS) in the evaluation of morphological features of branch duct intraductal papillary mucinous neoplasm (BD-IPMN). Pancreatology 2018, 18, 170-175.

51. De Jong, K.; Verlaan, T.; Dijkgraaf, M.G.; Poley, J.W.; Dullemen, H.; Bruno, M.J.; Fockens, P. Interobserver agreement for endosonography in the diagnosis of pancreatic cysts. Endoscopy 2011, 43, 579-584.

52. Catalano, M.F.; Sivak, M.V., Jr.; Rice, T.; Gragg, L.A.; Van Dam, J. Endosonographic features predictive of lymph node metastasis. Gastrointest. Endosc. 1994, 40, 442-446.

53. Faige, D.O. EUS in patients with benign and malignant lymphadenopathy. Gastrointest. Endosc. 2001, 53, 593-598.

54. Eloubeidi, M.A.; Wallace, M.B.; Reed, C.E.; Hadzijahic, N.; Lewin, D.N.; Van Velse, A.; Leveen, M.B.; Etemad, B.; Matsuda, K.; Patel, R.S.; et al. The utility of EUS and EUS-guided fine needle aspiration in detecting celiac lymph node metastasis in patients with esophageal cancer: A single-center experience. Gastrointest. Endosc. 2001, 54, 714-719.

55. Okasha, H.; Elkholy, S.; Sayed, M.; Salman, A.; Elsherif, Y.; El-Gemeie, E. Endoscopic ultrasound-guided fine-needle aspiration and cytology for differentiating benign from malignant lymph nodes. Arab. J. Gastroenterol. 2017, 18, 74-79.

56. Chin, Y.K.; Iglesias-Garcia, J.; de la Iglesia, D.; Larino-Noia, J.; Abdulkader-Nallib, I.; Lázare, H.; Olmedo, S.R.; Dominguez-Muñoz, J.E. Accuracy of endoscopic ultrasound-guided tissue acquisition in the evaluation of lymph nodes enlargement in the absence of on-site pathologist. World J. Gastroenterol. 2017, 23, 5755-5763.

57. Chen, V.K.; Eloubeidi, M.A. Endoscopic ultrasound-guided fine needle aspiration is superior to lymph node echofeatures: A prospective evaluation of mediastinal and peri-intestinal lymphadenopathy. Am. J. Gastroenterol. 2004, 99, 628-633.

58. Takasaki, Y.; Irisawa, A.; Shibukawa, G.; Sato, A.; Abe, Y.; Yamabe, A.; Arakawa, N.; Maki, T.; Yoshida, Y.; Igarashi, R.; et al. New endoscopic ultrasonography criteria for malignant lymphadenopathy based on interrater agreement. PLoS ONE 2019, 14, e0212427.

59. Du Melo, A.W.; Panjala, C.; Crespo, S.; Diehl, N.N.; Woodward, T.A.; Raimondo, M.; Wallace, M.B. Interobserver agreement on the endosonographic features of lymph nodes in aerodigestive malignancies. Dig. Dis. Sci. 2011, 56, 3204-3208.

60. Xia, Y.; Kitano, M.; Kudo, M.; Imai, H.; Kamata, K.; Sakamoto, H.; Komaki, T. Characterization of intra-abdominal lesions of undetermined origin by contrast-enhanced harmonic EUS (with videos). Gastrointest. Endosc. 2010, 72, 637-642.

61. Giovannini, M.; Hookey, L.C.; Bories, E.; Pesenti, C.; Monges, G.; Delpero, J.R. Endoscopic ultrasound elastography: The first step towards virtual biopsy? Preliminary results in 49 patients. Endoscopy 2006, 38, 344-348.

62. Janssen, J.; Dietrich, C.F.; Will, U.; Greiner, L. Endosonographic elastography in the diagnosis of mediastinal lymph nodes. Endoscopy 2007, 39, 952-957.

63. Larsen, M.H.; Fristrup, C.W.; Mortensen, M.B. Intra- and interobserver agreement of endoscopic sonoelastography in the evaluation of lymph nodes. Ultraschall Med. 2011, 32, E45-E50.

64. Muguruma, N.; Okamura, S.; Ichikawa, S.; Tsujigami, K.; Suzuki, M.; Tadatsu, M.; Kusaka, Y.; Okita, Y.; Yano, M.; Ito, S. Endoscopic ultrasonography in the diagnosis of gallbladder wall lesions in patients with gallstones. J. Clin. Ultrasound. 2001, 29, 395-400.

65. Kim, H.J.; Park, J.H.; Park, D.I.; Cho, Y.K.; Sohn, C.I.; Jeon, W.K.; Kim, B.I.; Choi, S.H. Clinical usefulness of endoscopic ultrasonography in the differential diagnosis of gallbladder wall thickening. Dig. Dis. Sci. 2012, 57, 508-515.

66. Choi, J.H.; Seo, D.W.; Choi, J.H.; Park, D.H.; Lee, S.S.; Lee, S.K.; Kim, M.H. Utility of contrast-enhanced harmonic EUS in the diagnosis of malignant gallbladder polyps (with videos). Gastrointest. Endosc. 2013, 78, 484-493.

67. Park, C.H.; Chung, M.J.; Oh, T.G.; Park, J.Y.; Bang, S.; Park, S.W.; Kim, H.; Hwang, H.K.; Lee, W.J.; Song, S.Y. Differential diagnosis between gallbladder adenomas and cholesterol polyps on contrast-enhanced harmonic endoscopic ultrasonography. Surg. Endosc. 2013, 27, 1414-1421.

68. Imazu, H.; Mori, N.; Kanazawa, K.; Chiba, M.; Toyoizumi, H.; Torisu, Y.; Koyama, S.; Hino, S.; Ang, T.L.; Tajiri, H. Contrast-enhanced harmonic endoscopic ultrasonography in the differential diagnosis of gallbladder wall thickening. Dig. Dis. Sci. 2014, 59, 1909-1916.

69. Sakamoto, H.; Kitano, M.; Matsui, S.; Kamata, K.; Komaki, T.; Imai, H.; Dote, K.; Kudo, M. Estimation of malignant potential of GI stromal tumors by contrast-enhanced harmonic EUS. Gastrointest. Endosc. 2011, 73, 227-237.

70. Gress, F.; Schmitt, C.; Savides, T.; Faigel, D.O.; Catalano, M.; Wassef, W.; Roubein, L.; Nickl, N.; Ciaccia, D.; Bhutani, M.; et al. Interobserver agreement for EUS in the evaluation and diagnosis of submucosal masses. Gastrointest. Endosc. 2001, 53, 71-76.

71. Guo, W.; Han, W.; Liu, J.; Jin, L.; Li, J.S.; Zhang, Z.T.; Wang, Y. Intrapancreatic accessory spleen: A case report and review of the literature. World J. Gastroenterol. 2009, 15, 1141-1143.

72. Marques, S.; Bispo, M.; Noia, L. Intrapancreatic accessory spleen: A diagnosis not to forget! Case Rep. Gastroenterol. 2016, 10, 749-754.

73. Rodriguez, E.; Netto, G.; Li, Q.K. Intrapancreatic accessory spleen: A case report and review of literature. Diagn. Cytopathol. 2013, 41, 466-469.

74. Kim, G.E.; Morris, J.D.; Anand, N.; DePalma, F.; Greenwald, B.D.; Kim, R.E.; Laczek, J.; Lee, W.J.; Papadopoulas, I.; Uradomo, L.; et al. Recognizing intrapancreatic accessory spleen via EUS: Interobserver variability. Endosc. Ultrasound. 2019, 8, 392-397.

75. Caletti, G.C.; Ferrari, A.; Bocus, P.; Togliani, T.; Scalorbi, C.; Barbara, L. Endoscopic ultrasonography in gastric lymphoma. Schweiz Med. Wochenschr. 1996, 126, 819-825.

76. Suekane, H.; Iida, M.; Yao, T.; Matsumoto, T.; Masuda, Y.; Fujishima, M. Endoscopic ultrasonography in primary gastric lymphoma: Correlation with endoscopic and histologic findings. Gastrointest. Endosc. 1993, 39, 139-145.

77. Palazzo, L.; Roseau, G.; Ruskone-Fourmestraux, A.; Rougier, P.; Chaussade, S.; Rambaud, J.C.; Couturier, D.; Paolaggi, J.A. Endoscopic ultrasonography in the local staging of primary gastric lymphoma. Endoscopy 1993, 25, 502-508.

78. Van Dam, J. The role of endoscopic ultrasonography in monitoring treatment: Response to chemotherapy in lymphoma. Endoscopy 1994, 26, 772-773.

79. Caletti, G.; Ferrari, A.; Brocchi, E.; Barbara, L. Accuracy of endoscopic ultrasonography in the diagnosis and staging of gastric cancer and lymphoma. Surgery 1993, 113, 14-27.

80. Fusaroli, P.; Buscarini, E.; Peyre, S.; Federici, T.; Parente, F.; Angelis, C.D.; Bonanno, G.; Meroni, E.; Napolitano, V.; Pisani, A.; et al. Interobserver agreement in staging gastric malt lymphoma by EUS. Gastrointest. Endosc. 2002, 55, 662-668.

81. Yanai, H.; Harada, T.; Okamoto, T.; Hirano, A.; Takeo, N.; Yoshida, T.; Okita, K.; Kawano, T. Prognostic value and interobserver agreement of endoscopic ultrasonography for superficial squamous cell carcinoma of the esophagus: A prospective study. Int. J. Gastrointest. Cancer 2003, 34, 1-8.

82. Burtin, P.; Rabot, A.F.; Heresbach, D.; Carpentier, S.; Rousselet, M.C.; Le Berre, N.; Boyer, J. Interobserver agreement in the staging of rectal cancer using endoscopic ultrasonography. Endoscopy 1997, 29, 620-625.

83. Roubein, L.D.; Lynch, P.; Glober, G.; Sinicrope, F.A. Interobserver variability in endoscopic ultrasonography: A prospective evaluation. Gastrointest. Endosc. 1996, 44, 573-577.

84. Meining, A.; Rösch, T.; Wolf, A.; Lorenz, R.; Allescher, H.D.; Kauer, W.; Dittler, H.J. High interobserver variability in endosonographic staging of upper gastrointestinal cancers. Z. Gastroenterol. 2003, 41, 391-394.

85. Kuwahara, T.; Hara, K.; Mizuno, N.; Haba, S.; Okuno, N.; Koda, H.; Miyano, A.; Fumihara, D. Current status of artificial intelligence analysis for endoscopic ultrasonography. Dig. Endosc. 2020. Online ahead of print.

86. Marya, N.B.; Powers, P.D.; Chari, S.T.; Gleeson, F.C.; Leggett, C.L.; Abu Dayyeh, B.K.; Chandrasekhara, V.; Iyer, P.G.; Majumder, S.; Pearson, R.K.; et al. Utilisation of artificial intelligence for the development of an EUS-convolutional neural network model trained to enhance the diagnosis of autoimmune pancreatitis. Gut 2020. Online ahead of print.

87. Minoda, Y.; Ihara, E.; Komori, K.; Ogino, H.; Otsuka, Y.; Chinen, T.; Tsuda, Y.; Ando, K.; Yamamoto, H.; Ogawa, Y. Efficacy of endoscopic ultrasound with artificial intelligence for the diagnosis of gastrointestinal stromal tumors. J. Gastroenterol. 2020. Online ahead of print.

88. Kuwahara, T.; Hara, K.; Mizuno, N.; Okuno, N.; Matsumoto, S.; Obata, M.; Kurita, Y.; Koda, H.; Toriyama, K.; Onishi, S.; et al. Usefulness of Deep Learning Analysis for the Diagnosis of Malignancy in Intraductal Papillary Mucinous Neoplasms of the Pancreas. Clin. Transl. Gastroenterol. 2019, 10, 1-8.