Polyp Detection and Segmentation from Video Capsule Endoscopy: A Review

Abstract

Video capsule endoscopy (VCE) is used widely nowadays for visualizing the gastrointestinal (GI) tract. Capsule endoscopy exams are prescribed usually as an additional monitoring mechanism and can help in identifying polyps, bleeding, etc. To analyze the large scale video data produced by VCE exams, automatic image processing, computer vision, and learning algorithms are required. Recently, automatic polyp detection algorithms have been proposed with various degrees of success. Though polyp detection in colonoscopy and other traditional endoscopy procedure based images is becoming a mature field, due to its unique imaging characteristics, detecting polyps automatically in VCE is a hard problem. We review different polyp detection approaches for VCE imagery and provide systematic analysis with challenges faced by standard image processing and computer vision methods.

1. Introduction

Video capsule endoscopy (VCE) is an innovative diagnostic imaging modality in gastroenterology, which acquires digital photographs of the gastrointestinal (GI) tract using a swallowable miniature camera device with LED flash lights [1,2]. The capsule transmits images of the gastrointestinal tract to a portable recording device. The captured images are then analyzed by gastroenterologists, who locate and detect abnormal features such as polyps, lesions, bleeding, etc. and carry out diagnostic assessments. A typical capsule exam consists of more than 50,000 images, during its operation time, which spans a duration of 8 to 10 h. Hence, examining each image sequence produced by VCE is an extremely time-consuming process. Clearly, an efficient and accurate automatic detection procedure would relieve the diagnosticians of the burden of analyzing a large number of images for each patient.

Detecting polyps from VCE imagery is one of the foremost problems in devising automated computer-aided detection and diagnosis systems. The polyps to be detected in the images are characterized by physicians according to human perception of their distinctive shapes, and also in some cases, by their color and texture in these geometric objects. In effect, according to medical information, the geometry of colonic polyps can be classified essentially into two types: pedunculate polyps, which are mushroom-like structures attached by a thin stalk to the colon mucosa, and sessile polyps, which are like caps (mushroom structures with no stalk). Their colors are generally red, reddish or rose, and their textures can be very similar to a human brain. Figure 1 shows example polyps across different regions of the gastrointestinal (GI) tract, illustrating the variability in shape, color, and texture.

Figure 1. Variability of appearance in colonic polyps under the video capsule endoscopy (VCE) imaging system. Images are taken from PillCam® COLON capsule(Given Imaging Inc., Yoqneam, Israel) based exams on different patients. Notice the blurring, color, texture and geometric features (or lack of) at various levels. Other interferences for feature detectors are turbid gastrointestinal (GI) liquids. Trash which is present due to VCE exams does not involve colon cleaning unlike traditional colonoscopy imaging techniques.

Many previous studies have focused on detecting polyps on classical imaging techniques of colonoscopies (not capsule endoscopy images) [3,4,5,6,7,8,9,10]. Polyp detection approaches in colonoscopy imagery include using elliptical features [6], texture [3,11], color and position features [12,13] (see [14] for a review of polyp detection methods in colonoscopy images). Polyp detection schemes applicable to colonoscopy and Computed Tomography (CT) colonography use mainly geometry based techniques (see, for example, [15]). However, due to the different imaging modality in VCE, images have different characteristics and hence require unique methods for efficient polyp detection across various frames. Several shape based schemes that were proposed to find polyps in virtual colonoscopy or computed tomography colonography have been addressed (see, e.g., [15,16,17,18,19,20]). Most of these methods take the already reconstructed surface representing the colon’s interior or rely on some specific imaging techniques (see [21,22] for reviews). In contrast, VCE comes with an un-aided, uncontrolled photographic device, which moves automatically and is highly susceptible to illumination saturation due to near-field lighting [23]. Moreover, the images from VCE differs significantly from images obtained with the traditional colonoscopy. For example, the liquid material in the lumen section is less in colonoscopy and hence the images look more specular. However, in VCE images, the mucosa tissue looks diffusive under the presence of liquid and additionally the trash and turbidity can hinder the view of the mucosal surface [23]. Due to the unaided movement of the capsule camera, blurring effects make the image looks less sharper. Moreover, the color of mucosal tissue under VCE has some peculiar characteristics [24]. Due to these reasons particular to VCE, its sensitivity for detecting colonic lesions is low compared with the use of optical colonoscopy as noted in [25]. Nevertheless, a recent meta-analysis showed that capsule endoscopy is effective in detecting colorectal polyps [26] (at-least in the colon capsules, though the jury is still out on the small-bowel and esophagus). Newer advances in sensors, camera system results in second generation capsule endoscopes and sensitivity and specificity for detecting colorectal polyps was improved [27,28]. However, the increased imaging complexity and higher frame rates, though they provide more information, inevitably put more burden on the gastroenterologists. Thus, having efficient, robust automatic computer aided detection and segmentation of colorectal polyps is of great importance and needed now.

In this comprehensive survey paper, we provide an overview on different automatic image/video data based polyp detection (localization) and segmentation methods proposed in the literature so far (up to December 2016, and we refer the reader to the project website that is updated continuously with links to all the papers presented here to obtain more details about this research area [29]). and discuss the challenges that remain. We organized the rest of the paper as follows: Section 2 provides a review of polyp detection, segmentation and holistic techniques from the literature. In Section 3, we discuss the outlook in this field along with challenges that need to be tackled by future research.

2. Review of Polyp Detection and Segmentation in VCE

Variable lighting and rare occurrence of polyps in a given (full) VCE video creates immense difficulties in devising robust and data-driven methods for reliable detection and segmentation. We can classify polyp detection/segmentation methods into two categories: (a) polyp detection—finding where the polyp frame (polyp frames may contain more than one polyp. We do not make a distinction of detecting one or multiple polyps in a given image) occurs, not necessarily the location of the polyp within that frame; (b) polyp segmentation—once a frame that contains polyp(s) is given a segment mucosal area in which the polyp appears. Note that the first task is a much harder problem than the latter due to the large number of frames that the automatic algorithm needs to sift through in order to find the polyp frames, which is usually a rare occurrence. Naturally, machine learning based approaches are essential for both categories, especially for polyp detection since the occurrence of polyp frames, frames where at least one polyp is visible, are very few in contrast to the typical full length of video frames (typically greater than 50,000) in VCE imagery. The task is more complicated since colorectal polyps do not have common shape, texture, and color features even within a single patient’s video. Nevertheless, there have been efforts in identifying polyp frames using automatic data-driven algorithms. For polyp segmentation, it is relatively an easier problem since the automatic algorithms need only to analyze a given polyp frame to find and localize the polyps that are present. We next review polyp detection and segmentation methods studied so far in the literature and discuss the key techniques used with relevant results.

2.1. Polyp Detection in Capsule Endoscopy Videos

There are two main classes of polyps with respect to their appearance in shape: pedunculated and sessile. In Figure 2, we show some example polyps (selected from images shown in Figure 1) in 3D using the shape from shading techniques [30], indicating the amount of protrusion out of the mucosa surface (visualizations as 3D figures are available in the Supplementary Materials and also at the project website). Figure 3 shows a pedunculated polyp that appears in consecutive frames from a VCE exam of a patient. The appearance changes drastically (rotation, translation, and scale changes), and the polyp that is not so clear in the first few frames (Figure 3 (top row)) then becomes completely visible (Figure 3 (middle row)) and then becomes smaller before vanishing from view. In the last few frames (Figure 3 (bottom row)), the stalk of the polyp is not visible, thus appearing like a sessile polyp attached to the mucosal fold at the bottom. In addition, please note that the texture feature (due to vascularization) on top of the polyp is distinctively different from the surrounding mucosal folds that do contain small scale textures.

Figure 2. Shapes or geometry of polyps significantly differ along the GI tract. Examples of polyps: (a) pedunculated/stalked or subpedunculated; (b) sessile. We show a 3D representation of VCE frames obtained using shapes from shading techniques [30].

Figure 3. Shapes of polyps vary across different frames, within a single patient VCE exam, due to the unconstrained movement of the camera. (top row): data from PillCam® COLON 2 polyps occurring in consecutive neighboring frames ( #117 #117 to 122); (bottom two rows): other frames ( #123 #123 , 128, 133, 138, 143, 148, and 153, 158, 163, 168, 173, 178) from the polyp sequence where the shape and color undergo changes.

One of the earliest works in VCE image processing, and especially in detecting polyps automatically, is by Kodogiannis et al. [31], who studied an adaptive neuro-fuzzy approach. By utilizing texture spectrum from six channels (red-green-blue (RGB) and hue-saturation-value (HSV) color spaces) with an adaptive fuzzy logic system based classifier, they obtained 97% 97%sensitivity in 140 images with 70 polyp frames.

Li et al. [32] compared two different shape features to discriminate polyps from normal regions. They utilized an MPEG-7 shape descriptor (angular radial transform—ART), and Zernike moments as features along with a multi-layer perceptron (MLP) neural network as the classifier. Due to invariance to rotation, translation, and scale change, Zernike moments are well-suited for polyp detection in VCE, which has unconstrained movement of the camera. They tested their approach on 300 representative images, out of which 150 contained polyps, and achieved an accuracy of 86.1% 86.1% . However, their approach only compared two specific shape features and discarded color and texture information entirely. In a related work, Li et al. [33] exploited combined color and shape features for polyp detection with HSI (hue, saturation, intensity) color space. They used compressed 2D chromaticity histogram (from hue and saturation), and Zernike moments (from intensity) and tested MLP and support vector machines (SVM) classifiers. In 300 representative images, out of which 150 contain polyps, a 2D chromaticity histogram plus 5th order Zernike moments with MLP obtained 94.20% 94.20% accuracy when compared to color wavelet covariance feature based results of 70.50% 70.50%with SVM. The chromaticity histogram is compressed with discrete cosine transform and lower frequency coefficients. This quantization scheme affects the performance with respect to color discriminability.

Karargyris and Bourbakis [34] performed Log–Gabor filter based segmentation along with a smallest univalue segment assimilating nucleus (SUSAN) edge detector [35], curvature clusters, and active contour segmentation to identify polyp candidates. On a 50-frame video containing 10 polyp frames, they achieved a sensitivity of 100% 100% (all 10 frames were selected as polyp candidates by the scheme). The method relies heavily on a geometric rule that assumes that the polyp region is an ellipse close to a circle, which limits its applicability to detecting polyps of different shapes. Moreover, due to the small test cases of polyps, it is not clear how the proposed approach performs in a full VCE exam where the same polyp can have different geometric boundaries across frames. Karargyris and Bourbakis [36] later extended their previous work [34] by adding an SVM classifier. Hwang and Celebi [37] used watershed segmentation with initial markers selected using Gabor texture features and K-means clustering that avoided the requirement of accurate edge detection considered in [34]. On a set of 128 images with 64 polyp frames, this method achieved a sensitivity of 100% 100% . However, a similar assumption as [34] about the elliptical or circular shape was made and curvature clusters were used as an indicator for polyp candidates.

Nawarathna et al. [38] considered texton histograms for identifying abnormal regions with different classifiers such as SVM and K-nearest neighbors (K-NN). With Schmid filter bank based textons and SVM classifier, an accuracy of 95.27% 95.27% was obtained for polyp detection. Nawarathna et al. [39] later extended this approach with the addition of an local binary patterns (LBP) feature. In addition, a bigger filter bank (Leung–Malik), which includes Gaussian filters, were advocated for capturing texture more effectively. Note that these approaches rely only on texture features and do not include any color or geometrical features. The best results of 92% 92%accuracy was obtained for the Leung–Malik-LBP filter bank with K-NN classifier for 400 images with 25 polyps.

Figueiredo et al. [40] used a protrusion measure based on mean and Gaussian curvature for detecting polyps. They defined a novel protrusion measure which detected 80% 80% polyp frames accurately with localization of polyps. Unfortunately, not all polyps are protrusions (see Figure 2b for examples), and the curvature index derived in [40] fails to pick up VCE frames which contain sessile or flat polyps. Please also note that this method does not rely on any classification and is purely a geometry based approach.

Zhao and Meng [41] proposed using opponent color moments with LBP texture features computed over contourlet transformed images with an SVM classifier, and reported 97% 97%accuracy. Their work unfortunately did not mention how many total frames and polyp frames were used or how the color and texture features were fused.

Zhao et al. [42] studied a supervised classification approach with a hidden Markov model (HMM) by integrating temporal information for polyp detection in VCE videos. They utilized a combination of color, edge and texture features that resulted in a 118-D feature vector that was reduced to 13-D via a Laplacian eigenmap method along with a K-NN classifier. Their approach was tested on 400 images with 200 polyp frames and obtained an accuracy of 83.3% 83.3% . However, the true strength of their method lied in testing image sequences, and, for this purpose, they utilized a second dataset of 1120 images (224 videos of 5-frame duration), out of which there were 560 polyp frames. The obtained accuracy of 91.7% 91.7% for image sequences of 5-frame length indicates a strong potential for utilizing temporal information. In a related work, Zhao et al. [43] used the same features along with a boosted classifier and conducted evaluations using 1200 VCE images with 90% 90%classification accuracy.

Hwang [44] used a bag of visual words model from computer vision literature by treating polyp regions as positive documents and normal regions as the negative documents. The author used speeded up robust features (SURF) and quantized them with K-means clustering to generate the codebook to represent images as histograms of visual words. These feature vectors were then fed into an SVM classifier, and the results showed 90% 90% sensitivity in a total of 120 images with 60 polyp frames. A very similar approach was undertaken in [45] with color features from HSI space added in addition. Tests were conducted on 250 images with 50 polyp frames and 66% 66%sensitivity was obtained.

Li and Meng [46] applied the uniform LBP on discrete wavelet transformed sub-images to capture texture features on polyps. On a dataset of 1200 images with 600 polyp frames, an accuracy of 91.6% 91.6%was obtained with uniform LBP on 24 circular set members and radii of 3 with an SVM classifier. The method is based only on texture features and can fail to detect flat polyps with little texture. In addition, there was no mention of the dimension of the feature set based on uniform LBPs, which is usually high.

Condessa and Bioucas-Dias [47] studied an extension of the protrusion measure in [40] by utilizing a two-stage approach. The first stage involved multichannel segmentation, local polynomial approximation, and the second stage extracted contours and curvature features with an SVM classifier. They obtained 92.31% 92.31%sensitivity. The authors advocated using the temporal information via recursive stochastic filtering, though this has not yet been considered by other researchers.

David et al. [48] utilized the protrusion measure from [40]. However, they observed that not all polyps exhibit high curvature values. By augmenting these high curvature peak based locations with color and illumination segmentation, they applied a training stage with histograms of oriented Gaussians (HOG) features (computed on those locations) with an MLP classifier. Results on 30,540 30,540 with 540 polyp frames indicated that an accuracy of 80% 80%was achieved with this approach.

Figueiredo et al. [49] combined the protrusion measure [40] along with multiscale and uniform LBP texture features. They obtained an accuracy of 98.50% 98.50% in a Commission internationale de l’éclairage (CIE) Lab color space with monogenic LBP and a linear SVM classifier in 400 images with 200 polyp frames. The protrusion measure alone achieved 65.5% 65.5% accuracy, indicating that the reliance on polyp protrusion only is not a good assumption. Furthermore, with an addition of basic LBP features, the accuracy increased to 97.25% 97.25%, indicating the importance of texture.

Yuan and Meng [50] used scale invariant feature transform (SIFT) feature vectors with K-means clustering for bag of features representation of polyps. By integrating histograms in both saliency and non-saliency regions, the authors calculated weighted histograms of the visual words. These were fed into an SVM classifier and experiments on 872 images with 436 polyp frames showed that 92% 92% detection accuracy was obtained. In a related work, Yuan and Meng [51] tested Gabor filter and monogenic LBP features with an SVM classifier. Their results indicated an accuracy of 91.43% 91.43% on the same dataset. Yuan et al. [52] later extended these approaches with the addition of LBP, uniform LBP, complete LBP, and HOG features for capturing texture information along with SIFT features. They tested with SVM and Fisher’s linear discriminant analysis (FLDA) classifiers with different combinations of local features. Their method with SIFT + complete LBP with an SVM classifier achieved top classification accuracy of 93.2% 93.2%on a set of 2500 images that contained 500 polyp frames.

Mamonov et al. [53] proposed a system based on sphere fitting for VCE frames. The pipeline consisted of considering the gray-scale images and applying a cartoon + texture decomposition. The texture part of the given frame is enhanced with nonlinear convolution and then mid pass filtered to obtain binary segments of possible polyp regions. Then, a binary classifier uses a best-fit ball radius as an important decision parameter to decide whether there is a polyp present in the frame or not. This decision parameter is based on the assumption that the polyps are characterized as protrusions that are mostly round in shape, hence polyps which violate this do not get detected. The method tested on 18,968 18,968 frames with 230 polyp frames obtained 81.25% 81.25%sensitivity per polyp. Here, per polyp basis was computed as correct detection of at least one polyp frame in the corresponding sequence by the automatic algorithm.

Jia et al. [54] used geometrical features via ellipse fitting (area, ratio of major and minor axis), and multiscale rotation invariant LBP, and HOG texture features. With support vector regression, they reported a 64.8% 64.8% true positive rate on a total number of 27,984 27,984 frames with 12,984 12,984 polyp frames. The huge number of polyp frames were obtained by perturbing original 541 polyp frames to obtain 12,443 12,443extra samples for training.

Zhou et al. [55] utilized RGB averaging with variance for polyp localization and radius measurement of suspected protruding polyps using a statistical region merging approach. On a total of 359, this approach with an SVM classifier obtained 75% 75% sensitivity. Gueye et al. [56] used SIFT features and a Bag of Features (BoF) method. On 800 frames with 400 polyp frames, their SVM classifier obtained a classification rate 61.83% 61.83% , and on 400 frames with 200 polyp frames, the rate increased to 98.25% 98.25%.

Yuan and Meng [57] recently proposed a saliency driven model for polyp recognition. They used a global and local saliency coding method to encode with SIFT features computed in patches. Overall, their proposed method with an SVM classifier obtained 92.51% 92.51% sensitivity, 89.81% 89.81% specificity and 91.13% 91.13% accuracy in 1080 frames with 540 polyp frames. Yuan et al. [58] later extended this by adding local constrained linear coding for detecting not only polyps, but also bleeding and ulcers. They applied SIFT on the HSV color space and a feature coding based on saliency and an adaptive locality constrained linear approach. With an SVM classifier with max pooling on 17 polyp videos (containing 500 polyp frames), 83.50% 83.50%accuracy was reported.

In summary, there have been a small number of automatic polyp detection methods, though, unfortunately, the full dataset description is lacking in many of these published works, which makes it hard for us to benchmark them. Table 1 summarizes all of the polyp detection methods covered so far with main techniques, classifiers utilized along with tested dataset details (whenever available). It can be seen that the popular classifier is linear SVM with radial basis function as kernel due to its simplicity and ease of use. However, it is our belief that the majority of these methods either overfit or underfit, as the proposed methods are tuned to obtain best possible detection accuracy results for their corresponding datasets. All of the aforementioned methods highlight the accuracy of polyp detection based on how many polyp frames are detected out of all the input frames given. However, as mentioned by Mamonov et al. [53], per polyp accuracy over per frame basis is very important. This is because detecting a minimum of a single polyp frame in (typically) consecutive frames based sequences will suffice to alert the gastroenterologist/clinician. Figure 3 shows different frames of a single patient VCE exam wherein a sequence of length 55 contains a pedunculated polyp (original video available in the Supplementary Materials section). It is clear that detecting any one of the frames as a polyp frame is enough, as the gastroenterologist can inspect the neighboring frames manually. This prospective automatic polyp alert system can reduce the burden on gastroenterologists, as the number of frames to be inspected can be dramatically reduced, from tens of thousands of frames to a few hundred possible polyp sequences.

[ Table omitted. See PDF. ]

2.2. Polyp Localization or Segmentation within a VCE Frame

Polyp localization or segmentation of polyps in a single frame of VCE is an (relatively easier) object identification problem. The general task falls under the category of image segmentation, which is a well-studied problem in various biomedical imaging domains. As we have seen before, color, texture, and shape features individually are not discriminative enough to obtain reliable polyp segmentations from VCE frames. There have been a number of efforts in polyp segmentation from VCE which we classify based on which segmentation techniques are utilized. Note that the majority of the previously discussed polyp detection algorithms employ a polyp segmentation step to identify candidate polyp frames, although accurately segmenting the polyp region is not required for subsequent polyp detection in VCE.

1. Localization

1.

Filtering: Karargyris and Bourbakis [34] performed Log–Gabor filter based segmentation. Prasath and Kawanaka [59] proposed using a novel vascularization feature, which is based on Frangi vesselness filter.

2. Geometry: Hwang and Celebi [37] derive a localization approach using curvature center ratio on K-means clustering based segmented polyp frames. Figueiredo et al. [40] used a protrusion measure based on Gaussian and mean curvature to highlight possible polyp locations in a given VCE image.

3. Hybrid: Jia [60] used K-means clustering and localizing region-based active contour segmentation. The results shown in this paper seems to be from colonoscopy imagery and not VCE.

2. Accurate boundaries, segmentation

1.

Active contours: Meziou et al. [61] used alpha divergence based active contours segmentation, though their approach is a general segmentation, and not for polyps in particular (see also [62]). Prasath et al. [30] used an active contours without edges method for identifying mucosal surface in conjunction with shape from shading technique (see also [65]). Eskandari et al. [63] used a region based active contour model to segment the polyp region from a given image that contain a polyp (see also [64]).

Figure 4 highlights some example images and results obtained with the aforementioned automatic polyp localization and segmentation methods. Note that, in localization approaches, though polyp bounding boxes or circles were obtained, no crisp boundary segmentations are available. Obtaining accurate boundaries and segmentations of polyps in a frame requires a robust method that can avoid the pitfalls of bubbles, trash, and illumination artifacts. Table 2 summarizes the methods available for polyp localization and segmentation so far. A preliminary mucosa segmentation [30,65,66,67] may be required before applying the polyp segmentation step in order to avoid these non-polyp pixels.

Figure 4. Example polyps localization and segmentations. (top row): localization methods, (a) Log–Gabor filter [34]; (b) K-means, curvature [37]; (c) protrusion measure [40]; (d) vascularization [59]; (bottom row): accurate boundaries and segmentation methods; (e–h) variants of active contours [30,62,63,64].

[ Table omitted. See PDF. ]

2.3. Holistic Systems

Romain et al. [68] studied a preliminary multimodal VCE for detecting polyps by utilizing active stereo vision within the capsule itself. It is based on two steps: (1) 2D identification polyp regions of interest (ROI) using simple geometric shape features; and (2) classification of polyps using 3D parameters computed on ROI using active stereo vision. Although the authors of [68] perform a prototype made of silicone for in vitro learning, the experiments were limited by the fact that there are no liquids (GI juices), trash, or a distinct lack of distention of the colon walls, which affect the performance of automatic image analysis methods in real VCE exams. Similar efforts have been reported in [69,70].

Table 3 summarizes these holistic system approaches proposed so far. Note that the polyp detection testing is done using traditional colonoscopy images for now. Apart from these technological constraints, these approaches require robust embeddable computer vision systems that need to operate under a strict energy budget (battery constraints). However, a computer vision embedded VCE system can revolutionize the diagnosis procedures that have been done manually through tedious processes so far.

[ Table omitted. See PDF. ]

3. Discussion and Outlook

Automatic polyp detection and segmentation is a nascent area which requires various computer vision methodologies such as geometrical primitives, color spaces, texture descriptors, features matching, along with strong machine learning components. A standard assumption made in classical colonoscopy imagery based polyp detection methods is that polyps have high geometrical features such as well-defined shape [20] or protrusion out of mucosal surfaces [15]. Thus, curvature measures are widely utilized in detecting polyps and the adaptation to VCE imagery is undertaken with limited success [40,47,48,49]. Texture or color features based schemes such as [11,12] applied to colonoscopy images do not work well in WCE when used on their own, as we have seen in Section 2.1. This is because, though the polyps appear to have different textural characteristics than the surrounding mucosa, bubble, trash and capsule motion induced by uncontrollable peristalsis have a strong effect on texture features. In such a scenario, classifying with texture features alone is highly error prone. For example, the polyp shown in Figure 3 (top row) has a unique texture pattern, whereas for the polyp in Figure 3 (middle row), texture appears to be uniformly spread across the mucosal structure surrounding it. Moreover, in a video, the neighboring frames show a partial view of the polyp with mucosa folds having a similar texture. Similarly, the color of the polyp is not homogeneous across different polyp frames within a patient and highly variable across different exams from patients. In a comprehensive learning framework, one can try to incorporate local and global texture features for discerning polyps along with vascularization, and color information. Existing approaches reviewed in Section 2 are plagued by various factors. One of the main problems is the existence of trash and bubbles since no colon cleaning is required in VCE exams. Robust polyp detection approaches must combine efficient trash, and bubble detectors to avoid false positives. To conclude, we believe, a holistic view of combining, motion, geometry, color, and texture with a strong machine learning paradigm may prove to be successful for robust, efficient automatic polyp detection in VCE imagery.

Based on the detailed description of the current state-of-the-art polyp detection and segmentation methods, we observe the following salient points for future outlook:

1. Recent excitement generated by deep learning is a very promising direction where massively trained neural network based classifiers can be used to better differentiate polyp frames from normal frames. However, deep learning networks in general require a huge amount of training data, in particular labeled data of positive (polyp frame) and negative (normal frame) samples. One possible remedy for an imbalanced data problem is to use data augmentation, therein one can increase the polyp frames by artificial perturbation (rotation, reflection, ...) (see e.g., [54], for an attempt to create a higher number of polyp frames for training). There have been some works in the last two years on endoscopy image analysis with deep learning [71,72,73]. Other approaches include using deep sparse feature selection (see, e.g., [74]).

2. Similar to the Arizona State University (ASU)-Mayo Clinic polyp database for colonoscopy polyp detection benchmarking, VCE polyp detection requires a well-defined database with multiple expert gastroenterologists marked polyp regions. This will make the benchmarking and testing of different methodologies for automatic polyp detection and segmentation standardized.

3. Sensor improvements with novel capsule systems [75,76], such as more control in terms of higher image resolution, standardized illumination/contrast, controlled capsule speed, and variable image capturing mechanisms, can help automatic image analysis.

4. Finally, embedding the image analysis part within capsule endoscopy imaging systems [70,77] is an exciting research area which will enable the gastroenterologists to make real-time decisions. However, there are a lot of challenges that remain for essential progress [78].

Acknowledgments

This work was initiated while the author was a post doctoral fellow at the Department of Mathematics, University of Coimbra, Portugal under a UTAustin|Portugal program and a Fundação para a Ciência e a Tecnologia (FCT) fellowship. The author sincerely thanks Isabel N. Figueiredo (Mathematics, University of Coimbra, Coimbra, Portugal), Yen-Hsai Tsai (Mathematics, University of Texas, Austin, TX, USA), and Pedro N. Figueiredo (Gastroenterology, University Hospital of Coimbra, Coimbra, Portugal) for initial collaborations in video capsule endoscopy image processing and analysis. The author sincerely thanks the reviewers for their comments, which improved this review.

Conflicts of Interest

The author declares no conflict of interest.

References

1. Iddan, G.; Meron, G.; Glukhovsky, A.; Swain, F. Wireless capsule endoscopy. Nature 2000, 405, 417.

2. Li, Z.; Liao, Z.; McAlindon, M. Handbook of Capsule Endoscopy; Springer: Dordrecht, The Netherlands, 2014.

3. Iakovidis, D.K.; Maroulis, D.E.; Karkanis, S.A.; Brokos, A. A comparative study of texture features for the discrimination of gastric polyps in endoscopic video. In Proceedings of the 18th IEEE Symposium on Computer-Based Medical Systems, Washington, DC, USA, 23–24 June 2005; pp. 575–580.

4. Iakovidis, D.K.; Maroulis, D.E.; Karkanis, S.A. An intelligent system for automatic detection of gastrointestinal adenomas in video endoscopy. Comput. Biol. Med. 2006, 36, 1084–1103.

5. Jiang, Y.; Jaffer, N. A novel segmentation and navigation method for polyps detection using mathematical morphology and active contour models. In Proceedings of the 6th IEEE International Conference on Cognitive Informatics, Lake Tahoe, CA, USA, 6–8 August 2007; pp. 357–363.

6. Hwang, S.; Oh, J.; Tavanapong, W.; Wong, J.; De Groen, P.C. Polyp detection in colonoscopy video using elliptical shape feature. In Proceedings of the 2007 IEEE International Conference on Image Processing, San Antonio, TX, USA, 16 September–19 October 2007; Volume 2, pp. 465–468.

7. Park, S.Y.; Sargent, D.; Spofford, I.; Vosburgh, K.G.; A-Rahim, Y. A colon video analysis framework for polyp detection. IEEE Trans. Biomed. Eng. 2012, 59, 1408–1418.

8. Bernal, J.; Sánchez, J.; Vilarino, F. Towards automatic polyp detection with a polyp appearance model. Pattern Recognit. 2012, 45, 3166–3182.

9. Bernal, J.; Sánchez, F.J.; Fernández-Esparrach, G.; Gil, D.; Rodríguez, C.; Vilariño, F. WM-DOVA maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians. Comput. Med. Imaging Gr. 2015, 43, 99–111.

10. Tajbakhsh, N.; Gurudu, S.R.; Liang, J. Automated polyp detection in colonoscopy videos using shape and context information. IEEE Trans. Med. Imaging 2016, 35, 630–644.

11. Cheng, D.C.; Ting, W.C.; Chen, Y.F.; Pu, Q.; Jiang, X. Colorectal polyps detection using texture features and support vector machine. In Advances in Mass Data Analysis of Images and Signals in Medicine, Biotechnology, Chemistry and Food Industry; Springer: Berlin/Heidelberg, Germany, 2008; pp. 62–72.

12. Alexandre, L.A.; Nobre, N.; Casteleiro, J. Color and position versus texture features for endoscopic polyp detection. In Proceedings of the 2008 International Conference on BioMedical Engineering and Informatics, Sanya, China, 27–30 May 2008; Volume 2, pp. 38–42.

13. Alexandre, L.A.; Casteleiro, J.; Nobre, N. Polyp detection in endoscopic video using SVMs. In Knowledge Discovery in Databases: PKDD 2007; Springer: Berlin/Heidelberg, Germany, 2007; pp. 358–365.

14. Ameling, S.; Wirth, S.; Paulus, D. Methods for Polyp Detection in colonoscopy Video: A Review; Technical Report 14; University of Koblenz–Landau: Koblenz Landau, Germany, 2008.

15. Yoshida, H.; Näppi, J. Three-dimensional computer-aided diagnosis scheme for detection of colonic polyps. IEEE Trans. Med. Imaging 2001, 20, 1261–1274.

16. Gokturk, S.B.; Tomasi, C.; Acar, B.; Beaulieu, C.F.; Paik, D.S.; Jeffrey, R.B., Jr.; Yee, J.; Napel, S. A statistical 3-D pattern processing method for computer-aided detection of polyps in CT colonography. IEEE Trans. Med. Imaging 2001, 20, 1251–1260.

17. Paik, D.S.; Beaulieu, C.F.; Rubin, G.D.; Acar, B.; Jeffrey, R.B., Jr.; Yee, J.; Dey, J.; Napel, S. Surface normal overlap: A computer-aided detection algorithm with application to colonic polyps and lung nodules in helical CT. IEEE Trans. Med. Imaging 2004, 23, 661–675.

18. Konukoglu, E.; Acar, B.; Paik, D.S.; Beaulieu, C.F.; Rosenberg, J.; Napel, S. Polyp enhancing level set evolution of colon wall: Method and pilot study. IEEE Trans. Med. Imaging 2007, 26, 1649–1656.

19. Van Wijk, C.; Van Ravesteijn, V.F.; Vos, F.M.; Van Vliet, L.J. Detection and segmentation of colonic polyps on implicit isosurfaces by second principal curvature flow. IEEE Trans. Med. Imaging 2010, 29, 688–698.

20. Ruano, J.; Martinez, F.; Gomez, M.; Romero, E. Shape estimation of gastrointestinal polyps using motion information. In Proceedings of the IX International Seminar on Medical Information Processing and Analysis, Mexico City, Mexico, 11–14 November 2013.

21. Liedlgruber, M.; Uhl, A. Computer-aided decision support systems for endoscopy in the gastrointestinal tract: A review. IEEE Rev. Biomed. Eng. 2011, 4, 73–88.

22. El Khatib, A.; Werghi, N.; Al-Ahmad, H. Automatic polyp detection: A comparative study. In Proceedings of the 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Milano, Italy, 25–29 August 2015; pp. 2669–2672.

23. Prasath, V.B.S.; Delhibabu, R. Automatic contrast enhancement for wireless capsule endoscopy videos with spectral optimal contrast-tone mapping. In Computational Intelligence in Data Mining; Jain, L., Behera, H.S., Mandal, J.K., Mohapatra, D.P., Eds.; Springer: Odisha, India, 2015; Volume 1, pp. 243–250.

24. Prasath, V.B.S. On fuzzification of color spaces for medical decision support in video capsule endoscopy. In Proceedings of the 26th Modern Artificial Intelligence and Cognitive Science Conference, Greensboro, NC, USA, 25–26 April 2015; pp. 1–5.

25. Van Gossum, A.; Munoz-Navas, M.; Fernandez-Urien, I.; Carretero, C.; Gay, G.; Delvaux, M.; Lapalus, M.G.; Ponchon, T.; Neuhaus, H.; Philipper, M.; et al. Capsule endoscopy versus colonoscopy for the detection of polyps and cancer. N. Engl. J. Med. 2009, 361, 264–270.

26. Spada, C.; Hassan, C.; Marmo, R.; Petruzziello, L.; Riccioni, M.E.; Zullo, A.; Cesaro, P.; Pilz, J.; Costamagna, G. Meta-analysis shows colon capsule endoscopy is effective in detecting colorectal polyps. Clin. Gastroenterol. Hepatol. 2010, 8, 516–522.

27. Spada, C.; Hassan, C.; Munoz-Navas, M.; Neuhaus, H.; Deviere, J.; Fockens, P.; Coron, E.; Gay, G.; Toth, E.; Riccioni, M.E.; et al. Second-generation colon capsule endoscopy compared with colonoscopy. Gastrointest. Endosc. 2011, 74, 581–589.

28. Spada, C.; De Vincentis, F.; Cesaro, P.; Hassan, C.; Riccioni, M.E.; Grazioli, L.M.; Bolivar, S.; Zurita, A.; Costamagna, G. Accuracy and safety of second-generation PillCam COLON capsule for colorectal polyp detection. Ther. Adv. Gastroenterol. 2012, 5, 173–178.

29. Polyp Detection and Segmentation in Video Capsule Endoscopy: A Review. Available online: http://goo.gl/eAUWKJ (accessed on 21 December 2016).

30. Prasath, V.B.S.; Figueiredo, I.N.; Figueiredo, P.N.; Palaniappan, K. Mucosal region detection and 3D reconstruction in wireless capsule endoscopy videos using active contours. In Proceedings of the 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, San Diego, CA, USA, 28 August–1 September 2012; pp. 4014–4017.

31. Kodogiannis, V.; Boulougoura, M. An adaptive neurofuzzy approach for the diagnosis in wireless capsule endoscopy imaging. Int. J. Inf. Technol. 2007, 13, 46–56.

32. Li, B.; Meng, M.Q.; Xu, L. A comparative study of shape features for polyp detection in wireless capsule endoscopy images. In Proceedings of the 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Piscataway, NJ, USA, 3–6 September 2009; pp. 3731–3734.

33. Li, B.; Fan, Y.; Meng, M.Q.; Qi, L. Intestinal polyp recognition in capsule endoscopy images using color and shape features. In Proceedings of the 2009 international conference on Robotics and biomimetics, Guilin, China, 19–23 December 2009; pp. 1490–1494.

34. Karargyris, A.; Bourbakis, N. Identification of polyps in Wireless Capsule Endoscopy videos using log Gabor filters. In Proceedings of the 2009 IEEE/NIH Life Science Systems and Applications Workshop, Arlington, TX, USA, 9–10 April 2009; pp. 143–147.

35. Smith, S.M.; Brady, J.M. SUSAN—A new approach to low level image processing. Int. J. Comput. Vis. 1997, 23, 45–78.

36. Karargyris, A.; Bourbakis, N. Detection of small bowel polyps and ulcers in wireless capsule endoscopy videos. IEEE Trans. Biomed. Eng. 2011, 58, 2777–2786.

37. Hwang, S.; Celebi, M.E. Polyp detection in wireless capsule endoscopy videos based on image segmentation and geometric feature. In Proceedings of the 2010 IEEE International Conference on Acoustics, Speech and Signal Processing, Dallas, TX, USA, 14–19 March 2010; pp. 678–681.

38. Nawarathna, R.D.; Oh, J.; Yuan, X.; Lee, J.; Tang, S.J. Abnormal image detection using texton method in wireless capsule endoscopy videos. In Medical Biometrics; Springer: Berlin/Heidelberg, Germany, 2010; pp. 153–162.

39. Nawarathna, R.; Oh, J.; Muthukudage, J.; Tavanapong, W.; Wong, J.; De Groen, P.C.; Tang, S.J. Abnormal image detection in endoscopy videos using a filter bank and local binary patterns. Neurocomputing 2014, 144, 70–91.

40. Figueiredo, P.N.; Figueiredo, I.N.; Prasath, S.; Tsai, R. Automatic polyp detection in Pillcam colon 2 capsule images and videos: Preliminary feasibility report. Diagn. Ther. Endosc. 2011, 2011, 182435.

41. Zhao, Q.; Meng, M.Q.H. Polyp detection in wireless capsule endoscopy images using novel color texture features. In Proceedings of the 9th World Congress on Intelligent Control and Automation, Taipei, Taiwan, 21–25 June 2011; pp. 948–952.

42. Zhao, Q.; Dassopoulos, T.; Mullin, G.; Hager, G.; Meng, M.Q.; Kumar, R. Towards integrating temporal information in capsule endoscopy image analysis. In Proceedings of the 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Boston, MA, USA, 30 August–3 September 2011; pp. 6627–6630.

43. Zhao, Q.; Dassopoulos, T.; Mullin, G.E.; Meng, M.; Kumar, R. A decision fusion strategy for polyp detection in capsule endoscopy. In Studies in Health Technology and Informatics, Proceedings of the Medicine Meets Virtual Reality conference (MMVR19), Newport Beach, CA, USA, February 2012; Westwood, J., Westwood, S., Fellander-Tsai, L., Haluck, R., Robb, R., Senger, S., Vosburgh, K., Eds.; IOS Press: Hong Kong, China, 2012; Volume 173, pp. 559–565.

44. Hwang, S. Bag-of-visual-words approach based on SURF features to polyp detection in wireless capsule endoscopy videos. In World Congress in Computer Science, Computer Engineering, and Applied Computing; Department of Computer Science, University of Illinois at Springfield: Springfield, IL, USA, 2011.

45. Hwang, S. Bag-of-visual-words approach to abnormal image detection in wireless capsule endoscopy videos. In Advances in Visual Computing; Springer: Berlin/Heidelberg, Germany, 2011; pp. 320–327.

46. Li, B.; Meng, M.Q.H. Automatic polyp detection for wireless capsule endoscopy images. Expert Syst. Appl. 2012, 39, 10952–10958.

47. Condessa, F.; Bioucas-Dias, J. Segmentation and detection of colorectal polyps using local polynomial approximation. In Image Analysis and Recognition; Springer: Berlin/Heidelberg, Germany, 2012; pp. 188–197.

48. David, E.; Boia, R.; Malaescu, A.; Carnu, M. Automatic colon polyp detection in endoscopic capsule images. In Proceedings of the International Symposium on Signals, Circuits and Systems (ISSCS), Iasi, Romania, 11–12 July 2013; pp. 1–4.

49. Figueiredo, I.N.; Kumar, S.; Figueiredo, P.N. An intelligent system for polyp detection in wireless capsule endoscopy images. Comput. Vision Med. Image Process. IV VIPIMAGE 2013, 2013, 229–235.

50. Yuan, Y.; Meng, M.Q.H. Polyp classification based on bag of features and saliency in wireless capsule endoscopy. In Proceedings of the 2014 IEEE International Conference on Robotics and Automation (ICRA), 31 May–7 June 2014; pp. 3930–3935.

51. Yuan, Y.; Meng, M.Q.H. A novel feature for polyp detection in wireless capsule endoscopy images. In Proceedings of the 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2014), Chicago, IL, USA, 14–18 September 2014; pp. 5010–5015.

52. Yuan, Y.; Li, B.; Meng, M.Q.H. Improved bag of feature for automatic polyp detection in wireless capsule endoscopy images. IEEE Trans. Autom. Sci. Eng. 2016, 13, 529–535.

53. Mamonov, A.V.; Figueiredo, I.N.; Figueiredo, P.N.; Tsai, Y.H.R. Automated polyp detection in colon capsule endoscopy. IEEE Trans. Med. Imaging 2014, 33, 1488–1502.

54. Jia, J.; Sun, S.; Terrence, T.; Wang, P. Accurate and Efficient Polyp Detection in Wireless Capsule Endoscopy Images. U.S. Patent App. 14/471,143, 5 March 2014.

55. Zhou, M.; Bao, G.; Geng, Y.; Alkandari, B.; Li, X. Polyp detection and radius measurement in small intestine using video capsule endoscopy. In Proceedings of the 2014 7th International Conference on Biomedical Engineering and Informatics (BMEI), Dalian, China, 14–16 October 2014; pp. 237–241.

56. Gueye, L.; Yildirim-Yayilgan, S.; Cheikh, F.A.; Balasingham, I. Automatic detection of colonoscopic anomalies using capsule endoscopy. In Proceedings of the 2015 IEEE International Conference on Image Processing (ICIP), Quebec City, QC, Canada, 27–30 September 2015; pp. 1061–1064.

57. Yuan, Y.; Meng, M.Q.H. A novel global and local saliency coding method for polyp recognition in WCE videos. In Proceedings of the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, Korea, 9–14 October 2016; pp. 2394–2399.

58. Yuan, Y.; Li, B.; Meng, M.Q.H. WCE abnormality detection based on saliency and adaptive locality-constrained linear coding. IEEE Trans. Autom. Sci. Eng. 2016, 1–11.

59. Prasath, V.B.S.; Kawanaka, H. Vascularization features for polyp localization in capsule endoscopy. In Proceedings of the 2015 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Washington, DC, USA, 9–12 November 2015; pp. 1740–1742.

60. Jia, Y. Polyps auto-detection in wireless capsule endoscopy images using improved method based on image segmentation. In Proceedings of the 2015 IEEE International Conference on Robotics and Biomimetics (ROBIO), Zhuhai, China, 6–9 December 2015; pp. 1631–1636.

61. Meziou, L.; Histace, A.; Dray, X.; Romain, O.; Granado, B. Segmentation of video capsule endoscopic images using alpha-divergence based active contour method. In Computer Assisted Radiology and Surgery; Springer: Pisa, Italy, 2012; pp. S325–S326.

62. Meziou, L.; Histace, A.; Precioso, F.; Romain, O.; Dray, X.; Granado, B.; Matuszewski, B.J. Computer-assisted segmentation of videocapsule images using alpha-divergence-based active contour in the framework of intestinal pathologies detection. J. Biomed. Imaging 2014, 2014, 428583.

63. Eskandari, H.; Talebpour, A.; Alizadeh, M.; Soltanian-Zadeh, H. Polyp detection in Wireless Capsule Endoscopy images by using region-based active contour model. In Proceedings of the 2012 19th Iranian Conference of Biomedical Engineering (ICBME), Ehran, Iran, 20–21 December 2012; pp. 305–308.

64. Alizadeh, M.; Zadeh, H.S.; Maghsoudi, O.H. Segmentation of small bowel tumors in wireless capsule endoscopy using level set method. In Proceedings of the 2014 IEEE 27th International Symposium on Computer-Based Medical Systems (CBMS), New York, NY, USA, 27–29 May 2014; pp. 562–563.

65. Prasath, V.B.S.; Figueiredo, I.N.; Figueiredo, P.N. Colonic Mucosa Detection in Wireless Capsule Endoscopic Images and Videos. In Proceedings of the Congress on Numerical Methods in Engineering (CMNE 2011), Coimbra, Portugal, 14–17 June 2011.

66. Prasath, V.B.S.; Delhibabu, R. Automatic image segmentation for video capsule endoscopy. In Computational Intelligence in Medical Informatics; Muppalaneni, N.B., Gunjan, V.K., Eds.; Briefs in Applied Sciences and Technology; Springer: Visakhapatnam, India, 2015; pp. 73–80.

67. Prasath, V.B.S.; Delhibabu, R. Automatic mucosa detection in video capsule endoscopy with adaptive thresholding. In Computational Intelligence in Data Mining; Springer: Bhubaneswar, India, 2016; Volume 1, pp. 95–102.

68. Romain, O.; Histace, A.; Silva, J.; Ayoub, J.; Granado, B.; Pinna, A.; Dray, X.; Marteau, P. Towards a multimodal wireless video capsule for detection of colonic polyps as prevention of colorectal cancer. In Proceedings of the 2013 IEEE 13th International Conference on Bioinformatics and Bioengineering (BIBE), Chania, Greece, 10–13 November 2013; pp. 1–6.

69. Silva, J.; Histace, A.; Romain, O.; Dray, X.; Granado, B.; Pinna, A. Towards real-time in situ polyp detection in WCE images using a boosting-based approach. In Proceedings of the 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Osaka, Japan, 3–7 July 2013; pp. 5711–5714.

70. Silva, J.; Histace, A.; Romain, O.; Dray, X.; Granado, B. Toward embedded detection of polyps in WCE images for early diagnosis of colorectal cancer. Int. J. Comput. Assist. Radiol. Surg. 2014, 9, 283–293.

71. Zhu, R.; Zhang, R.; Xue, D. Lesion detection of endoscopy images based on convolutional neural network features. In Proceedings of the 8th International Congress on Image and Signal Processing (CISP), Shenyang, China, 14–16 October 2015.

72. Bae, S.H.; Yoon, K.J. Polyp detection via imbalanced learning and discriminative feature learning. IEEE Trans. Med. Imaging 2015, 34, 2379–2393.

73. Yu, L.; Chen, H.; Dou, Q.; Qin, J.; Heng, P.A. Integrating online and offline 3D deep learning for automated polyp detection in colonoscopy videos. IEEE J. Biomed. Health Inform. 2016.

74. Cong, Y.; Wang, S.; Liu, J.; Cao, J.; Yang, Y.; Luo, J. Deep sparse feature selection for computer aided endoscopy diagnosis. Pattern Recognit. 2015, 48, 907–917.

75. Hatogai, K.; Hosoe, N.; Imaeda, H.; Rey, J.F.; Okada, S.; Ishibashi, Y.; Kimura, K.; Yoneno, K.; Usui, S.; Ida, Y.; et al. Role of enhanced visibility in evaluating polyposis syndromes using a newly developed contrast image capsule endoscope. Gut Liv. 2012, 6, 218–222.

76. Filip, D.; Yadid-Pecht, O.; Andrews, C.N.; Mintchev, M.P. Self-stabilizing colonic capsule endoscopy: Pilot study of acute canine models. IEEE Trans. Med. Imaging 2011, 30, 2115–2125.

77. Angermann, Q.; Histace, A.; Romain, O.; Dray, X.; Pinna, A.; Granado, B. Smart videocapsule for early diagnosis of colorectal cancer: Toward embedded image analysis. In Computational Intelligence in Digital and Network Designs and Applications; Springer: Cham, Switzerland, 2015; pp. 325–350.

78. Iakovidis, D.K.; Koulaouzidis, A. Software for enhanced video capsule endoscopy: Challenges for essential progress. Nat. Rev. Gastroenterol. Hepatol. 2015, 12, 172–186.