Pathology Image Analysis Using Segmentation Deep Learning Algorithms
2019; Elsevier BV; Volume: 189; Issue: 9 Linguagem: Inglês
10.1016/j.ajpath.2019.05.007
ISSN1525-2191
AutoresShidan Wang, Donghan M. Yang, Ruichen Rong, Xiaowei Zhan, Guanghua Xiao,
Tópico(s)Medical Imaging and Analysis
ResumoWith the rapid development of image scanning techniques and visualization software, whole slide imaging (WSI) is becoming a routine diagnostic method. Accelerating clinical diagnosis from pathology images and automating image analysis efficiently and accurately remain significant challenges. Recently, deep learning algorithms have shown great promise in pathology image analysis, such as in tumor region identification, metastasis detection, and patient prognosis. Many machine learning algorithms, including convolutional neural networks, have been proposed to automatically segment pathology images. Among these algorithms, segmentation deep learning algorithms such as fully convolutional networks stand out for their accuracy, computational efficiency, and generalizability. Thus, deep learning–based pathology image segmentation has become an important tool in WSI analysis. In this review, the pathology image segmentation process using deep learning algorithms is described in detail. The goals are to provide quick guidance for implementing deep learning into pathology image analysis and to provide some potential ways of further improving segmentation performance. Although there have been previous reviews on using machine learning methods in digital pathology image analysis, this is the first in-depth review of the applications of deep learning algorithms for segmentation in WSI analysis. With the rapid development of image scanning techniques and visualization software, whole slide imaging (WSI) is becoming a routine diagnostic method. Accelerating clinical diagnosis from pathology images and automating image analysis efficiently and accurately remain significant challenges. Recently, deep learning algorithms have shown great promise in pathology image analysis, such as in tumor region identification, metastasis detection, and patient prognosis. Many machine learning algorithms, including convolutional neural networks, have been proposed to automatically segment pathology images. Among these algorithms, segmentation deep learning algorithms such as fully convolutional networks stand out for their accuracy, computational efficiency, and generalizability. Thus, deep learning–based pathology image segmentation has become an important tool in WSI analysis. In this review, the pathology image segmentation process using deep learning algorithms is described in detail. The goals are to provide quick guidance for implementing deep learning into pathology image analysis and to provide some potential ways of further improving segmentation performance. Although there have been previous reviews on using machine learning methods in digital pathology image analysis, this is the first in-depth review of the applications of deep learning algorithms for segmentation in WSI analysis. Optical microscopy of pathology slides captures the histologic details of tissues in high resolution. With the rapid advance of technology, whole slide imaging (WSI) is becoming part of the routine procedure for clinical diagnosis of many diseases. The emergence of digital pathology1Jara-Lazaro A.R. Thamboo T.P. Teh M. Tan P.H. Digital pathology: exploring its applications in diagnostic surgical pathology practice.Pathology. 2010; 42: 512-518Abstract Full Text PDF PubMed Scopus (77) Google Scholar, 2Webster J.D. Dunstan R.W. Whole-slide imaging and automated image analysis: considerations and opportunities in the practice of pathology.Vet Pathol. 2014; 51: 211-223Crossref PubMed Scopus (107) Google Scholar provides new opportunities to develop algorithms and software tools that can assist pathologists in clinical diagnosis and researchers in studying disease mechanisms. The digitalized pathology slides are often called images in the computer vision field, and can benefit from many image analysis algorithms. As an example, the common task where pathologists locate and recognize tissue components can also be achieved by image segmentation and recognition algorithms. Nowadays, digital pathology is making rapid progress owing to the success of deep learning.3LeCun Y. Bengio Y. Hinton G. Deep learning.Nature. 2015; 521: 436-444Crossref PubMed Scopus (42675) Google Scholar Before the application of deep learning algorithms, digital pathology, due to its high complexity, achieved limited success with laborious modeling.4Janowczyk A. Madabhushi A. Deep learning for digital pathology image analysis: a comprehensive tutorial with selected use cases.J Pathol Inform. 2016; 7: 29Crossref PubMed Scopus (700) Google Scholar Since 2012, deep learning has made significant improvements in all image recognition benchmarks.3LeCun Y. Bengio Y. Hinton G. Deep learning.Nature. 2015; 521: 436-444Crossref PubMed Scopus (42675) Google Scholar, 5Krizhevsky A. Sutskever I. Hinton G.E. ImageNet classification with deep convolutional neural networks.Commun ACM. 2017; 60: 84-90Crossref Scopus (10391) Google Scholar, 6Goodfellow I. Bengio Y. Courville A. Bengio Y. Deep Learning. MIT Press, Cambridge, MA2016Google Scholar The applications of deep learning algorithms in digital pathology have had remarkable success in traditional pathology tasks. For example, deep learning algorithms achieved performance comparable to pathologists in interpreting whole slide images for the detection of tumor regions7Liu Y. Gadepalli K. Norouzi M. Dahl G.E. Kohlberger T. Boyko A. Venugopalan S. Timofeev A. Nelson P.Q. Corrado G.S. Detecting cancer metastases on gigapixel pathology images.arXiv. 2017; (arXiv:1703.02442)Google Scholar, 8Wang D. Khosla A. Gargeya R. Irshad H. Beck A.H. Deep learning for identifying metastatic breast cancer.arXiv. 2016; (arXiv:1606.05718)Google Scholar, 9Wang S. Chen A. Yang L. Cai L. Xie Y. Fujimoto J. Gazdar A. Xiao G. Comprehensive analysis of lung cancer pathology images to discover tumor shape and boundary features that predict survival outcome.Sci Rep. 2018; 8: 10393Crossref PubMed Scopus (60) Google Scholar and lymph node metastases.10Ehteshami Bejnordi B. Veta M. van Diest P.J. van Ginneken B. Karssemeijer N. Litjens G. et al.Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer.JAMA. 2017; 318: 2199-2210Crossref PubMed Scopus (1444) Google Scholar Although this comparable performance may not generalize to all task domains, advanced methodology is anticipated to solve or aid in common challenges faced by pathologists, including locating neoplasia within a tissue and quantifying specific features such as mitoses and inflammation. To understand how deep learning excels in these areas, we build conceptual connections of deep learning in the machine learning literature. In essence, deep learning is a special kind of artificial neural network (ANN), which is one category of machine learning algorithm. Deep learning and other ANNs are inspired by biological neural networks and mathematically construct a network model with multiple connected layers. The first network layer (called the input layer) receives inputs (eg, slide images). It has a set of parameters and can use them to compute outputs. Similarly, each successive network layer receives inputs from its previous layers, uses its parameters, and computes outputs. At the end, the last network layer (called the output layer) calculates the outputs of the whole model. The layers between the input and output layers are not visible because they do not directly receive model input or generate model outputs, and thus are called the hidden layers. The structure of a segmentation neural network is illustrated in Figure 1A. In this process, prediction outputs from a good neural network can well approximate the observed outputs. Although ANNs claim excellent performances based on theoretical work,11Cybenko G. Approximation by superpositions of a sigmoidal function.Math Control Signal Syst. 1989; 2: 303-314Crossref Scopus (8650) Google Scholar historically, it has been notoriously hard to calculate the network parameters when the total number of network layers exceeded three, which limited the performance of the model. Fortunately, this is no longer a severe bottleneck, owing to the advancements in computational hardware, the scale of data accumulation, and the improvements in algorithms. Nowadays, popular ANNs can have hundreds of layers. The machine learning community refers to these algorithms as deep learning to distinguish them from the conventional shallow ANN algorithm. In this review, the application of deep learning algorithms in pathology image analysis is the focus. Convolutional neural networks (CNNs) are introduced, which have been widely used for image classification and pathology image analysis, such as tumor region and metastasis detection.7Liu Y. Gadepalli K. Norouzi M. Dahl G.E. Kohlberger T. Boyko A. Venugopalan S. Timofeev A. Nelson P.Q. Corrado G.S. Detecting cancer metastases on gigapixel pathology images.arXiv. 2017; (arXiv:1703.02442)Google Scholar, 8Wang D. Khosla A. Gargeya R. Irshad H. Beck A.H. Deep learning for identifying metastatic breast cancer.arXiv. 2016; (arXiv:1606.05718)Google Scholar, 9Wang S. Chen A. Yang L. Cai L. Xie Y. Fujimoto J. Gazdar A. Xiao G. Comprehensive analysis of lung cancer pathology images to discover tumor shape and boundary features that predict survival outcome.Sci Rep. 2018; 8: 10393Crossref PubMed Scopus (60) Google Scholar In essence, a CNN can have a series of convolution layers as the hidden layers and thus make the network deep. This network structure enables the extraction of representational features for prediction. The design of CNN is inspired by the functional mechanism of the visual cortex3LeCun Y. Bengio Y. Hinton G. Deep learning.Nature. 2015; 521: 436-444Crossref PubMed Scopus (42675) Google Scholar: instead of using all outputs from the previous layer, a convolution kernel only focuses on a certain area, the so-called receptive field, to compute a feature at the corresponding spatial position. By spatially sliding the receptive field along the input dimensions (eg, along the width and height directions for two-dimensional images), a feature map is computed as the outputs from the convolution layer. This process is illustrated in Figure 1B. Because the number of parameters is determined by the size of the receptive field, convolution layers have many fewer parameters than the image size. This design thus effectively reduces the number of parameters within a neural network and greatly improves its computational efficiency. In addition to image classification, CNNs have also been implemented for pathology image segmentation.12Xu J. Luo X. Wang G. Gilmore H. Madabhushi A. A deep convolutional neural network for segmenting and classifying epithelial and stromal regions in histopathological images.Neurocomputing. 2016; 191: 214-223Crossref PubMed Scopus (324) Google Scholar To perform image segmentation for large data (eg, whole slide pathology images), the image is first divided into many small patches. A CNN is trained to classify these patches, and all patches in the same class are combined into one segmented area. Fine spatial resolution of segmentation can be achieved by using patches of small sizes; however, the patches need to be large enough that they can be classified accurately. To overcome the tradeoff between segmentation resolutions and patch size, instead of cutting the image region into adjacent patches, a moving window with a small step size is generally used to generate patches with a certain degree of overlap. In this procedure, the spatial resolution (determined by the step size instead of patch size) is largely improved. However, this demands a substantial amount of computing time and memory, which largely limits the computation speed. In recent years, several deep learning algorithms have been developed specifically for segmentation tasks, which can segment the image at pixel resolution and at a relatively high speed. In this review, the segmentation deep learning algorithms refer to semantic or instance segmentation algorithms, which are derivatives of CNNs. Compared with patch-based CNNs, segmentation deep learning algorithms are more computationally efficient in pixel classification and thus serve as powerful tools to extract detailed image information at pixel resolution.13Shelhamer E. Long J. Darrell T. Fully convolutional networks for semantic segmentation.IEEE Trans Pattern Anal Mach Intell. 2017; 39: 640-651Crossref PubMed Scopus (5329) Google Scholar, 14Yi F. Yang L. Wang S. Guo L. Huang C. Xie Y. Xiao G. Microvessel prediction in H&E stained pathology images using fully convolutional neural networks.BMC Bioinformatics. 2018; 19: 64Crossref PubMed Scopus (22) Google Scholar Compared with multilabel classification algorithms, which can identify15Wang J. Yang Y. Mao J.H. Huang Z.H. Huang C. Xu W. CNN-RNN: a unified framework for multi-label image classification.in: Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, Los Alamitos, CA2016: 2285-2294Crossref Scopus (810) Google Scholar and locate16Chen T. Wang Z. Li G. Lin L. Recurrent attentional reinforcement learning for multi-label image recognition.arXiv. 2017; (arXiv:1712.07465)Google Scholar objects of different types, the segmentation deep learning algorithms detect not only the objects but also the segmentation boundaries. Compared with conventional image segmentation algorithms that are not based on deep learning, deep learning algorithms have robust performances under different staining conditions17Sheikhzadeh F. Ward R.K. van Niekerk D. Guillaud M. Automatic labeling of molecular biomarkers of immunohistochemistry images using fully convolutional networks.PLoS One. 2018; 13: e0190783Crossref PubMed Scopus (32) Google Scholar because they do not heavily rely on staining intensity or hand-crafted (ie, manually defined) features,18Ruiz A. Kong J. Ujaldon M. Boyer K. Saltz J. Gurcan M. Pathological image segmentation for neuroblastoma using the GPU.Proc IEEE Int Symp Biomed Imaging. 2008; 2008: 296-299Google Scholar, 19Kong H. Gurcan M. Belkacem-Boussaid K. Partitioning histopathological images: an integrated framework for supervised color-texture segmentation and cell splitting.IEEE Trans Med Imaging. 2011; 30: 1661-1677Crossref PubMed Scopus (174) Google Scholar, 20Alsubaie N. Trahearn N. Raza S.E.A. Snead D. Rajpoot N.M. Stain deconvolution using statistical analysis of multi-resolution stain colour representation.PLoS One. 2017; 12: e0169875Crossref PubMed Scopus (49) Google Scholar, 21Ma Z. Shiao S.L. Yoshida E.J. Swartwood S. Huang F. Doche M.E. Chung A.P. Knudsen B.S. Gertych A. Data integration from pathology slides for quantitative imaging of multiple cell types within the tumor immune cell infiltrate.Diagn Pathol. 2017; 12: 69Crossref PubMed Scopus (18) Google Scholar, 22Gonzalez R.C. Woods R.E. Digital Image Processing. Pearson Education, Upper Saddle River, NJ2002Google Scholar and they can utilize neighborhood structural information. Thus, deep learning–based segmentations are anticipated to become an important tool in WSI analysis. In this review, the detailed process of deep learning–based pathology image segmentation is described, including data preparation, image preprocessing, model selection and construction, post-processing, and feature extraction and association with disease (Figure 2). The goals of this review are to provide quick guidance for implementing deep learning–based segmentation for pathology images and to provide some potential ways of further improving the segmentation performance for experienced investigators. Although there have previously been in-depth reviews on using machine learning methods, including deep learning, in digital pathology image analysis,4Janowczyk A. Madabhushi A. Deep learning for digital pathology image analysis: a comprehensive tutorial with selected use cases.J Pathol Inform. 2016; 7: 29Crossref PubMed Scopus (700) Google Scholar, 23Komura D. Ishikawa S. Machine learning methods for histopathological image analysis.Comput Struct Biotechnol J. 2018; 16: 34-42Crossref PubMed Scopus (413) Google Scholar this is the first review of the implementations of segmentation deep learning algorithms for WSI analysis. Pathology images are usually as large as giga-pixels. Because the memory associated with a central processing unit (CPU) or a graphic processing unit (GPU) is often limited, the pathology images should first be chopped into small patches and/or resized, if necessary, to fit the CPU or GPU memory (Figure 2). Common image patch sizes range from 256 × 256 to 512 × 512 pixels. The size is often chosen to be divisible by 2 to avoid the trouble of padding for pooling layers. Here, padding means adding pixels to the upper, bottom, left, and right sides of the image, respectively. The value of the added pixels is usually set to 0, which is the so-called zero-padding. Pooling means extracting one representative pixel in each receptive field to reduce the size of the feature map. Max-pooling is the most common pooling method, which uses the maximum value to represent a receptive field. Although, in theory, image segmentation neural networks do not require input images to be of the same size, it is preferable for the images to be cropped into the same size to speed up the algorithm and to fully utilize parallel computations on CPU/GPU. Otherwise, the image patches should be resized or padded into the same size before being fed into the neural network. In this case, zero-padding (adding zero-value pixels to the image boundaries) and symmetric-padding (adding pixels with values symmetrical to the original pixel values along the image boundaries) are commonly used. When the input image patches are padded, simply removing the padding region from the segmentation output can yield a result of the same size as the original image. Training a neural network for image segmentation is a supervised learning process. Thus, to construct a training set for segmentation, the next step is to manually annotate (ie, label) the ground truth. Pathology expertise is essential in this step. Several tools are available for annotating images, and their features are summarized in Table 1. The annotations are exported into one or a group of image masks, which are single-channel binary images of the same size as the input image patch. The contents of masks can be combined to describe the category of each pixel, which is usually coded as categorical data (eg, a natural number, 0, 1, 2, 3 …).Table 1Summary of Tools for Mask PreparationSoftware name: descriptionSupport multiclass labeling?Area selection methodResourceActive∗Tools are considered under active development if they were updated after October 2017.Photoshop (Adobe, San Jose, CA): Sophisticated commercial product to draw masksYesFlexible, including polygonal, brush, and fillinghttps://www.adobe.com/products/photoshop.htmlYesMATLAB Image Segmenter (MathWorks, Natick, MA): To create binary masks with the help of multiple image processing methodsNoPolygonalhttps://www.mathworks.com/help/images/image-segmentation-using-the-image-segmenter-app.htmlYesQuPath: A Java-based pathology image analysis toolYesManual labeling; automated cell detectionhttps://github.com/qupath/qupathNoJS Segment Annotator24Tangseng P. Wu Z. Yamaguchi K. Looking at outfit to parse clothing.arXiv. 2017; (arXiv:1703.01386)Google Scholar: A web-based image annotation toolYesArea-based clickinghttps://github.com/kyamagu/js-segment-annotatorNoLabelMe25Russell B.C. Torralba A. Murphy K.P. Freeman W.T. LabelMe: a database and web-based tool for image annotation.Int J Comput Vis. 2008; 77: 157-173Crossref Scopus (2282) Google Scholar: A web-based image annotation tool providing iPhone/iPad (Apple, Cupertino, CA) applicationYesPolygonalhttps://github.com/CSAILVision/LabelMeAnnotationToolYesLabelme: A Python-based image annotation toolYesPolygonalhttps://github.com/wkentaro/labelmeNoOpenSurfaces26Bell S. Upchurch P. Snavely N. Bala K. OpenSurfaces: a richly annotated catalog of surface appearance.ACM Trans Graph. 2013; 32 (Article 11)Crossref PubMed Scopus (132) Google Scholar: A web-based image annotation toolYes, different labels will be assigned for each objectPolygonalhttps://github.com/seanbell/opensurfaces-segmentation-uiNoLabelImg: A Python-based image annotation toolYesPolygonal and brushhttps://github.com/lzx1413/LabelImgToolYesLabelbox (San Francisco, CA): A commercial product to annotate images and customize user interfaceYesPolygonalhttps://github.com/labelbox/labelboxYes∗ Tools are considered under active development if they were updated after October 2017. Open table in a new tab To accelerate the training phase and improve model generalizability, image preprocessing is needed right before feeding the image patches into the segmentation neural networks (Figure 2). In preprocessing, image normalization is necessary for both training and application phases, whereas augmentation is necessary only for the training phase. Feature normalization is commonly used in the machine learning field to ensure that different features have a similar effect on the response. For a step-wise gradient descent algorithm,3LeCun Y. Bengio Y. Hinton G. Deep learning.Nature. 2015; 521: 436-444Crossref PubMed Scopus (42675) Google Scholar which is widely used to train deep-learning models, feature normalization also helps accelerate convergence. There are several common ways to perform image normalization. One option is to simply rescale the pixel value into [0, 1] or [−1, 1]. For example, dividing an 8-bit image by 255 will rescale it to the range [0, 1]. Another option is standardization, which refers to transforming the signal from each image channel into a random variable with mean 0 and variance 1:X˜=X−μσ,(1) where μ is the sample mean and σ is the sample variance. Pathology images are usually not stationary. Here, stationary means that a certain channel from each image follows the same pixel value distribution. Therefore, in the standardization step, using the mean and variance of the whole training set instead of using the statistics calculated from each individual image is often recommended. Because there are millions of parameters to train, using a limited training set will easily cause the neural network to memorize rather than learn how to segment. Image data augmentation is an important step, because it could greatly increase the size of the training set, reduce overfitting, and improve generalizability. There are multiple image shape augmentation methods. Using the projection matrix is fast and effective. A projective transformation simultaneously enables scaling, translation, rotation, and affine transformations under this formula:[scale×aspectratio×cosθ−sinθtranslationinxsinθscale×cosθaspectratiotranslationinygh1][xy1]=[uvw](2) In this transformation, θ is the counterclockwise rotation degree; g and h control keystone distortions (an image distortion that distorts the rectangle shape into a trapezoid). After the projective transformation, the position of the pixel (x,y) on the original image will be mapped to (u/w,v/w) on the transformed image. Other transformations include horizontal flipping, vertical flipping, and piecewise affine transformation.27Pitiot A. Malandain G. Bardinet E. Thompson P.M. Piecewise affine registration of biological images.Biomed Image Registration. 2003; 2717: 91-101Crossref Scopus (18) Google Scholar If shape augmentation is used, it is critical to apply the same transformation to both the image and its corresponding mask. Because pathology images may look very different due to different staining conditions and slide thicknesses, it is important to make the deep learning algorithm learn to adapt. One possible solution is to normalize pathology images to a uniform scale. Several pathology image standardization methods have been reported.20Alsubaie N. Trahearn N. Raza S.E.A. Snead D. Rajpoot N.M. Stain deconvolution using statistical analysis of multi-resolution stain colour representation.PLoS One. 2017; 12: e0169875Crossref PubMed Scopus (49) Google Scholar However, they are usually time-consuming and may diminish some intrinsic information. For example, hematoxylin and eosin–stained renal cell carcinoma pathology images are often classified into eosinophilic and basophilic subtypes, which are prone to be stained by eosin or hematoxylin, respectively, and thus have intrinsically different color distributions.28Lopez-Beltran A. Scarpelli M. Rodolfo M. Kirkali Z. 2004 WHO classification of the renal tumors of the adults.Eur Urol. 2006; 49: 798-805Abstract Full Text Full Text PDF PubMed Scopus (693) Google Scholar A comparatively easier solution in the deep learning context is to use color augmentation to mimic practical differences. By adding a random mean and multiplying a random variation to each channel of each image, the sample size is largely augmented. Thus, the neural network can learn to ignore the systematic biases raised from the pathology slide-making process. It is worth noting that the range, mean, and variance of augmentation parameters should be chosen carefully to reduce distorting image-intrinsic features. Other color augmentation methods include adding Gaussian noise, introducing salt-and-pepper noise, and blurring. Although deep learning algorithms can be implemented in C/C++, MATLAB (MathWorks, Natick, MA), R (R Foundation for Statistical Computing, Vienna, Austria), or Julia, Python is still the most commonly used language in the deep learning field. There are several open-source Python libraries to choose from: Caffe,29Jia Y. Shelhamer E. Donahue J. Karayev S. Long J. Girshick R. Guadarrama S. Darrell T. Caffe: convolutional architecture for fast feature embedding.in: Proceedings of the 22nd ACM International Conference on Multimedia. ACM, New York, NY2014: 675-678Crossref Scopus (7626) Google Scholar TensorFlow,30Abadi M. Barham P. Chen J. Chen Z. Davis A. Dean J. Devin M. Ghemawat S. Irving G. Isard M. Tensorflow: a system for large-scale machine learning.in: Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’16). USENIX Association, Berkeley, CA2016: 265-283Google Scholar Keras (https://keras.io, last accessed March 2019), and PyTorch (https://pytorch.org, last accessed July 2019).31Paszke A, Gross S, Chintala S, Chanan G, Yang E, DeVito Z, Lin Z, Desmaison A, Antiga L, Lerer A: Automatic differentiation in PyTorch. 31st Conference on Neural Information Processing Systems (NIPS 2017), December 4–9, 2017, Long Beach, CAGoogle Scholar Some software, such as Aperio GENIE (Leica Biosystems, Buffalo Grove, IL; https://pdfs.semanticscholar.org/28bd/c353c500d08b67eff9871d7f659c85321696.pdf), also incorporate a machine learning–based segmentation function. These software tools greatly decrease the coding effort, but the cost is that the models are less flexible in regard to both model structure and training phase. Thus, the following parts are based on using a model that can be easily customized to best suit the goal of pathology image analysis. To best suit the needs of pathology image segmentation, it is important to choose a proper neural network structure. Select a semantic or instance segmentation algorithm and the corresponding loss function (Loss Function) should be done first, whereas encoder backbone selection and layer manipulation are necessary only when improving the performance. For a quick start, Python implementations of segmentation deep learning algorithms can be easily found on GitHub, which have been summarized online (eg, https://github.com/mrgloom/awesome-semantic-segmentation, last accessed March 2019). Currently, several image segmentation models have been reported. On the basis of the design, these models can be divided into two main categories: semantic segmentation and instance segmentation (Figure 3). It is worth noting that either semantic or instance segmentation can be converted into each other through twisting the models, including changing the prediction target and adding post-processing steps, so the models are classified here according to their original implementation. The goal of semantic segmentation is to segment image parts with different meanings. The first end-to-end and pixel-to-pixel semantic segmentation neural network is the Fully Convolutional Network (FCN).13Shelhamer E. Long J. Darrell T. Fully convolutional networks for semantic segmentation.IEEE Trans Pattern Anal Mach Intell. 2017; 39: 640-651Crossref PubMed Scopus (5329) Google Scholar In FCN, the last fully connected layer in CNN is replaced with a deconvolutional layer to efficiently classify each pixel. The summation of deconvolutional layers and pooling layers enables FCN to do fine structure segmentation with respect to coarse structure information. Different modifications have been made to FCN to further improve the segmentation performance. For example, U-net greatly increases the number of deconvolutional layers to propagate information to higher resolutions.32Ronneberger O. Fischer P. Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation.in: Navab N. Hornegger J. Wells W. Frangi A. Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. Lecture Notes in Computer Science. vol 9351. Springer, Cham, Switzerland2015: 234-241Crossref Scopus (33612) Google Scholar SegNet refines the deconvolutional layers by using indices generated from max-pooling layers.33Badrinarayanan V. Kendall A. Cipolla R. SegNet: a deep convolutional encoder-decoder architecture for image segmentation.IEEE Trans Pattern Anal Mach Intell. 2017; 39: 2481-2495Crossref PubMed Scopus (8895) Google Scholar Recen
Referência(s)