Published

3-11-2021

Type of Work

Article - Journal

Abstract

Digital pathology is gaining prominence among the researchers with developments in advanced imaging modalities and new technologies. Generative adversarial networks (GANs) are a recent development in the field of artificial intelligence and since their inception, have boosted considerable interest in digital pathology. GANs and their extensions have opened several ways to tackle many challenging histopathological image processing problems such as color normalization, virtual staining, ink removal, image enhancement, automatic feature extraction, segmentation of nuclei, domain adaptation and data augmentation. This paper reviews recent advances in histopathological image processing using GANs with special emphasis on the future perspectives related to the use of such a technique. The papers included in this review were retrieved by conducting a keyword search on Google Scholar and manually selecting the papers on the subject of H&E stained digital pathology images for histopathological image processing. In the first part, we describe recent literature that use GANs in various image preprocessing tasks such as stain normalization, virtual staining, image enhancement, ink removal, and data augmentation. In the second part, we describe literature that use GANs for image analysis, such as nuclei detection, segmentation, and feature extraction. This review illustrates the role of GANs in digital pathology with the objective to trigger new research on the application of generative models in future research in digital pathology informatics.

References

  1. HA Alturkistani, FM Tashkandi, ZM Mohammedsaleh Histological stains: A literature review and case study Glob J Health Sci, 8 (2015), pp. 72-79
  2. N Farahani, AV Parwani, L Pantanowitz Whole slide imaging in pathology: Advantages, limitations, and emerging perspectives Pathol Lab Med Int,7 (2015), Article 4321
  3. D Lichtblau, C Stoean Cancer diagnosis through a tandem of classifiers for digitized histopathological slides PloS One, 14 (2019), Article e0209274
  4. KH Yu, C Zhang, GJ Berry, RB Altman, C Ré, DL Rubin, et al. Predicting non-small cell lung cancer prognosis by fully automated microscopic pathology image features Nat Commun, 7 (2016), Article 12474
  5. F Aeffner, MD Zarella, N Buchbinder, MM Bui, MR Goodman, DJ Hartman, et al. Introduction to digital image analysis in whole-slide imaging: A white paper from the digital pathology association J Pathol Inform, 10 (2019), p. 9
  6. A Di leva, E Bruner, G Widhalm, G Minchev, M Tschabitscher, F Grizzi Computer-assisted and fractal-based morphometric assessment of microvascularity in histological specimens of gliomas Sci Rep, 2 (2012), p. 429
  7. AN Esgiar, RN Naguib, BS Sharif, MK Bennett, A Murray Fractal analysis in the detection of colonic cancer images IEEE Trans Inf Technol Biomed, 6 (2002), pp. 54-58
  8. A Di Ieva Fractal analysis of microvascular networks in malignant brain tumors Clin Neuropathol, 31 (2012), pp. 342-351
  9. MK Niazi, AV Parwani, MN Gurcan Digital pathology and artificial intelligence Lancet Oncol, 20 (2019), pp. e253-e261
  10. BA Richards, TP Lillicrap, P Beaudoin, Y Bengio, R Bogacz, A Christensen, et al. A deep learning framework for neuroscience Nat Neurosci, 22 (2019), pp. 1761-1770
  11. D Ravi, C Wong, F Deligianni, M Berthelot, J Andreu-Perez, B Lo, et al.  Deep learning for health informatics IEEE J Biomed Health Inform, 21 (2017), pp. 4-21
  12. A Janowczyk, A Basavanhally, A Madabhushi  Stain normalization using sparse autoencoders (StaNoSA): Application to digital pathology Comput Med Imaging Graph, 57 (2017), pp. 50-61
  13. Y Zhou, H Chang, KE Barner, B Parvin Nuclei segmentation via sparsity constrained convolutional regression. IEEE 12th International Symposium on Biomedical Imaging (ISBI) (2015), pp. 1284-1287
  14. K Sirinukunwattana, SE Ahmed Raza, Tsang Yee-Wah, DR Snead, IA Cree, NM Rajpoot Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images IEEE Trans Med Imaging, 35 (2016), pp. 1196-1206
  15. N Coudray, PS Ocampo, T Sakellaropoulos, N Narula, M Snuderl, D Fenyö, et al. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning Nat Med, 24 (2018), pp. 1559-1567
  16. W Li, J Li, KV Sarma, KC Ho, S Shen, BS Knudsen, et al. Path R-CNN for prostate cancer diagnosis and gleason grading of histological images IEEE Trans Med Imaging, 38 (2019), pp. 945-954
  17. IJ Goodfellow, J Pouget-Abadie, M Mirza, B Xu, D Warde-Farley, S Ozair, et al. Generative Adversarial Networks. Proceedings of the International Conference on Neural Information Processing Systems (NIPS 2014) (2014), pp. 672-680
  18. X Yi, E Walia, P Babyn Generative adversarial network in medical imaging: A review Med Image Anal, 58 (2019), Article 101552
  19. T Xu, P Zhang, Q Huang, H Zhang, Z Gan, X Huang, et al. AttnGAN: Fine-Grained Text to Image Generation With Attentional Generative Adversarial Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018), pp. 1316-1324
  20. C Ledig, L Theis, F Huszar, J Caballero, A Cunningham, A Acosta, et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)(2017), pp. 4681-4690
  21. JY Zhu, T Park, P Isola, AA Efros Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV)(2017), pp. 2223-2232
  22. JM Wolterink, T Leiner, MA Viergever, I Isgum Generative adversarial networks for noise reduction in low-dose CT IEEE Trans Med Imaging, 36 (2017), pp. 2536-2545
  23.  Y Chen, Y Shi, F Christodoulou, AG Xie, Y Zhou, Z Li, et al. Efficient and Accurate MRI Super-Resolution Using a Generative Adversarial Network and 3D Multi-level Densely Connected NetworkMedical Image Computing and Computer Assisted Intervention – MICCAI (2018), pp. 91-99
  24. M Frid-Adar, I Diamant, E Klang, M Amitai, J Goldberger, H Greenspan GAN-based synthetic medical image augmentation for increased CNN performance in liver lesion classification Neurocomputing, 321 (2018), pp. 321-331
  25. M Gadermayr, L Gupta, V Appel, P Boor, BM Klinkhammer, D Merhof  Generative adversarial networks for facilitating stain-independent supervised and unsupervised segmentation: A study on kidney histology IEEE Trans Med Imaging, 38 (2019), pp. 2293-2302
  26. J Ren, I Hacihaliloglu, EA Singer, DJ Foran, X Qi Unsupervised domain adaptation for classification of histopathology whole-slide images Front Bioeng Biotechnol, 7 (2019), p. 102
  27. Z Xu, CF Moro, B Bozóky, Q Zhang GAN-Based Virtual Re-Staining: A Promising Solution for Whole Slide Image Analysis. arXiv 2019 arXiv: 1901.04059
  28. AC Quiros, R Murray-Smith, K Yuan PathologyGAN: Learning deep Representations of Cancer Tissue Proceedings of the Third Conference on Medical Imaging with Deep Learning, Vol. 121, PMLR (2020), pp. 669-695
  29. N Bayramoglu, M Kaakinen, L Eklund, J Heikkila Towards virtual H&E staining of hyperspectral lung histology images using conditional generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision Workshops (2017), pp. 64-71
  30. JF Nash Equilibrium points in N-person games Proc Natl Acad Sci U S A, 36 (1950), pp. 48-49
  31. M Mirza, S Osindero Conditional Generative Adversarial Nets. arXiv arXiv: 1411.1784 (2014)
  32. T Iqbal, H Ali  Generative adversarial network for medical images (MI-GAN)J Med Syst, 42 (2018), p. 231
  33. F Mahmood, R Chen, D Borders, GN McKay, K Salimian, A Baras, et al.  Adversarial U-net with spectral normalization for histopathology image segmentation using synthetic data. Proc. SPIE 10956, Medical Imaging 2019 Digit Pathol (2019), Article 109560N doi: 10.1117/12.2512918
  34. Y Jin, J Zhang, M Li, Y Tian, H Zhu, Z Fang, et al. Towards the Automatic Anime Characters Creation with Generative Adversarial Networks arXiv Preprint arXiv: 1708.05509 (2017)
  35. B Hu, Y Tang, EI Chang, Y Fan, M Lai, Y Xu Unsupervised learning for cell-level visual representation in histopathology images with generative adversarial networks IEEE J Biomed Health Inform, 23 (2019), pp. 1316-1328
  36. Ş Öztürk, B Akdemir  Effects of histopathological image pre-processing on convolutional neural networks Procedia Comput Sci, 132 (2018), pp. 396-403
  37. MT Shaban, C Baur, N Navab, S Albarqouni Staingan: Stain style transfer for digital histological images. In 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019) IEEE (2019), pp. 953-956
  38. G Çelik, MF Talu  Resizing and cleaning of histopathological images using generative adversarial networks Physica A Stat Mech Appl, 554 (2019), Article 122652
  39. FG Zanjani, S Zinger, BE Bejnordi, JV Laak Histopathology stain-color normalization using deep generative models. 1st Conference on Medical Imaging with Deep Learning (MIDL 2018) (2018), pp. 1-11
  40. S Ali, NK Alham, C Verrill, J Rittscher  Ink Removal from Histopathology Whole Slide Images by Combining Classification, Detection and Image Generation Models. IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019)(2019), pp. 928-932
  41. A Mikołajczyk, M Grochowski  Data augmentation for improving deep learning in image classification problem. In 2018 International Interdisciplinary PhD Workshop (IIPhDW)IEEE (2018), pp. 117-122
  42. D Komura, S Ishikawa Machine learning methods for histopathological image analysis Comput Struct Biotechnol J, 16 (2018), pp. 34-42
  43. S Kothari, JH Phan, TH Stokes, MD Wang Pathology imaging informatics for quantitative analysis of whole-slide images J Am Med Inform Assoc, 20 (2013), pp. 1099-1108
  44. K Clark, B Vendt, K Smith, J Freymann, J Kirby, P Koppel, et al. The Cancer Imaging Archive (TCIA): Maintaining and operating a public information repository J Digit Imaging, 26 (2013), pp. 1045-1057
  45. A Tabesh, M Teverovskiy, HY Pang, VP Kumar, D Verbel, A Kotsianti, et al. Multifeature prostate cancer diagnosis and Gleason grading of histological images IEEE Trans Med Imaging, 26 (2007), pp. 1366-1378
  46. O Sertel, UV Catalyurek, H Shimada, MN Gurcan Computer-aided prognosis of neuroblastoma: Detection of mitosis and karyorrhexis cells in digitized histological images Annu Int Conf IEEE Eng Med Biol Soc, 2009 (2009), pp. 1433-1436
  47. MD Zarella, DE Breen, A Plagov, FU Garcia An optimized color transformation for the analysis of digital images of hematoxylin & eosin stained slides J Pathol Inform, 6 (2015), p. 33
  48.  E Reinhard, M Adhikhmin, B Gooch, P Shirley  Color transfer between imagesIEEE Comput Graph Appl, 21 (2001), pp. 34-41
  49. H Cho, S Lim, G Choi, H Min  Neural Stain-Style Transfer Learning Using Gan for Histopathological Images. arXiv arXiv: 1710.08543 (2017)
  50. A Bentaieb, G Hamarneh Adversarial stain transfer for histopathology image analysis  IEEE Trans Med Imaging, 37 (2018), pp. 792-802
  51. A Rana, G Yauney, A Lowe, P Shah Computational Histological Staining and Destaining of Prostate Core Biopsy RGB Images with Generative Adversarial Neural Networks. 17th IEEE International Conference on Machine Learning and Applications (ICMLA) IEEE (2018)
  52. N Zhou, D Cai, X Han, J Yao Enhanced Cycle-Consistent Generative Adversarial Network for Color Normalization of H&E Stained Images. Medical Image Computing and Computer Assisted Intervention – MICCAI (2019), pp. 694-702
  53. S Cai, Y Xue, Q Gao, M Du, G Chen, H Zhang, et al. Stain style transfer using transitive adversarial networks. Machine learning for medical Image Reconstruction. MLMIR 2019. Lecture Notes in Computer Science Vol. 11905, Springer, Cham (2019)
  54.  A Shrivastava, W Adorno, Y Sharma, L Ehsan, SA Ali, SR Moore, et al. Self-Attentive Adversarial Stain Normalization arXiv Preprint arXiv: 1909.01963 (2019)
  55. P Salehi, A Chalechale  Pix2Pix-based Stain-to-Stain Translation: A Solution for Robust Stain Normalization in Histopathology Images Analysis. International Conference on Machine Vision and Image Processing (MVIP)(2020), pp. 1-7
  56.  H Zhang, I Goodfellow, D Metaxas, A Odena Self-attention generative adversarial networks. In International Conference on Machine Learning. PMLR (2019), pp. 7354-7363
  57.  J Long, E Shelhamer, T Darrell Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015), pp. 3431-3440
  58. C Cong, S Liu, A Di Ieva, M Pagnucco, S Berkovsky, Y Song Texture Enhanced Generative Adversarial Network for Stain Normalisation in Histopathology Images. International Symposium on Biomedical Imaging(ISBI) IEEE (2021)
  59. Y Rivenson, T Liu, Z Wei, Y Zhang, K de Haan, A Ozcan  PhaseStain: The digital staining of label-free quantitative phase microscopy images using deep learningLight Sci Appl, 8 (2019), p. 23
  60. Y Rivenson, H Wang, Z Wei, K de Haan, Y Zhang, Y Wu, et al. Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning Nat Biomed Eng, 3 (2019), pp. 466-477
  61.  L Fujun, P Sylvain, S Eli, B Kavita  Deep Photo Style Transfer. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(2017), pp. 4990-4998
  62.  Z Wang, AC Bovik, HR Sheikh, EP Simoncelli  Image quality assessment: From error visibility to structural similarityIEEE Trans Image Process, 13 (2004), pp. 600-612
  63. M Combalia Escudero, J Pérez Ankar, A García Herrera, L Alos, V Vilaplana Besler, F Marqués Acosta, et al. Digitally stained confocal microscopy through deep learning. International Conference on Medical Imaging with Deep Learning: 8-10 July 2019, London, United Kingdom: proceedings of Machine Learning Research (2019), pp. 121-129
  64. YN Nygate, M Levi, SK Mirsky, NA Turko, M Rubin, I Barnea, et al.  Holographic virtual staining of individual biological cells Proc Natl Acad Sci U S A, 117 (2020), pp. 9223-9231
  65.  T Abraham, A Todd, DA Orringer, R Levenson Applications of artificial intelligence for image enhancement in pathology. Artificial Intelligence and Deep Learning in Pathology Elsevier, USA (2021), pp. 119-148
  66. U Uddeshya, P Awate Suyash A Mixed-Supervision Multilevel GAN Framework for Image Quality Enhancement. Medical Image Computing and Computer Assisted Intervention – MICCAI (2019), pp. 556-564
  67. F Shahidi  Breast Cancer Histopathology Image Super-Resolution Using Wide-Attention GAN with Improved Wasserstein Gradient Penalty and Perceptual Loss IEEE Access, 9 (2021), pp. 32795-32809
  68. M Niethammer, D Borland, J Marron, J Woosley, NE Thomas Appearance normalization of histology slides. International Workshop on Machine Learning in Medical Imaging Springer, Berlin, Heidelberg (2010), pp. 58-66
  69. B Venkatesh, T Shaht, A Chen, S Ghafurian Restoration of Marker Occluded Hematoxylin and Eosin Stained Whole Slide Histology Images Using Generative Adversarial Networks. IEEE 17th International Symposium on Biomedical Imaging (ISBI) (2020), pp. 591-595
  70. WS Wong, M Amer, T Maul, IY Liao, A Ahmed Conditional Generative Adversarial Networks for Data Augmentation in Breast Cancer Classification. Recent Advances on Soft Computing and Data Mining. SCDM 2020. Advances in Intelligent Systems and Computing Vol. 978, Springer, Cham (2020)
  71. S Pandey, PR Singh, J Tian An image augmentation approach using two-stage generative adversarial network for nuclei image segmentation Biomed Signal Process Control, 57 (2020), Article 101782
  72.  Y Xue, Q Zhou, J Ye, LR Long, S Antani, C Cornwell Synthetic Augmentation and Feature-Based Filtering for Improved Cervical Histopathology Image Classification. Medical Image Computing and Computer Assisted Intervention – MICCAI (2019), pp. 387-396
  73. S Liu, Z Shah, A Sav, C Russo, S Berkovsky, Y Qian, et al. Isocitrate dehydrogenase (IDH) status prediction in histopathology images of gliomas using deep learning Sci Rep, 10 (2020), Article 7733
  74. MN Gurcan, LE Boucheron, A Can, A Madabhushi, NM Rajpoot, B Yener  Histopathological image analysis: A review IEEE Rev Biomed Eng, 2 (2009), pp. 147-171
  75. OC Koyun, T Yildirim  Adversarial Nuclei Segmentation on H&E Stained Histopathology Images. IEEE International Symposium on INnovations in Intelligent SysTems and Applications (INISTA)(2019), pp. 1-5
  76. T Wollmann, CS Eijkman, K Rohr Adversarial Domain Adaptation to Improve Automatic Breast Cancer Grading in Lymph Nodes. IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018) (2018), pp. 582-585
  77. F Xing, L Yang Robust nucleus/cell detection and segmentation in digital pathology and microscopy images: A comprehensive review IEEE Rev Biomed Eng, 9 (2016), pp. 234-263
  78. H Höfener, A Homeyer, N Weiss, J Molin, CF Lundström, HK Hahn Deep learning nuclei detection: A simple approach can deliver state-of-the-art results Comput Med Imaging Graph, 70 (2018), pp. 43-52
  79. J Li, W Shao, Z Li, W Li, D Zhang  Residual Attention Generative Adversarial Networks for Nuclei Detection on Routine Colon Cancer Histology Images. Machine Learning in Medical Imaging MLMI (2019), pp. 142-150
  80. F Mahmood, D Borders, RJ Chen, GN Mckay, KJ Salimian, A Baras, et al. Deep adversarial training for multi-organ nuclei segmentation in histopathology images IEEE Trans Med Imaging, 39 (2020), pp. 3257-3267
  81. E Yuan, J Suh Neural Stain Normalization and Unsupervised Classification of Cell Nuclei in Histopathological Breast Cancer Images. arXiv arXiv: 1811.03815 (2018)
  82. L Gupta, BM Klinkhammer, P Boor, D Merhof, M Gadermayr GAN-Based Image Enrichment in Digital Pathology Boosts Segmentation Accuracy. Medical Image Computing and Computer Assisted Intervention – MICCAI 2019. MICCAI (2019), pp. 631-639
  83. NJ Schork  Artificial intelligence and personalized medicine. Precision Medicine in Cancer Therapy. Cancer Treatment and ResearchVol. 178, Springer, Cham (2019), pp. 265-283
  84. D Saxena, J Cao  Generative Adversarial Networks (GANs): Challenges, Solutions, and Future Directions ACM Computing Surveys (2021)

Citation

Jose, L., Liu, S., Russo, C., Nadort, A., & Di Ieva, A. (2021). Generative adversarial networks in digital pathology and histopathological image processing: A review. Journal of Pathology Informatics, 12(1), 43. https://doi.org/10.4103/jpi.jpi_103_20

Download

Share

COinS