Combating COVID-19 Using Generative Adversarial Networks and Artificial Intelligence for Medical Images: Scoping Review

Background: Research on the diagnosis of COVID-19 using lung images is limited by the scarcity of imaging data. Generative adversarial networks (GANs) are popular for synthesis and data augmentation. GANs have been explored for data augmentation to enhance the performance of artificial intelligence (AI) methods for the diagnosis of COVID-19 within lung computed tomography (CT) and X-ray images. However, the role of GANs in overcoming data scarcity for COVID-19 is not well understood. Objective: This review presents a comprehensive study on the role of GANs in addressing the challenges related to COVID-19 data scarcity and diagnosis. It is the first review that summarizes different GAN methods and lung imaging data sets for COVID-19. It attempts to answer the questions related to applications of GANs, popular GAN architectures, frequently used image modalities, and the availability of source code. Methods: A search was conducted on 5 databases, namely PubMed, IEEEXplore, Association for Computing Machinery (ACM) Digital Library, Scopus, and Google Scholar. The search was conducted from October 11-13, 2021. The search was conducted using intervention keywords, such as “generative adversarial networks” and “GANs,” and application keywords, such as “COVID-19” and “coronavirus.” The review was performed following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews (PRISMA-ScR) guidelines for systematic and scoping reviews. Only those studies were included that reported GAN-based methods for analyzing chest X-ray images, chest CT images, and chest ultrasound images. Any studies that used deep learning methods but did not use GANs were excluded. No restrictions were imposed on the country of publication, study design, or outcomes. Only those studies that were in English and were published from 2020 to 2022 were included. No studies before 2020 were included. Results: This review included 57 full-text studies that reported the use of GANs for different applications in COVID-19 lung imaging data. Most of the studies (n=42, 74%) used GANs for data augmentation to enhance the performance of AI techniques for COVID-19 diagnosis. Other popular applications of GANs were segmentation of lungs and superresolution of lung images. The cycleGAN and the conditional GAN were the most commonly used architectures, used in 9 studies each. In addition, 29 (51%) studies used chest X-ray images, while 21 (37%) studies used CT images for the training of GANs. For the majority of the studies (n=47, 82%), the experiments were conducted and results were reported using publicly available data. A secondary evaluation of the results by radiologists/clinicians was reported by only 2 (4%) studies. Conclusions: Studies have shown that GANs have great potential to address the data scarcity challenge for lung images in COVID-19. Data synthesized with GANs have been helpful to improve the training of the convolutional neural network (CNN) models trained for the diagnosis of COVID-19. In addition, GANs have also contributed to enhancing the CNNs’ performance through the superresolution of the images and segmentation. This review also identified key limitations of the potential transformation of GAN-based methods in clinical applications.


Background
In December 2019, the COVID-19 broke out and spread at an unprecedented rate, given the highly contagious nature of the virus.As a result, the World Health Organization (WHO) declared it a global pandemic in March 2020 [1].Therefore, a response to combat the spread through speedy diagnosis became the most critical need of the time.A common method for diagnosing COVID-19 is the use of a realtime reverse transcription-polymerase chain reaction (RT-PCR) test.However, with the increasing number of cases worldwide, the health care sector was overloaded as it became challenging to cope with the requirements of the tests with the available testing facilities.Besides, research studies showed that RT-PCR may result in false negatives or fluctuating results [2].Hence, diagnosis through computed tomography (CT) and X-Ray images of lungs may supplement the performance.Motivated by this need, alternative methods such as automatic diagnosis of COVID-19 from lungs images were explored and encouraged.In this regard, it is well understood that Artificial Intelligence (AI) techniques could help inspect chest CTs and X-rays within seconds and augment the public health care sector.The use of properly trained AI models for diagnosis of COVID-19 is promising for scaling up the capacity and The results of this review will be helpful for researchers and professionals in the medical imaging and healthcare domain who are considering using GANs methods to address challenges related to COVID-19 imaging data and to address the challenge of improving the automatic diagnosis using radiology images.

Methods
In this work, a scoping review was conducted following the guidelines of "Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews" (PRISMA-ScR) [16].The methods for performing the study are described below.

Search Sources
A search was conducted between 11 October to 13 October 2021.The search was performed on the following five databases: Pubmed, IEEEXplore, ACM Digital This is a pre-print of the paper accepted for publication in JMIR Medical Informatics.Final version will be available from JMIR.doi: 10.2196/37365 Library, Scopus, and Google Scholar.In the case of Google Scholar, only the first 99 results were retained as the results beyond 99 items were highly irrelevant to the scope of the study.Similarly, in the case of ACM Digital Library, the first 100 results were retained as a lack of relevancy to the study was obvious in results beyond 100.

Search Terms
The search terms used in this study were chosen from the literature with guidance from experts in the field.The terms were chosen based on the intervention (for example, generative adversarial networks, GANs, cycleGANs) and the target application (COVID-19, coronavirus, corona pandemic).The exact search strings used in the search for this study are available in Appendix 1.

Search Eligibility Criteria
This study focused on the applications of GANs in radiology images of lungs for COVID-19 used for any purpose such as data augmentation or synthesis, diagnosis, super-resolution, and prognosis.Only those studies were included that reported GANs based methods for chest x-ray images, chest CT images and chest ultrasound images.Studies that reported GANs based methods for non-lungs images were removed.Any studies that used deep learning methods but did not use GANs were also excluded.Studies reporting GANs for non-image data were also excluded.To provide a list of reliable studies, only peer-reviewed articles, conference papers and book chapters were included.Preprints, conference abstracts, short letters, and commentaries were excluded.Similarly, review articles were also excluded.No restrictions were imposed on the country of publication, study design, or outcomes.This is a pre-print of the paper accepted for publication in JMIR Medical Informatics.Final version will be available from JMIR.doi: 10.2196/37365 Studies that were written in English and were published from 2020 to 2022 were included.No studies before 2020 were included.

Study Selection
Two reviewers namely authors HA and ZS screened the titles and abstracts of the search results.Initial screening by the two reviewers was performed independently.
The disagreement occurred for 9 articles only.The disagreement was resolved through mutual discussion and consensus.For measuring the disagreement, Cohen Kappa [17] was calculated to be 0.89, which shows good agreement between the two independent reviewers.Appendix 2 shows the matrix for the agreement between the two independent reviewers.

Data Extraction
Appendix 3 shows the form for extraction of the key characteristics.The form was pilot-tested and refined in two rounds, firstly by data extraction for five studies and then by data extraction for another five studies.This refinement of the form ensured that only relevant data is extracted from the studies.The two reviewers (HA and ZS) extracted the data from the included studies, related to the GANs-based method, applications, and data sets.Any disagreement between the reviewers was resolved through mutual consensus and discussions.As the disagreements at the study selection stage were resolved through careful and lengthy discussions, the disagreement at the data extraction was only minor.

Data Synthesis
After extraction of the data from the full text of the identified studies, a narrative approach was used to synthesize the data.The use of GANs methods was classified This is a pre-print of the paper accepted for publication in JMIR Medical Informatics.Final version will be available from JMIR.doi: 10.2196/37365 in terms of the application of GAN (for example, augmentation, segmentation of lungs, etc.), the type of GAN architecture, if reported (for example, conditional GAN or cycleGAN), and the modality of the imaging data for which the GAN was used (for example, CT or X-Ray imaging).Similarly, the studies were classified based on the availability of the dataset (for example, public or private), the size of the dataset (for example, the number of images in the original images and the number of images after augmentation with GAN, if applicable), and the proportion of the training and test sets as well as the type of cross-validation.The data synthesis was managed and performed using Microsoft Excel (Microsoft Corporation) workbook.This is a pre-print of the paper accepted for publication in JMIR Medical Informatics.Final version will be available from JMIR.doi: 10.2196/37365

Search Results
From five online databases, a total of 348 studies were retrieved (see Figure 1).Out of the 348 studies, 81 duplicates were removed.The title and abstracts of the remaining 267 studies were carefully screened as per the criteria of inclusion and exclusion.The screening of the titles and abstracts resulted in the exclusion of 208 studies (see Figure 1 for reasons of exclusion).After the full-text reading of the remaining 59 studies, 02 studies were excluded following the inclusion/exclusion

Demographics of the included studies
Among the included studies (n=57), 37 studies were published articles in peerreviewed journals, 18 studies were published in conference proceedings, and 2 studies were published as book chapters.No thesis publication was found relevant to the scope of this review.Around one-fourth of the studies (n=15) were published in the year 2020.Most of the studies were published in 2021 (n=41).The included studies were published in 14 countries.The largest number of publications were from China (n=12), followed by India (n=10).Both USA and Egypt published the same number of studies (n=6).The characteristics are summarized in (Table 1).
(Figure 2) shows the demographics of the included studies along with the modality of the chest images used.
Figure 3. Major applications of GANs in the included studies.The number of publications for each application is reflected by the size of the circle in the second-last layer.The numbers S1 through S57 refer to the included studies, as per ( 18) through ( 74) in the references.
Different variants have been proposed for GANs architectures since their inception.
Out of 57 studies, only ten studies [18,19], [26,27], [30], [34], [43], [61 -73] reported changes to the architecture of the GAN they were using.For the rest of the studies, no major changes were reported to the architecture of the GAN.
Majority of the studies reported the size of the data set in terms of the number of images.The number of images used was greater than 10,000 in only seven studies This is a pre-print of the paper accepted for publication in JMIR Medical Informatics.Final version will be available from JMIR.doi: 10.2196/37365 [20], [22], [30], [39], [63], [66], [74].Three studies used images between 5000 and 10,000 [33], [47], [64].The most common range for the number of images used was between 1000 to 5000 images used in 15 studies.Around one-fifth of the studies (n=11) used the number of images between 500 and 1000.In 11 studies, the number of images used was less than 500.No study reported a number of images less than 100.The maximum number of images was 84971 used by [22].Only a few of the studies reported the number of patients for whom the data has been used.
After augmentation using GANs, the studies have increased the number of images to several thousand with a maximum number of 21295 [54].In six studies using GANs for data augmentation, the number of images was increased to more than 10,000.In three studies, the number of images was increased between 5000 to 10,000.In nine studies, the increased number of images was between 1000 to 5000 and in two studies, the increased number of images was between 500 and 1000.No study reported data augmentation output below 500 images.
The numbers do not add up as many studies used more than one metric for evaluation.Besides the metrics mentioned above, only one study [22] used additional metrics, namely concordance index and relative absolute error, to evaluate prognosis and survival prediction for COVID-19 affected individuals.

Reproducibility and Secondary Evaluation
This review also summarizes the studies for which the authors provided the implementation code.Only seven [19,20], [34], [47,48], [66], [70] out of the 57 studies provided links for their code.Only two studies [19,45] reported a secondary evaluation by radiologists/doctors/experts by presenting the outcome of the results obtained by their model.One study [19] presented their results of an end-to-end diagnosis COVID-19 from CT images to three radiologists for a second opinion.One study [45] presented the synthetic X-Ray images to two radiologists for a second opinion on the quality of the generated X-Ray images.

Principal Results
In this review, a significant rise in the number of studies on the topic was found in Interestingly, the same number of studies (n=6) were published from USA and Egypt.The correlation mapping in (Figure 5) shows that most of the studies published in 2020 originated from China, India, Egypt, and Canada.However, in 2021, many other countries also contributed to the published research.The number of journals papers was twice of the conference papers.This is surprising as journal publications would typically require more time in article processing than conferences.It can be possible that many authors turned to journals submissions as, during the start of the pandemic, many conferences were suspended initially before moving to the online (virtual) mode.
In the majority of the included studies (n=39), the main task was to perform diagnosis of COVID-19 using lungs CT or X-Ray images.For these studies, GAN was volumes synthesis of CT.This is not surprising as the synthesis of 3D volumes using 3D GANs is computationally very expensive.The computations for the 3D synthesis of CT volumes may exceed the available resources of the GPU.
Since there are many variants of GANs, this review also looked at the most commonly used GAN architecture in the included studies.The most common choice of GAN in the included studies was the cycleGAN used in nine studies.The cycleGAN is a GAN architecture that comprises two generators and two discriminators and does not require pair-to-pair training data [11].Hence, it was a popular choice to generate COVID-19 positive images from normal images.
This review analyzed the common imaging modality for the different applications related to COVID-19.As chest X-Ray imaging and CT scans are the most popular imaging methods for studying the infection in individuals, the studies included in this review also used these two imaging modalities.Specifically, 35 studies used X- The majority of the included studies (n=47) used data that is available publicly on Github, Kaggle, or other publicly accessible websites.These data are acquired from multiple sources (for example, collected from more than one hospital or through crowdsourcing) which makes them more diverse and hence more useful for training of GANs models.Similarly, it is hoped that the use of publicly accessible data will also encourage other researchers to conduct experiments on the data sets.The rise of publications in 2021 can also be linked to the availability of publicly available data sets that continued to rise as the number of COVID-19 infected cases continued to grow.A few of the included studies (n=10) used private or proprietary data sets, and hence, the details about those data sets are only limited to what has been described in the corresponding studies.
Only 13 studies provided information on the number of individuals whose data was used in the included studies.Amongst these, only one study [26] used data for more than 1000 individuals, and two studies [29], [42] used data for more than 500 individuals.The remaining ten studies used data for less than 500 individuals.Given the size of the population infected with the COVID-19 (418+ million as of writing Similarly, researchers can also add to the existing dataset on Github by uploading their data to the current data repositories.An example of crowdsourcing of data is the COVIDx images repository for lungs X-ray images (see Table 3).
This review identified that the code to reproduce the results was not available for the majority of the studies.Only seven of the included studies provided a public link to the code.Availability of a public repository to reproduce the results for diagnosis or augmented data can help in advancing the research as well as increase the trust This is a pre-print of the paper accepted for publication in JMIR Medical Informatics.Final version will be available from JMIR.doi: 10.2196/37365 and reliance on the reported results in terms of the quality of the generated images or the accuracy reports for the diagnosis.Besides, the reproducibility by this code is not assessed by this review as it was beyond the scope of this review.Careful and responsible studies are needed to make an assessment of the published methods for transformation into clinical applications.
The majority of the included studies (n=43) did not provide information on the number of patients, although they did mention the number of images used in the experiments.So, it is unclear that how many images were used per individual.
Hence, the lack of information limits the ability of the readers to evaluate the performance in the context of the number of patients.Moreover, for public data set with crowd-sourced contributions, it is challenging to trace back the number of images to the number of individuals.
Validation of the performance of GANs in terms of the quality/usability of the generated images has a significant role in promoting the acceptability of the methods.In the included studies, only two studies reported that the results were presented to radiologists/clinicians for a secondary validation.For one study on the synthesis of X-Ray images, the radiologists agreed that the quality of the X-Rays has improved but falls short of diagnostic quality for use in clinics [45].While using GANs methods in COVID-19 is tempting for many researchers, the lack of evaluation by radiologists or using GANs based methods without radiologists and clinicians in the loop will hinder the acceptability of these methods for clinical applications.
Besides, it is beyond the scope of this review to evaluate a study based on reporting of secondary evaluation by the radiologists, though a secondary assessment by the This is a pre-print of the paper accepted for publication in JMIR Medical Informatics.Final version will be available from JMIR.doi: 10.2196/37365 radiologists would have added value to the studies and increased their acceptability.
The lack of details related to the individuals whose COVID-19 data were used in these studies may also hinder their acceptance for transformation into clinical applications.The training of GANs is usually computationally demanding, requiring GPUs.More edge computing-based implementations are needed for clinical applications to make these models compatible for implementation on low-power devices.This will increase the acceptability of these methods in clinical devices.

Strengths
Though several reviews can be found on the applications of AI techniques in COVID-19, no review was found that focused on the potential of GANs based methods to combat COVID-19.Compared to other reviews [3,4], [6,7] where the scope is too broad as they attempted to cover many different AI models, this review provides a comprehensive analysis of the GANs based approaches used primarily on lungs CT and X-Ray images.Similarly, many reviews cover the applications of GANs in medical imaging [10], [12][13][14][15]; their applications in lungs images for COVID-19 have not been reviewed before.So, this review may be considered the first comprehensive review that covers all the GANs methods used for COVID-19 imaging data for different applications in general and data augmentation in particular.Thus, it is helpful for the readers to understand how GANs based approaches were used to address the problem of data scarcity and how the synthetic data (generated by GANs) was used to improve the performance of CNNs for COVID-19.This review This is a pre-print of the paper accepted for publication in JMIR Medical Informatics.Final version will be available from JMIR.doi: 10.2196/37365 provided a thorough list of the various publicly available datasets of lungs CT, lungs X-Ray, and lungs ultrasound images, along with the public URL.Hence, this can serve as a single point of contact for the readers to explore these data set resources and use them in their research work.This review is consistent with the guidelines of PRISMA-ScR for scientific reviews [16].

Limitations
This review included studies from five databases: Pubmed, IEEEXplore, ACM Digital Library, Scopus, and Google Scholar.Hence, it is possible that literature might have been left out if it is not indexed in these libraries.However, given the coverage by these popular databases, the included studies form a comprehensive representation of the applications of GANs in COVID-19.The review, for practical reasons, included studies published in English only and did not include studies in other languages.
Since the scope of this review was limited to lungs images only, the potential of GANs for other types of medical data such as electronic health records, textual data, and audio data (recordings of coughing) is not covered in this review.The results and interpretations presented in this review are derived from the available information in the included studies.Since different studies may have variations and even missing details in their reporting of the dataset, the training and test sets, the validation mechanism, a direct comparison of the results might not be possible.

1 .
What were the common applications of GANs proposed for challenges related to COVID-19? 2. Which architectures of GANs are most commonly applied for data augmentation tasks related to COVID-19? 3. Which imaging modality is the popular choice for the diagnosis of COVID-19?4. What were the most commonly used datasets of CT and X-Ray images for COVID-19? 5. What studies were conducted with open source code to reproduce the results?6.What studies were conducted and presented to radiology experts for evaluation of the suitability towards future use in clinical applications?

Figure 1 .
Figure 1.PRISMA-ScR flowchart for the search outcomes and selection of studies

Figure 4 .
Figure 4. Venn diagram showing the number of studies using CT vs X-Ray images.Only one study reported the use of ultrasound images (not reflected here).

Figure 5 .
Figure 5. Mapping of correlation between publications from each country vs year of publication.Studies in 2020 are originating mostly from China, India, Egypt, Canada only.In 2021, many other countries are also contributing to the published research.
the year 2021 as compared to 2020.This makes sense as the first half of the year 2020 saw only initial cases of the COVID-19 infections, and research on the use of GANs for COVID-19 had yet to gain a pace.Lung radiology images data for COVID-19 positive examples gradually became available during this period and increased only in the latter part of 2020.The highest number of studies were published from China and India (n=22).There can be two possible reasons for this.Firstly, the two countries hold the top two spots on the rankings of the world's most populous countries.Secondly, the COVID-19 pandemic started in China, hence, prompting earlier research efforts there.
Inconsistent information on the number of images, the training mechanism for GANs, and the selection of test set examples may have affected the findings of this review.In addition, by modern standards of training deep learning models, the size of data reported in most included studies is too small.So, the results reported in the This is a pre-print of the paper accepted for publication in JMIR Medical Informatics.Final version will be available from JMIR.doi: 10.2196/37365 studies in terms of diagnosis accuracy may not generalize well.The findings and the discussions of this review are mainly based on the authors' understanding of GANs (and other AI methods) and do not necessarily reflect the comments and feedback of the doctors and clinicians.ConclusionThis scoping review provided a comprehensive review of 57 studies on the use of GANs for COVID-19 lungs images data.Similar to other deep learning and AI methods, GANs have demonstrated outstanding potential in research on addressing COVID-19 diagnosis.However, the most significant application of GANs has been the data augmentation by generating synthetic chest CT or X-Ray images data from the existing limited size data as the synthetic data showed a direct bearing on the enhancement of the diagnosis.Although GANs based methods have demonstrated great potential, their adoption in COVID-19 research is still in a stage of infancy.Notably, the transformation of GANs based methods into clinical applications is still limited due to the limitations in the validation of the results, the generalization of the results, the lack of feedback from radiologists, and the limited explainability offered by these methods.Nevertheless, GANs based methods can assist in the performance enhancement of COVID-19 diagnosis even though they should not be used as independent tools.Besides, more research and advancements are needed towards the explainability and clinical transformations of these methods.This will pave the way for a broader acceptance of GANs based methods in COVID-19 applications.

Table 1 .
This is a pre-print of the paper accepted for publication in JMIR Medical Informatics.Final version will be available from JMIR.doi: 10.2196/37365 Characteristics of the included studies.Demographics are shown for type of publication, country of publication and year of publication.

Table 2 .
Applications of using GANs based method and type of GANsThis is a pre-print of the paper accepted for publication in JMIR Medical Informatics.Final version will be available from JMIR.doi: 10.2196/37365 This is a pre-print of the paper accepted for publication in JMIR Medical Informatics.Final version will be available from JMIR.doi: 10.2196/37365 This is a pre-print of the paper accepted for publication in JMIR Medical Informatics.Final version will be available from JMIR.doi: 10.2196/37365 used as a sub-module of the overall framework, and the diagnosis was performed with the help of variants of convolutional neural networks such as ResNet, VGG16, Inception-net, etc.In the included studies, GANs were used for seven different purposes: data augmentation, segmentation of lungs within chest radiology images, super-resolution of lungs images to improve the quality of the images, diagnosis of COVID-19 within the images, features extraction, prognosis studies related to COVID-19, and synthesis of 3D volumes of CT.Around 73% of the included studies used the GANs methods for data augmentation to address the data scarcity challenge of COVID-19.It is not unexpected as data augmentation is the most popular application of GANs.Only one study used the 3D variant of GAN for 3D This is a pre-print of the paper accepted for publication in JMIR Medical Informatics.Final version will be available from JMIR.doi: 10.2196/37365 Ray images, and 21 studies used CT images.Some of the studies (n=6) also used both the CT and X-Ray images for diagnosis by training different models or for the transformation of images from X-Ray to CT.Though ultrasound imaging is not prevalent in the clinical diagnosis of COVID-19, one study reported using ultrasound images to diagnose COVID-19 with GANs.No other modality of imaging was used by the included studies.
This is a pre-print of the paper accepted for publication in JMIR Medical Informatics.Final version will be available from JMIR.doi: 10.2196/37365 this, reported from John Hopkins University Coronavirus Resource Center 1 ), the need for experiments with much more extensive data is obvious.As a result of having more data, learning inherent features within the radiology images by using GANs will become more generalized with training on larger data.There is still more need to contribute to publicly accessible data.Practical and Research ImplicationsThis review presented the different studies that used GANs for various applications in COVID-19.Data augmentation of COVID-19 images data was the most common application in the included studies.The augmented data can significantly improve the training of AI methods, particularly deep learning methods used for COVID-19diagnosis.This review found that for most of the studies, the current CT and X-Ray images data (even if smaller in size) is already available through publicly accessible links on Github, Kaggle, or institutional websites.This should encourage more researchers to build upon the available data sets and train more variants of deep learning and GANs methods to speed up the research progress on COVID-19.