2.1.1. Data annotation module

The data annotation module provides numerous manual and semi-automatic annotation tools to support various annotation tasks such as segmentation, classification, target detection, key point positioning, etc., along with customizable multi-level tags for annotation content management and a well-equipped multi-blind review mechanism. The annotation toolbox provides common annotation functions such as stipple, line, ellipse, polygon, rectangle, key point, classification label, and measurements, and more advanced functions such as brush, magic wand, curve sketching, contour instance measurements, etc. It also comes along with semi-automatic annotation tools, including adaptive fitting, interpolation and Lasso tool. Thus, users do not need to switch to other annotation software. Figure 1 shows the UI of data annotation tools for region of interest (ROI) segmentation.

Figure 1

Semi-automatic ROI annotation.

This module also supports data management. It supports a batch upload of data with breakpoint continuation and a direct connection to imaging equipment/PACS, followed by the cloud distributed storage with a build-in desensitization tool for the anonymization of personal data, such as patient's name, institute's name, etc. After being uploaded, the original data including PET/CT/MR/DR/US images and structured clinical information (diagnosis results/pathological grades/gene protein) data can be grouped by area and disease type, and the records of the uploading, distribution, and the utilization history are kept for future references. The data is managed according to the projects, advanced users can allocate data to multi-level projects and set review permissions within the platform, with isolated access and real-time record of the annotation progress. For an individual project, annotation can be performed from different devices and stored securely, while a database is shared to make research/communication in a common annotation environment for a collaborative project.

After an annotation task is completed, annotated data can be assigned to reviewers for assessment. The reviewers can approve or modify the annotation results. Final annotation and reviewers' marking results are mutually inferable and traceable, which forms a flexible mechanism for double-blind and multi-blind annotations. The corrected annotated data after reviewing can be set as gold-standard annotations by administrator to be used for further research or education purposes.

The annotation module can prevent the loss/damage of annotated data by hosting the service on cloud, allowing efficient co-operative annotation without the limitations of computer systems and the restriction of tools. Its data management function enables the management of project groups, data, and collaborations, and ensures the communication work in a safe and secure cyberspace. The customization option for annotation tags suiting for various clinical scenarios can satisfy the demand for flexibility.

2.1.2. ML-based radiomics module

After the data are annotated, the radiomics module will carry out feature extraction and feature selection and ML-based prediction. Figure 2 demonstrated the pipeline of this module.

Figure 2

An overview of the ML-based radiomics workflow.

Figure 3

A workflow of deep learning medical image analysis.

2.1.3. DL-based medical image analysis module

The DL-based medical image analysis module provides a much easier way for medical image analysis. Different from ML-based radiomics, which extracted predefined features, DL approaches can extract more complicated and high-dimensional features from multi-layer of neural networks. Figure 3 illustrates a workflow of DL-based analysis. The DL models include segmentation models (DeepLab, U-Net, Mask-RCNN), detection models (YOLO, FPN, ResNet series and Faster-RCNN), and classification models (Inception, VGG). The DL module mainly involves the following eight steps:

1. Dataset collection: upload annotated/un-annotated medical images and related clinical information to the platform;

2. Data annotation: annotation module described in Section 2.1.1;

3. Data split: training, validation, and testing dataset division at random or predefined groups;

4. Model selection: select a DL model from the model library of the platform for training from scratch, or importing internally pretrained and saved models or format-compatible external models;

5. Hyperparameter configuration: GPU index, epoch number, batch size, training times per epoch, optimizer, learning rate, loss function, anchor ratios if applicable, etc.;

6. Model training: perform and monitor model training through visualizing the loss function curvature and the landscapes of other indices during the training process;

7. Model test and validation: test and validate the model with predefined validation and test datasets;

8. Output: output the trained model in a given format (such as HDF5, JASON, YAML).

With 2 * GPU (1080 TI 12GB), 1 * CPU(intel Xeon(R)), training 100 samples(epoch = 50, batch = 4) takes 2–6 hours for classification task and 4–12 hours for segmentation and detection tasks also differs depending on the different frameworks applied.

3.1.1. Lung

The feature extraction module in the RadCloud platform have been used for extracting more than one thousand radiomic features from 2D/3D ROIs of CT and PET images to improve the prediction of the lymphovasular invasive status and prognosis of patients with lung adenocarcinoma [7,8]. It was also used to extract radiomic features of lung tumors from CT component of PET/CT images for distinguishing benign and malignant lung lesions [9].

Radiomics-based predictive models with RF and KNN classifiers have been used for identifying glucocorticoid-sensitive patients with connective tissue disease (CTD)-related interstitial lung disease (ILD) for optimizing glucocorticoid treatment [10].

3.1.2. Breast

Following the workflow of radiomics analysis built in RadCloud, a multivariate logistic regression based radiomics model was developed to predict benign and malignant breast tumors (<1 cm) from contrast-enhanced spectral mammography (CESM) images [11]. The model showed a good prediction performance with an area under the curve (AUC) of 0.940 in a retrospective study of 139 patients. To identify whether the existence of HER-2 oncogene was associated with the poor prognosis and responses to certain chemotherapies (e.g., trastuzumab), the radiomics models with SVM and LR classifiers using mammography (MG) radiomics features, were developed and validated on an MG dataset of 526 patients [12]. In addition, a LR-based MR radiomics model was used for predicting the Ki-67 proliferation index in patients with invasive ductal breast cancer from MR images [13].

Similarly, radiomics models with three ML classifiers (LR, XGBoost, and SVM) were constructed based on dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) of primary breast cancer patients and used to predict the existence of axillary SLN metastasis, an important biomarker for determining the cancer stage and adjuvant treatment plan decision after surgery. The SVM classifier achieved the highest classification performance with an accuracy of 0.85 and an AUC of 0.83 in a clinical study of 62 patients [14].

3.1.3. Liver

To preoperatively differentiate focal nodular hyperplasia (FNH), the second most common benign liver tumor, from HCC in the noncirrhotic liver for facilitating clinical decision-making of treatment, a radiomics-based regression model was implemented and validated, in which the feature extraction module of RadCloud was used for radiomic feature extraction from CT images [15]. To predict microvascular invasion (MVI) of HCC, six radiomics-based classifiers were constructed. Among them, SVM, XGBoost, and LR classifiers showed great diagnostic performance with AUCs > 0.93 [16].

In addition, four radiomics models with ML classifiers (MLP, SVM, LR, and KN) supported by the RadCloud platform have been successfully established for the preoperative diagnosis of dual-phenotype hepatocellular carcinoma (DPHCC) and the prediction of patient prognosis from MR images [17]. Moreover, the radiomics-based classification of HCC and hepatic hemangioma (HH) using MR images showed a good performance equivalent to an experienced radiologist with 10 years experience.

3.1.4. Kidney

Clear cell renal cell carcinoma (ccRCC) is an extremely aggressive urologic neoplasm, and ccRCC patients with synchronous distant metastasis (SDM) have poor prognosis. Early identification of SDM can ensure the reasonable and effective treatment. In a retrospective study of 201 patients, the RadCloud platform was used to construct an MR radiomics model with multivariable logistic regression for the individualized identification of SDM in patients with ccRCC. The model achieved a convincing predictive performance with an AUC of 0.854 [18].

In addition, the RadCloud platform help build CT radiomics models for distinguishing benign rental angiomyolipoma without visible fat from malignant ccRCC [19], discriminating low-grade and high-grade ccRCCs [20,21], differentiating sarcomatoidrenal cell carcinoma (SRCC) and ccRCC [22], and distinguishing benign fat-poor angiomyolipoma (fpAML) from malignant chromophobe renal cell carcinoma (chRCC) [23].

3.1.5. Rectum

The RadCloud platform has been applied for rectal cancer (RC) studies. For example, in a retrospective trial of 166 consecutive RC cases, an XGBoost-based MR radiomics model was constructed to differentiate local recurrence of RC from nonrecurrence lesions at the site of anastomosis, with an AUC of 0.864 [24]. In a retrospective study of 165 consecutive patients with locally advanced rectal cancer (LARC), an LR-based MR radiomics model was developed to predict the pathological complete response (pCR) after neoadjuvant chemoradiotherapy (nCRT) in order to identify candidates for optimal treatments [25]. In a study of 61 patients with renal tumors, CT radiomics models with five ML classifiers were implemented for differential diagnosis of malignant renal chromophobe cell carcinoma (chRCC) and benign renal oncocytoma (RO) from CT images [26]. MR radiomics models with SVM and LRT were developed to differentiate benign rectal adenoma and adenoma with canceration [27].

The RadCloud platform has also been applied in grading tumor studies. MR radiomics models with different ML classifiers (such as MLP, LR, SVM, DT, RF, and KNN, etc.) were developed for predicting tumor stages simplified as T(T1-T2) and N(N1-N2) stages [28], and tumor regression grades (TRGs) for evaluating the treatment response of LARC after neoadjuvant chemoradiotherapy (nCRT) of RC [29].

3.1.6. Others

The RadCloud platform has also been employed in studies on tissues as well through constructing MR radiomics models for discriminating high-risk from low-risk lesions in the anterior mediastinum [30], or predicting the histopathological grades of soft tissue sarcomas [31].