2.1. Related Works

Various applications of machine learning techniques have been applied to the phishing detection problem. Several machine learning approaches are used for combating phishing attacks. These approaches include Support Vector Machine (SVM), Bayesian additive regression trees, Random Forests (RF), classification and regression trees, Logistic Regression (LR), and Artificial Neural Networks (ANN) [6]. A nonlinear regression strategy was used in Rao and Pais [7] for detecting a phishing web site. They preferred the use of a harmony search and support vector machine metaheuristic algorithms for training the system. The authors Basnet et al. [8] and Jain and Gupta [9] used specific detection algorithms with selected feature patterns in their specific domain.

Two machine learning aided approaches for static analysis of Android malware were proposed in Chin et al. [10]. The first approach was based on the permissions and the other was based on the source code analysis utilizing a bag-of-words representation model. A phishing detection and mitigation approach with software-defined networking was proposed to identify the phishing activities through an e-mail and a web-based communication [11]. A phishing detection model based on deep belief networks was presented in Yi et al. [12] that works on real IP flows from Internet Service Provider (ISP).

Each of these phishing detection approaches showed the acceptable detection accuracy, the detection accuracy and performance are limited to the use of features and machine learning techniques [13]. Again, some ensemble method in the form of a combination of multiple classifiers was used in Toolan and Carthy [14], Hota et al. [15] and “A Comparative Study of Phishing Websites Classification Based on Classifier Ensembles” [16]. Although, these ensemble method-based detection system could improve the accuracy as well as the efficiency, they suffered from concept drift issues because they could not support the automatic adaptability to the various input features.

A phishing detection system using a case-based reasoning was presented in Abutair et al. [17], by using the features of phishing Uniform Resource Locator (URLs) as the case and classifying the phishing URL. Their work concentrated on the URL feature extraction and the keyword similarities in the main domain name and the remaining keywords in the URL such as the path, subdomains, and query part. CBR was used in Abutair et al. [17] as a classifier for the new incoming web link URL to detect the phishing website. In our previous work [4], CBR was used to adaptively select the most suitable classifier among existing classifiers to solve the current case.

2.2. Case-based Reasoning

In the CBR technique [18], the new problems were solved by re-usable or adapted solutions which were previously used in the past. Under the situation of solving the unfamiliar problem in the domains that are not well understood, CBR can perform well, by using the knowledge on what worked in the past. The experiences from the previously solved problems are encoded as cases and each case contains the feature characteristic of the problem and the corresponding solution. The case-base maintains the collection of cases and it can be used as the knowledge base to handle the new problems. Thus, CBR is used to handle the concept drift, for endorsing the optimized machine learning in our previous work [4].

In our CBR-based automatic adaptable mobile phishing detecting model, the feature pattern of the mobile app and the corresponding machine learning algorithm which can support the optimized accuracy and efficiency will be encoded into cases and stored in our case-base. For a new feature input, the case-base will endorse the best machine learning scheme for making a detection. The general structure of the CBR as a cyclical process [19] which can be divided into the sub processes is represented in Figure 1.

Figure 1

Case-based reasoning process.

The four main sub-processes are existed in a case-based reasoning process:

1. (i)

   Searching the most similar case to the current problem.

2. (ii)

   Re-applying the similar case solution to solve the current problem.

3. (iii)

   Feeding back the proposed solution to the system, if requires.

4. (iv)

   Maintaining the solution for future uses.

Basically, a new problem is symbolized as a case, and it is compared with the stored cases in the case-base. Then, the most similar case or cases are selected according to the comparative results of the case representation features. These cases will be combined or reused to suggest a possible solution for the new problem. The suggested solution is evaluated and corrected if a revision is needed. The confirmed solution will be maintained by adding it back as a new case to the case-base or as an amendment to existing cases in the case-base for a future problem.

In our previous work [4], various machine learning algorithms are applied to the problem of mobile phishing detections and the CBR technique is applied. Thus, an approach for retaining the solution of the new case is proposed in this work to enhance the previous work in adapting to the new case by learning it and combine it into our system in a systematical way. Incremental learning can be defined as learning a concept incrementally by processing predefined training examples in a one-by-one manner [20].

2.3. Proposed Adaptive Phishing Detection Model

An adaptive mobile phishing detection model that will work on a variation of input feature patterns using a CBR technique is proposed in our prior work [4]. The overall system design of the proposed adaptive phishing detection system is shown in Figure 2.

Figure 2

System design of adaptive phishing detection.

The extracted features from the active Android apps are fed to the detection system to find the most similar case in the case-base. When the extracted features from the active Android apps are received by the detection system, the first process is selecting the most similar case from the case-base and retrieving the similar case. The details of the case-base setting up process were presented in our previous work [4]. According to the information of the retrieved case, the corresponding classification techniques will be performed on the extracted features by the adaptive classification.

When the extracted features of the active Android apps do not match with the features of the stored cases in the case-base, the incremental learning process will be performed on the extracted features using multiple classifiers. Then, the optimal classification technique with the maximum detection accuracy is selected and the corresponding information including the features with the corresponding classifiers is stored in the case-base. Then, the result of the active Android application classification will be displayed to the user.

2.4. Incremental Learning to Update the Case-base

Our prior work proposed to create a phishing detection system based on the CBR approach to automatically adapt the classifiers depending on the features input patterns. Combining the good performance of all classification methods appropriately, the proposed phishing detection system could provide the best performance by addressing the optimal selection of the appropriate classifier to the input features using a case-based reasoning approach. However, the details of the learning process to update the case-base for handling the concept drift were lacking in our prior work [4]. In this work, the threshold-based incremental learning model for the CBR-based adaptive phishing detection is proposed and evaluated.

The incremental learning model for the case-based updating process constitutes the learning process for the new cases. The learning process will be launched in two possible cases; first, the learning process will be launched when the output from the reused process meets the threshold acceptable point; second, the learning process will be launched when an uncertain output is observed [21]. The solution have to be revised and modified to become a confirmed solution and the appropriate retaining or adding to the case-base are included for further problems [22]. The case-base updating process involves two steps; the proposed solution is evaluated and repaired, if needed. The goodness and competence of the proposed solution to the case-base is judged in the evaluation step.

Generally, there are two categories of learning techniques in the case-base updating policy. The first one is a continuous case-base updating with training examples that have been misclassified by the system. The second one is a periodic retraining of the classifier to reselect the features. Performing these two approaches of learning, the system will be more predictive in finding the most appropriate detection algorithm.

The evaluation process can be performed in a number of different methods such as based on the results of the simulations when applying the solution. The evaluation outcomes could suggest the further adaptation to the case-base or the repairing of the proposed solution. When the further adaptation is performed on the solution, the revised case will be added to the case base. Thus, the case-base updating process is the ongoing argument of the case-base with the revised case thereby allowing the system to learn. Successful new solutions are added to the case-base memory to facilitate any similar problem in the future and failures can be added to avoid repeating the same mistakes. Increasing the number of example situations in the case-base means that the system covers more domains and will work better. In this work, updating the solution of the new case is addressed for our case-based reasoning approach to enhance the automatic adaptation of classifiers in a mobile phishing detection domain.

3.1. Feature Extraction

The feature extraction process will extract the features of the active Android application for the detection process. Features can be extracted and collected from various sources in the Android environment by using static analysis techniques (e.g., AndroidManifest.xml and class files) and dynamic analysis techniques. For example, the required permissions and the sensitive APIs will be extracted through the static analysis, whereas the dynamic behaviors will be extracted through the dynamic analysis.

Under the static analysis technique, the installation file (i.e., *.apk file) of each Android app are decompressed and the resulted AndroidManifest.xml and Classes.dex files are parsed. By parsing the Android Manifest.xml file, the permissions required by the apps are obtained. By parsing the classes.dex file with the disassembler, the usages of the API functions are extracted.

Under the dynamic analysis technique, the sandboxing software (i.e., CuckooDroid Sandbox) needs to be installed and processed on each apps for automating analysis of the suspicious files. The monitoring process will take place by hooking the chosen Dalvik API calls for extracting the behavioral information such as file input/output, short message services, mobile phone calls, and information leaks. Feature extraction process performed by the system is discussed in previous work [4]. When our phishing detection system received the extracted features of android apps, the similarity scores of these features are calculated by our system.

3.2. Feature Selection

About 1065 frequently used mobile phishing features were used in our prior work [4]. These features were extracted from more than 10,000 Android samples, collected from public Android malware repositories. The mobile phishing features are classified into 10 classes including Android components, Android API counts, API usage action, security sensitive data flow, hardware components, intent actions, permissions, shell command and strings, contents and visual, and URLs. Feature selection techniques on these ten feature sets are performed in this work to reduce the feature space dimension, to reduce the over fitting issue as well as to improve the accuracy and efficiency of the detection. The comparative analysis of two feature selection methods including correlation-based feature selection and information gain based feature selection, are conducted in our prior work [4].

In this work, the information gain attribute evaluation-based feature selection with a ranker search method is performed on the ten feature sets. Then, the highest ranked attributes with the maximum gain ratio are selected and the lower ranked attributes are excluded for conducting the experiments in this work. After performing the information gain attribute evaluation based feature selection technique on the dataset, the total number of features in the reduced features sets are 552 features.

3.3. Case Retrieval with Cosine Similarity Calculation

Finding the stored cases closet to the current problem in the case-base is the case retrieving process. The closet case gets the opportunity to be a suitable solution for the new problem. The finding process of the most suitable case was believed to be the core of CBR [22]. One challenge related to the case retrieving process is the computation of the similarity score during the case retrieving process. The usefulness of a retrieved case in a new problem solving determines the effectiveness of the similarity measurement. In this work, the total similarity scores of the potential cases are calculated with the cosine similarity measure. The cosine similarity calculates the similarity metric between two vectors of an inner product space by comparing the input target case to the cases in the case-base. It is measured by the cosine of the angle between two vectors and determined whether two vectors are roughly pointing in the same direction. This technique is often used to measure the document similarity in text analysis problem [23].

The similarity between a stored case vector C and the test set as a query vector T is calculated using the cosine similarity function where the cases deal with the feature attributes name (textual information). This ratio is defined as the cosine of the angle between the vectors, within the values between 0 and 1 and can be calculated by Equations (1) and (2).

The closet measurements of each feature of the stored cases to the new case are formed the similarity metric. These closing measurements are presented as the similarity scores and the score is assigned depending on the difference between the two features. That is, the closer the features the higher the similarity scores. The features according to their similarity score are ranked for the most relevance to the current problem to the least relevance to the current problem. The cases with the highest similarity values will be retrieved. The prediction result of the new problem can then be derived from the solution of these most similar cases.

4.1. Experimental Setup

To verify the ability of handling the concept drift, our proposed model is evaluated with an experiment using the new unseen dataset containing 43 features that are extracted from 32,342 Android apps samples. This experiment was conducted on a prototype system which is developed with Java SE Development Kit 8, Eclipse Integrated Development Environment Version 4.7.3a (Oxygen) with Google Core Library, Guava, Simmetrics Core Library 4.1.1, and WEKA 3.8.1 package running on a Laptop computer with Windows 8.1 64-bit operating system on core i7 processor with 8 GB RAM.

For the adaptive classifier evaluation in our prior work [4], three combination rules of voting are performed with six classification algorithms including voting with average probability, voting with the maximum probability, and the majority voting methods. Nine classification techniques were used for this experiment. In this paper, three more classification techniques are added to the existing nine classification techniques including a stacking with a logistic regression as the meta-classifier, a Neural network with two different hidden layers selection methods. Totally 12 classification techniques are executed in this experiment. The maximum detection accuracy from the 12 classification results is chosen for the final decision. Two parts of experiments are arranged to demonstrate the incremental learning ability of our proposed model on the incomplete or missing features, and the new set of features to the case-base. The detail information of the case-base editing process is explained in the following sections [24].

4.2. Dataset

An experiment is performed on our proposed model using five testing features as shown in Table 1 to verify the ability of the incremental learning for handling the concept drift. There are two categories on this testing data. First, four testing feature groups contain test feature 1–4 which were collected from the previously used dataset in our prior work [4]. This first category is collected with the aim of testing the incomplete or missing features in the case-base.

Second category contains the feature group test 5 which consists of 43 features extracted from over 24,000 malicious apps and 7535 benign apps, from 32,342 Android unique applications samples. These Android samples were gathered from the Virus Total Malware Intelligence Service [25], and Google Play. The extracted features are collected from Amos et al. [26]. This second category is collected with the aim of testing the new features to the case-base.

4.3. Adding New Cases to the Case-base

In this work, 15 new cases as shown in Table 2 are added to the case-base of our proposed system. These 15 new cases are the reduced feature set. In our previous work [4], the proposed system was evaluated on the complete feature set for the first 10 cases. In this work, the reduced feature sets, selected by the feature selection process, are used instead. Furthermore, the same five random feature combinations used in the previous work [4] are also used here as the last five cases. These five random feature combinations are created to mimic the real-world apps and defined as cases. These five combinations are randomly extracted from the 10 feature sets that form the list of frequently used features by existing anti-phishing solutions.

As same as the experimental setting of the previous work, the input features must pass through different machine learning classifiers for the case defining process and the case adding process. There are nine classifiers used in this initial step. Only nine out of 12 classifiers are used for the case defining and the case adding processes because the additional three classifiers will be used for demonstrating the benefit of the incremental learning model proposed in this work. The nine classifiers consist of six individual classifiers and three ensembles of classifiers. The six individual classifiers include C4.5 (J48), Decision Table (DT), k-Nearest Neighbors (IBK), Logistic Regression (LR), Naïve Bayes (NB), and SVM. Each of the three ensembles of the six classifiers used a different voting technique to produce its answer including the average of probabilities (AVG), the majority voting (MAJ), and the maximum probability (MAX) or winner-take-all. The results of all nine classifiers on all fifteen cases are shown in Table 3.

The bolder italicized values shown in Table 3 represent the maximum detection accuracy among nine classifiers for each case. Then, the input features, the classifiers with their parameters, the activation function, and the final result are stored in the case-base (knowledge base) as a new case. According to the detection results shown in Table 3, the detection accuracy of each classification technique is heavily depends on the features. Similar with the experimental results shown in our previous work [4], ensembles of classifiers perform well on many cases.

4.4. Similarity Scores of Testing Data to Case-base

The proposed adaptive phishing detection system will choose the suitable classifier based on the similarity scores. If the test data found the matching case with a high similarity score (the larger the similarity score value the better match), the corresponding classifier is assigned for that test data according to the adaptive classification process. If the matching case is not found by the system, the test data will undergo to the incremental learning section in order to get an optimal solution for that test data. Then, the features, the solution, the accuracy, and the performance of that test data are stored in the case-base as the new case which is called the case-base updating process or the system learning process.

Table 4 shows the similarity scores of all the five test cases with the currently stored cases (15 cases in total). In addition, the resulting similar cases, and the suggested classifier are also presented in Table 4 when applicable.

According to the results shown in Table 4, test case 1, 2 and 4 will be proceeded with retrieving the most similar case as the solution. The most similar cases are extracted from the case-base with the highest similarity score, and the solution for that case will be used for the detection process. That is, test case 1 and 2 will use J48 as the solution while test case 4 will use the ensemble of classifiers with an average voting technique. After performing the detection process using the suggested classifier, the detection accuracy is 95.22%, 95.93% and 97.51% for test case 1, 2 and 4 respectively.

Since there is no similar case presented for test case 3 and 5, both cases will be sent to the incremental learning process. Test case 3 will be sent to the incremental learning process under the low similarity score condition while test case 5 will be sent to the incremental learning process under the completely new case since the similarity score is very low. The process of handling the test case 3 and 5 is called the case revising process presented in the next section.

4.5. Editing Case-base for Comprehensiveness of Adaptive Phishing Detection

Before adding any new case to the case-base, the features of the new case must pass through the feature selection process in order to prevent the storing of ineffective features. The feature selection process used in our prior work [4] is applied in this work. Therefore, the features of test case 3 and the features of test case 5 will be sent through the feature selection process. After performing the feature selection process, each test case results in a set of reduced feature. That is, there are 160 features and 26 features for test case 3 and 5, respectively.

Next, each test case will be processed by multiple classifiers including all twelve classifiers. The twelve classifiers include six individual classifies (C4.5 (J48), DT, IBK, LR, NB, and SVM), three voting ensembles (the average, the majority voting, and the maximum (winner-take-all)), a stacking ensemble with logistic regression as meta-classifier and two ANN with different numbers of hidden layers. The first ANN is with ‘a’ hidden layers where ‘a’ is the number of attributes while the second ANN is with ‘o’ hidden layers where ‘o’ is the number of classes in the case-base.

Then, the most appropriated classifiers will be selected and added into the case-base. The detection results of testing the reduced feature sets of test case 3 and 5 are shown in Table 5. According to the results presented in Table 5, the NB will be selected for test case 3 while the ANN option-a will be selected for test case 5.

After the case revising process, the new feature set and its corresponding classifiers are stored as a new case in the case-base. Thus, the number of cases in the case-base is now increased from 15 to 17 cases. That is, the information of the test case 3 is stored as case 16 in the case-base while the information of the test case 5 is stored as case 17 in the case-base.

4.6. Evaluating the Newly Added Cases

As the main challenge faced by using the CBR in the phishing detection work is bringing up to date the case-base to handle the concept drift of mobile phishing. This section, the newly added case 16 and 17 will be evaluated by using 4 test cases intentionally crafting as a mismatch feature of case 16 and 17 in the case-base. Table 6 shows the profiles of the four test cases namely test case 6–9. Test case 6 testing feature groups are newly collected as the incomplete feature patterns of case 16 in the case-base whereas test case 7–9 testing feature groups are newly collected as the incomplete feature patterns of case 17 in the case-base.

The similarity scores of the four test cases on the current system with 17 cases in the case-base are presented in Table 7. While, Table 8 shows the selected case from our system to produce the answer to each test case. As can be seen that the current system can detect the correct similar case from the case-base for each test case. According to the results shown in Table 8, the accuracy of the selected classifier from the case-base is 89.34%, 99.94%, 97.32% and 97.35% for test case 6–9, respectively. While the runtime for most test cases is lower than 4 s except the runtime for test case 7 which takes almost 21 s because test case 7 contains 4041 instances.

4.7. Confusion Matrix

In order to measure the usefulness of our proposed incremental learning model, the confusion matrix evaluation is applied. The amount of correct phishing apps prediction is represented by the True Positive (TP); the amount of incorrect phishing apps prediction is represented by the False Positive (FP); the amount of correct benign apps prediction is represented as the True Negative (TN); and the amount of incorrect benign apps prediction is represented as the False Negative (FN). These four represented outcomes are designed the confusion matrix as shown in Table 9.

The effectiveness of our adaptive phishing detection is represented with three performance measurements including accuracy expressed by Equation (3), precision expressed by Equation (4) and sensitivity expressed by Equation (5). The sensitivity expresses the ability of the proposed system to find all relevant instances in the dataset. The precision expresses the proportion of the instances that our model predicts as positive and they are actually positive. The following formulas represent their definitions, Equations (3)–(5).

The evaluation of effectiveness on our proposed model by means of accuracy, precision and sensitivity is described in Table 10. According to the results shown in Table 10, our adaptive model with incremental learning achieves a good detection accuracy for the mobile phishing. That is, most cases (14 out of 17), the accuracy is more than 90%. While, the accuracy of two cases (case 5 and 6) is more than 85% and the accuracy of case 8 is 75.4%. Also, most cases (14 out of 17), the precision is more than 80%. The precision of two cases (case 6 and 08) is more than 75% and the precision of case 5 is 59%. The sensitivity of most cases (14 out of 17) is more than 70%. The sensitivity of case 14 is 62%. The sensitivity of case 6 is 51% and the sensitivity of case 5 is 3%.

According to the results shown in Table 10, it can be seen that our adaptive model with incremental learning achieves a good detection accuracy for all cases. The proposed system can adapt to an incoming of newly and missing mobile phishing features. That is an ability of our proposed case-based reasoning system with incremental learning model to resist the concept drift in the mobile phishing context.

5.1. Scope of the Work

The scope of the proposed adaptive mobile phishing detection model that worked on a variety of features using a CBR technique summarizes as follows:

1. (i)

   This work investigates the phishing detection on Android platform.

2. (ii)

   The proposed adaptive mobile phishing detection model initially developed and verified. The detection performance on variety of features patterns with 1065 features from 10 feature groups which are collected from 10,000 Android apps are presented. Nine classifiers are used in our previous work including six classification algorithms, and three ensembles of six base classifiers.

3. (iii)

   The incremental learning ability of the proposed adaptive phishing detection model is evaluated with a total number of 759 features which are collected from 32,342 Android apps. In addition to the previously used nine classifiers, another one ensemble technique (stacking) and two types of neural network with different numbers of hidden layers are also used in this experiment. Totally, 12 classifiers are used in this paper for the coverage of different classification techniques.

4. (iv)

   The evaluation of incremental learning model is performed with a total of nine testing feature groups for two real world circumstances containing the new strange features to the case-base and the incomplete or missing features cases.

5. (v)

   The experiment was conducted by running Weka 3.8 and Eclipse Java Oxygen 4.7.3a with Guava-package_19, simmetrics-core package_4.1.1, WekaMod and commons-codec-1.10 on a Laptop computer with core i7 processor, 8 GB RAM, and Windows 8.1 64-bit operating system.

6. (vi)

   Information gain feature selection is applied on the dataset to reduce the size of the features for overcoming the efficiency degradation.

5.2. Future Work

This work performed the continuously case-base updating with new cases that are not matched with the stored cases of our proposed detection system and reused these cases as the training examples to the case-base. The implementation of the periodic retraining of the classifier to adapt the new cases with a strange features will be addressed as the future work of this research for the sustainable learned case-base. That implementation will contribute to become more predictive of the most appropriate phishing detection algorithm.