2.1. IT Best Practices

IT Best Practices^{4, 5} is a document developed by the Bank of Thailand (BOT) and commercial banks in Thailand as a recommendation of security solutions to control risk that could occur to banking information systems. The recommendation is based on cybersecurity frameworks developed by NIST, ISO 27005, COBIT etc. and covers operation procedures, operation controls, and information systems. The risk control part of the IT Best Practices specifies baseline requirements for banking information systems and is the most relevant in the context of this paper. The baseline requirements address five domains of information systems: 1) Core Banking Application, 2) ATM Application Control, 3) ATM Machine, 4) Internet Banking Application Control, and 5) Internet Banking Security.

2.2. CAPEC Attack Pattern

Attack patterns document reusable attack knowledge to bridge the knowledge gap and assist with attack analysis.⁷ The Common Attack Pattern Enumeration and Classification (CAPEC)⁶ is a taxonomy of cyber security attacks developed by MITRE corporation. It has been incrementally built, starting from 2007, and includes a collection of attack patterns. Each attack pattern captures knowledge about how specific parts of an attack are designed and executed, and gives guidance on how to mitigate the effectiveness of the attack. Each attack pattern description includes several topics, e.g., Summary, Attack Execution Flow, Typical Severity, Typical Likelihood of Exploit, Methods of Attack, Examples-Instances, Attackers Skills and Knowledge Required, Resources Required, Solutions and Mitigations, Related Weaknesses, Relevant Security Requirements, Confidentiality/Integrity/Availability Impact, Technical Context. Among these, we use Solutions and Mitigations, Relevant Security Requirements, Typical Severity, and Typical Likelihood of Exploit information for risk assessment.

4.1. Prepare Standard Security Requirements (SSRB)

First, we extract standard security requirements of banking (SSRB) from the IT Best Practices.^{4, 5} There are 52 standard requirements under five domains (i.e., Core Banking Application, ATM Application Control, ATM Machine, Internet Banking Application Control, and Internet Banking Security). We give each requirement an ID for future reference and define a security category for each requirement. There are 12 categories¹⁵: Identification, Authentication, Authorization, Immunity, Integrity, Intrusion Detection, Non-repudiation, Privacy, Security Auditing, Survivability, Physical Protection, and System Maintenance Security.

4.2. Match SSRB with Attack Patterns

In this step, we study the SSRBs and collect CAPEC attack patterns⁶ that involve software and whose contents are related to the SSRBs. We filter out a number of attack patterns that address the implementation level of the software and keep 24 attack patterns whose details are at the requirement level. Table 1 shows an example. Since each SSRB is the security control that should be in place, we consider the Solutions and Mitigations and Relevant Security Requirements information in each attack pattern description to match them with the SSRB. For example:

SSRB_002: The system shall enforce strong password (contain a mix of alphabetic and non-alphabetic characters).

matches the Solutions and Mitigations of CAPEC_ID 16: Create a strong password policy and ensure that your system enforces this policy.

and the Solutions and Mitigations of CAPEC_ID 49: Put together a strong password policy and make sure that all user created passwords comply with it.

Therefore, the SSRB_002 and the CAPEC_ID 16 and 49 are identified as a match. We have the matching results reviewed by 12 BOT’s security engineers and network engineers, with 2–10 years of experience. An example is in Table 2. We associate the Typical Severity (SEV) and Likelihood of Exploit (LOE) of the attack patterns with each SSRB. Typical Severity refers to the typical severity of impact on the software if this attack occurs. Typical Likelihood of Exploit means the likelihood of this attack typically succeeding considering the weakness attack surface, skills and resources required, available blocking solutions etc. We map the ordinal scale of Very Low, Low, Medium, High, and Very High of SEV and LOE to a numerical scale of 1–5 respectively for later calculation of the risk index. In the case that the SSRB is matched with more than one attack pattern, we use the principle of High Water Mark, i.e., the maximum level, to determine the SEV and LOE associated with the SSBR. For example, given Tables 1 and 2, the associated SEV of SSRB_004 is max(Very Low, High) = High = 4 and LOE is max(High, Medium) = High = 4.

4.3. Prepare Bank Security Requirements (SRB)

Given a security requirements document of a commercial bank, we prepare a list of bank security requirements (SRB), give each requirement an ID for future reference, and organize them into the same five domains as in the case of the SSRBs. An example is in Table 3.

4.4. Calculate Difference Score of Each SSRBSRB Pair

To determine how compliant the SRBs are with the SSRBs, we calculate the difference score of each SSRB and SRB requirement pair based on a text similarity measure. We preprocess the SSRB and SRB texts before similarity comparison as follows.

1. (i)

   Segment words and remove stop words: For SSRBs and SRBs, perform word segmentation on each requirement and remove stop words taken from a list of 619 words from http://countwordsfree.com/stopwords, plus 39 common domain words, e.g., system, application, customer.

2. (ii)

   Change to lowercase letters: Change all capitalized words to lowercase except for words with specific meanings, e.g., SSL, HTTP, SNA.

3. (iii)

   Remove punctuation marks: Remove punctuation marks, e.g., full stop (.), comma (,), brackets(()) etc.

4. (iv)

   Remove suffixes: Remove suffixes of words by truncating their suffixes using the Porter stemming algorithm.

5. (v)

   Determine difference between each requirement pair based on degree of similarity: Using the vector space model, we represent each SSRB and SRB requirement as a weighted term vector, where each element w_i of the vector is the weight of the term (or word) i that appears in that requirement. We follow the Cosine coefficient to determine similarity between each SSRB-SRB requirement pair by using the Term Frequency-Inverse Document Frequency weight (TF-IDF).⁹ Since the similarity measure is bounded in [0, 1], it can be adapted to calculate the degree of difference D_q,r between an SSRB requirement q from the IT Best Practices and an SRB requirement r of a bank by

   (1)Dq,r=1-∑i=1N(wq,i*wr,i)∑i=1Nwq,i2*∑i=1Nwr,i2

   where

   w_q,i =

   weight of word i in SSRB requirement q,

   w_r,i =

   weight of word i in SRB requirement r,

   N =

   number of distinct words in q and r, and

   D_q,r

   is in [0, 1].

Note that the following weight w_s,i is used to determine the weight of word i in document s (i.e., each requirement q or r):

These D_q,r values are calculated for every pair of SSRB and SRB requirements. Given the SSRBs in Table 2 and SRBs in Table 3, the difference scores are shown in Table 4.

4.5. Calculate Risk Index with Respect to Each SSRB

A risk index is a product of the probability of a risk (i.e., likelihood) and the severity of impact caused by the risk (i.e., consequence). In our context, there is the probability of risk of attacks associated with an SSRB requirement when it is not met by any SRBs of a bank. We represent this probability of risk by the minimum D_q,ri of each SSRB requirement q, i.e., the difference of the SRB requirement r_i that is the best match with the SSRB q. This probability is also weighted by the LOE of the attacks that could occur in the absence of that SSRB q from the banking information system. We use the SEV of those attacks as the severity impact caused by the risk. Thus, a risk index R_q with respect to any SSRB requirement q is calculated by

For example, given Tables 4 and 2, the risk index with respect to each SSRB is in Table 5. We define a threshold (0.25 in this example) such that if the minimum difference score is greater than the threshold, we consider the SSRB missing from, i.e., not met by, the SRB requirements. Therefore, there is a degree of risk associated with the missing SSRB. On the other hand, if the minimum difference score falls below the threshold, we reset it to 0 and consider the SSRB not missing as an SRB that can meet that SSRB requirement is found.

4.6. Determine Risk Index of SRB by Category or Whole Document

In the previous section, an SRB that is the best match with an SSRB is identified, and a risk index is calculated with respect to how well the SRB can meet the SSRB. We then can determine the overall risk index for each of the 12 categories of the SRBs based on the risk indices associated with all SRBs under that category. Likewise, the overall risk index of the whole SRB document can be determined by the risk indices associated with all SRBs of a bank. Using the High Water Mark, we can represent the risk index R_c, where c is either a security category context or the whole document context, by the maximum risk index R_q associated with that context as in

Given Table 5, the risk index by each category of the SRBs is shown in Table 6.

5.1. Performance of Compliance Checking

We use the security requirements documents of nine commercial banks in Thailand to evaluate how well the proposed method can identify which SSRBs are missing from the SRBs of the banks. The method is applied to calculate the difference score of each SSRB-SRB requirement pair, find the SRB that is the best match, and determine if there are SSRBs that are not met by any SRBs with regard to a threshold. We use the results of bank requirements validation from the audits as the solution against which the performance of the method is evaluated. The solution tells which SSRBs are considered by the auditors as not met by, or missing from, the SRB documents of the banks.

Tables 7, 8, and 9 show the performance of the method in terms of F-measure of the two predicted classes of the SSRBs (i.e., Missing and Not Missing) and the overall accuracy respectively. The result is depicted graphically in Fig. 2. At the threshold of 0.4, all average performance reaches 80% and the best performance is when the threshold is 0.5. Starting at the threshold of 0.54, false negatives begin to show (i.e., the method predicts missing SSRBs as Not Missing) and the F-measure of the Missing class subsequently drops. We consider the false negatives as risky and not desirable, and so the threshold between 0.4–0.5 is recommended. Note that, at the threshold of 0.75, the F-measure of the Missing class for Doc#4, Doc#8 and Doc#9 is not applicable (N/A) because there is no SSRB with the difference score of at least 0.75 and predicted as Missing. So the precision, and hence the F-measure, of the Missing class cannot be calculated.

Fig. 2.

Average F-measure and accuracy with regard to all SRB documents.

On taking a closer look at Doc#2 and Doc#5, we notice that the performance is particularly high even at the low threshold of 0.05–0.25, compared with the other seven documents. As the SSRBs usually mention technical security terms as the examples of the techniques that should be implemented, Doc#2 and Doc#5 which are written in more technical terms, match better with the SSRBs than the other seven documents which are written in more general terms and do not contain many technical terms. For example, an SSRB states that “The system shall encrypt confidential information (e.g., user ID, password encryption key, database user id, database password) by strong encryption (e.g., AES 128 bits, AES 256 bits, RSA 2048).” While Doc#1 is written as “The application shall apply strong encryption to encrypt confidential information and store in a database or configuration file with appropriate access control.”, Doc#5 is written as “The application shall use AES 256 bits encryption to encrypt system user id, password, database user id, and password before storing those hashes in a configuration file.” The difference score of Doc#5 with more technical term is lower, and this goes along with the view of the auditors who are less likely to agree with a requirement as a general statement. The auditors usually expect the banks to be explicit about the techniques used whenever possible.

We further experiment by dividing the SRB documents into two groups, i.e., Doc#1, Doc#3, Doc#4, Doc#6, Doc#7, Doc#8 and Doc#9 that use less technical terms and Doc#2 and Doc#5 that use more technical terms. The performance with regard to the two groups is in Fig. 3 and Fig. 4. For the non-technical terms group, the best performance is still when the threshold is 0.5 as there are false negatives when the threshold is higher. For the technical terms group, the performance is best even at the low threshold of 0.25 and continues so until the threshold reaches 0.5 where the performance subsequently drops and false negatives appear. This experiment suggests the requirements engineers who use the proposed method to consider the writing style of the SRB document and may adjust the threshold accordingly.

Fig. 3.

Average F-measure and accuracy with regard to SRB documents that use less technical terms.

Fig. 4.

Average F-measure and accuracy with regard to SRB documents that use more technical terms.

5.2. Validity of Risk index

To validate the risk index, we use Spearman’s rank order correlation to determine the correlation between the risk index of each SRB document as calculated by the method and the ordinal risk level determined by 12 security engineers and network engineers as in Table 10. We map the ordinal risk level of Very Low, Low, Medium, High, and Very High, given by the engineers, to a numerical scale of 1–5 respectively. For the risk index by the method whose value is in [0, 25], we map the range [0, 5] to 1, (5, 10] to 2, (10, 15] to 3, (15, 20] to 4, and (20, 25] to 5.

The hypotheses are

• H₀: There is no monotonic correlation between the risk level by the engineers and the risk index by the method (ρ_s = 0).

• H₁: There is a monotonic correlation between the risk level by the engineers and the risk index by the method (ρ_s ≠ 0).

The calculated correlation coefficient r_s = 0.74186. Since r_s is not less than r_critical = 0.7000 at the significance level α = 0.05 and n = 9, we reject H₀ and accept H₁. There is a monotonic correlation between the risk level by the engineers and the risk index by the method at α = 0.05. The correlation is strong and positive.

To experiment further, we consider the total of 61 missing SSRBs from all nine SRB documents, together with their associated risk indices, to test a monotonic correlation with the risk levels given by the engineers. The distribution of the mapped risk levels of all 61 SSRBs is shown in Fig. 5. In this case, since r_s = 0.63824 is not less than r_critical = 0.252 at the significance level α = 0.05 and n = 61, we again reject H₀ and accept H₁. For the case of missing SSRBs, there is also a monotonic correlation between the risk level by the engineers and the risk index by the method at α = 0.05. Again, the correlation is strong and positive.

Fig. 5.

Average F-measure and accuracy with regard to all SRB documents.