2.1. Contribution to Source File Development and Maintenance

A source code repository of a software project stores the source files and their change history. From the repository, we can easily obtain the following information: “when and who created a source file,” “when and who changed the source file,” “which parts of the source file were changed,” etc. For instance, Fig. 1 presents a part of commit log from the repository of an OSS project^‡. The log is obtained by running git log command with -p option.

Fig. 1.

An example of a repository log (the author’s name and address are masked by dummy ones, and the file pathes are partially omitted).

From Fig. 1, we see that “XXX YYYY” committed InnerHitBuilder.java at 11:59:59 (+0200) on July 16, 2018. The commit is identified by the hash fa59bb10999b20d487ff12a890ee8cfa0f414826. Through the commit, two lines in this file were changed: the heads of changed lines were denoted by symbols “−” or “+,” and a “−” line and a “+” line show the contents before and after the code change, respectively.

By analyzing these data, we can observe the contributions of developers (authors in the example shown in Fig. 1) to the development and maintenance of the source file. For example, given a source file, we can investigate “who have been involved in its development and maintenance” and “how many source lines of code have been added, deleted or changed by each of the developers.” They form a history of contributions by developers to a source file. Different source files have different histories of developers’ contributions. While one source file may have been developed and maintained by only a certain developer, another source file may have been done by many different developers. Moreover, for the latter kind of source file, there would be many variations in the structure for cooperation among developers. For example, suppose two developers d₁ and d₂ have contributed to a set of source files. While one source file may have been dominantly maintained by d₁ and the contribution by d₂ is a little, another source file may have been evenly maintained by both d₁ and d₂. Our research interest is to analyze how such differences of contributions affect the code quality, especially, the fault-proneness of source files.

Notice that we focus only on the change history stored in the code repository in order to see a developer’s contribution; there are also other types of contributions such as the code reviews, tests and discussions. Although it is ideal to take into account all kinds of contributions, it is hard to collect all of such data from any OSS project. Since the code repository is commonly available at any OSS project, we will limit our focus of “contribution” to the code changes in this paper. That is to say, we measure a developer’s contribution by counting the lines of code which the developer has been involved in.

From the code repository, we can also easily investigate who is the author of each line: Fig. 2 shows an example output of git blame command. Each line in the figure consists of the hash (short version), the file path, the author name, the commit date, the line number and the line contents, respectively. The commit hash and date correspond to the last change on the line. That is to say, by checking the data, we can see who made each line of the current version of the source file. For example, there are three different developers in the figure, and “XXX YYYY,” “AAA BBBB” and “CCC DDDD” are authors of ten lines, one line and two lines, respectively. A straightforward way of measuring these developers’ contributions is to use the above lines of code on which they play the authors. Indeed, some previous work^13,14,15 adopted such a measure and conducted various studies.

Fig. 2.

An example of a line author investigation (the authors’ names are masked by dummy ones, and the file paths are partially omitted).

However, the above measurement has a lack of consideration in regard to the history of code changes. To explain the missing point, let us consider a simple example shown in Fig. 3. This example represents the updates of a source file, where there are five versions of the file and the updates are made by three developers d₁, d₂ and d₃; their update events are given in Table 1. In the figure, the hatched parts signify the lines whose final author is d₂, i.e., d₂ is the developer who last touched these lines. At the current version (version 5), d₂ becomes the final author of all lines of this source file. Although three developers d₁, d₂ and d₃ have contributed to the development and maintenance of this source file, this file looks a product made by only d₂ if we measure the developer’s contribution by counting the lines of touched code at the current version. In other words, the contributions by d₁ and d₃ are not taken into account in the measurement.

Fig. 3.

An example of source file updates (the parts finally made by d₂ are hatched).

Hence, we propose to use the “cumulative” lines of code changed by each developer in the contribution measurement. There are two key reasons why we focus on the cumulative changed lines rather than the current lines. One reason is that we consider a deletion of source code to be a contribution as well. Even if a developer just deleted some source code from a source file at a commit, it is one of essential modifications to build the current version of the source file. Thus, we should not avoid considering a code deletion in order to evaluate a developer’s contribution. Another reason is that there can be two different developers before and after a code modification. Suppose developer d_i modified a source code which was originally written by different developer d_j. Although the modification is a d_i’s contribution, it may be based on the original code and that is a d_j’s contribution. Thus, it would be reasonable to take into account not only d_i but also d_j for evaluating their contributions.

We formally define the developer’s contribution.

To see the difference of the contribution measurements between the proposed metric and the previous one—the cumulative lines of touched code through all versions vs. the lines of touched code at the current version—, Table 2 presents both of them in the above example shown in Fig. 3 and Table 1.

While d₃ made code deletions when the update to version 4, his/her contribution is not expressed by the previous metric and it continues to be zero for all versions. On the other hand, the proposed metric counts the contribution (see Table 2). Since the proposed metric cumulatively counts the lines of touched code, the difference between two metric values gets larger as more upgrades are made. At the current version (version 5), although the previous metric shows that d₂ is the only developer who contributed to the file, the proposed metric evaluates that the file has been developed and maintained by mainly d₁ and d₂, and a contribution by d₃ is also not null. This example shows an importance of our viewpoint and our motivation in this paper.

2.2. Contribution Entropy

We have just defined a developer’s contribution to a source file as the cumulative number of source code lines which the developer has added to or deleted from the file. When n (⩾ 2) developers have contributed to a source file, there are a lot of variations in the balance of contributions among n developers. For example, one of the developers may have dominantly contributed to the source file and the contributions by the remaining n − 1 developers may be really small. For another example, all of n developers may have approximately equally contributed to the source file. The difference of these two example cases may cause a remarkable difference in the quality of source files. Now we propose to evaluate the balance of developers’ contributions by introducing a metric using the notion of entropy.

The notion of entropy is well-known as a measure of information randomness in the information theory²⁰. When there are n possible events and the i-th event occurs with probability p_i (for i = 1, …, n), the entropy H is computed by

In this equation, log₂(1/p_i) represents the information content which can be intuitively interpreted as a degree of “surprisal” in regard to the occurrence of the i-th event. Thus, an occurrence of an event having a lower probability produces a larger surprisal. The entropy (H) can be regarded as the expected value (mean) of the surprisal. If the occurrences of possible events are biased to a certain event (the probabilities are highly imbalanced), we can expect that the biased event would occur. On the other hand, if the occurrences of possible events are equally expected for the all ones (the probabilities are perfectly balanced), we cannot predict which event would occur and the expected value (mean) of the surprisal is large. To illustrate the differences of entropies, we take four examples (i)–(iv) shown in Fig. 4, where there are three possible events^¶.

Fig. 4.

Examples of balanced and imbalanced distributions of p₁, p₂ and p₃.

For each example of (i)–(iv), the entropy (H) is computed as follows.

1. (i)

   H=3×(13log23)=log23≃1.58.

2. (ii)

   H=0.5×log210.5+0.4×log210.4+0.1×log210.1≃1.36.

3. (iii)

   H=0.98×log210.98+0.01×log210.01+0.01×log210.01≃0.16.

4. (iv)

   H=1×log211=0.

The entropy gets higher as the probabilities are more balanced. That is to say, we can use the entropy as a measure of balance among things. The examples (i) and (iv) are the maximum case and the minimum case, respectively.

Next, we apply the notion of entropy to a measurement of developers’ contribution balance. By regarding the rate of a developer’s contribution to all developers’ contributions as the probability in the entropy equation, we can evaluate a balance of contributions among developers. We will call it “contribution entropy” and formally define as follows.

This metric focuses on the source lines which have been changed by each developer, and quantifies the balance of these changed lines among developers as an “entropy.” When f has been developed and maintained by a single developer, it means n = 1 so we have H(f) = 0. When f has been developed and maintained by two or more developers, H(f) varies from 0 to 1 in accordance with the balance of p_i(f) among n developers. If all p_i(f)’s are equal, i.e., all developers’ contributions to f are truly equal, we get H(f) = 1. As the contributions get more imbalanced among developers, H(f) becomes lower. That is to say, as the contribution structure of f gets closer to a certain developer’s monopoly, H(f) becomes closer to 0.

As a previous study, Taylor et al.¹³ proposed the “author entropy.” They focused on “who is the author of each source line” in a source file, and the author entropy evaluates the balance of source lines among authors (developers). While the author entropy is an attractive measure of developers’ contributions to a source file, there is a lack of consideration in regard to the code change history—it focuses only on the current version of the source file—as we already explained with using examples shown in Fig. 3 and Tables 1, 2.

In the example case shown in Table 1, we have c(f, d₁) = 140, c(f, d₂) = 126 and c(f, d₃) = 30 just after version 5. From Eqs.(2) and (3), we get

and then

While all lines of f just after version 5 look to be ones made by only d₂ (see Fig. 3), d₁ has more contributions than d₂ if we focus on not only the current state but also the change history of f; if we focused only on the current state (version 5) of f, the contribution is considered to be a monopoly of d₂ and H(f) becomes zero.

2.3. Research Questions

In this section, we have introduced metrics for measuring the developer’s contribution and for evaluating the balance of contributions among developers. By using these metrics, we can quantitatively observe a structure of contributions and cooperations among developers in the development and maintenance of a source file. Here, we have the following simple question in regard to the relationship between the observed structure of developers’ contributions and the source file quality: “what type of contribution structure is the riskiest in terms of the code quality?” If we can see that a certain type of structure— characterized by a certain value range of contribution entropy—is risky, we can properly alert risky source files, which need more careful code reviews, during the maintenance activities, and it would be a help for an efficient and successful quality management.

Since the notion of code quality covers a broad range of topics, we will focus on the fault-proneness of source files in analogy with a lot of previous MSR studies. That is to say, a risky source file means a fault-prone source file in this paper.

To make the aim of our study in this paper clear, we set up our research questions (RQs) as follows. For the sake of convenience, we will call a source file which has been developed and maintained by two or more developers as a “multi-developer file,” and a source file which has been done by a single developer as a “single-developer file,” respectively. Similarly, we call the former-styled maintenance as the “multi-developer maintenance,” and the latter one as the “single-developer maintenance,” respectively.

• RQ1: Is there a difference in the fault-proneness between multi-developer files and single-developer files?

• RQ2: When a source file is a multi-developer file, can the balance of the developers’ contributions be useful for predicting the fault-prone source files?

We explain the reasons why we set the above questions.

RQ1 is a fundamental question if the difference of developers’ contributions to source files is related to the fault-proneness. Needless to say, a source file can be classified into two categories: 1) multi-developer files, and 2) single-developer files. An involvement of more developers to the maintenance of a source file might be better to enhance the code quality because the code would be reviewed by more people. However, there might also be an opposite effect: such an involvement might cause unnecessary confusion in the coding. Shortly, RQ1 is our simple question: “which is better” in terms of the fault-proneness, multi-developer maintenance or single-developer maintenance?

Next, RQ2 is a further question on the relationship of developers’ contributions with the fault-proneness of source files. As mentioned above, there would be various structures for cooperation in the maintenance of multi-developer files, i.e., there are variations in the balance of developers’ contributions. The aim of RQ2 is to clarify if the balance of developers’ contributions can be a useful metric for predicting fault-prone source files.

3.1. Procedure

We conducted our data collection and analysis in the following procedure.

1. 1)

   Make a copy of the repository from the GitHub by using git clone command.^||

2. 2)

   Get the development history of each source file from the repository.

   For each source file included in the current version, we investigate its development history (change logs) by using git log command. In the investigation, we trace the renaming operations to the file as well.

3. 3)

   Extract data items required for our analysis.

   By analyzing the commit logs that we got at step 2), we extract the following items for each commit of each source file: (a) commit hash, (b) whether it is aimed at fixing a fault or not, (c) author’s name, (d) author’s e-mail address, and (e) changed source lines.

   The item (b) is decided if the corresponding commit message contains a fault-fix-related keyword or not²².

4. 4)

   Organize the data collected at step 3).

   Since there are source files which were renamed through commits, we assign unique file IDs to source files in order to identify them.

   There may be an author who uses two or more different names or e-mail addresses on the repository. We integrated the duplicated author data by the following set of rules—it is a simpler version of the algorithm proposed by Bird et al.²³: (1) if two authors have different addresses but the same name, then we regard them as the same author; (2) if two authors have the same address but different names, then we regard them as the same author.

5. 5)

   Analyze the collected data on RQs.

   • For RQ1:

     1. (i)

        For each source file f, decide whether f is a single-developer file or a multi-developer file at the end of the observation period (see Fig. 5), and check if a fault fix is occurred at f during the examination period or not. Here, the observation period signifies the days in which we check the development and maintenance of the source file, and the examination period corresponds to the days in which we decide the file is faulty or not. Since a new source file may be immature and require some fixes, we empirically set the observation period as one year from the day that f was newly created, and the examination period as three months from the end of the observation period, respectively.

     2. (ii)

        Randomly select the same numbers of source files from each project, and build out dataset for the analysis. As shown in Table 3, the numbers of source files vary widely among the investigated OSS projects. If we use all of them in our analysis, the results would be highly influenced by the large-scaled projects’ trends. To mitigate such a bias, we decided to randomly select the same number of source files from each of the projects. In consideration of the minimum-sized project (Butter Knife), we select 100 source files from each project. Since there are 10 projects, our dataset consists of 1000 source files.

     3. (iii)

        Classify the set of source files into two subsets: the set of single-developer files and that of multi-developer ones. Then, compare their rates of faulty source files (the fault rates: FRs). A higher FR value means a higher fault-proneness of source files in the corresponding subset.

   • For RQ2:

     1. (i)

        For each source file f included in our dataset (built in the step RQ1(ii)), compute its contribution entropy, H(f), at the end of the observation period (see Fig. 5). Then, we check if a fault fix is occurred at f during the examination period or not.

     2. (ii)

        Classify the set of source files into five subsets according to their contribution entropy: C0, C1, C2, C3 and C4. C0 is the set of source files such that H(f) = 0, i.e., the set of single-developer files. The remaining source files are categorized into C1 – C4 corresponding to equally divided ranges of H(f): C1, C2, C3 and C4 correspond to 0 < H(f) ⩽ 1/4, 1/4 < H(f) ⩽ 2/4, 2/4 < H(f) ⩽ 3/4 and 3/4 < H(f) ⩽ 1, respectively (Table 4).

        After that, compare FRs among C0 – C4.

Fig. 5.

Observation period and examination period.

3.2. Results for RQ1

On RQ1, we compared the set of single-developer source files and that of multi-developer ones in terms of the fault-proneness. Table 5 and Fig. 6 show the results.

Fig. 6.

Comparison of the fault-proneness: multi-developer source files vs. single-developer ones.

As a result, the multi-developer files are about two times more likely to be faulty than the single-developer one: 16.2% vs. 7.1%, and their difference was statistically significant at a 1% significance level.^**

3.3. Discussions for RQ1

In order to examine a difference in the fault-proneness between multi-developer files and single-developer files, we compared the fault-proneness (fault rate: FR) of the multi-developer source files with that of the single-developer ones. Then, we got the result showing that a multi-developer file tends to be more fault-prone than a single-developer one, and the FR of multi-developer files is about two times higher than that of single-developer ones—16.2% vs. 7.1%. In general, it is not easy to properly update a program written by other people. Hence, such a difficulty may cause the above striking difference between the two types of source file.

Here, we have a concern that the program size might be a confounding factor in our results. Since a larger program is more likely to be faulty in many cases, we cannot omit the impact of the program size. Thus, we perform the above FR comparison for the dataset stratified by the program size as well: we categorize source files by their program size, and compare the FRs between multi-developer files and single-developer ones within each category; We use the lines of code (LOC) as our metric of program size since it is well-known metric which is highly related to the fault-proneness of programs²⁴. According to the distribution of LOC values in our dataset, we decided to categorize the source files into the following four categories:

1. (1)

   “small” whose LOC ⩽ 17;

2. (2)

   “relatively small” whose LOC > 17 and ⩽ 48;

3. (3)

   “relatively large” whose LOC > 48 and ⩽ 119;

4. (4)

   “large” whose LOC > 119,

where values 17, 48 and 119 are the 25 percentile, the median and the 75 percentile of the LOC distribution, respectively. Table 6 and Fig. 7 show the FR comparison results.

Fig. 7.

Comparisons of the fault-proneness within LOC categories.

As Fig. 7 shows, the magnitude relationship of FR between multi-developer files and single-developer files is the same as the total result shown in Fig. 6, except for LOC category (1)—small-sized source files. Moreover, the gap of FR between file types gets larger when the source file becomes larger which are in LOC categories (3) or (4). The differences of FRs are statistically significant at 1% significance level in LOC categories (3) and (4), but not significant in categories (1) and (2)^††.

As mentioned above, a successful update of other programmer’s code is not easy task. The difficulty would increase as the program gets larger since the programmer has to properly comprehend more source lines and moreover the program structure would be more complex. Therefore, it seems to be natural that the gap of FR becomes larger in the LOC categories (3) and (4)—relatively-large or large source files. Similarly, as the program gets smaller, a proper comprehension and update of code would become easier even if the code was written by other programmers. Furthermore, in general, a smaller program originally tends to be less fault-prone. Hence, it would be a natural result that there is no significant difference of the FR values between the small-sized multi-developer files and the small-sized single-developer ones.

Notice that the above results are derived from the data of “fault fix” but not of “fault creation.” There may be a case such that: a fault was introduced into a source file when the file was a single-developer one, and then the fault was detected and fixed after the file became a multi-developer one by a participation of another developer. One of key reasons why more faults were detected and fixed in multi-developer files might be that more developers reviewed these files. In other words, a single-developer file might have latent faults which have not been detected yet because the file has been maintained by a certain developer only. Although our focus on whether a source file is a single-developer file or not seems to be noteworthy through this data analysis, we have to do a more detailed analysis in the future: a further analysis on when the fixed faults were introduced.

3.4. Results for RQ2

On RQ2, we classified the source files into five categories C0, C1,…, C4 by their contribution entropy, and compared these categories in terms of the fault-proneness. Table 7 and Fig. 8 show the results.

Fig. 8.

Comparison of source files categorized by their contribution entropy.

As a result, the least fault-prone category is C0 whose contribution entropy is zero, i.e., the category of the single-developer files. In contrast, the most fault-prone category is C1 which is right next to C0. Category C1 corresponds to source files in which the developers’ contributions are highly imbalanced. The fault-proneness (the FR value) decreases as the contribution entropy gets higher, i.e., as the structure of contributions gets away from the most imbalanced state.

3.5. Discussions for RQ2

In order to investigate the impact of contribution balance among developers on the fault-proneness, we categorized source files by their contribution entropy into five categories C0 – C4, and compared their FR values. As a result, multi-developer source files in category C1 are the most fault-prone. In such a source file, there are two or more developers who have had contributed to the file, but their contributions are highly-imbalanced and one of the corresponding developers is dominant. That state of source file would be made when the source file was just changed from the single-developer maintenance to the multi-developer maintenance. By a participation of another developer in the maintenance of a source file, it seems to become unstable in terms of the code quality and the most fault-prone.

Since the trend of FR values shows a monotonically decreasing from C1 to C4, we can predict that a multi-developer source file is likely to be more fault-prone as its contribution entropy gets smaller.

We can consider the following two potential reasons why C1 is a fault-prone category of source file: (1) a fault was created by a new developer through a code modification, or (2) a fault was detected by a new developer. In general, a code modification has a risk of creating a fault^25,26. Since it is not easy task to accurately comprehend and properly change a source code written by other programmers, the fault-creation risk would get higher when a code change is made by a new developer. In another perspective, a participation of a new developer includes another code review. Then, a latent fault might be detected as a result of code review by a new developer. In both cases, more faults may be detected from source files of category C1. On the other hand, source files in categories C2, C3 or C4 tend to be more changed and reviewed more times by two or more developers. Through those code changes and reviews, the source files may be polished and become less fault-prone. Because our data is based on fault-fix events in the repository, we cannot clearly analyze the above potential reasons in this study. In order to examine the reason why C1 is so fault-prone and the fault-proneness gets lower as the contribution entropy becomes higher, we need to perform a further analysis using data of “fault creation” in the future.

As with the above discussion on RQ1 (Section 3.3), it would be better to analyze the impact of the program size (LOC) on the results as well. Although we originally considered dividing each of C0 – C4 into four subcategories in accordance with the above four LOC categories, the devision made small-sized categories—e.g., the subcategory of C1 corresponding to the small-sized programs (LOC category (1)) has only 9 source files. Since there were such small-sized categories, we considered it is hard to properly compare the subcategories by using their FR values. That is to say, it is difficult to discuss the change of fault-proneness over the contribution entropy while stratifying source files by their LOC. Thus, as a yet another way of analysis, we build a fault-prone prediction model whose explanatory variables are LOC and the proposed entropy metric. If the impact of LOC is dominant in the fault-prone prediction, the proposed entropy metric cannot contribute to the prediction when it is used together with LOC.

While there are many types of prediction models, we build a model using the random forest method which is a promising way of predicting fault-prone programs²⁷. The random forest-based prediction model use two metrics as its explanatory variables— the LOC of a source file and the contribution entropy of the file—, and predicts whether the source file is faulty or not. The random forest can present the importance of each explanatory variable according to the Breiman’s method²⁸ in a quantitative form. Table 8 shows the importance of two metrics in the random forest model, where a larger value means a higher importance. In the table, the dataset “all” signifies the all source files included in our dataset, i.e., the importance values on the row are computed from the random forest model which was built by using not only the multi-developer files but also the single-developer ones; the dataset “multi-developer file only” means that the corresponding random forest model was built by using only the multi-developer files.

As the results, the importance values showed that the proposed metric—the contribution entropy— plays an important role in the fault-prone prediction even if it is used together with the LOC. Especially, when we focus only on the multi-developer source files, the proposed metric outperforms the LOC. Since all single-developer files’ entropies are zero, such a data separation would enhance the performance of the entropy metric. While the LOC metric is one of promising metrics to predict fault-prone programs, the proposed entropy metric can contribute to the prediction as well. As we saw in Fig. 7, the contribution entropy can be useful in detecting fault-prone programs when they are large-sized ones. Moreover, category C1 would be a hot spot of fault-prone programs as shown in Fig. 8. These characteristics of the proposed contribution entropy metric would help a more precise prediction of fault-prone programs.

3.6. Threats to Validity

We analyzed popular OSS projects whose main-development language is Java. Since our data collection method does not depend on any Java-specific feature, we can perform a similar analysis to other projects developed in other languages without any changes of our method if their code is maintained with Git. Since OSS projects written in other languages may show different trends, it is a threat to the validity regrading the generality of our findings, and our significant future work.

In order to ensure quality of code, some OSS projects have strict checking (reviewing) systems for a code change proposed by a developer. Such a checking system may have a large impact on our results of data analysis. To mitigate such a threat to validity, we collected data from various OSS projects in different domains. Moreover, to avoid any project-specific bias, we randomly selected the same number of source files from each of the projects. We plan to perform a further analysis focusing on project-specific features in the future.

To examine if the proposed metric can contribute to a fault-prone program prediction, we leveraged the random forest model. It is a popular and widely-used model in the fault-prone prediction studies. However, its computation (model construction) requires many parameters and these parameters can affect the computational results. That is to say, if we use a different set of parameters in the model construction, a different result might be obtained in our analysis. Since the decision of their parameters can be a yet another threat to validity, we used default parameters (settings) of randomForest package of R in our computation to avoid any bias regarding the parameter decision. A further study on a better parameter turning is our future work.