Associate Editor: Sarah Nadi, University of Alberta, Canada (@sarahnadi)
Continuous integration (CI) systems automate the compilation, building, and testing of software. A recent large survey of software development professionals, found that 50% of respondents use CI [1]. Despite the widespread usage of CI, it has received almost no attention from the research community. This gap has left a lot of unanswered questions around CI. For example, how widely is CI used in practice, and what are some costs and benefits associated with CI? To answer these questions, we studied the usage, costs, and benefits of CI in open-source projects. We examined the 34,000 most popular open-source GitHub projects to evaluate which CI service(s) they use. From that list, we collected over 1.5 Million builds from over 600 of the most popular projects and then surveyed 442 of the developers of these projects. In the following, we answer some of your burning questions about CI.
Usage of CI
Is CI actually used? We found that over 40% of all open-source projects that we examined use CI. Note that since we cannot identify private CI servers that may be used by some projects, 40% should be considered as the floor of CI usage.
What is the most popular CI for open source? The most commonly used CI service in our data was Travis CI. Over 90% of all open-source projects using CI use Travis CI.
Are the ``big names’’ in open source using CI? We sorted the projects by popularity and found that for the 500 most popular projects, 70% use CI. As the projects become less popular, the percentage that use CI goes down.
Is CI a passing fad? We asked developers in our survey if they plan on using CI for their next project. The top two options, 'Definitely' and 'Most Likely' account for 94% of all our survey respondents. Even among respondents who are not currently using CI, over 50% said they would use CI for their next project.
Costs of CI
Why then do some projects not use CI? The most common reason reported in our survey is that other developers in a project are not familiar with CI. The second most popular reason was that the project does not have automated tests.
Will I have to be continuously fixing the configuration files? We found that the median project changes their CI configuration files 12 times over the lifetime of the project, so there is not a lot of churn in CI configurations.
Benefits of CI
So what exactly do developers like about CI? When we asked developers why they use CI, the most common answer was “CI helps us worry less about breaking our builds”. This was reported by 87% of our respondents. The second most common answer was that “CI helps catch bugs earlier”.
Can CI help me release more often? We found that projects with CI do in fact release faster then projects that do not use CI (.54 releases per month with CI versus .24 releases per month without CI). To control for project type, we also looked at only projects that use CI and compared how often they released before and after introducing CI to the project. Before CI, they released at a rate of .34 releases per month, but after introducing CI, the release rate rose to .54 releases per month.
Can CI help me spot problems before they happen? Projects that use CI accepted less pull requests then projects that do not use CI. This could be because CI helps identify subtle issues that a quick human review would not catch.
Will CI help me save time? We found that projects with CI accept pull requests (PRs) on average 1.6 hours faster than projects that do not use CI. This could be because of the time saved by letting the CI review the PR, as opposed to manually reviewing it.
Conclusion
CI can save you time, help you release more often, and help you worry less about breaking your builds. We hope these results motivate developers to continue to adopt CI, but also to provide a call to action for the research community to continue investigating this important aspect of the development process.
If you are interested in more details about our results, please read our full paper [2].
[1] Version One. 10th annual state of Agile development survey. https://versionone.com/pdf/VersionOne-10th-Annual- State-of-Agile-Report.pdf, 2016.
[2] Usage, Costs, and Benefits of Continuous Integration in Open-Source Projects. Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (ASE 2016). To appear.
No comments:
Post a Comment