Participation of Women in Free/Libre/Open Source Software. What is the current status and how much has changed in the last 10 years?

Participation of Women in Free/Libre/Open Source Software. What is the current status and how much has changed in the last 10 years?

by Gregorio Robles, Laura Arjona Reina, Jesús M. González-Barahona and Santiago Dueñas Domı́nguez.
Associate editor: Stefano Zacchiroli (@zacchiro)

It is well known that women are generally underrepresented in the IT sector. However, in FLOSS (free, libre, and open source) projects the number of women reported is even lower (from 2% to 5% according to several surveys, such as FLOSS 2002 [1] or FLOSSPols [2]). The FLOSS community is aware of this situation, and some projects, such as GNOME and Debian, have promoted the attraction of female participants.

As the previous surveys date back from the early 2000s, we designed a new web survey in 2013 which was answered by more than 2,000 FLOSS contributors, of which more than 200 were women. From the analysis of the answers provided by men and women, mainly those ones that are on their involvement with FLOSS, their educational level and background and personal status, we wanted to shed some light on how the status of female participation in FLOSS is. This blog post shows a glimpse of this. The survey responses are publicly available and documented for further analysis by third parties [3].

We have found that women begin to collaborate at a later age than men. Interestingly enough, as can be seen from Figure 1, while both peak at the age of 21, the tail for women is not that abrupt than the one for men. So, women that start in their thirties are 70% of the ones in the 20s, while for men this number decreases to 30%.

Figure 1 - Age of first contribution to FLOSS. Source [4].

Women perform more diverse tasks than men. While the latter are mostly concentrated on coding tasks (only slightly above 20% of men perform other tasks), with almost 45% "other type of contributions" is the main task chosen by women. The percentage of women who mainly code is 31%, as can be seen from Figure 2.

Figure 2 - Type of contributions to FLOSS projects. Source [4]

Figure 3 shows that while a majority of FLOSS participants does not have children, the number of those who have varies largely depending on their gender. So, the number of women with children (19%) is almost half the number of men with children (34%) as shown in Figure 3.

Figure 3 - Answers to the question Do you have children?.

Graphs have different scales. Source [4]

If we have a look at how much time FLOSS contributors devote to FLOSS, we obtain Figure 4. At first sight the distribution might seem very similar, but a closer look shows that the share of men devoting less than 5 hours/week (50%) is lower than for women (54%), as is the the amount of men working 40 or more hours per week (12% for men and 15% for women). So, contributions to FLOSS projects by women are over-represented among the less active and the full-time professional contributors, in the latter case being probably hired by an industrial software company.

Figure 4 - Number of hours per week devoted to contributing to FLOSS projects. 

Graphs have different scales. Source [4]

All in all, our study confirms (and extends) the results from FLOSS [1] and FLOSSPols [2], even if almost 10 years have passed between both. Even if it is possible that the amount of women is now slightly higher, many contextual patterns have remained the same. So, a study of GitHub developers from 2015 found that only around 6% were women [5], but this increase could be due to the major involvement of the software industry in FLOSS. The current situation is far from what ten years ago was set as a goal, so we may speak of a "lost decade" in the inclusion of women in FLOSS.

A pre-print copy of the full paper "Women in Free/Libre/Open Source Software: The situation in the 2010s" [4] can be found at https://flosshub.org/sites/flosshub.org/files/paper-pre.pdf.

References

[1] Rishab A Ghosh, Ruediger Glott, Bernhard Krieger, and Gregorio Robles. Free/libre and open source software: Survey and study, 2002.
[2] Dawn Nafus, James Leach, and Bernhard Krieger. Gender: Integrated report of findings. FLOSSPOLS, Deliverable D, 16, 2006.
[3] Gregorio Robles, Laura Arjona Reina, Alexander Serebrenik, Bogdan Vasilescu, and Jesús M González-Barahona. FLOSS 2013: A survey dataset about free software contributors: Challenges for curating, sharing, and combining. In MSR, pages 396-399, 2014. DOI: https://doi.org/10.1145/2597073.2597129
[4] Gregorio Robles, Laura Arjona Reina, Jesús M. González-Barahona, and Santiago Dueñas Domı́nguez. Women in Free/Libre/Open Source Software: The situation in the 2010s. In OSS 2016, IFIP AICT 472, pp. 163–173, 2016. DOI: https://doi.org/10.1007/978-3-319-39225-7_13
[5] https://www.toptal.com/open-source/is-open-source-open-to-women

The shape of a cryptographic keyring: How is the Debian project social structure like?

By: Gunnar Wolf 
Associate editor: Stefano Zacchiroli (@zacchiro)

Quite often, we can be quite familiar with a set of data; But looking at it with a different viewpoint reveals a completely unexpected reality.

I am part of the keyring-maint group in Debian: The developers that manages the cryptographic keyring through which developers identify themselves for most of our actions — Most important, voting in project decisions and uploading software.

There are two main models for establishing trust via cryptographic signatures: The centralized, hierarchical model based on Certification Authorities (CAs) from where all trust and authority stems and flows only downwards, and a decentralized one, the Web of Trust where each participant in the network can sign other participants' public keys; the first one is most often used in HTTPS (encrypted Web), and our project uses a specific mode of the second one, which we have termed the Curated Web of Trust [1].

Cryptographic signatures are way more secure as identifications to the omnipresent but very weak username/password scheme. However, technological advances must be factored in. Being already 24 years old, Debian is a very long-lived free software project, and it sports a great retention: Many of its developers have been active for over ten years. In the late 1990s, the recommended key size was 1024 bits — Public key cryptography is very expensive computationally, and said key size was perceived secure enough for the foreseeable future. However, according to a study on algorithms and keysizes in 2012, [2] this key size is today good only for Short-term protection against medium organizations, medium-term protection against small organizations — Clearly not below our required standards.

Our group had started pushing for migration to stronger keys back in 2009. By 2014, as the Figure 1 shows, we had achieved the migration of half of the developers to stronger keys; But the pace of migration was really insufficient. At the Debian Developers' Conference that year (DebConf14), we announced that by January 1st, 2015, we would remove all keys shorter than 2048 bits.

Figure 1: Number of keys by length in the Debian Developers keyring, between mid 2008 and late 2015

The migration process was hard and intensive. Given the curated Web of Trust model followed, our policy is that, for a key replacement, a new key must be signed by two currently trusted keys in our keyrings; being Debian a globally distributed project, many people are simply unable to meet other developers. This migration process resulted in us losing close to a fourth of all keys (that is, a fourth of all Debian Developers could no longer perform their work without asking somebody to "sponsor" their actions), we felt it to be quite successful. This migration prompted a deeper analysis into what the keyrings were able to tell about the developers themselves.

Trying to find any emergent properties, we graphed the signatures in the keyring at different points in time. Although most times we got just a useless huge blob, we observed a very odd division starting around 2011; Figure 2 shows the graph close to the point where this split was maximum: January 2012.

Figure 2: Trust relationships in the Debian keyring near the maximum /split/, January 2012

Then, trying to find the meaning of this split, we colored the edges according to their relative age — How long before each of the snapshots was each of the signatures made. This is shown for the above presented keyring in Figure 3.

Figure 3: Trust relationships in the Debian keyring near the maximum /split/, January 2012, graphed by signature age: (Blue: less than one year old; green: one to two years old; yellow: two to three years old; orange: three to four years old; red: over four years old)

Our hypothesis was that the red blob mostly represents an earlier generation of Debian Developers, who have mostly faded from project activity. We presented our findings last May at the 13th International Conference on Open Source Systems [3], together with a first foray into a statistical analysis on key aging and survival.

This rich data set can still yield much more information on the Debian project's social interactions. It's basically just a matter of finding other angles from which to read it!

[1] Wolf, G. E., & Gallegos-García, G. (2016). Strengthening a curated web of trust in a geographically distributed project. Cryptologia, 1-16.
[2] ECRYPT, I. (2012). Yearly Report on Algorithms and Keysizes (2011-2012), Rev. 1.0. ECRYPT II Network of Excellence (NoE), funded within the Information Societies Technology (IST) Programme of the European Commission’s Seventh Framework Programme (FP7).
[3] Wolf, G., & Quiroga, V. G. (2017, May). Progression and Forecast of a Curated Web-of-Trust: A Study on the Debian Project’s Cryptographic Keyring. In IFIP International Conference on Open Source Systems (pp. 117-127). Springer, Cham.

Performance testing in Java-based open source projects

by Cor-Paul Bezemer (@corpaul), Queen's University, Canada
Associate Editor: Zhen Ming (Jack) Jiang, York University, Canada

From a functional perspective, the quality of open source software (OSS) is on par with comparable closed-source software [1]. However, in terms of nonfunctional attributes, such as reliability, scalability, or performance, the quality is less well-understood. For example, Heger et al. [2] stated that performance bugs in OSS go undiscovered for a longer time than functional bugs, and fixing them takes longer.

As many OSS libraries (such as apache/log4j) are used almost ubiquitously across a large span of other OSS or industrial applications, a performance bug in such a library can lead to widespread slowdowns. Hence, it is of utmost importance that the performance of OSS is well-tested.

We studied 111 Java-based open source projects from GitHub to explore to what extent and how OSS developers conduct performance tests. First, we searched for projects that included at least one of the keywords 'bench' or 'perf' in the 'src/test' directory. Second, we manually identified the performance and functional tests inside that project. Third, we identified performance-sensitive projects, which mentioned in the description of their GitHub repository that they are the 'fastest', 'most efficient', etc. For a more thorough description of our data collection process, see our ICPE 2017 paper [3]. In the remainder of this blog post, the most significant findings of our study are highlighted.

Finding # 1 - Performance tests are maintained by a single developer or a small group of developers. 
In 50% of the projects, all performance test developers are one or two core developers of the project. In addition, only 44% of the test developers worked on the performance tests as well.

Finding # 2 - Compared to the functional tests, performance tests are small in most projects. 
The median SLOC (source lines of code) in performance tests in the studied projects was 246, while the median SLOC of functional tests was 3980. Interestingly, performance-sensitive projects do not seem to have more or larger performance tests than non-performance-sensitive projects.

Finding # 3 - There is no standard for the organization of performance tests. 
In 52% of the projects, the performance tests are scattered throughout the functional test suite. In 9% of the projects, code comments are used to communicate how a performance test should be executed. For example, the RangeCheckMicroBenchmark.java file from the nbronson/snaptree project contains the following comment:

/*

* This is not a regression test, but a micro-benchmark.

*

* I have run this as follows:

*

* repeat 5 for f in -client -server;

* do mergeBench dolphin . jr -dsa\

*       -da f RangeCheckMicroBenchmark.java;

* done

*/

public class RangeCheckMicroBenchmark {

...

}

In four projects, we even observed that code comments were used to communicate the results of a previous performance test run.

Finding # 4 - Most projects have performance smoke tests. 

We identified the following five types of performance tests in the studied projects:

1. Performance smoke tests: These tests (50% of the projects) typically measure the end-to-end execution time of important functionality of the project.
2. Microbenchmarks: 32% of the projects use microbenchmarks, which can be considered performance unit tests. Stefan et al. [4] studied microbenchmarks in depth in their ICPE 2017 paper.
3. One-shot performance tests: These tests (15% of the projects) were meant to be executed once, e.g., to test the fix for a performance bug.
4. Performance assertions: 5% of the projects try to integrate performance tests in the unit-testing framework, which results in performance assertions. For example, the TransformerTest.java file from the anthonyu/Kept-Collections project asserts that one bytecode serialization method is at least four times as fast as the alternative.
5. Implicit performance tests: 5% of the projects do not have performance tests, but simply yield a performance metric (e.g., the execution time of the unit test suite). 

The different types of tests show that there is a need for performance tests at different levels, ranging from low-level microbenchmarks to higher-level smoke tests.

Finding # 5 - Dedicated performance test frameworks are rarely used. 

Only 16% of the studied projects used a dedicated performance test framework, such as JMH or Google Caliper. Most projects use a unit test framework to conduct their performance tests. One possible explanation is that developers are trying hard to integrate their performance tests into the continuous integration processes. 

The main takeaway of our study

Our observations imply that developers are currently missing a “killer app” for performance testing, which would likely standardize how performance tests are conducted, in the same way as JUnit standardized unit testing for Java. An ubiquitous performance testing tool will need to support performance tests on different levels of abstraction (smoke tests versus detailed microbenchmarking), provide strong integration into existing build and CI tools, and support both, extensive testing with rigorous methods as well as quick-and-dirty tests that pair reasonable expressiveness with being fast to write and maintain even by developers who are not experts in software performance engineering.

References

[1] M. Aberdour. Achieving quality in open-source software. IEEE Software. 2007.

[2] C. Heger, J. Happe, and R. Farahbod. Automated Root Cause Isolation of Performance Regressions During Software Development. In Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering (ICPE). 2013.

[3] P. Leitner and C.-P. Bezemer. An exploratory study of the state of practice of performance testing in Java-based open source projects. In Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering (ICPE). 2017. 

[4] P. Stefan, V. Horky, L. Bulej, and P. Tuma. Unit testing performance in Java projects: Are we there yet? In Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering (ICPE). 2017.

If you like this article, you might also enjoy reading:

[1] Jose Manuel Redondo, Francisco Ortin. A Comprehensive Evaluation of Common Python Implementations. IEEE Software. 2015.

[2] Yepang Liu, Chang Xu, Shing-Chi Cheung. Diagnosing Energy Efficiency and Performance for Mobile Internetware Applications. IEEE Software. 2015.

[3] Francisco Ortin, Patricia Conde, Daniel Fernández Lanvin, Raúl Izquierdo. The Runtime Performance of invokedynamic: An Evaluation with a Java Library. IEEE Software. 2014.

On the rise of Casual Contributions in postmodern Software Development - How does it impact Software Quality

By: Naveen N Kulkarni
Associate Editor: Sridhar Chimalakonda (@ChimalakondaSri)

Postmodernism, a term used in variety of disciplines like arts, literature, fashion, music, movies and technology, is used to describe a tendency of philosophical movements and resulting shifts where there is a rejection of the predecessor. In case of software engineering too, the postmodernistic views have challenged the rigidity of traditional approaches; much like agile methods challenged the need for upfront requirement gathering, perpetual beta cycles where users are considered co-developers or heterogeneous application stack replaced with homogeneous stacks like Node.js and Vert.x. In case of collaboration in software development we see breaking the norm of controlled contributions to shared contributions to fragmented contributions combined with social underpinnings.

In context of Open Source Software (OSS), empirical studies have shown that developers aim to rapidly deliver ’credible promises’ to keep their projects viable and active [Sojer, Haefliger]. With social interactions gaining popularity on the Internet, casual contributions (popularly known as pull requests on platforms such as, GitHub and BitBucket) to OSS is increasing. Empirical studies suggest that such contributions are not trivial, but involves bug fixes, refactoring and new features. While an OSS can deliver its promise faster with such requests, it can can quickly overwhelm the developers evaluating them whether to accept or not in case there are many. In order to deliver faster, it is intuitive that developers choose project relevance and code quality as the primary factors to evaluate the pull requests (substantiated in a recent survey by Gousios et al.). To minimize the effort, many OSS projects use strict policies that include discussions, linting, clean merge, tests and code reviews; most of which are automated through continuous integration tools. As the casual contributions can be unplanned and opportunistic, attempts are made by the development 1 2 team to understand the context of contribution through prior discussions. For example, in case of Scala, pull requests are accepted only if it is discussed on ’scala-internals’ mailing list. Also, mandatory discussions with more than one developers is required depending on the complexity of the contribution. All these are undertaken as a part of the quality assessment of the casual contribution. However, with such extensive checks and large volumes, the pull requests can quickly get stale, making it even harder for developers and contributors to understand the impact of composing multiple parallel contribution.

Survey of work practices in managing and integrating pull request suggest that every contribution requires multiple code review sessions. This is substantiated by the fact that code reviews require ”code reading” to find defects, which is greatly dependent on individual expertise. Unfortunately, though they are perceived as a best practice, they are often localized and informal. Today’s code review session include interactive sessions, discussions and in-line comments. A study of modern code review process show that 35% of the review suggestions are discarded and 23% of the changes are applied after review (Beller M et al.). It is suggested that this process can be more people independent in postmodern software engineering. On the quality, there are very few qualitative evidences relating to code review process. Recent study by McIntosh et al. suggest that review coverage as an important factor to ensure quality. However, quality criteria are subjective during code reviews. Quality criteria is often restricted to statically analyzed source code using popular continuous integration tools (like Sonar or Squale), coding style and test cases.

Despite the challenges, we observe casual contributions are raising. For example, at the time of writing this post, the rails and redis projects have nearly 500 open pull requests. The bootstrap, npm, tensorflow and docker have nearly 100 open pull requests. More detailed analysis on the growth of pull requests can be found at http://ghtorrent.org/pullreq-perf/ and https://octoverse.github.com/. As we adopt different approaches to compose software with frequent fragmented contributions, we are faced with challenges, such as emergent behavior, constraint violations and conflicting assumptions. These challenges arise due to the parallel and isolated nature of the contributions. Though these challenges are not new, we believe the current practices on quality assessments are inadequate to address them. The techniques used today help developers quickly spot local mismatches but they are not effective for comprehending global mismatches that originate due to many complex relationship among the software elements. In our initial study, we found similar evidence when a opportunistically reused external module not only violated the existing constraints, but also added unanticipated behavior making the software restrictive.

Attempts to address the global mismatches using architectural reasoning and design decisions approaches have so far has met with limited success. Also, during code reviews are insufficient to highlight the ramifications of a contribution on the global constraints and mismatches. To overcome this issue, the merge process should extend beyond syntactic merge to include techniques for identifying constraint mismatches. We believe, by validating the constraint synthesized as predicates from the hot paths (critical execution flow) can optimally help validating the global mismatches. In past, predicate abstractions are effectively used for software verification (SLAM). On one hand, the predicate abstraction can facilitate theoretical guarantees, synthesizing the predicates can be overwhelming due to state explosion. To overcome this we can use descriptive statistical models (created from the features available in the source code) to choose a subset of predicates sufficient for the verification. Alternatively, mining of software repository techniques can play pivotal role in developing alternative approaches where they not only provide factual data, but help in decision process. Mining constraints as a set of sequential patterns (SPADE, a popular sequence mining algorithm) can be used where comparing sequence for dissimilarities between the source codes can suggest mismatches.

There is a need for lightweight processes to cope with the dramatic shifts in the postmodern software engineering. In case of casual contributions, defining code quality can be confounding due to their inherent isolated nature. So, code reviews are critical and there is a need for alternative approaches that account for the various postmodern shifts. Without this the software quality in postmodern software development will remain elusive.

References

• SojerM, Henkel J. Code reuse in open source software development: quantitative evidence, drivers, and impediments. Journal of the Association for Information Systems 2010; 11(12):868901.
•Stanciulescu S, Schulze S,Wasowski A. Forked and integrated variants in an open-source firmware project. Proceedings of the IEEE Int’l Conference on Software Maintenance and Evolution, ICSME 15, Bremen, Germany, 2015; 151160.
• Perry DE, Siy HP, Votta LG. Parallel changes in large-scale software development: an observational case study. ACM Transactions on Software Engineering and Methodology (TOSEM) 2001; 10(3):308337.
• Gousios G, Zaidman A, Storey MA, Deursen Av. Work Practices and Challenges in Pull-Based Development: The Integrators Perspective. Proceedings of the 37th International Conference on Software Engineering, 2015, vol. 1, pp. 358368.
• Scala Pull request Policy, http://docs.scala-lang.org/scala/pull-request-policy.html, accessed 15-Mar-2017.
• Pinto G, Steinmacher I, Gerosa MA. More common than you think: An in-depth study of casual contributors. Proceedings of the 23rd IEEE Intl Conference on Software Analysis, Evolution, and Reengineering, SANER 16, Suita, Osaka, Japan, 2016; pp. 112123. 4
• Beller M, Bacchelli A, Zaidman A, Juergens E. Modern Code Reviews in Open-Source Projects: Which Problems Do They Fix? In Proceedings of Mining Software Repository, MSR’14, Hyderabad, India, 2014.

When and Which Version to Adopt a Library: A Case Study on Apache Software Foundation Projects

By: Akinori Ihara, Daiki Fujibayashi, Hirohiko Suwa, Raula Gaikovina Kula, and Kenichi Matsumoto (Nara Institute of Science and Technology, Japan)
Associate editor: Stefano Zacchiroli (@zacchiro)

Are you currently using a third-party library in your system? How did you decide the version and when to adapt the library? Is it the latest version or the older (reliable) version that you adopted? Do you plan to update and would you trust the latest version? These are all tough questions with no easy answers.

A software library is a collection of reusable programs, used by both industrial and open software client projects to help achieve shorter development cycles and higher quality software [1]. Often enough, most active libraries release newer and improved versions of their libraries to fix bugs, keep up with the latest trends and showcase any new enhancements. Ideally, any client user of a library would adopt the latest version of that library immediately. Therefore, it is recommended that a client project should upgrade their outdated versions as soon a new release becomes available.

Developers do not always select to adopt the latest version over previous versions

As any practitioner is probably well-aware, adoption of the latest version is not as trivial as it sounds, and may require additional time and effort (i.e., adapting code to facilitate the new API and testing) to ensure successful integration into their existing client system. Developers of client projects are especially wary of library projects that follow a rapid-release style of development, since such library projects are known to delay bug fixes [2]. In a preliminary analysis, we identified two obstacles that potentially demotivate client users from updating:

1. Similar client users are shown not to adopt new version shortly after it is released and that
2. there is a delay between the library release and its adoption by similar clients.

These insights may indicate client users are likely to 'postpone' updating until a new release is deemed to become 'stable'. In this empirical study, we aim to investigate how libraries are selected in relation to their release cycles.

Design: We analyze when and which library versions are being adopted by client users. From 4,815 libraries, our study focuses on the 23 most frequent Apache Software Foundation (ASF) libraries used by 415 software client projects [3].

Figure 1: distribution of the periods between releases in each library

When to adapt a library?: We find that not all 23 libraries were yearly released (see Figure 1). Some library projects (e.g., jetty-server, jackson-mapper-asl, mockito-all) often release new versions within the year (defined as quick-release libraries), while others (e.g., commons-cli, servlet.api, commons-logging) take over a year to come out with a release (defined as late-release libraries). We found that these more traditional and well-established (i.e., older than 10 years) projects were the late-release libraries, while newer, beginner projects belonged to the quick-release libraries.

Figure 2: Percentage of client users to select the latest version (gray) and the previous version (black)

Which version to adopt?: Software projects do not always adopt new library versions in their projects (se Figure 2). Interestingly, we found that some client users of a late-release library would first select the latest version as soon as it was released, only to later on downgrade to a previous version (Figure 2: Red box and blue box shows the percentage of client users which performed downgrade after adapting the latest version or the previous version).

Lessons Learnt: From our study, we find that client users may postpone updates until a library is deemed to become stable and reliable. Although quality of most open source software would often improve by minor and micro release changes, the study finds that client projects may wait, especially in the case of a late-release library. Our study validates the notion that library updates is not trivial. We find that practitioners are indeed careful when it comes to adopting the latest version, as they may include dependency problems and potentially untested bugs.

We presented this study in International Conference on Open Source Systems (OSS'17). For more details, please see the preprint and the presentation from our website: http://akinori-ihara.jpn.org/oss2017/

[1] Frank McCarey, Mel Ó Cinnéide, and Nicholas Kushmerick, "Knowledge reuse for software reuse," Journal of Web Intelligence and Agent Systems, pp.59-81, Vol.6, Issue.1, 2008.
[2] Daniel Alencar da Costa, Surafel Lemma Abebe, Shane McIntosh, Uira Kulesza, Ahmed E Hassan "An Empirical Study of Delays in the Integration of Addressed Issues," In Proc. of the 30th IEEE International Conference on Software Maintenance and Evolution (ICSME'14), pp.281-290, 2014.
[3] Akinori Ihara, Daiki Fujibayashi, Hirohiko Suwa, Raula Gaikovina Kula, and Kenichi Matsumoto, "Understanding When to Adapt a Library: a Case Study on ASF Projects," In Proc. of the International Conference on Open Software Systems (OSS'17), pp.128-138, 2017.

EVOSS: simulating FOSS distribution upgrades

By: Davide Di Ruscio (@DDiRuscio) and Patrizio Pelliccione (@pellicciTwit)

Associate editor: Stefano Zacchiroli (@zacchiro)

Upgrading software is a critical and error prone task that might have dramatic consequences. This is the case of the Japanese flagship astronomical satellite Hitomi that was successfully launched on 17 February 2016 but that, most probably because of a software upgrade, was declared lost on March 26th, 2016. The cost of the satellite was $286 million.

In this article we focus on how to manage the upgrade of Free and Open Source Software (FOSS) distributions. Distributions are complex systems, composed of thousand components (software packages) evolving rapidly and independently from each other. A FOSS distribution can be seen as a consistent and functional collection of software packages comprising a complete operating system. The management of FOSS distributions evolution is very challenging due to its community-centric nature and to the frequent releases of components [1].

Typically, distributions offer automated tools, called package managers, for managing the components they are made of, and for managing system upgrades. State of the art package managers are able to successfully manage a limited set of upgrade types and, specifically, they are only aware of static dependencies among packages that can influence upgrades. The most important information available to package managers concern the specification of inter-package relationships such as dependencies (i.e., what a package needs in order to be correctly installed and to function correctly), and conflicts (i.e., which other packages should not be present on the system in order to avoid malfunctioning). Package managers completely ignore relevant dynamic aspects, such as potential faults of configuration scripts that are executed during upgrade deployment. Thus, it is not surprising that an apparently innocuous package upgrade can end up with a broken system state [2].

EVOSS

EVOSS (EVolution of free and Open Source Software), proposed within the context of the Mancoosi EU project, is an approach to support the upgrade of FOSS systems (see Figure 1). The approach is based on Model-driven Engineering (MDE) [3] which refers to the systematic use of models as first class entities throughout the software engineering life cycle. In MDE, Domain-Specific Modeling Languages (DSMLs) are used to describe the system. DSMLs use metamodels to define the main language concepts such as the relations among domain concepts and their semantics. More precisely, a metamodel is an abstraction that highlights the properties of the models, which are said to conform to its metamodel like a program conforms to the grammar of the programming language. DSMLs are used to build a system model according to its metamodel semantics and constraints. In MDE it is common to have a set of transformation engines and generators that produce various types of artefacts. There are many model-to-model transformation approaches, as surveyed in [4]. It is important to note that model transformations are defined once for all at the metamodeling level; therefore, practitioners act as users of model transformations, which are defined by MDE and domain experts.

Figure 1: Overview of the EVOSS approach working in a real system

In order to make upgrade prediction more accurate, EVOSS considers both static and dynamic aspects of packages and their upgrades. The main dynamic aspects considered are those related to the behaviour of package configuration scripts (AKA maintainer scripts) which are executed during upgrade deployment.

Maintainer scripts are executed during upgrades and they are full-fledged programs usually written in the POSIX shell language. Moreover, they are run with the system administrator rights and may therefore perform arbitrary changes to the whole system. They are expected to complete without errors: their failures, usually signalled by returning non-0 exit codes, automatically trigger upgrade failures and may easily lead the system to a non-coherent state.

EVOSS defines a DSL to specify the behaviour of maintainer scripts: this is a way to make the effect of the scripts predictable. This also limits the expressive power of the language used in maintainer scripts, without however reducing important functionality of the scripts themselves. The DSL includes a set of high level clauses with a well-defined transformational semantics expressed in terms of system state modifications: each system state is given as a model and the script behaviour is represented by corresponding model transformations.

The idea of EVOSS is to exploit the DSL semantics to better understand how the system evolves in a new configuration. This is obtained by simulating system upgrades via an upgrade simulator (see Figure 1) that allows system users to discover upgrade failures due to fallacious maintainer scripts before performing the upgrade on the real system. The simulator takes into account both fine-grained static aspects (e.g., configuration incoherencies) and dynamic aspects (e.g., the execution of maintainer scripts). The simulator is based on model-driven techniques and makes use of a model-based description of the system to be upgraded. More specifically, the configuration model represents a model of the system configuration and package models represent a model of packages to be installed, removed, or updated. The simulator is able to simulate also pre- and post-installation scripts that come with distribution packages. In order to apply the proposed simulation approach system configuration and package models have to be extracted automatically from existing artefacts.

In order to build the system's configuration and package models, EVOSS makes use of model injectors that are apt to extract models from existing artefacts. The outcome of the system injection is a model that represents, in a homogeneous form, different aspects of a running system, such as installed packages, users and groups, MIME type handlers, file alternatives, implicit dependencies, etc. The outcome of package injection contains modelling elements encoding both the considered package and its scripts (as DSL statements).

The fault detector is then used to check system configurations for incoherencies. The coherence of a configuration model is evaluated by means of queries which are embodied in the fault detector. In particular, for each detectable fault, a corresponding Object Constraint Language (OCL) expression is defined and used to query models and search for model elements denoting faults. OCL is a declarative language that provides constraint and object query expressions on models and meta-models. Obviously it is not possible to define once forever a complete catalogue of faults because they are based on experience and acquired know-how. Therefore, the fault detector has been designed to be open and extensible so that new queries can be added whenever new classes of faults are identified.
Implications

EVOSS represents an advancement, with respect to the state of the art of package managers, in the following aspects: (i) it provides a homogeneous representation of the whole system’s configuration in terms of models, including relevant system elements that are currently not explicitly represented, (ii) it supports the upgrade simulation with the aim of discovering failures before they can affect the real system, (iii) it proposes a fault detector module able to discover problems on the configuration reached by the simulation.

The EVOSS approach has been investigated and exploited by the Caixa Magica GNU/Linux distribution to support selective roll-back. More details might be found here on YouTube.

More information about EVOSS as well as its source code can be found on the EVOSS website.

If you like this article, you might also enjoy reading:

Overall description of the EVOSS approach:

• Roberto Di Cosmo, Davide Di Ruscio, Patrizio Pelliccione, Alfonso Pierantonio, Stefano Zacchiroli (2011) Supporting software evolution in component-based FOSS systems, Science of Computer Programming 76: 12. 1144-1160. http://dx.doi.org/10.1016/j.scico.2010.11.001

Details about the simulator component:

• Davide Di Ruscio, Patrizio Pelliccione (2014) Simulating upgrades of complex systems: The case of Free and Open Source Software, Information and Software Technology 56: 4. 438-462 April. http://dx.doi.org/10.1016/j.infsof.2014.01.006

Details about the fault detector component:

• Davide Di Ruscio, Patrizio Pelliccione (2015) A model-driven approach to detect faults in FOSS systems, Journal of Software: Evolution and Process 27: 4. 294-318 April. http://dx.doi.org/10.1002/smr.1716

Details about the tool:

• Davide Di Ruscio, Patrizio Pelliccione, Alfonso Pierantonio. EVOSS: a tool for managing the Evolution of Free and Open Source Software systems. 34th International Conference on Software Engineering (ICSE), Zurich, Demo paper, 2012, pp. 1415-1418. http://dx.doi.org/10.1109/ICSE.2012.6227234

References

[1] Eric S. Raymond. The cathedral and the bazaar. O'Reilly, 2001.
[2] Olivier Crameri, Nikola Knezevic, Dejan Kostic, Ricardo Bianchini, and Willy Zwaenepoel. Staged deployment in mirage, an integrated software upgrade testing and distribution system. SIGOPS Oper. Syst. Rev., 41(6):221-236, 2007.
[3] D. C. Schmidt. Guest Editor's Introduction: Model-Driven Engineering. Computer, 39(2):25-31, 2006.
[4] K. Czarnecki and S. Helsen. Feature-based survey of model transformation approaches. IBM Syst. J., 45:621-645, July 2006.

Release management in Open Source projects

By: Martin Michlmayr (@MartinMichlmayr)
Associate editor: Stefano Zacchiroli (@zacchiro)

Open source software is widely used today. While there is not a single development method for open source, many successful open source projects are based on widely distributed development models with many independent contributors working together. Traditionally, distributed software development has often been seen as inefficient due to the high level of communication and coordination required during the software development process. Open source has clearly shown that successful software can be developed in a distributed manner.

The open source community has over time introduced many collaboration systems, such as version control systems and mailing lists, and processes that foster this collaborative development style and improve coordination. In addition to implementing efficient collaboration systems and processes, it has been argued that open source development works because it aims to reduce the level of coordination needed. This is because development is done in parallel streams by independent contributors who work on self-selected tasks. Contributors can work independently and coordination is only required to integrate their work with others.

Relatively little attention has been paid to release management in open source projects in the literature. Release management, which involves the planning and coordination of software releases and the overall management of releases throughout the life cycle, can be studied from many different aspects. I investigated release management as part of my PhD from the point of view of coordination theory. If open source works so well because of various mechanism to reduce the level of coordination required, what implications does this have on release management which is a time in the development process when everyone needs to come together to align their work?

Complexity of releases

As it turns out, my study on quality problems has highlighted that release management can be a very problematic part in the production of open source. Several projects described constant delays with their releases, leading to software which is out-of-date or which has other problems. Fundamentally, release management relies on trust. Contributors have to trust release managers and deadlines imposed by them, otherwise the deadlines are simply ignored. This leads to a self-fulfilling prophecy: because developers don't believe a release will occur on time, they continue to make changes and add new code, leading to delays with the release. It's very hard to break such a vicious circle.

It's important to consider why creating alignment in open source projects can be a challenge. I identified three factors that made coordination difficult:

1. Complexity: alignment is harder to achieve in large projects and many successful open source projects have hundreds of developers.
2. Decentralization: many open source projects are distributed, which can create communication challenges.
3. Voluntary nature: it's important to emphasize that this does not mean that contributors are unpaid. While some open source contributors are unpaid, increasingly open source development is done by developers employed or sponsored by corporations. The Linux kernel is a good example with developers from hundreds of companies, such as Intel and Red Hat. What I mean by voluntary nature is that the project itself has no control over the contributors. The companies (in the case of paid developers) define what work gets done and unpaid contributors generally "scratch their own itch". What this means is that it's difficult for a release manager or a project leader to tell everyone to align their work at the same time.

Time-based releases

While my research has shown problems with release management in many projects, it has also identified a novel approach to release management employed by an increasing number of projects. Instead of doing releases based on new features and functionality, which has historically been the way releases are done, releases are made based on time. Time-based releases are like a release train: there is a clear timetable by which developers can orient themselves to plan their work. If they want to make the next train (release), they know when they have to be ready.

Time-based releases work particularly well if the releases are predictable (for example, every X months) and frequent. If a release is predictable, developers can plan accordingly. If a release is fairly frequent, it doesn't matter if you miss a release because you can simply target the next one. You can think of a train station with a train that is leaving. Should you run? If you know the next train will leave soon and you can trust that the next train will leave on time, there is no need to hurry — you can avoid a dangerous sprint across the platform. Similarly, if the next release is near and predictable, you don't need to disrupt the project by making last minute changes.

Additionally, frequent releases give developers practice making releases. If a project does releases only every few years, it's very hard to make the release process an integral process of the development cycle. When releases are done on a regular basis (say every three or six months), the release process can work like a machine — it becomes part of the process.

Time-based releases are a good mechanism for release management in large open source projects. Speaking in terms of coordination theory, time-based releases decrease the level of coordination required because the predictable timetable allows developers to plan for themselves. The timetable is an important coordination mechanism in its own right.

Looking at various open source projects, time-based release management is implemented in different ways. For example, GNOME and Ubuntu follow a relatively strict frequency of six months. This is frequent, predictable and easy to understand. The Linux kernel employs a 2 week "merge window" in which new features are accepted. This is followed by a number of release candidates with bug fixes (but no new features) until the software is ready for release, typically after 7 or 8 release candidates. Debian follows a model where the freeze date is announced in advance. The time between freeze and release depends on how fast defects get fixed. Debian's freeze dates are more than 2 years apart, which in my opinion leads to challenges because the release process is not performed often enough to become a routine.

Recent changes

There have been many changes recently in the industry and development community that have influenced release management. One change relates to the release frequency, which has been going up in several projects (such as U-Boot, which moved from a release every three months to one every two months in early 2016). This may stem from the need to serve updates to users more frequently because the technology is changing so rapidly. It could also be because the release train is working so well and the cost of doing releases has gone down. Frequent releases lead to a number of questions, though. For example, do users prefer small, incremental updates over larger updates? Furthermore, how can you support old releases or are users expect to upgrade to the latest version immediately (a model employed by various app stores for mobile devices)?

In addition to more frequent releases, I also observe that some projects have stopped making releases altogether. In a world of Continuous Integration (CI) and Continuous Deployment (CD), does it make sense to offer the latest changes to users because you can assure that every change has been sufficiently tested? Is there still a value of performing separate releases when you have CI/CD?

I believe more research is needed to understand release management in our rapidly changing world, but one thing is clear to me: studying how contributors with different interests come together to produce and release software is fascinating!

References

• Michlmayr, M., Fitzgerald, B. (2012). Time-Based Release Management in Free and Open Source (FOSS) Projects. International Journal of Open Source Software and Processes, 4(1). 1-19.
• Michlmayr, M., Fitzgerald, B., Stol, K. (2015). Why and How Should Open Source Projects Adopt Time-Based Releases? IEEE Software, 32(2). 55-63.
• Erenkrantz, J. R. (2003). Release management within open source projects. In Proceedings of the 3rd Workshop on Open Source Software Engineering, Portland, OR, USA, 51-55. ICSE.
• LWN, Some 4.5 Development statistics, March 9, 2016
• Michlmayr, M. (2007). Quality Improvement in Volunteer Free and Open Source Software Projects: Exploring the Impact of Release Management. PhD dissertation, University of Cambridge.
• Chen, W., Krishnan, V., Zhu, K. (2013), "Release Early, Release Often"? An Empirical Analysis of Release Strategy in Open Source Software Co-Creation. PACIS 2013 Proceedings. Paper 11.
• International Workshop on Release Engineering

Why not use open source code examples? A Case Study of Prejudice in a Community of Practice

by Ohad Barzilay, Tel Aviv University (ohadbr@tau.ac.il)
Associate Editor: Christoph Treude (@ctreude)

With so much open source code available online, why do some software developers avoid using it? That was the research question guiding a qualitative grounded-theory study recently published [1]. We analyzed the perceptions of professional software developers as manifested in the LinkedIn online community, and used the theoretical lens of prejudice theory to interpret their answers in a broader context.

We focused on developers’ perception of (re)using code examples - existing code snippets that are used in a new context. Our definition of a code ‘example’ is broad; some of the code examples which appear on the Internet were not written in order to be reused. Code examples may accompany answers on Q&A sites [2], illustrate an idea in an online tutorial, or even be extracted from an open source project [7].

We suggest that developers’ approach with respect to using code examples is dominated by their personality, and affected by concerns such as their community identity, ownership and trust. We find that developers’ perception of such reuse goes beyond the activities and practices, and that some developers associate the use of code examples with negative character. Some of these developers stereotype habitual example users as inferior and unprofessional.

It should be noted that not only human aspects are associated with example usage – there are some other issues involved in this activity such as engineering aspects (e.g. search techniques and tools) and legal issues (e.g. copyright and licensing). These issues are outside of the scope of our discussion; however, we believe that these challenges can be mitigated with proper tools [9], training and organizational support (e.g. leveraging upon social media cues [8], teaching developers which code could they use, and under what circumstances).

Code Writers vs. Copy-and-Paste Monkeys

Some software developers perceive themselves as code writers, and feel strongly about it. Their identity and sometimes even self-esteem are derived from perceiving themselves that way. As suggested by Bewer [5] this may result in the creation of ingroup bias, and can be in turn used as a platform for hate of the outgroup – in this domain, example users. For the code writers (virtual) group, new code is the unit of progress, a sign of productivity (however misleading it may sometimes be). Copying, on the other hand, is perceived as a devalued shortcut – an imitation rather than a creation. In most university courses, the students are not allowed to share their work with fellow students, but are expected to write their own code.

Having ingroup bias often limits boundaries of trust and cooperation [6], which may explain why some developers avoid copy and paste at all cost. They do not trust other programmers enough to take responsibility for and ownership of their code. These programmers find it difficult to understand existing code, they feel that they cannot identify fallacies in someone else's code nor test it thoroughly. They prefer to write the code by themselves and take responsibility for it rather than trust others, and perhaps lose control over their code.

Furthermore, we find that example usage opponents do not conform to organizational goals, and specifically the need for speed. They do not acknowledge the required dexterity and practices for effective example usage, and they aspire to be held in high regard (as opposed to "merely plumbers" [4], suggesting that the essence of the software engineering job boils down to putting the components together and making small non-glorious fixes). After all, some of them might have chosen programming as their profession because of its status.

Implications

In a commercial context, revealing implicit prejudice and disarming it may allow developers to leverage further benefit of code reuse, and may improve the collaboration of individuals, teams and organizations. Moreover, prejudice may interfere with achieving organizational goals, or while conducting an organizational change. Some of these concerns may be mitigated by providing a comprehensive ecosystem of tools, practices, training and organizational support [3]. Having the prejudice lens in mind, one may incorporate methods which were proven effective in addressing prejudice in different context (racism, sexism, nationalism) as part of the software engineering management tools.

Finally, this study may also be considered in the broader context of the changing software engineering landscape. The recent availability of information over the Web, and in our context – availability of source code, is challenging the way software is produced. Some of the main abstractions used in the software domain, namely development and construction, do not adequately describe the emerging practices involving pragmatic and opportunistic reuse. These practices favor composing over constructing and finding over developing. In this context, prejudice can be perceived as a reaction to change and an act resulting from fear of the new and unknown.

References

[1] O. Barzilay and C. Urquhart. Understanding reuse of software examples: A case study of prejudice in a community of practice. Information and Software Technology 56, pages 1613-1628, 2014.
[2] O. Barzilay, C. Treude, and A. Zagalsky. Facilitating crowd sourced software engineering via Stack Overflow. In S. E. Sim and R. E. Gallardo-Valencia, editors, Finding Source Code on the Web for Remix and Reuse, pages 289–308. Springer New York, 2013.
[3] O. Barzilay. Example embedding. In Proceedings of the 10th SIGPLAN symposium on New ideas, new paradigms, and reflections on programming and software, pages 137-144, 2011, ACM.
[4] O. Barzilay, A. Yehudai, and O. Hazzan. Developers attentiveness to example usage. In Human Aspects of Software Engineering, HAoSE ’10, pages 1–8, New York, NY, USA, 2010. ACM.
[5] M. B. Brewer. The psychology of prejudice: Ingroup love and outgroup hate? Journal of social issues, 55(3):429–444, 1999.
[6] S. L. Jarvenpaa and A. Majchrzak. Knowledge collaboration among professionals protecting national security: Role of transactive memories in ego-centered knowledge networks. ORGANIZATION SCIENCE, 19(2):260–276, 2008.
[7] S. E. Sim and R. E. Gallardo-Valencia, editors. Finding Source Code on the Web for Remix and Reuse. Springer, 2013.
[8] C. Treude and M. P. Robillard. Augmenting API documentation with insights from Stack Overflow. Forthcoming ICSE ’16: 38th Int’l. Conf. on Software Engineering, 2016.
[9] A. Zagalsky, O. Barzilay, and A. Yehudai. Example overflow: Using social media for code recommendation. In Proceedings of the Third International Workshop on Recommendation Systems for Software Engineering, pages 38-42, 2012, IEEE Press.

Dissecting The Myth That Open Source Software Is Not Commercial

By: Karl Fogel (@kfogel)
Associate editor: Stefano Zacchiroli (@zacchiro)

Writing a myth-debunking piece for such an informed audience poses a certain risk. The readers of the IEEE Software Blog already know what open source software is, and many have probably written some. How can I be sure that anyone reading this even holds the belief about to be debunked?

Well, I can't be completely sure, but can at least say that this myth is one I still encounter frequently among software specialists, including people who themselves use free software on a daily basis. (By the way, I will use the terms "free" — as in "free software" — and "open source" interchangeably here, because they are synonyms in the sense that they refer to the same set of software and the same set of pro-sharing software licenses.) The continued prevalence of this myth in many organizations is an obstacle to the adoption and production of open source software.
First, to state it clearly:

Myth: Open source is not commercial, or is even anti-commercial, and is driven mostly by volunteerism.

That's really two myths, but they're closely related and it's best to address them together.

In mainstream journalism, open source is still almost always portrayed as a spare-time activity pursued by irrepressible programmers who band together for the love of coding and for the satisfaction they get from releasing free tools to the world. (See the second letter here for one example, but there are many, many more examples like that.) Surprisingly, this portrayal is widespread within the software industry too, and in tech journalism. There is, to be fair, a grain of truth to the legend of the volunteers: especially in the early days of open source — from the mid 1980s until the late 1990s (a period when it wasn't even called "open source" yet, just "free software") — a higher proportion of open source development could legitimately have been called volunteer than is the case today.

But still not as high a proportion as one might think. Much free software activity was institutionally funded even then, although the institutions in question weren't always aware of it. Programmers and sysadmins frequently launched shared collaborative projects simply to make their day jobs easier. Why should each person have to write the same network log analyzer by themselves, when a few people could just write it once, together, and then maintain it as a common codebase? That's cheaper for everyone, and a lot more enjoyable.

In any case, intentional investment in open source by for-profit outfits started quite early on, and such investment has only been growing since (indeed, to the point now where meta-investment is happening: for example, my company, Open Tech Strategies, flourishes commercially by doing exclusively open source development and by advising other organizations on how to run open source projects). For a long time now, a great deal of widely-used open source software has been written by salaried developers who are paid specifically for their work on that software, and usually paid by for-profit companies. There is not space here to discuss all their business models in depth, nor how erstwhile competitors manage to collaborate successfully on matters of shared concern (though note that no one ever seems to wonder how they manage this when it comes to political lobbying). Suffice it to say that there are many commercial organizations in whose interests it is to have this growing body of code be actively maintained, and who have no need to "own" or otherwise exercise monopolistic control over the results.

A key ingredient in this growth has been the fact that all open source licenses are commercial licenses. That is, they allow anyone to use the software for any purpose, including commercial purposes. This has always been part of the very definition of a "free software" or "open source" license, and that's why there is no such thing as software that is "open source for non-commercial use only", or "open source for academic use only", etc.

An important corollary of this is that open source software automatically meets the standard government and industry definition of "Commercial Off-The-Shelf" (COTS) software: software that is commercially available to the general public. COTS doesn't mean you must pay money — though you might choose to purchase a support contract, which is a fee for service and is very different from a licensing fee. COTS essentially just means something that is equally available to all in the marketplace, and open source software certainly fits that bill.

So: open source is inherently commercial, and the people who write it are often paid for their work via normal market dynamics.

Why, then, is there a persistent belief that open source is somehow non-commercial or anti-commercial, and that it's developed mainly by volunteers?

I think this myth is maintained by several mutually reinforcing factors:

• Open source's roots are as an ideologically-driven movement (under the name "free software"), opposed to monopoly control over the distribution and modification of code. Although that movement has turned out to be successful in technical and business terms as well, it has not shed its philosophical roots. Indeed, I would argue, though will not here due to space limitations, that its philosophical basis is an inextricable part of its technical and business success. (It is worth considering deeply the fact that merely being anti-monopoly is enough to get a movement a reputation for being anti-commercial; perhaps it is the roots of modern capitalism as actually practiced that need closer examination, not the roots of open source.)
• For a time, various large tech companies whose revenues depend mainly on selling proprietary software on a fee-for-copying basis made it a standard part of their marketing rhetoric to portray open source as being either anti-commercial or at any rate unconcerned with commercial viability. In other words: don't trust this stuff, because there's no one whose earnings depend on making sure your deployment is successful. This tactic has become less popular in recent years, as many of those companies start to have open-source offerings themselves. I hope to see it gradually fade away entirely, but its legacy lives on in the many corporate and government procurement managers who were led to believe that open source is the opposite of commercial.
• Many companies now offer software-as-a-service based on open source packages with certain proprietary additions — those additions being their "value-add" (or, less formally, their "secret sauce"), the thing that distinguishes their SaaS offering from you just deploying the open source package on which it is based, and the thing that not coincidentally has the potential to lock you in to that provider.
  Unfortunately, companies with such offerings almost always refer to the open source base package as the "community edition", and their proprietized version as the "commercial edition" or sometimes "enterprise edition". A more accurate way to label the two editions would be "open source" and "proprietary", of course. But, from a marketing point of view, that has the disadvantage of making it plain what is going on.
• Software developers have multiple motivations, and it's true that in open source, some of their motivation is intrinsic and not just driven by salary. It's actually quite common for open source developers to move from company to company, being paid to work on the same project the whole time; their résumé and work product are fully portable, and they take advantage of that. Open source means that one cannot be alienated from the fruits of one's labor, even when one changes employers. There is nothing anti-commercial about this — indeed, it could be viewed as the un-distortion of a market — but one can certainly see how observers with particular concerns about the mobility of labor might be inclined to fudge that distinction.

Finally, I think people also want to believe in a semi-secret worldwide army of happy paladins acting for the good of humanity. It would be so comforting to know they're out there. But what's actually going on with open source is much more complex and more interesting, and is firmly connected to commerce.

References

• Free Software Definition
• Open Source Definition
• GNU Manifesto
• Cathedral and the Bazaar
• FLOSS is Commercial Software
• Brief history of free and open source software