On the rise of Casual Contributions in postmodern Software Development - How does it impact Software Quality

By: Naveen N Kulkarni
Associate Editor: Sridhar Chimalakonda (@ChimalakondaSri)

Postmodernism, a term used in variety of disciplines like arts, literature, fashion, music, movies and technology, is used to describe a tendency of philosophical movements and resulting shifts where there is a rejection of the predecessor. In case of software engineering too, the postmodernistic views have challenged the rigidity of traditional approaches; much like agile methods challenged the need for upfront requirement gathering, perpetual beta cycles where users are considered co-developers or heterogeneous application stack replaced with homogeneous stacks like Node.js and Vert.x. In case of collaboration in software development we see breaking the norm of controlled contributions to shared contributions to fragmented contributions combined with social underpinnings.

In context of Open Source Software (OSS), empirical studies have shown that developers aim to rapidly deliver ’credible promises’ to keep their projects viable and active [Sojer, Haefliger]. With social interactions gaining popularity on the Internet, casual contributions (popularly known as pull requests on platforms such as, GitHub and BitBucket) to OSS is increasing. Empirical studies suggest that such contributions are not trivial, but involves bug fixes, refactoring and new features. While an OSS can deliver its promise faster with such requests, it can can quickly overwhelm the developers evaluating them whether to accept or not in case there are many. In order to deliver faster, it is intuitive that developers choose project relevance and code quality as the primary factors to evaluate the pull requests (substantiated in a recent survey by Gousios et al.). To minimize the effort, many OSS projects use strict policies that include discussions, linting, clean merge, tests and code reviews; most of which are automated through continuous integration tools. As the casual contributions can be unplanned and opportunistic, attempts are made by the development 1 2 team to understand the context of contribution through prior discussions. For example, in case of Scala, pull requests are accepted only if it is discussed on ’scala-internals’ mailing list. Also, mandatory discussions with more than one developers is required depending on the complexity of the contribution. All these are undertaken as a part of the quality assessment of the casual contribution. However, with such extensive checks and large volumes, the pull requests can quickly get stale, making it even harder for developers and contributors to understand the impact of composing multiple parallel contribution.

Survey of work practices in managing and integrating pull request suggest that every contribution requires multiple code review sessions. This is substantiated by the fact that code reviews require ”code reading” to find defects, which is greatly dependent on individual expertise. Unfortunately, though they are perceived as a best practice, they are often localized and informal. Today’s code review session include interactive sessions, discussions and in-line comments. A study of modern code review process show that 35% of the review suggestions are discarded and 23% of the changes are applied after review (Beller M et al.). It is suggested that this process can be more people independent in postmodern software engineering. On the quality, there are very few qualitative evidences relating to code review process. Recent study by McIntosh et al. suggest that review coverage as an important factor to ensure quality. However, quality criteria are subjective during code reviews. Quality criteria is often restricted to statically analyzed source code using popular continuous integration tools (like Sonar or Squale), coding style and test cases.

Despite the challenges, we observe casual contributions are raising. For example, at the time of writing this post, the rails and redis projects have nearly 500 open pull requests. The bootstrap, npm, tensorflow and docker have nearly 100 open pull requests. More detailed analysis on the growth of pull requests can be found at http://ghtorrent.org/pullreq-perf/ and https://octoverse.github.com/. As we adopt different approaches to compose software with frequent fragmented contributions, we are faced with challenges, such as emergent behavior, constraint violations and conflicting assumptions. These challenges arise due to the parallel and isolated nature of the contributions. Though these challenges are not new, we believe the current practices on quality assessments are inadequate to address them. The techniques used today help developers quickly spot local mismatches but they are not effective for comprehending global mismatches that originate due to many complex relationship among the software elements. In our initial study, we found similar evidence when a opportunistically reused external module not only violated the existing constraints, but also added unanticipated behavior making the software restrictive.

Attempts to address the global mismatches using architectural reasoning and design decisions approaches have so far has met with limited success. Also, during code reviews are insufficient to highlight the ramifications of a contribution on the global constraints and mismatches. To overcome this issue, the merge process should extend beyond syntactic merge to include techniques for identifying constraint mismatches. We believe, by validating the constraint synthesized as predicates from the hot paths (critical execution flow) can optimally help validating the global mismatches. In past, predicate abstractions are effectively used for software verification (SLAM). On one hand, the predicate abstraction can facilitate theoretical guarantees, synthesizing the predicates can be overwhelming due to state explosion. To overcome this we can use descriptive statistical models (created from the features available in the source code) to choose a subset of predicates sufficient for the verification. Alternatively, mining of software repository techniques can play pivotal role in developing alternative approaches where they not only provide factual data, but help in decision process. Mining constraints as a set of sequential patterns (SPADE, a popular sequence mining algorithm) can be used where comparing sequence for dissimilarities between the source codes can suggest mismatches.

There is a need for lightweight processes to cope with the dramatic shifts in the postmodern software engineering. In case of casual contributions, defining code quality can be confounding due to their inherent isolated nature. So, code reviews are critical and there is a need for alternative approaches that account for the various postmodern shifts. Without this the software quality in postmodern software development will remain elusive.

References

• SojerM, Henkel J. Code reuse in open source software development: quantitative evidence, drivers, and impediments. Journal of the Association for Information Systems 2010; 11(12):868901.
•Stanciulescu S, Schulze S,Wasowski A. Forked and integrated variants in an open-source firmware project. Proceedings of the IEEE Int’l Conference on Software Maintenance and Evolution, ICSME 15, Bremen, Germany, 2015; 151160.
• Perry DE, Siy HP, Votta LG. Parallel changes in large-scale software development: an observational case study. ACM Transactions on Software Engineering and Methodology (TOSEM) 2001; 10(3):308337.
• Gousios G, Zaidman A, Storey MA, Deursen Av. Work Practices and Challenges in Pull-Based Development: The Integrators Perspective. Proceedings of the 37th International Conference on Software Engineering, 2015, vol. 1, pp. 358368.
• Scala Pull request Policy, http://docs.scala-lang.org/scala/pull-request-policy.html, accessed 15-Mar-2017.
• Pinto G, Steinmacher I, Gerosa MA. More common than you think: An in-depth study of casual contributors. Proceedings of the 23rd IEEE Intl Conference on Software Analysis, Evolution, and Reengineering, SANER 16, Suita, Osaka, Japan, 2016; pp. 112123. 4
• Beller M, Bacchelli A, Zaidman A, Juergens E. Modern Code Reviews in Open-Source Projects: Which Problems Do They Fix? In Proceedings of Mining Software Repository, MSR’14, Hyderabad, India, 2014.

Diversity in software teams: buzzword or necessity?

Associate Editor: Bogdan Vasilescu, University of California, Davis. USA (@b_vasilescu) 

"I've had some bad experiences, particularly based on my gender. [...] a lot of time I'll join an online team and sometimes the other team members are very rude. In the past I have used a fake GitHub handle (my normal GitHub handle is my first name, which is a distinctly female name) so that people would assume I was male. I wouldn't lie if asked directly about my gender, but I know a lot of people assume by default others on the internet are male, and so I use a unisex handle [...] so people assume I am male without asking. I'm not happy with this solution, but sometimes I really want to participate [...] and I don't have the energy to defend myself for being female." [open source developer, USA]

This is not a singular occurrence. Despite its core meritocratic values, the open source "hacker" culture is known to be male-dominated and unfriendly to women [1, p.194]. Some authors go as far as to say that sexist behavior in open source is "as constant as it is extreme" [2]. In open source, women represent only around 10% of the all contributors [3], considerably more underrepresented than in the big tech companies; Google, for example, reports 18% women in technical jobs; Microsoft is close, with 16.6%.

Yet, open source software are well recognised for their high quality, and are becoming increasingly popular. According to a recent survey, 78% of companies run their operations on open source software; 66% create software for their customers on top of open source software. Surely, open source software teams know what they’re doing. By staying homogeneous, teams limit their members’ potential differences in values, norms, and communication styles; it also better shields them from stereotyping, cliquishness, and conflict. It becomes much less likely for team members to disagree if they all think (and look) alike, with the majority class being the Western young white male. 

"Clones" by Nick Royer (CC BY-SA)

However, teams who inadvertently act like this may be missing out. Increased team diversity results in more varied backgrounds and ideas, which provide a team with access to broader information and enhanced creativity, adaptability, and problem solving skills. This fact has been documented countless times in "traditional" teams, that interact face-to-face. Does it also hold online, in the meritocratic open source, where "code sees no color or gender" [4]? After all, on the Internet nobody knows you're a dog.

We set out to investigate how diversity affects team performance on GitHub, the nexus of open source development, using a mixture of qualitative and quantitative research methods. First, through a survey with more than 800 respondents [4], we found that individual demographic attributes are surprisingly salient: while developers are, above all, aware of one another’s programming skills, they are also well aware of each other’s gender, real name, and country of residence. Their opinions on diversity are, however, split. Most respondents (62.5%) seem to view diversity as always positive. They acknowledge that diversity in terms of demographics and technical background is often associated with new ideas and approaches to solve problems, access to different networks, lively discussions around issues and pull requests, and ultimately better code. Still, some respondents (30%) report on occasional negative effects due to diversity, such as the developer quoted at the beginning of this blog post, referring to a gender-related incident.

Second, we extracted data from a large sample of 23,000 active projects on GitHub, spanning six years of activity and including a multitude of variables. We focused on two facets of diversity: gender (having a more balanced male-female team; we used people’s names to infer their gender) and tenure (having a mixture of seniors and juniors; we estimated a person’s programming experience from across all their GitHub contributions). We then used regression analysis to model effects associated with diversity on the outputs produced by teams per unit time (we counted the number of commits to a project by team members during a quarter), as a measure of the teams' effectiveness [5]. After controling for team size, project age, social activity, and other confounds, our models show that both gender and tenure diversity are positive and significant predictors of productivity, together explaining a small but significant fraction of the data variability.

Together, the two analyses paint a complete picture: diversity is more than a buzzword, it’s a necessity! More diverse teams perform better. On a larger, economic and societal scale, these findings also suggest that added investments in educational and professional training efforts and outreach for female programmers will likely result in added overall value.

[1] Turkle, S. The Second Self: Computers and the Human Spirit. MIT Press, 2005
[2] Nafus, D. ‘Patches don’t have gender’: What is not open in open source software. New Media & Society 14, 4 (2012), 669–683.
[3] FLOSS 2013 survey, http://floss2013.libresoft.es/results.en.html
[4] Vasilescu, B., Filkov, V., and Serebrenik, A. Perceptions of Diversity on GitHub: A User Survey. In 8th International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE, IEEE (2015), 50–56.
[5] Vasilescu, B., Posnett, D., Ray, B., Brand, M.G.J. van den, Serebrenik, A., Devanbu, P., and Filkov, V. Gender and tenure diversity in GitHub teams. In ACM SIGCHI Conference on Human Factors in Computing Systems, CHI, ACM (2015), 3789–3798.

If you liked this post, you might also enjoy reading:

• Begel, A.; Bosch, J.; Storey, M.-A., "Social Networking Meets Software Development: Perspectives from GitHub, MSDN, Stack Exchange, and TopCoder," in Software, IEEE , vol.30, no.1, pp.52-66, Jan.-Feb. 2013
• Dabbish, L.; Stuart, C.; Tsay, J.; Herbsleb, J., "Leveraging Transparency," in Software, IEEE , vol.30, no.1, pp.37-43, Jan.-Feb. 2013