Gazes to Traces: Tracing Developers’ Eyes to Automate Traceability Link Recovery

by Bonita Sharif, Youngstown State University (@shbonita), and Huzefa Kagdi, Wichita State University

Associate Editor: Christoph Treude (@ctreude)

There is no denying that large-scale software systems evolve.  Software developers are routinely faced with new features and bugs that drive essential changes.  Oftentimes, they are not even the original authors of the code they need to change. They need to piece together many aspects to realize these changes whilst juggling various project parameters such as quality assurance and deadlines. The knowledge space for this existential activity typically spans several artifacts from the problem (e.g., requirements) and solution (design and implementation) domains.  The explicit connectivity among these artifacts is often missing in practice [1, 2], which further adds to the developer’s plea. Questions that are left to be answered include “Where is the relevant code to this feature/bug?” or “What would be the design impact due to a change request?”.  The promise of a software traceability tool is to establish (recover) and maintain (evolve) links among artifacts as software evolves [3]. Traceability benefits are projected in several key software development tasks such as program comprehension [4, 5] and impact analysis [6, 7], which address the aforementioned questions.

Over the years, software engineering researchers have proposed techniques for automatically recovering and maintaining (explicit) traceability links among software artifacts.  The application of Information Retrieval (IR) techniques is a popular and heavily experimented choice. It is an artifact-centric approach that has an underlying model of textual similarities in software artifacts.   The expressiveness, effectiveness, and usefulness rely on the artifacts, and their state of availability and quality during the software’s lifecycle. 

What We Did

Our research direction was to investigate a human-centric approach to traceability.  The underlying premise of our approach is based on what humans look at while they are performing software-engineering tasks, including those of bug fixes and implementing new features.   The prerequisite to such an approach is that it should be unobtrusive to developers and blend into the background.  We use an eye tracker to collect developers' gazes on software artifacts while they work on their tasks within the IDE (we use Eclipse; however, the same concept applies to others). Our eye tracking infrastructure is called iTrace [8, 9]. It seamlessly works within the IDE to map eye gazes to source code elements on large files in the presence of scrolling and context switching between files. A preliminary version of iTrace is available at https://github.com/Sereslab/iTrace-Archive. An enhanced release with additional support for IDEs and Web browsers is planned for the near future.

We first did a pilot study [10] to determine the feasibility of using eye gaze for traceability link recovery. After seeing promising results, we conducted a larger, realistic study [11] with thirteen software developers who were asked to perform bug-localization tasks for eight submitted bug reports in Jabref (an open-source reference management system). The gaze-link algorithm uses gaze data of developers during the session using certain heuristics and weights. In a bug fixing task for example, the main premise is that as you get closer to fixing a bug you will focus on the part(s) of the code that is most related to the bug report, thus having more weight. We compared trace link results of our gaze-link algorithm with IR methods such as Latent Semantic Indexing (LSI) and Vector Space Model (VSM). 

What We Found

The gaze-link algorithm outperforms both LSI and VSM in terms of precision and recall with respect to the commit oracle (how the JabRef developers fixed the bug). We recorded an average precision of 55% and recall of 67% for all tasks. The gaze-link algorithm outperformed in 6 out of the 8 tasks.  Another set of developers found the links generated with iTrace to be significantly more useful than the IR links in a majority of the tasks. The gaze-link algorithm underperforms when the developer prematurely attempts to fix the bug without the adequate understanding of the bug or its solution. Interestingly and perhaps surprisingly, the gazes captured from a developer in cases where they did not quite fix the bug, but was close enough, ended up being helpful to another developer as a starting point towards eventually fixing the bug. 

See Table 1 for an example of links generated for a bug report across the different approaches at both class and method level granularity. The gaze algorithm is crucial to weeding out irrelevant and stray glances (shown in ET_raw). Results from the gaze link algorithm (ET_weighted) are more specific than the rankings from current IR methods (LSI and VSM). 

Table 1. An example of class and method level source code elements found linked to Bug ID: 1489454 titled “Acrobat Launch fails on Win 98”. Variables are shown in square brackets. ET_raw represents raw eye tracking data before the algorithm is run, ET_weighted is the result of running gaze-link on the gazes. The Commit is our oracle representing the fix by the JabRef developer.

Concluding Remarks

The eye tracker captures the gaze data without any additional effort on the part of the developers. This property allows it to be as effortless as we can get to provide traceability under the hood while developers work.  An added benefit of using the eye tracker is that we also learn about those hidden links that information retrieval methods have a hard time finding as they are related to tacit developer knowledge in various related code entities. Imagine a software development world where your gazes could inform things you do. The transparency and minimal effort required by developers makes gaze tracking an attractive possibility. It is reasonable to imagine a future in which eye trackers capture information while we work to help us with many more tasks than just software traceability. 

We refer interested readers to the research article titled “Eye Movements in Software Traceability Link Recovery” [11].

References

[1] B. Ramesh, "Factors Influencing Requirements Traceability Practice," Communications of the ACM, vol. 41, pp. 37-44, December 1998.

[2] P. Mader, O. Gotel, and I. Philippow, "Motivation matters in the traceability trenches," in RE, 2009, pp. 143-148.

[3] D. J. Palmer, "Traceability," in Software Engineering, M. Dorfman and R. H. Thayer, Eds., ed: Wiley-IEEE Computer Society Press, Los Alamitos, California, 1996, pp. 266-276.

[4] A. De Lucia, D. P. Massimiliano, and R. Oliveto, "Improving Source Code Lexicon via Traceability and Information Retrieval," IEEE Transactions on Software Engineering pp. 205-227, 2011.

[5] A. De Lucia, M. D. Penta, R. Oliveto, and F. Zurolo, "Improving Comprehensibility of Source Code via Traceability Information: a Controlled Experiment," in 14th IEEE International Conference on Program Comprehension (ICPC'06), Athens, Greece, 2006, pp. 317-326.

[6] G. Antoniol, G. Canfora, G. Casazza, and A. De Lucia, "Identifying the starting impact set of a maintenance request," in 4th CSMR 2000, pp. 227–230.

[7] A. von Knethen, "Change-Oriented Requirements Traceability. Support for Evolution of Embedded Systems," in International Conference on Software Maintenance (ICSM 02), 2002.

[8] B. Sharif and J. I. Maletic, "iTrace: Overcoming the Limitations of Short Code Examples in Eye Tracking Experiments," in 32nd IEEE International Conference on Software Maintenance and Evolution (ICSME), Technical Briefing, Raleigh, NC, USA, 2016, p. 647.

[9] T. Shaffer, J. Wise, B. Walters, S. Müller, M. Falcone, and B. Sharif, "iTrace: Enabling Eye Tracking on Software Artifacts Within the IDE to Support Software Engineering Tasks," in ESEC/FSE 2015, Bergamo, Italy, 2015, pp. 954-957.

[10] B. Walters, T. Shaffer, B. Sharif, and H. Kagdi, "Capturing Software Traceability Links from Developers' Eye Gazes," in 22nd International Conference on Program Comprehension (ICPC), Hyderabad, India, 2014, pp. 201-204.

[11] B. Sharif, J. Meinken, T. Shaffer, and H. Kagdi, "Eye Movements in Software Traceability Link Recovery," Empirical Software Engineering Journal, vol. 22, pp. 1063-1102, 2017.

Can We Trust the Stack Overflow Netiquette? Evidence-based Guidelines for Asking Good Technical Questions

by Fabio Calefato, University of Bari, ITALY (@fcalefato), Filippo Lanubile, University of Bari, ITALY (@lanubile), and Nicole Novielli, University of Bari, ITALY (@NicoleNovielli)
Associate Editor: Christoph Treude (@ctreude)

Several thousands of developers daily head to Stack Overflow (SO) for asking technical questions, hoping to receive swift help and fix the issues that they have been facing. To increase the chances of getting help from others, the SO community provides members with detailed guidelines on how to write more effective questions (e.g., see [9]). These official recommendations also include those provided by Jon Skeet, the highest reputation member, whose guidelines have become over time a de facto standard for the community [10].

For example, SO states that the site is “all about getting answers. It's not a discussion forum. There's no chit-chat.” Thus, askers are recommended to avoid “greetings and sign-offs […], as they’re basically a distraction,” which are also supposed to be edited out by other users [10]. Still, many askers finish their questions showing gratitude in advance towards potential helpers. Why do they go against this explicit recommendation? Are they just unaware of it or do they feel that having a positive attitude may attract more potential solutions?

In our work [5], we provide an evidence-based netiquette for writing effective questions by empirically validating several SO guidelines, retrieved from both the community and previous empirical studies on Q&A sites. Specifically, we analyzed a dataset of 87K questions by combining a logistic regression analysis with a user survey, first, to estimate the effect of these guidelines on the probability of receiving a successful answer and, then, to compare their actual effectiveness to that perceived by SO users.

Actionable Factors for Good Questions

Figure 1. Our conceptual framework of success factors for writing good questions in Stack Overflow. The success of a question is defined as the probability of receiving an answer that is accepted as solution. Affect, Presentation Quality, and Time are the three actionable success factors of interest, while Reputation (non-actionable) serves as a control factor. Each success factor is related to several metrics used in turn to inform guidelines.

We developed a conceptual framework (Figure 1) for the analysis of the aspects that may influence the success of a question. We focused on 9 actionable metrics that can be acted upon by developers when writing a question and, therefore, are useful to inform guidelines. These metrics are grouped into 3 success factors, concerning affect (i.e., the positive/negative tone conveyed by a question), presentation quality (i.e., the readability and comprehensibility of its text), and time (i.e., when to post it). As per the asker’s reputation, it is included in our model, but only as a non-actionable control factor. While reputation has been already found to help people receive more and fasters answers in other communities such as Reddit [1], one can’t really do anything to increase their score just when posting a question.

Findings: Evidence vs. User Perception

Table 1. The evidence-based netiquette for effective question writing on Stack Overflow. Guidelines are shown in bold when supported by evidence.

#	Guideline	Success factor	Empirical support	User perception	Source
1	Write questions using a neutral emotional style	Affect	Yes	Effective	Skeet [10], SO Help Center [9], Kucuktunc et al. [7], Bazelli et al. [3]
2	Provide sample code and data	Presentation quality	Yes	Effective	Skeet [10], Asaduzzaman et al. [2], Duijn et al. [6], Treude et al. [11]
3	Use capital letters where appropriate	Presentation quality	Yes	Effective	Skeet [10]
4	Be concise	Presentation quality	Yes	Ineffective	Skeet [10]
5	Use short, descriptive question titles	Presentation quality	No	Ineffective	Skeet [10]
6	Provide context through tags	Presentation quality	No	Effective	Skeet [10]
7	Provide context through URLs	Presentation quality	No	Effective	Ponzanelli et al. (partially) [8]
8	Be aware of low-efficiency hours	Time	Yes	Ineffective	Bosu et al. [4]

Table 1 reports the main findings of our study, from which some interesting observations emerge. For 4 out of the 8 guidelines studied, we found that the perceived and actual effectiveness match. In particular, SO users are correct to think that minding their tone (#1), providing snippets with examples (#2), and avoiding the inappropriate use of capital letters (#3) all increase the probability to receive a successful answer to their questions. Instead, the use of short titles is neither perceived nor found to be an effective guideline (#5).

Perhaps even more interesting considerations arise from the remaining cases of mismatch, though. First, SO users seem to be unaware that writing questions concisely (#4) and posting them during GMT evening hours (#8) increase the chances of a question to be resolved. Regarding efficiency times, Bosu et al. [4] have speculated that these time slices are the most successful because they correspond to the working time in the USA. Second, contrary to users’ perception, we found that providing context for questions by adding extra tags (#6) and including in the text URLs to external resources (#7) have no positive effects on the chance of getting a successful answer.

Final Remarks

One of the greatest issues with Stack Overflow is the sheer number of unresolved questions. Currently, the site hosts almost 15 million questions of which about 7 million are still unresolved. Helping users ask “better” questions can increase the number of those resolved. One way to do so is increasing the awareness among community members about the existence of effective question-writing guidelines while also trimming down the list to only those supported by evidence. 

For more information about our study, please refer to our paper “How to Ask for Technical Help? Evidence-based Guidelines for Writing Questions on Stack Overflow” [5].

References

[1] T. Althoff, T., Danescu-Niculescu-Mizil, C., and Jurafsky, D. How to Ask for a Favor: A Case Study on the Success of Altruistic Requests. In Proc. of the 8th Int’l Conf. on Weblogs and Social Media (ICWSM’14), 2014.
[2] M. Asaduzzaman, A.S Mashiyat, C.K. Roy, K.A. Schneider. 2013. Answering questions about unanswered questions of Stack Overflow. In Proc. of the 10th IEEE Working Conf. on Mining Software Repositories (MSR 2013), pp. 97-100.
[3] B. Bazelli, A. Hindle, and E. Stroulia. On the Personality Traits of StackOverflow Users. In Proc. of Int’l Conference on Software Maintenance (ICSM '13), 2013, pp. 460-463.
[4] A. Bosu, C.S. Corley, D. Heaton, D. Chatterji, D.J. Carver, and N.A. Kraft. Building Reputation in StackOverflow: An Empirical Investigation. In Proc. of 10th IEEE Working Conf. on Mining Software Repositories (MSR’13), 2013, pp. 89-92.
[5] F. Calefato, F. Lanubile, N. Novielli. How to Ask for Technical Help? Evidence-based Guidelines for Writing Questions on Stack Overflow. Information and Software Technology, 2017.
[6] M. Duijn, A. Kučera, and A. Bacchelli. Quality questions need quality code: classifying code fragments on stack overflow. In Proc. of 12th Working Conf. on Mining Software Repositories (MSR '15), 2015, pp. 410-413.
[7] O. Kucuktunc, B.B. Cambazoglu, I. Weber, and H. Ferhatosmanoglu. A large-scale sentiment analysis for Yahoo! answers. In Proc. of 5th Int’l Conf. on Web Search and Data Mining (WSDM '12), 2012, pp. 633-642.
[8] L. Ponzanelli, A. Mocci, A. Bacchelli, and M. Lanza. Understanding and Classifying the Quality of Technical Forum Questions. In Proc. of 14th Int’l Conf. on Quality Software (QSIC’14), 2014, pp. 343-352.
[9] Stack Overflow Help Center, Be nice (last accessed: Nov. 2017).
[10] J. Skeet, 2010. Writing the Perfect Question (last accessed: Nov. 2017).
[11] C. Treude, O. Barzilay, and M-A. Storey. How do programmers ask and answer questions on the web? (NIER track). In Proc. of the 33rd Int’l Conf. on Software Engineering (ICSE '11), 2011, pp. 804-80.

Performance testing in Java-based open source projects

by Cor-Paul Bezemer (@corpaul), Queen's University, Canada
Associate Editor: Zhen Ming (Jack) Jiang, York University, Canada

From a functional perspective, the quality of open source software (OSS) is on par with comparable closed-source software [1]. However, in terms of nonfunctional attributes, such as reliability, scalability, or performance, the quality is less well-understood. For example, Heger et al. [2] stated that performance bugs in OSS go undiscovered for a longer time than functional bugs, and fixing them takes longer.

As many OSS libraries (such as apache/log4j) are used almost ubiquitously across a large span of other OSS or industrial applications, a performance bug in such a library can lead to widespread slowdowns. Hence, it is of utmost importance that the performance of OSS is well-tested.

We studied 111 Java-based open source projects from GitHub to explore to what extent and how OSS developers conduct performance tests. First, we searched for projects that included at least one of the keywords 'bench' or 'perf' in the 'src/test' directory. Second, we manually identified the performance and functional tests inside that project. Third, we identified performance-sensitive projects, which mentioned in the description of their GitHub repository that they are the 'fastest', 'most efficient', etc. For a more thorough description of our data collection process, see our ICPE 2017 paper [3]. In the remainder of this blog post, the most significant findings of our study are highlighted.

Finding # 1 - Performance tests are maintained by a single developer or a small group of developers. 
In 50% of the projects, all performance test developers are one or two core developers of the project. In addition, only 44% of the test developers worked on the performance tests as well.

Finding # 2 - Compared to the functional tests, performance tests are small in most projects. 
The median SLOC (source lines of code) in performance tests in the studied projects was 246, while the median SLOC of functional tests was 3980. Interestingly, performance-sensitive projects do not seem to have more or larger performance tests than non-performance-sensitive projects.

Finding # 3 - There is no standard for the organization of performance tests. 
In 52% of the projects, the performance tests are scattered throughout the functional test suite. In 9% of the projects, code comments are used to communicate how a performance test should be executed. For example, the RangeCheckMicroBenchmark.java file from the nbronson/snaptree project contains the following comment:

/*

* This is not a regression test, but a micro-benchmark.

*

* I have run this as follows:

*

* repeat 5 for f in -client -server;

* do mergeBench dolphin . jr -dsa\

*       -da f RangeCheckMicroBenchmark.java;

* done

*/

public class RangeCheckMicroBenchmark {

...

}

In four projects, we even observed that code comments were used to communicate the results of a previous performance test run.

Finding # 4 - Most projects have performance smoke tests. 

We identified the following five types of performance tests in the studied projects:

1. Performance smoke tests: These tests (50% of the projects) typically measure the end-to-end execution time of important functionality of the project.
2. Microbenchmarks: 32% of the projects use microbenchmarks, which can be considered performance unit tests. Stefan et al. [4] studied microbenchmarks in depth in their ICPE 2017 paper.
3. One-shot performance tests: These tests (15% of the projects) were meant to be executed once, e.g., to test the fix for a performance bug.
4. Performance assertions: 5% of the projects try to integrate performance tests in the unit-testing framework, which results in performance assertions. For example, the TransformerTest.java file from the anthonyu/Kept-Collections project asserts that one bytecode serialization method is at least four times as fast as the alternative.
5. Implicit performance tests: 5% of the projects do not have performance tests, but simply yield a performance metric (e.g., the execution time of the unit test suite). 

The different types of tests show that there is a need for performance tests at different levels, ranging from low-level microbenchmarks to higher-level smoke tests.

Finding # 5 - Dedicated performance test frameworks are rarely used. 

Only 16% of the studied projects used a dedicated performance test framework, such as JMH or Google Caliper. Most projects use a unit test framework to conduct their performance tests. One possible explanation is that developers are trying hard to integrate their performance tests into the continuous integration processes. 

The main takeaway of our study

Our observations imply that developers are currently missing a “killer app” for performance testing, which would likely standardize how performance tests are conducted, in the same way as JUnit standardized unit testing for Java. An ubiquitous performance testing tool will need to support performance tests on different levels of abstraction (smoke tests versus detailed microbenchmarking), provide strong integration into existing build and CI tools, and support both, extensive testing with rigorous methods as well as quick-and-dirty tests that pair reasonable expressiveness with being fast to write and maintain even by developers who are not experts in software performance engineering.

References

[1] M. Aberdour. Achieving quality in open-source software. IEEE Software. 2007.

[2] C. Heger, J. Happe, and R. Farahbod. Automated Root Cause Isolation of Performance Regressions During Software Development. In Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering (ICPE). 2013.

[3] P. Leitner and C.-P. Bezemer. An exploratory study of the state of practice of performance testing in Java-based open source projects. In Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering (ICPE). 2017. 

[4] P. Stefan, V. Horky, L. Bulej, and P. Tuma. Unit testing performance in Java projects: Are we there yet? In Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering (ICPE). 2017.

If you like this article, you might also enjoy reading:

[1] Jose Manuel Redondo, Francisco Ortin. A Comprehensive Evaluation of Common Python Implementations. IEEE Software. 2015.

[2] Yepang Liu, Chang Xu, Shing-Chi Cheung. Diagnosing Energy Efficiency and Performance for Mobile Internetware Applications. IEEE Software. 2015.

[3] Francisco Ortin, Patricia Conde, Daniel Fernández Lanvin, Raúl Izquierdo. The Runtime Performance of invokedynamic: An Evaluation with a Java Library. IEEE Software. 2014.

On the rise of Casual Contributions in postmodern Software Development - How does it impact Software Quality

By: Naveen N Kulkarni
Associate Editor: Sridhar Chimalakonda (@ChimalakondaSri)

Postmodernism, a term used in variety of disciplines like arts, literature, fashion, music, movies and technology, is used to describe a tendency of philosophical movements and resulting shifts where there is a rejection of the predecessor. In case of software engineering too, the postmodernistic views have challenged the rigidity of traditional approaches; much like agile methods challenged the need for upfront requirement gathering, perpetual beta cycles where users are considered co-developers or heterogeneous application stack replaced with homogeneous stacks like Node.js and Vert.x. In case of collaboration in software development we see breaking the norm of controlled contributions to shared contributions to fragmented contributions combined with social underpinnings.

In context of Open Source Software (OSS), empirical studies have shown that developers aim to rapidly deliver ’credible promises’ to keep their projects viable and active [Sojer, Haefliger]. With social interactions gaining popularity on the Internet, casual contributions (popularly known as pull requests on platforms such as, GitHub and BitBucket) to OSS is increasing. Empirical studies suggest that such contributions are not trivial, but involves bug fixes, refactoring and new features. While an OSS can deliver its promise faster with such requests, it can can quickly overwhelm the developers evaluating them whether to accept or not in case there are many. In order to deliver faster, it is intuitive that developers choose project relevance and code quality as the primary factors to evaluate the pull requests (substantiated in a recent survey by Gousios et al.). To minimize the effort, many OSS projects use strict policies that include discussions, linting, clean merge, tests and code reviews; most of which are automated through continuous integration tools. As the casual contributions can be unplanned and opportunistic, attempts are made by the development 1 2 team to understand the context of contribution through prior discussions. For example, in case of Scala, pull requests are accepted only if it is discussed on ’scala-internals’ mailing list. Also, mandatory discussions with more than one developers is required depending on the complexity of the contribution. All these are undertaken as a part of the quality assessment of the casual contribution. However, with such extensive checks and large volumes, the pull requests can quickly get stale, making it even harder for developers and contributors to understand the impact of composing multiple parallel contribution.

Survey of work practices in managing and integrating pull request suggest that every contribution requires multiple code review sessions. This is substantiated by the fact that code reviews require ”code reading” to find defects, which is greatly dependent on individual expertise. Unfortunately, though they are perceived as a best practice, they are often localized and informal. Today’s code review session include interactive sessions, discussions and in-line comments. A study of modern code review process show that 35% of the review suggestions are discarded and 23% of the changes are applied after review (Beller M et al.). It is suggested that this process can be more people independent in postmodern software engineering. On the quality, there are very few qualitative evidences relating to code review process. Recent study by McIntosh et al. suggest that review coverage as an important factor to ensure quality. However, quality criteria are subjective during code reviews. Quality criteria is often restricted to statically analyzed source code using popular continuous integration tools (like Sonar or Squale), coding style and test cases.

Despite the challenges, we observe casual contributions are raising. For example, at the time of writing this post, the rails and redis projects have nearly 500 open pull requests. The bootstrap, npm, tensorflow and docker have nearly 100 open pull requests. More detailed analysis on the growth of pull requests can be found at http://ghtorrent.org/pullreq-perf/ and https://octoverse.github.com/. As we adopt different approaches to compose software with frequent fragmented contributions, we are faced with challenges, such as emergent behavior, constraint violations and conflicting assumptions. These challenges arise due to the parallel and isolated nature of the contributions. Though these challenges are not new, we believe the current practices on quality assessments are inadequate to address them. The techniques used today help developers quickly spot local mismatches but they are not effective for comprehending global mismatches that originate due to many complex relationship among the software elements. In our initial study, we found similar evidence when a opportunistically reused external module not only violated the existing constraints, but also added unanticipated behavior making the software restrictive.

Attempts to address the global mismatches using architectural reasoning and design decisions approaches have so far has met with limited success. Also, during code reviews are insufficient to highlight the ramifications of a contribution on the global constraints and mismatches. To overcome this issue, the merge process should extend beyond syntactic merge to include techniques for identifying constraint mismatches. We believe, by validating the constraint synthesized as predicates from the hot paths (critical execution flow) can optimally help validating the global mismatches. In past, predicate abstractions are effectively used for software verification (SLAM). On one hand, the predicate abstraction can facilitate theoretical guarantees, synthesizing the predicates can be overwhelming due to state explosion. To overcome this we can use descriptive statistical models (created from the features available in the source code) to choose a subset of predicates sufficient for the verification. Alternatively, mining of software repository techniques can play pivotal role in developing alternative approaches where they not only provide factual data, but help in decision process. Mining constraints as a set of sequential patterns (SPADE, a popular sequence mining algorithm) can be used where comparing sequence for dissimilarities between the source codes can suggest mismatches.

There is a need for lightweight processes to cope with the dramatic shifts in the postmodern software engineering. In case of casual contributions, defining code quality can be confounding due to their inherent isolated nature. So, code reviews are critical and there is a need for alternative approaches that account for the various postmodern shifts. Without this the software quality in postmodern software development will remain elusive.

References

• SojerM, Henkel J. Code reuse in open source software development: quantitative evidence, drivers, and impediments. Journal of the Association for Information Systems 2010; 11(12):868901.
•Stanciulescu S, Schulze S,Wasowski A. Forked and integrated variants in an open-source firmware project. Proceedings of the IEEE Int’l Conference on Software Maintenance and Evolution, ICSME 15, Bremen, Germany, 2015; 151160.
• Perry DE, Siy HP, Votta LG. Parallel changes in large-scale software development: an observational case study. ACM Transactions on Software Engineering and Methodology (TOSEM) 2001; 10(3):308337.
• Gousios G, Zaidman A, Storey MA, Deursen Av. Work Practices and Challenges in Pull-Based Development: The Integrators Perspective. Proceedings of the 37th International Conference on Software Engineering, 2015, vol. 1, pp. 358368.
• Scala Pull request Policy, http://docs.scala-lang.org/scala/pull-request-policy.html, accessed 15-Mar-2017.
• Pinto G, Steinmacher I, Gerosa MA. More common than you think: An in-depth study of casual contributors. Proceedings of the 23rd IEEE Intl Conference on Software Analysis, Evolution, and Reengineering, SANER 16, Suita, Osaka, Japan, 2016; pp. 112123. 4
• Beller M, Bacchelli A, Zaidman A, Juergens E. Modern Code Reviews in Open-Source Projects: Which Problems Do They Fix? In Proceedings of Mining Software Repository, MSR’14, Hyderabad, India, 2014.

Insights for practitioners into empirical software architecture research

By: Matthias Galster, Danny Weyns

Associate Editor: Mehdi Mirakhorli (@MehdiMirakhorli)

Software architecture is an applied discipline. Therefore, research in that area should eventually offer practitioners useful and trustworthy results.

What we did: To find out more about the state of empirical software architecture research and its relevance for practitioners, we looked at research papers published between 1999 and 2015 at the most popular architecture conferences:

·       CBSE (International ACM SIGSOFT Symposium on Component-based Software Engineering)

·       ECSA (European Conference on Software Architecture)

·       QoSA (International ACM SIGSOFT Conference on the Quality of Software Architectures)

·       WICSA (Working IEEE/IFIP Conference on Software Architecture)

We went through a total of 667 “full” papers out of which 115 (i.e., around 17%) turned out to be “empirical”. In other words, 17% presented research based on systematically collected evidence from observation or experience [1], using methods such as case studies, experiments or surveys (the remaining 83% of papers present solutions or new methods without evaluation, experience reports, tools, etc.). The full study has been published as a research paper at WICSA 2016 [2] and a preprint is available [5]. In this post, we briefly highlight some of the insights.

Insight 1 – Amount of empirical work is increasing: Since around 2010, there is an increasing trend to publish empirical research in software architecture. The figure below shows the relative contribution of empirical papers compared to all papers published in a year.

Empirical methods are used equally to (a) to study phenomena (e.g., to understand how architecture decisions are made in practice) and (b) to evaluate approaches (e.g., to evaluate a new approach for capturing architectural knowledge). Therefore, current empirical research helps us understand software architecture practice (for example, to identify the real problems that research should tackle), but also contributes to increasing the confidence in the validity of newly proposed approaches and methods.

Insight 2 – Involvement of practitioners as researchers/co-authors is limited: On average, every two papers have one industry author but five authors from academia. Around 70% of papers have no industry author at all. Overall, empirical papers have roughly the same number of authors from industry than non-empirical papers. Little active involvement of practitioners (assuming that co-authors are actively involved in the research presented in a paper) might be due to challenges related successful industry-academia collaborations (e.g., champions in industry organizations, buy-in support commitment to contribute to industry needs) [3].

Insight 3 – Involvement of practitioners as study participants increases: Half of the empirical studies involve humans as subjects or study participants (e.g., in as participants in case studies or respondents of surveys). Most of these studies (68%) involve practitioners from industry (with the exception of experiments where students are the dominating participants). This involvement of practitioners is one way to increase the applicability and relevance of research findings for practice.

Insight 4 – Trustworthiness and rigor vary: Most studies (60%) discuss validity threats and acknowledge limitations. On the other hand, we should be careful when interpreting research findings: Many so-called “evaluations” presented in papers are often only illustrations using toy examples. Also, replications to confirm or refute previous findings and to build our confidence in research outcomes are almost non-existent in software architecture research. There might be a misconception in the research community that replications have little scientific value.

Insight 5 – There is a potential mismatch between research topics and trends in industry: Component selection and composition, architecture reasoning and decisions are dominating themes in empirical papers in the period 1999-2015. Michael Keeling in his reflection on emerging trends at SATURN [4] found architecting for DevOps, flexible design, lightweight architecture design methods and a renewed interest in architecture fundamentals as trends at SATURN. We couldn’t find these topics as trends in empirical papers.

Summary: Applying empirical methods puts an increasing demand on planning, conducting and evaluating research and its results, and presenting it in a way that is accessible and meaningful for practitioners. On the other hand, empirical research allows practitioners to get actively involved in collaborations with academics, both to explore emerging phenomena of interest (and to find out about the real problems in software architecture practice) and to evaluate new approaches that address practically relevant problems.

References:

1. D. Sjoberg, T. Dyba, and M. Jorgensen, "The Future of Empirical Methods in Software Engineering Research," in Future of Software Engineering (FOSE) Minneapolis, MN: IEEE Computer Society, 2007, pp. 358-378.
2. M. Galster and D. Weyns, "Empirical Research in Software Architecture - How far have we come?," in 13th Working IEEE/IFIP Conference on Software Architecture (WICSA) Venice, Italy: IEEE Computer Society, 2016.
3. C. Wohlin, A. Aurum, L. Angelis, L. Philips, Y. Dietrich, T. Gorschek, H. Grahn, K. Henningsson, S. Kagstrom, G. Low, P. Rovegard, P. Tomaszewski, C. Van Toorn, and J. Winter, "The Success Factors Powering Industry-Academia Collaboration," IEEE Software, vol. 29, pp. 67-73, 2012.
4. M. Keeling, "Lightweight and Flexible - Emerging Trends in Software Architecture from the SATURN Conferences," IEEE Software, vol. 32, pp. 7-11, 2015.
5. https://people.cs.kuleuven.be/danny.weyns/papers/2016WICSA.pdf

You may also like:

• F. Shull, "Research 2.0?," in IEEE Software, vol. 29, no. 6, pp. 4-8, Nov.-Dec. 2012.
  doi: 10.1109/MS.2012.164.
• Colin Potts. 1993. Software-Engineering Research Revisited. IEEE Software 10, 5 (September 1993), 19-28. DOI=http://dx.doi.org/10.1109/52.232392
• Tore Dyba, Barbara A. Kitchenham, and Magne Jorgensen. 2005. Evidence-Based Software Engineering for Practitioners. IEEE Software, 22, 1 (January 2005), 58-65. DOI=http://dx.doi.org/10.1109/MS.2005.6

Has empiricism in Software Engineering amounted to anything?

Associate Editor: Bogdan Vasilescu, University of California, Davis. USA (@b_vasilescu) 

A 2014 talk by Jim Larus, currently (after a long tenure at Microsoft Research) Dean of the School of Computer and Communication Sciences at EPFL, was circulated recently over email among some colleagues. In the talk, Larus reflects on success and failure stories of software engineering (SE) tools used at Microsoft over the years: interestingly, he lists the empirical work on data mining and failure prediction, from Microsoft Research, as one of the biggest successes, together with Windows error reporting; in contrast, Larus mentions both PREfast (static analysis) and SDV (formal verification) as examples of tools that didn't matter as much.

Needless to say, this sparked quite the controversy among participants in my email thread, about the value (or lack thereof) of empiricism in SE. I found the discussion thoughtful and useful, and I’d like to share it. This blog post is an attempt to summarize two main arguments, drawing mostly verbatim from the email thread participants. Active participants included, at various times, Alex Wolf, David Rosenblum, Audris Mockus, Jim Herbsleb, Vladimir Filkov, and Prem Devanbu.

Is empiricism just blind correlation?

The main criticism is that empirical SE work does not move past mere correlations to find deeper understanding, yet it's understanding that leads to knowledge, not correlation. The analysis of data should enable understanding, explanation, and prediction, not be an end in and of itself. The accessibility of correlation among statistical methods, coupled with the flood of data from open-source development and social programming websites, have created a recipe for endless paper milling. As a result, many empirical SE papers usually don’t ask any critically important research questions, that matter both to researchers and practitioners; instead, they present obvious, expected, and unactionable insights, that neither researchers could exploit to develop new or improved approaches, nor practitioners to produce guaranteed improvements to their development efforts.

In response to this criticism, three counterexamples:

• Are automated debugging techniques actually helping programmers?

C Parnin, A Orso. ISSTA ’11 PDF

• On the naturalness of software

A Hindle, ET Barr, Z Su, M Gabel, P Devanbu. ICSE ’12, CACM Research Highlights ’16 PDF

• Are mutants a valid substitute for real faults in software testing?

R Just, D Jalali, L Inozemtseva, MD Ernst, R Holmes, G Fraser. FSE ’14 PDF

Arguably, the successes at Microsoft and other places, with saving money on quality control using process metrics, have led to a sort of Gold Rush and probably some excesses. However, it is clear that empirical SE has advanced leaps and bounds since the early correlation days. Admittedly, earlier work was trying to understand the usefulness of the data and stats, and resulted in some fluff; perhaps like with a new gadget, people were more interested in the technology than how useful it is in daily lives. However, thanks in part to all that playing around, we now have lots of solid work, and openness of the community to scientific methods.

Moreover, these days, to pass muster at top conferences, one needs to have a good theoretical grounding, not only on past work in SE & PL, but (often) also in sociology, psychology, management science, and behavioral science -- creating and maintaining software is inherently human-centric. Theory helps to predict and explain results; frame future research questions; provide continuity across studies and research groups; accumulate results into something resembling knowledge; and it points toward important open questions. For example, there is a lot of theory in psychology about multitasking and context switching, that really helps to explain the quantitative results relating productivity to patterns of work across projects. Such theory can both push further theory development as well as work toward deeper explanations of the phenomena we study. What exactly happens when humans context switch, why is context switching costly, and why do costs vary in different conditions?

This is also why correlations have largely fallen out of favour. Careful consideration of the relevant phenomena (in the context of existing work) will lead one to use, e.g., more sophisticated mixed-effects / multivariate models and bootstrapped distributions, and to carefully argue why their experimental methods are well-suited to validate the theory. The use of these models is driven precisely by more sophisticated theories (not statistical, but rather SE / behavioral science theories).

Can empiricism be valuable without fully understanding the “why”?

Although the demand for understanding why something works is prima facie not entirely unreasonable, if one insisted on that, then many engineering disciplines would come to a halt: e.g., airfoil design, medicine, natural language processing (no one really understands why n-gram models work better than just about anything else). Knowing that something works or doesn’t can be quite enough to help a great many people. We have many extremely useful things in the world that work by correlation and “hidden” models. For example, theoretically sound approaches such as topic models (e.g., LDA) don’t work well, but highly engineered algorithms (e.g., word2vec) work fantastically. By insisting on why, one would: stop using SVM, deep learning, many drugs; not fly on airplanes; and not use Siri/Echo, because in many cases the why of these things is not well understood. Even such a trivial thing as GPS has a lot of engineering adjustments that have not been derived from first principles but just work.

However, without explicit understanding of why things work, it’s not clear how far one can go with engineering. For example, many ancient civilizations were able to predict the phases of the moon and positions of the stars without understanding the why of gravity and orbital mechanics, simply based on repeated measurement. But would that have gotten us satellite communication, planetary exploration, and the Higgs boson? Similarly, would you feel comfortable in a plane that lands using a machine-learned autoland system?

Closing remarks

In summary, it seems there is a central place for empirical work in SE; not as an end in itself, but rather the beginning of an inquiry leading to understanding and useful insights that can we can use to truly advance our discipline. See also Peggy Storey's Lies, Damned Lies and Software Analytics talk along these lines.

Using Biometrics to Assess a Developer's Cognitive and Emotional States

by Sebastian Müller (@smueller9, smueller@ifi.uzh.ch) and Thomas Fritz (fritz@ifi.uzh.ch)
Associate Editor: Christoph Treude (@ctreude)

What difficulties and emotions do software developers experience during their work? When is a developer "in the flow" and making a lot of progress? So far, little is known about these kinds of questions or the cognitive and emotional states of developers during work. Researchers have predominantly focused on the tangible output of developers' work, such as the source code, to identify bug-prone parts. Yet, a better understanding of what developers experience in their work would allow us to provide better and instantaneous support to developers, for instance by avoiding costly interruptions when a developer is in the flow or by intervening when a developer perceives particularly high difficulty levels and might be close to implementing a bug into the code.

Recent advances in biometric (aka psychophysiological) sensor technology in combination with research in psychology show promise that biometric data can fill this gap and provide answers to some of these kinds of questions. For instance, measures of a person's skin conductivity (aka EDA)—how much a person is sweating—can tell us something about the person's stress level and cognitive effort, but also perceived emotions.

In our research, we investigate the use of biometric measurements in the context of software engineering. In particular, we conduct studies ranging from short controlled lab studies to several week long field studies in which we examine the link between a developer's biometric measures and cognitive and emotional aspects of a developer while working on a change task, such as the perceived difficulty of source code elements or tasks, and the emotions and progress the developer experiences. Our vision is to use biometric measurements to gain a better understanding of what developers experience in their work and to use this improved understanding to better support developers.

To study the potential of biometrics to predict developers' perceived progress and emotions (valence), for example, we conducted a lab study [1] with 17 participants from two companies and one university. For this study, we collected biometric data, such as EDA, or the heart rate of participating developers while they were working on two different change tasks. At the same time, we asked developers in periodic intervals to self-report their perceived progress and emotions, specifically valence and arousal (on two scales from -200 to +200). The following figure highlights some of the collected data, with the biometric EDA and heart rate data on top and the self-reported valence and progress on bottom.

The figure also illustrates a visible difference in the biometric data, in particularly in the EDA signal, between the first episode that was rated as medium progress and higher valence compared to the last episode with the developer reporting being stuck and a lower valence. To analyze the data, we performed several steps of data cleaning, noise filtering and feature extraction, before we could feed these features into a machine learning classifier. The results of our analysis show that biometric data can be used well to distinguish between positive and negative emotions, as well as between phases of low and high progress, in most cases.

Overall, the results of our three studies ([1], [2], [3]) show that we are able to predict aspects of developers' cognitive and emotional states in real-time using biometrics. The results also show that we are able to make accurate predictions in the field in the common work environment of professional software developers and over multiple days. Not surprisingly, biometric data is very sensitive to the individual and certain biometric measures work better for some than for others. By using an approach that takes into account these individual differences more explicitly, we should be able to further improve our results in the future.

So what can we do with these insights? Similar to other metrics, such as code metrics or interaction metrics, biometrics can be leveraged to provide better support to developers, even in real-time. The results of our latest study [3] show that we can use biometrics to predict where quality concerns are or will occur while developers are still working on the change task, and that the biometrics even outperform more traditional metrics. These insights can now be used to enhance existing tool support, such as the automatic detection of quality concerns, to highlight and rank places in the code that should be reviewed and/or refactored, or even to indicate to the developer when he/she should reconsider a change to prevent him/her from ever submitting code with errors to the repository. Similarly, we can use biometrics to provide better interruption management support for developers or to provide recommendations when a developer perceives difficulty, ranging from talking to a coworker to taking a break.

In summary, the results of our studies are promising and provide initial evidence that biometrics can be used to assess cognitive and emotional states of a developer and thus help answer some of the open questions. As one example, our results show that we might be able to support developers when they are experiencing difficulties and help them to fix quality concerns early on. With the rapid and continuous advances in sensor technologies and the increasing pervasiveness of smart wearable devices that already collect biometric data, we might soon be able to leverage this kind of data for each professional software developer and fine-tune our classifiers to further increase their accuracy. At the same time, this opens up many ethical and privacy concerns that will have to be addressed and investigated in further studies.

References

1. Sebastian Müller and Thomas Fritz. Stuck and Frustrated or In Flow and Happy: Sensing Developers' Emotions and Progress. In Proceedings of the 37th International Conference on Software Engineering (ICSE 2015), Florence, Italy
2. Thomas Fritz, Andrew Begel, Sebastian Müller, Serap Yigitt-Elliott, and Manuela Züger. Using psycho-physiological measures to assess task difficulty in software development. In Proceedings of the 36th International Conference on Software Engineering (ICSE 2014), Hyderabad, India
3. Sebastian Müller and Thomas Fritz. Using Bio(Metrics) To Predict Code Quality Online. In Proceedings of the 38th International Conference of Software Engineering (ICSE 2016), Austin, USA

If you like this article, you might also enjoy reading:

• Graziotin, D., Wang, X., and Abrahamsson, P. Software Developers, Moods, Emotions, and Performance. In IEEE Software, vol.31, no.4. 2014
• Ikutani, Y. and Uwano, H. Brain activity measurement during program comprehension with NIRS. In Proceedings of the International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing. 2014.
• Nakagawa, T., Kamei, Y., Uwano, H., Monden, A., Matsumoto, K., and German, D. M. Quantifying programmers' mental workload during program comprehension based on cerebral blood flow measurement: A controlled experiment. In Companion Proceedings of International Conference on Software Engineering. 2014.
• Parnin, C. Subvocalization - toward hearing the inner thoughts of developers. In Proceedings of the International Conference on Program Comprehension. 2011.
• Siegmund, J., Kästner, C., Apel, S., Parnin, C., Bethmann, A., Leich, T., Saake, G., and Brechmann, A. Understanding understanding source code with functional magnetic resonance imaging. In Proceedings of the 36th International Conference on Software Engineering. 2014.

The impact of FOSS on software engineering research

by Jesus M. Gonzalez-Barahona
Associate Editor: Stefano Zacchiroli

Back in year 2000, during Christmas, I had some time to play. Some days earlier, by chance, I had read "Estimating Linux's Size", a study on the number of lines of code of all the software in Red Hat 6.2, one of the most popular Linux-based distribution of the age. I had noticed that the tool used in the study, sloccount, was FOSS (free, open source software), so I could use it in my playground. For some days, my laptop was busy downloading all of Debian GNU/Linux 2.2 (Potato) source code, running sloccount on it, and producing all kinds of numbers, tables, and charts. At that time Debian was one of the largest, if not the largest, coordinated collections of software ever compiled. That fun lead to a series of paper, starting with "Counting potatoes: the size of Debian 2.2", and to a pivot in my research career that put me in the track of mining software repositories.
This personal story is an illustration of two emerging trends that proved to be game-changers: the public availability of information about software development, including the source code, and the availability of FOSS tools to retrieve and analyze that information. Both were enabled by FOSS, but they have worked in very different, yet complementary, ways.

FOSS as a matter of study

Everybody knows nowadays that FOSS projects release publicly the source code they produce. This has tremendously lowered the barriers to study software products, when that study requires access to their source code. No longer it is compulsory to have special agreements with software companies to research in this area. For the first time in the software history, it is possible and relatively easy to run comparative studies on large and very large samples of products.
Of course, the impact of these new opportunities has been gradual. As FOSS has become more and more relevant, more "interesting", industry-grade products have been available for studying. With time, researchers have developed new techniques and tools, and even a kind of a new mindset, for analyzing this new, yet very rich, corpus of cases.
Among FOSS projects, there is a large subset using an open development model. They keep public repositories for source code management, issue tracking, mailing lists and forums, etc. For them not only the source code, but also a lot of very detailed information about the development process is available. From the researcher point of view, this means a heaven of data to explore.
Already in the late 1990s some researchers started to take advantage of this new situation, but it was in the early 2000s when this new approach started to take shape, with works such as "Evolution in open source software: a case study", "Results From Software Engineering Research Into Open Source Development Projects Using Public Data","A Case Study of Open Source Software Development: The Apache Server", and "Open source software projects as virtual organizations".
Soon, specialized venues that dealt mostly with the study of data from software development repositories emerged. Among them, the International Working Conference on Mining Software Repositories, held annually since 2004, was one of the first and is today probably the most well known. During the last decade, this research approach has permeated all major conferences and journals in software engineering.

FOSS tools and public datasets as enablers of research

The availability of FOSS tools has been an enabler in many research fields, and software engineering has been no exception. The availability of efficient and mature FOSS databases, analysis tools, etc., allows any researcher, even with very modest budgets, to carry on ambitious studies with very large datasets. In addition, more and more tools specifically developed to analyze software or software repositories are available as FOSS. Many researchers are taking as customary to publish as FOSS the tools they use to produce the results shown in the data, letting others modify or adapt them to extend the research results. Some of them are useful for practitioners too.
For example, several FOSS tool sets exist to retrieve information from software repositories and organize it in databases, ready for analysis, such as MetricsGrimoire. They allow researchers to focus on the analysis of the data, and not on the retrieval, a task which may be very effort-consuming, specially when a large number of projects are involved. Practitioners are using them as well, to track KPIs (Key Performance Indicators) of their projects.
One step further, in some cases the datasets resulting from large data retrieval efforts are made public, letting researchers directly jump to analyze them. This is for example the case of the Boa Infrastructure, which allows to execute queries against hundreds of thousands of software projects very efficiently, or FLOSSMole, a repository of FOSS project data coming from several software development forges.

The impact on software engineering research

These two trends are changing large areas in the field of software engineering research, by allowing researchers to produce results in ways closer to the scientific method than before. With relatively little effort, they can start from solid data, which is becoming more and more accessible, and rely on mature, adaptable tools, to produce results that take into account a large diversity of cases. Some concrete impacts are:

• Availability of datasets. Researchers can work on the same dataset with different techniques, which eases the comparison of results, and allows to determine to which extent a study advances the state of the art. They also allow researchers to focus on analysis, and not on the assembly of the dataset itself, which is complex, error-prone, and effort-consuming.
• Reproducibility of the studies. When the tools used by researchers are FOSS, and the datasets are public, reproducing previous results becomes possible and even easy. This is already improving the chances of validating results, which is fundamental to advance on solid ground.
• Incremental research. Now it is much easier to build on the shoulders of giants, by reusing FOSS tools and datasets produced by other research teams. Researchers no longer have to start from the ground up, they can incrementally improve precious results.

In short, FOSS is making empirical software research much easier. Some clear examples are the advances in the areas of detection of code clones, the impact of different release models, or the limits to software evolution. The next years will let us know to which extent this is going to translate into results useful to improve our knowledge of how software is developed, and to improve software development itself.

References

"Estimating Linux's Size", David A. Wheeler, published online, November 6, 2000.
"Counting potatoes: the size of Debian 2.2", González-Barahona, Jesus M.; Pérez-Ortuño, Miguel; de las Heras Quirós, Pedro; Centeno-González, José; Matellán-Olivera, Vicent. Upgrade Magazine, 2(6), 60-66 (2001).
"Evolution in open source software: a case study", Godfrey, M.W.; Qiang Tu. Proceedings od the International Conference on Software Maintenance, 11-14 Oct. 2000, San Jose, CA, USA, 131-142.
"Results From Software Engineering Research Into Open Source Development Projects Using Public Data", Koch, Stefan; Georg Schneider, published online, 2000.
"A Case Study of Open Source Software Development: The Apache Server", Mockus, Audris; Fielding Roy; Herbsleb James.Proceedings of the International Conference on Software Engineering (ICSE 2000), 4-11 Jun. 2000, Limerick, Ireland.
"Open source software projects as virtual organizations: Competency rallying for software development", Crowston, K.; Scozzi, B. Proceedings of IEE Software, Feb. 2002, 149(1), 3-17.