What should practitioners know about user involvement in software development?

By: Didar Zowghi, Muneera Bano (@DidarZowghi @DrMuneeraBano) 
Associate Editor: Muneera Bano (@DrMuneeraBano)

In the neo-humanist approach, software is designed to support and improve the working environment of its intended users. In practice, this means involving them in the development process and not to treat them as mere consumers of the software. 

But is it really important to involve users in software development? To what extent, and at what stages of software development, should they participate? How much power should the users be given in software development decision making?  These questions have been asked by researchers and practitioners for decades and together with many related subjects, they have been the topic of a great deal of research.  

Users’ involvement (UI) during software development has been claimed to link to users’ satisfaction with the resulting system, that in turn leads to the assertion of system success. However, much of the empirical evidence to date shows that this connection between UI and system success is not ubiquitous. Although much of research in this area has revealed that involving users in software development contributes positively to system success, it is also been observed that UI is indeed a double-edged sword and can equally create problems rather than benefits.

In theory, Users’ involvement in software development and system success’ is a complex combination of four different concepts that have been studied and analyzed separately and in combination. Over five decades of this investigation enormous amount of debate and disagreement has been published both in software engineering and information systems research literature. 

Like others, we have been curious about so much interest and conflicting results on the topic of users’ involvement in software development. In the last 6 years, we have conducted empirical longitudinal studies to explore: what has been reported in research literature? what are the problems and challenges of UI? how does users’ satisfaction with their involvement evolve? and what useful theories could be developed about the link between UI and system success?. Furthermore, we have been wondering about what impact has Agile software development had on addressing the problems and challenges of UI. So, we also explored the alignment of stakeholder Expectations about UI in agile development. Finally, being convinced that UI is clearly a multi-faceted and communication rich phenomenon, we conducted a case study of organizational power and politics in the context of UI in software development.

The overall findings from our longitudinal study have resulted in several key observations and the development of a theory.  First, we observed that system success is achievable even when there are problems and challenges in involving users. User satisfaction significantly contributes to the system success even when schedule and budget goals are not met. Second, there are additional factors that contribute to the evolution of user satisfaction throughout the project. Users’ satisfaction with their involvement and the resulting system are mutually constituted while the level of user satisfaction evolves throughout the stages of the development process. Third, organizational and project politics are significant factors often used to exert power and influence in decision-making processes. Communication channels are frequently exploited for political purposes. These contribute to the users’ dissatisfaction with their involvement thus impacting on the project outcome. Fourth, varying degrees of expectation misalignments exist between the development team and users in Agile software development projects.

Our findings enabled us to provide useful hints for practitioners. Understanding the nature of the problems related to UI helps the project managers to develop appropriate strategies for increasing the effectiveness of user involvement. These management strategies and appropriate level and extent of user representation are essential elements of maintaining an acceptable level of user satisfaction throughout the software development process. When there are multiple teams of stakeholders with different levels of power in decision-making, politics is inevitable and inescapable. Without careful attention, the political aspect of user involvement in software development can contribute to an unsuccessful project.

SEABED: An Open-Source Repository of Software Engineering Case Studies

by Veena Saini (NIT Jalandhar, India), Deepti Ameta (DAIICT, India), Ashish Sureka (Ashoka University, India), Paramvir Singh (NIT Jalandhar, India) and Saurabh Tiwari (DAIICT, India)

Associate Editor: Sridhar Chimalakonda (@ChimalakondaSri)

Case-Based Learning (CBL) is a non-traditional teaching methodology which helps in enhancing the learning skills. It helps in the deeper understanding of the concepts; involves students in active learning, enhances collaboration among team members, helps in nurturing analytic and interpersonal skills through well-defined cases. We looked for most prevalent teaching pedagogies being used in different domains like Medical, Law, and Business education that require brainstorming and we found CBL to be an effective approach in enhancing students learning levels by helping them to think in different perspectives for the same problem.

Why CBL for Software Engineering?

Software Engineering is a highly practice oriented field, concerned with the development, operation, and maintenance of software, hence providing practical knowledge for its applicability in real world becomes equally important. Software development process is primarily based on a set of well-defined requirements in which understanding and prioritizing them is important for the success of any product. It is very important to bridge the gap between the theoretical concepts at university level and practical implementation of the concepts at industry level, hence we found CBL to be an effective approach in teaching some of the SE concepts as it is a student-centric teaching methodology which focuses on learning objectives like critical thinking, brainstorming, discussions, and understanding real life scenarios with multiple perspective solutions.

Our progress to bring CBL into practice through SEABED

The backbone of SEABED [1] is the variety of cases and vibrant SEABED community. We are building up our SEABED community by interacting with SE practitioners, enthusiasts and researchers all over the globe. Gradually SEABED community is growing up with potential users. Along with all the researchers and SE enthusiasts, we also welcome thesis students to contribute and take this learning platform to a new level. Our idea was to create a web-based platform as a hub for SE CBL practitioners. Teachers, students, researchers, instructors, or anyone can be benefited from CBL and can find relevant cases from our open-source case repository in Case Collection section. Cases are categorized according to the various phases of Software Development Life Cycle (SDLC).

It allows the interested users to contribute with their own case problems as PDF document (Case Submission).  They can also upload the experience report about the CBL session, its empirical analysis and students’ feedback after conducting some CBL sessions in the University. They can revise any existing case available at SEABED, by providing proper justification document that tells the rationale behind the amendments (Case Evolution). 

Case Example: Twitter started as a side project of Odeon Inc. in 2006. It had immense growth nearly 1000% growth/year and soon became the micro blogging platform of choice for majority of Internet users. It had 400,000 tweets per quarter in 2007. This grew to 100 million tweets per quarter in 2008. By 2010 there were around 175 million users, 90 million tweets per day and around 500 million searches per day. Twitter was initially built with time to market in mind. So the architecture and technology to build twitter was chosen such that they can build the site in a very short time. Twitter was not designed with this kind of growth in mind. Your team has to come up with a new architecture that addresses the scalability problems of twitter.

Challenges 1: What are the architectural drivers, assumptions and major constraints? Give details of at least 5 decisions related to major architectural strategies. • 2: Give the architecture in terms of system decomposition (as a diagram and text), structure, and connector and component responsibilities.

This is a sample case [2] for Software Engineering (SE) case-based learning (CBL) practice with challenging questions.

CBL Implementation at different universities

Experiment 1 - IIIT Hyderabad, India: We begin our journey when CBL was first implemented at IIIT Hyderabad for practicing SE concepts. They proposed a case based learning environment - Case Oriented Software Engineering Education model (COSEEd). It embeds problem solving as a core skill with cases as the primary learning objects. We implemented 4 course offerings with COSEEd considering UG, PG, and PGSSP (Industry participants). In all the offerings, students felt that CBL helped them in gaining learning objects along with an improvement in their communications skills. This model was successful in covering cognitive goals in SE education. The research work can be reached at [3].

Experiment 2 - NIT Jalandhar, India:

Here CBL session was conducted for 89, 3^rd year UG students. CBL is found to be effective with an agreement of 74.53% students, who were able to understand all five learning principles of learning, critical thinking, engagement, communication skills, and team work. Experience report [4] of this implementation is available at SEABED. After this exercise, SEABED platform was created and we proposed a case writing template. The research work can be reached at [5].

The architecture of SEABED is given below:

Experiment 3 - DAIICT Gandhinagar, India:  

The CBL was introduced in two different SE courses, Software Testing and Requirements Engineering, with a number of CBL sessions. Firstly, we applied CBL for Software Testing Discipline and got appreciable responses at DAIICT, Gandhinagar, India. This was the first time when students were introduced with the concepts of CBL. This drove us to conduct more CBL sessions at DAIICT and the second time we applied CBL for practicing Requirements Engineering Concepts by providing two different RE cases [6][7] to student teams and asked them to bring solutions (PowerPoint presentations) to check the effectiveness of CBL by evaluating their results on different parameters. We also intend to perform several experiments on the same. After every CBL session we asked student teams and Teaching Assistants (TAs) to give feedback and we got positive responses this time too. We further aim to expand this by considering new research techniques on CBL implementation.

So the overall analysis of the CBL experiments conducted at different Universities reveals that CBL teaching methodology can be used in teaching some of the Software Engineering concepts efficiently, supported by well-designed cases. It helps students’ in in solving real world problems by engaging themselves in thinking and discussing activities.

What’s next (future opportunities): Case-based learning in Software Engineering domain is effective and there are several sub-areas that are still unexplored. Our platform does not only help students to assimilate the concepts well, but also provides a promising research area for SE practitioners and enthusiasts to submit and discuss efficient cases. We further aim to expand this work by conducting more useful sessions with other universities and experimenting CBL with more detailed observations, analysis and results. We also invite MS, M.Tech and PhD scholars to collaborate with us on CBL project.

References:
[1] http://seabed.in/

[2] http://seabed.in/upload/twitter-case-study.pdf

[3] Kirti Garg, Ashish Sureka and Vasudeva Verma. 2015. A Case Study on Teaching Software     Engineering Concepts using a Case-Based Learning Environment.

[4] http://seabed.in/upload/An-Experience-Report-for-AIW-Case.pdf.

[5]V. Saini, P. Singh and A. Sureka, "SEABED: An Open-Source Software Engineering Case-Based       Learning Database," 2017. IEEE 41^st Annual Computer Software and Applications Conference           (COMPSAC), Turin, 2017, pp. 426-431. doi: 10.1109/COMPSAC.2017.204

[6] http://seabed.in/case-study/Metro_Case.pdf.

[7] http://seabed.in/case-study/LIC_Case.pdf.

[8] D. Kundra and A. Sureka, “An Experience Report on Teaching Compiler Design Concepts Using       Case-Based and Project-Based Learning Approaches,” 2016 IEEE Eighth International                         Conference on Technology for Education (T4E), Mumbai, 2016, pp. 216-219. doi:                               10.1109/T4E.2016.052.

Gazes to Traces: Tracing Developers’ Eyes to Automate Traceability Link Recovery

by Bonita Sharif, Youngstown State University (@shbonita), and Huzefa Kagdi, Wichita State University

Associate Editor: Christoph Treude (@ctreude)

There is no denying that large-scale software systems evolve.  Software developers are routinely faced with new features and bugs that drive essential changes.  Oftentimes, they are not even the original authors of the code they need to change. They need to piece together many aspects to realize these changes whilst juggling various project parameters such as quality assurance and deadlines. The knowledge space for this existential activity typically spans several artifacts from the problem (e.g., requirements) and solution (design and implementation) domains.  The explicit connectivity among these artifacts is often missing in practice [1, 2], which further adds to the developer’s plea. Questions that are left to be answered include “Where is the relevant code to this feature/bug?” or “What would be the design impact due to a change request?”.  The promise of a software traceability tool is to establish (recover) and maintain (evolve) links among artifacts as software evolves [3]. Traceability benefits are projected in several key software development tasks such as program comprehension [4, 5] and impact analysis [6, 7], which address the aforementioned questions.

Over the years, software engineering researchers have proposed techniques for automatically recovering and maintaining (explicit) traceability links among software artifacts.  The application of Information Retrieval (IR) techniques is a popular and heavily experimented choice. It is an artifact-centric approach that has an underlying model of textual similarities in software artifacts.   The expressiveness, effectiveness, and usefulness rely on the artifacts, and their state of availability and quality during the software’s lifecycle. 

What We Did

Our research direction was to investigate a human-centric approach to traceability.  The underlying premise of our approach is based on what humans look at while they are performing software-engineering tasks, including those of bug fixes and implementing new features.   The prerequisite to such an approach is that it should be unobtrusive to developers and blend into the background.  We use an eye tracker to collect developers' gazes on software artifacts while they work on their tasks within the IDE (we use Eclipse; however, the same concept applies to others). Our eye tracking infrastructure is called iTrace [8, 9]. It seamlessly works within the IDE to map eye gazes to source code elements on large files in the presence of scrolling and context switching between files. A preliminary version of iTrace is available at https://github.com/Sereslab/iTrace-Archive. An enhanced release with additional support for IDEs and Web browsers is planned for the near future.

We first did a pilot study [10] to determine the feasibility of using eye gaze for traceability link recovery. After seeing promising results, we conducted a larger, realistic study [11] with thirteen software developers who were asked to perform bug-localization tasks for eight submitted bug reports in Jabref (an open-source reference management system). The gaze-link algorithm uses gaze data of developers during the session using certain heuristics and weights. In a bug fixing task for example, the main premise is that as you get closer to fixing a bug you will focus on the part(s) of the code that is most related to the bug report, thus having more weight. We compared trace link results of our gaze-link algorithm with IR methods such as Latent Semantic Indexing (LSI) and Vector Space Model (VSM). 

What We Found

The gaze-link algorithm outperforms both LSI and VSM in terms of precision and recall with respect to the commit oracle (how the JabRef developers fixed the bug). We recorded an average precision of 55% and recall of 67% for all tasks. The gaze-link algorithm outperformed in 6 out of the 8 tasks.  Another set of developers found the links generated with iTrace to be significantly more useful than the IR links in a majority of the tasks. The gaze-link algorithm underperforms when the developer prematurely attempts to fix the bug without the adequate understanding of the bug or its solution. Interestingly and perhaps surprisingly, the gazes captured from a developer in cases where they did not quite fix the bug, but was close enough, ended up being helpful to another developer as a starting point towards eventually fixing the bug. 

See Table 1 for an example of links generated for a bug report across the different approaches at both class and method level granularity. The gaze algorithm is crucial to weeding out irrelevant and stray glances (shown in ET_raw). Results from the gaze link algorithm (ET_weighted) are more specific than the rankings from current IR methods (LSI and VSM). 

Table 1. An example of class and method level source code elements found linked to Bug ID: 1489454 titled “Acrobat Launch fails on Win 98”. Variables are shown in square brackets. ET_raw represents raw eye tracking data before the algorithm is run, ET_weighted is the result of running gaze-link on the gazes. The Commit is our oracle representing the fix by the JabRef developer.

Concluding Remarks

The eye tracker captures the gaze data without any additional effort on the part of the developers. This property allows it to be as effortless as we can get to provide traceability under the hood while developers work.  An added benefit of using the eye tracker is that we also learn about those hidden links that information retrieval methods have a hard time finding as they are related to tacit developer knowledge in various related code entities. Imagine a software development world where your gazes could inform things you do. The transparency and minimal effort required by developers makes gaze tracking an attractive possibility. It is reasonable to imagine a future in which eye trackers capture information while we work to help us with many more tasks than just software traceability. 

We refer interested readers to the research article titled “Eye Movements in Software Traceability Link Recovery” [11].

References

[1] B. Ramesh, "Factors Influencing Requirements Traceability Practice," Communications of the ACM, vol. 41, pp. 37-44, December 1998.

[2] P. Mader, O. Gotel, and I. Philippow, "Motivation matters in the traceability trenches," in RE, 2009, pp. 143-148.

[3] D. J. Palmer, "Traceability," in Software Engineering, M. Dorfman and R. H. Thayer, Eds., ed: Wiley-IEEE Computer Society Press, Los Alamitos, California, 1996, pp. 266-276.

[4] A. De Lucia, D. P. Massimiliano, and R. Oliveto, "Improving Source Code Lexicon via Traceability and Information Retrieval," IEEE Transactions on Software Engineering pp. 205-227, 2011.

[5] A. De Lucia, M. D. Penta, R. Oliveto, and F. Zurolo, "Improving Comprehensibility of Source Code via Traceability Information: a Controlled Experiment," in 14th IEEE International Conference on Program Comprehension (ICPC'06), Athens, Greece, 2006, pp. 317-326.

[6] G. Antoniol, G. Canfora, G. Casazza, and A. De Lucia, "Identifying the starting impact set of a maintenance request," in 4th CSMR 2000, pp. 227–230.

[7] A. von Knethen, "Change-Oriented Requirements Traceability. Support for Evolution of Embedded Systems," in International Conference on Software Maintenance (ICSM 02), 2002.

[8] B. Sharif and J. I. Maletic, "iTrace: Overcoming the Limitations of Short Code Examples in Eye Tracking Experiments," in 32nd IEEE International Conference on Software Maintenance and Evolution (ICSME), Technical Briefing, Raleigh, NC, USA, 2016, p. 647.

[9] T. Shaffer, J. Wise, B. Walters, S. Müller, M. Falcone, and B. Sharif, "iTrace: Enabling Eye Tracking on Software Artifacts Within the IDE to Support Software Engineering Tasks," in ESEC/FSE 2015, Bergamo, Italy, 2015, pp. 954-957.

[10] B. Walters, T. Shaffer, B. Sharif, and H. Kagdi, "Capturing Software Traceability Links from Developers' Eye Gazes," in 22nd International Conference on Program Comprehension (ICPC), Hyderabad, India, 2014, pp. 201-204.

[11] B. Sharif, J. Meinken, T. Shaffer, and H. Kagdi, "Eye Movements in Software Traceability Link Recovery," Empirical Software Engineering Journal, vol. 22, pp. 1063-1102, 2017.

Can We Trust the Stack Overflow Netiquette? Evidence-based Guidelines for Asking Good Technical Questions

by Fabio Calefato, University of Bari, ITALY (@fcalefato), Filippo Lanubile, University of Bari, ITALY (@lanubile), and Nicole Novielli, University of Bari, ITALY (@NicoleNovielli)
Associate Editor: Christoph Treude (@ctreude)

Several thousands of developers daily head to Stack Overflow (SO) for asking technical questions, hoping to receive swift help and fix the issues that they have been facing. To increase the chances of getting help from others, the SO community provides members with detailed guidelines on how to write more effective questions (e.g., see [9]). These official recommendations also include those provided by Jon Skeet, the highest reputation member, whose guidelines have become over time a de facto standard for the community [10].

For example, SO states that the site is “all about getting answers. It's not a discussion forum. There's no chit-chat.” Thus, askers are recommended to avoid “greetings and sign-offs […], as they’re basically a distraction,” which are also supposed to be edited out by other users [10]. Still, many askers finish their questions showing gratitude in advance towards potential helpers. Why do they go against this explicit recommendation? Are they just unaware of it or do they feel that having a positive attitude may attract more potential solutions?

In our work [5], we provide an evidence-based netiquette for writing effective questions by empirically validating several SO guidelines, retrieved from both the community and previous empirical studies on Q&A sites. Specifically, we analyzed a dataset of 87K questions by combining a logistic regression analysis with a user survey, first, to estimate the effect of these guidelines on the probability of receiving a successful answer and, then, to compare their actual effectiveness to that perceived by SO users.

Actionable Factors for Good Questions

Figure 1. Our conceptual framework of success factors for writing good questions in Stack Overflow. The success of a question is defined as the probability of receiving an answer that is accepted as solution. Affect, Presentation Quality, and Time are the three actionable success factors of interest, while Reputation (non-actionable) serves as a control factor. Each success factor is related to several metrics used in turn to inform guidelines.

We developed a conceptual framework (Figure 1) for the analysis of the aspects that may influence the success of a question. We focused on 9 actionable metrics that can be acted upon by developers when writing a question and, therefore, are useful to inform guidelines. These metrics are grouped into 3 success factors, concerning affect (i.e., the positive/negative tone conveyed by a question), presentation quality (i.e., the readability and comprehensibility of its text), and time (i.e., when to post it). As per the asker’s reputation, it is included in our model, but only as a non-actionable control factor. While reputation has been already found to help people receive more and fasters answers in other communities such as Reddit [1], one can’t really do anything to increase their score just when posting a question.

Findings: Evidence vs. User Perception

Table 1. The evidence-based netiquette for effective question writing on Stack Overflow. Guidelines are shown in bold when supported by evidence.

#	Guideline	Success factor	Empirical support	User perception	Source
1	Write questions using a neutral emotional style	Affect	Yes	Effective	Skeet [10], SO Help Center [9], Kucuktunc et al. [7], Bazelli et al. [3]
2	Provide sample code and data	Presentation quality	Yes	Effective	Skeet [10], Asaduzzaman et al. [2], Duijn et al. [6], Treude et al. [11]
3	Use capital letters where appropriate	Presentation quality	Yes	Effective	Skeet [10]
4	Be concise	Presentation quality	Yes	Ineffective	Skeet [10]
5	Use short, descriptive question titles	Presentation quality	No	Ineffective	Skeet [10]
6	Provide context through tags	Presentation quality	No	Effective	Skeet [10]
7	Provide context through URLs	Presentation quality	No	Effective	Ponzanelli et al. (partially) [8]
8	Be aware of low-efficiency hours	Time	Yes	Ineffective	Bosu et al. [4]

Table 1 reports the main findings of our study, from which some interesting observations emerge. For 4 out of the 8 guidelines studied, we found that the perceived and actual effectiveness match. In particular, SO users are correct to think that minding their tone (#1), providing snippets with examples (#2), and avoiding the inappropriate use of capital letters (#3) all increase the probability to receive a successful answer to their questions. Instead, the use of short titles is neither perceived nor found to be an effective guideline (#5).

Perhaps even more interesting considerations arise from the remaining cases of mismatch, though. First, SO users seem to be unaware that writing questions concisely (#4) and posting them during GMT evening hours (#8) increase the chances of a question to be resolved. Regarding efficiency times, Bosu et al. [4] have speculated that these time slices are the most successful because they correspond to the working time in the USA. Second, contrary to users’ perception, we found that providing context for questions by adding extra tags (#6) and including in the text URLs to external resources (#7) have no positive effects on the chance of getting a successful answer.

Final Remarks

One of the greatest issues with Stack Overflow is the sheer number of unresolved questions. Currently, the site hosts almost 15 million questions of which about 7 million are still unresolved. Helping users ask “better” questions can increase the number of those resolved. One way to do so is increasing the awareness among community members about the existence of effective question-writing guidelines while also trimming down the list to only those supported by evidence. 

For more information about our study, please refer to our paper “How to Ask for Technical Help? Evidence-based Guidelines for Writing Questions on Stack Overflow” [5].

References

[1] T. Althoff, T., Danescu-Niculescu-Mizil, C., and Jurafsky, D. How to Ask for a Favor: A Case Study on the Success of Altruistic Requests. In Proc. of the 8th Int’l Conf. on Weblogs and Social Media (ICWSM’14), 2014.
[2] M. Asaduzzaman, A.S Mashiyat, C.K. Roy, K.A. Schneider. 2013. Answering questions about unanswered questions of Stack Overflow. In Proc. of the 10th IEEE Working Conf. on Mining Software Repositories (MSR 2013), pp. 97-100.
[3] B. Bazelli, A. Hindle, and E. Stroulia. On the Personality Traits of StackOverflow Users. In Proc. of Int’l Conference on Software Maintenance (ICSM '13), 2013, pp. 460-463.
[4] A. Bosu, C.S. Corley, D. Heaton, D. Chatterji, D.J. Carver, and N.A. Kraft. Building Reputation in StackOverflow: An Empirical Investigation. In Proc. of 10th IEEE Working Conf. on Mining Software Repositories (MSR’13), 2013, pp. 89-92.
[5] F. Calefato, F. Lanubile, N. Novielli. How to Ask for Technical Help? Evidence-based Guidelines for Writing Questions on Stack Overflow. Information and Software Technology, 2017.
[6] M. Duijn, A. Kučera, and A. Bacchelli. Quality questions need quality code: classifying code fragments on stack overflow. In Proc. of 12th Working Conf. on Mining Software Repositories (MSR '15), 2015, pp. 410-413.
[7] O. Kucuktunc, B.B. Cambazoglu, I. Weber, and H. Ferhatosmanoglu. A large-scale sentiment analysis for Yahoo! answers. In Proc. of 5th Int’l Conf. on Web Search and Data Mining (WSDM '12), 2012, pp. 633-642.
[8] L. Ponzanelli, A. Mocci, A. Bacchelli, and M. Lanza. Understanding and Classifying the Quality of Technical Forum Questions. In Proc. of 14th Int’l Conf. on Quality Software (QSIC’14), 2014, pp. 343-352.
[9] Stack Overflow Help Center, Be nice (last accessed: Nov. 2017).
[10] J. Skeet, 2010. Writing the Perfect Question (last accessed: Nov. 2017).
[11] C. Treude, O. Barzilay, and M-A. Storey. How do programmers ask and answer questions on the web? (NIER track). In Proc. of the 33rd Int’l Conf. on Software Engineering (ICSE '11), 2011, pp. 804-80.

There Ain't No Such Thing as a Free Build

By: Shane McIntosh (@shane_mcintosh)

Associate Editor: Abram Hindle (@abramh)

Modern software is developed at a rapid pace. Last May (2015), Mozilla processed 8,363 updates to the codebase (roughly 270 updates per day!) The widespread adoption of techniques like Continuous Delivery (CD) accelerates the rate at which these changes become visible to users. Google, LinkedIn, and Facebook release several times daily. In May 2011, Amazon engineers deployed new releases to production every 11.6 seconds. Indeed, CD appears to be here to stay.

While its easy to sing CD's praises, there is plenty of hard work that goes into producing a smooth CD pipeline. At the heart of CD is the build system, i.e., the scripts, specifications, and tools that define and automate the complex build process of large software systems. Build systems orchestrate hundreds (or thousands!) of tool invocations, preserving the finicky order in which build commands must be executed. Rapid release cycles would be too error-prone and risky without a reliable build system in place.

In our research, we analyze the dark side of CD—the overhead that build systems introduces on development and release teams, and their infrastructure—with a particular focus on how the overhead can be mitigated.

Build Systems Require Maintenance!

Really, they do. While that statement may seem obvious to some, it's important that we're on the same page here. CD does not come for free. Indeed, our prior work shows that up to 27% of source code changes (and 44% of test code changes) are accompanied by changes to the build system.

In recent work, we asked ourselves "what can be done to mitigate build maintenance overhead?" We began by analyzing the impact of build technology choice. For example, do projects that adopt more modern build technologies like Maven incur less maintenance activity than projects that adopt older build technologies like Ant?

Surprisingly, the answer is no. In fact, our analyses of a large sample of open source repositories (177,039!) suggests that more modern build technologies are accompanied by greater quantities of build maintenance activity than older technologies are! In a follow-up study, we also found that more modern technologies like Maven tend to be more prone to copy-pasting than older technologies are. While there are several reasons for migrating to a more modern build technology, our analyses suggest that lowering maintenance activity does not hold.

On the other hand, we observed that there are open source projects that keep maintenance activity and cloning rates much lower than their counterparts. A deeper analysis of these projects revealed a couple of commonly-adopted patterns of creative build system abstraction.

Pattern 1: XML Entity Expansion

Rather than duplicating repetitive XML in their build.xml files, creative build engineers store common logic in a single file, and load it as a macro using the following snippet:

<!-- Define references to files containing common targets -->
<!DOCTYPE project [
  <!ENTITY modules-common SYSTEM "../modules-common.ent">
]>

Later, the macro can be expanded in various locations:

<project name="bea" default="all">
  <!-- Include the file containing common targets. -->
  &modules-common;
</project>

Pattern 2: On-the-fly Build Spec Generation

Although less egregious, copy-pasting is still quite a frequently occurring phenomenon in the build systems of C/C++ projects. In our work, we have observed that the studied C/C++ systems with a low rates of copy-pasting avoid duplication by filling in template build specs during an initial step of the build process. This also helps to keep build maintenance activity localized, avoiding painful duplicate effort when maintenance is required.

Conclusions

To keep up with the pace of modern software development, a robust and reliable build system is required. While recent advances have been made, build systems still require a considerable investment.

In our research, we evaluate means of reducing the overhead that's introduced by the build system. In this blog, I've presented two interesting patterns that we observed in the build systems of projects that have low build maintenance activity. More detail can be found in our papers.

---

If you liked this post, you may also like to read the IEEE Software Special Issue on Release Engineering. [magazine]

Why not use open source code examples? A Case Study of Prejudice in a Community of Practice

by Ohad Barzilay, Tel Aviv University (ohadbr@tau.ac.il)
Associate Editor: Christoph Treude (@ctreude)

With so much open source code available online, why do some software developers avoid using it? That was the research question guiding a qualitative grounded-theory study recently published [1]. We analyzed the perceptions of professional software developers as manifested in the LinkedIn online community, and used the theoretical lens of prejudice theory to interpret their answers in a broader context.

We focused on developers’ perception of (re)using code examples - existing code snippets that are used in a new context. Our definition of a code ‘example’ is broad; some of the code examples which appear on the Internet were not written in order to be reused. Code examples may accompany answers on Q&A sites [2], illustrate an idea in an online tutorial, or even be extracted from an open source project [7].

We suggest that developers’ approach with respect to using code examples is dominated by their personality, and affected by concerns such as their community identity, ownership and trust. We find that developers’ perception of such reuse goes beyond the activities and practices, and that some developers associate the use of code examples with negative character. Some of these developers stereotype habitual example users as inferior and unprofessional.

It should be noted that not only human aspects are associated with example usage – there are some other issues involved in this activity such as engineering aspects (e.g. search techniques and tools) and legal issues (e.g. copyright and licensing). These issues are outside of the scope of our discussion; however, we believe that these challenges can be mitigated with proper tools [9], training and organizational support (e.g. leveraging upon social media cues [8], teaching developers which code could they use, and under what circumstances).

Code Writers vs. Copy-and-Paste Monkeys

Some software developers perceive themselves as code writers, and feel strongly about it. Their identity and sometimes even self-esteem are derived from perceiving themselves that way. As suggested by Bewer [5] this may result in the creation of ingroup bias, and can be in turn used as a platform for hate of the outgroup – in this domain, example users. For the code writers (virtual) group, new code is the unit of progress, a sign of productivity (however misleading it may sometimes be). Copying, on the other hand, is perceived as a devalued shortcut – an imitation rather than a creation. In most university courses, the students are not allowed to share their work with fellow students, but are expected to write their own code.

Having ingroup bias often limits boundaries of trust and cooperation [6], which may explain why some developers avoid copy and paste at all cost. They do not trust other programmers enough to take responsibility for and ownership of their code. These programmers find it difficult to understand existing code, they feel that they cannot identify fallacies in someone else's code nor test it thoroughly. They prefer to write the code by themselves and take responsibility for it rather than trust others, and perhaps lose control over their code.

Furthermore, we find that example usage opponents do not conform to organizational goals, and specifically the need for speed. They do not acknowledge the required dexterity and practices for effective example usage, and they aspire to be held in high regard (as opposed to "merely plumbers" [4], suggesting that the essence of the software engineering job boils down to putting the components together and making small non-glorious fixes). After all, some of them might have chosen programming as their profession because of its status.

Implications

In a commercial context, revealing implicit prejudice and disarming it may allow developers to leverage further benefit of code reuse, and may improve the collaboration of individuals, teams and organizations. Moreover, prejudice may interfere with achieving organizational goals, or while conducting an organizational change. Some of these concerns may be mitigated by providing a comprehensive ecosystem of tools, practices, training and organizational support [3]. Having the prejudice lens in mind, one may incorporate methods which were proven effective in addressing prejudice in different context (racism, sexism, nationalism) as part of the software engineering management tools.

Finally, this study may also be considered in the broader context of the changing software engineering landscape. The recent availability of information over the Web, and in our context – availability of source code, is challenging the way software is produced. Some of the main abstractions used in the software domain, namely development and construction, do not adequately describe the emerging practices involving pragmatic and opportunistic reuse. These practices favor composing over constructing and finding over developing. In this context, prejudice can be perceived as a reaction to change and an act resulting from fear of the new and unknown.

References

[1] O. Barzilay and C. Urquhart. Understanding reuse of software examples: A case study of prejudice in a community of practice. Information and Software Technology 56, pages 1613-1628, 2014.
[2] O. Barzilay, C. Treude, and A. Zagalsky. Facilitating crowd sourced software engineering via Stack Overflow. In S. E. Sim and R. E. Gallardo-Valencia, editors, Finding Source Code on the Web for Remix and Reuse, pages 289–308. Springer New York, 2013.
[3] O. Barzilay. Example embedding. In Proceedings of the 10th SIGPLAN symposium on New ideas, new paradigms, and reflections on programming and software, pages 137-144, 2011, ACM.
[4] O. Barzilay, A. Yehudai, and O. Hazzan. Developers attentiveness to example usage. In Human Aspects of Software Engineering, HAoSE ’10, pages 1–8, New York, NY, USA, 2010. ACM.
[5] M. B. Brewer. The psychology of prejudice: Ingroup love and outgroup hate? Journal of social issues, 55(3):429–444, 1999.
[6] S. L. Jarvenpaa and A. Majchrzak. Knowledge collaboration among professionals protecting national security: Role of transactive memories in ego-centered knowledge networks. ORGANIZATION SCIENCE, 19(2):260–276, 2008.
[7] S. E. Sim and R. E. Gallardo-Valencia, editors. Finding Source Code on the Web for Remix and Reuse. Springer, 2013.
[8] C. Treude and M. P. Robillard. Augmenting API documentation with insights from Stack Overflow. Forthcoming ICSE ’16: 38th Int’l. Conf. on Software Engineering, 2016.
[9] A. Zagalsky, O. Barzilay, and A. Yehudai. Example overflow: Using social media for code recommendation. In Proceedings of the Third International Workshop on Recommendation Systems for Software Engineering, pages 38-42, 2012, IEEE Press.

Should software developers be replaced by the crowd?

by Thomas D. LaToza (@ThomasLaToza) and André van der Hoek (@awvanderhoek)
Associate Editor: Christoph Treude (@ctreude)

We value our professional software developers, hire them in teams (local or remote), and set off on our merry way producing projects. This has been the status quo for a long time. But does it need to be? Could the crowd supplant the need for explicitly hired teams? More strongly yet: should the crowd perhaps replace hired employees?

These are tantalizing questions, both in terms of excitement about possibilities and in terms of 'fear' regarding the significant disruption that may play out in the workplace. Here, we consider several aspects of the issue of crowds versus hired employees.

1. Isn’t this just open source?

According to Howe [1], crowdsourcing is “the act of a company or institution taking a function once performed by employees and outsourcing it to an undefined (and generally large) network of people in the form of an open call”. In software, open source is one of the oldest and most successful crowdsourcing models. But all crowdsourcing is not open source. Platforms such as TopCoder employ a competition model, in which developers respond to a client request and clients select and reward a winner. Others such as trymyUI offer to the crowd short, well-defined tasks that, when taken together, compose into a larger whole – for instance comprehensively testing the usability of an application. Labor markets such as UpWork offer the promise of on-demand, fluid labor forces with the skills necessary for the job at hand, with workers ready to take on specialized work for an hour, a day, or maybe a week.

2. What about the speed of development?

As the old adage goes, many hands make light work. Fundamental to many crowdsourcing models is the decomposition of a large task into many small tasks. In principle, decomposed tasks enable work to be completed dramatically faster, as they enable parallel distribution to and completion of tasks by the crowd. But decomposition itself brings challenges: communicating tasks to developers, coordinating contributions made by the crowd, and ensuring smooth handoffs between tasks. In practice, these competing forces have given rise to a wide diversity of competing systems and platforms, using different granularities of tasks to crowdsource a variety of software development activities.

3. What about quality?

One common concern about opening contributions to anyone is the quality of the work that will result. As in traditional development, there are many approaches to managing quality. Work is almost always reviewed, either by the client, an agent of the client, or the crowd itself. Many platforms explicitly track reputation, allowing clients to identify those who have demonstrated quality work on similar tasks in the past and motivating developers to do high quality work to maintain their reputation. Competition platforms use the crowd to generate competing alternatives, allowing the client to select the highest quality solution. Through these mechanisms, it may, in fact, be possible to achieve higher quality through crowdsourcing, as some of our work has demonstrated [2]. On the other hand, it has been observed that, without proper management, quality may well suffer; care, thus, must be taken to focus on quality from the start [3].

4. What about crowdsourcing environments?

A significant innovation many crowdsourcing approaches bring is their dedicated support for performing crowdsourced work. Crowdsourcing platforms often provide, directly in the environment, support for contributors to browse and identify tasks matching their interests or that have the greatest chance of a reward. Clients may browse reputation information about potential workers, browse submitted contributions, and use the platform to make and manage payments. Systems that offer fine-grained tasks may go further still, offering workers self-contained tasks that can be quickly completed within the environment itself. For example, CrowdCode [4] offers an online environment for completing programming microtasks.



5. What about more complex development tasks?

Another common question is how a complex task such as architecture or design—requiring knowledge of the project as a whole or with dependencies making decomposition difficult—could ever be crowdsourced? Of course, one approach is simply to not decompose such tasks. For example, some TopCoder tasks are architectural in nature, asking competing workers to create design documents. An experienced crowd worker called a “co-pilot” is then responsible for creating, coordinating, and managing the tasks to implement the architecture. However, this can bring challenges in imposing a waterfall development process, where an architecture is built up-front, independent of future programming work [3]. Alternatively, it may be possible for the crowd itself to be more involved, decomposing architectures into networks of connected decisions [5].

6. Can I help?

Crowdsourcing is already penetrating software development practice today [6][7]. TopCoder has hosted more than 427,000 software design, development, and data science competitions. More than 100,000 testers freelance on uTest. But we believe that this may be just the beginning [8]. Just as open source changed how many organizations do software development and the nature of organizations themselves, crowdsourcing opens the door to new types of contributions and new ways stakeholders may interact that may lead to new models of software development. To help understand where these models may fail and where they may succeed, we have begun to study crowdsourcing approaches through a variety of experiments, from debugging to programming to designing. If you’d like to see how such approaches may work and help discover new ways of developing software, sign up here.

References

[1] James Surowiecki. (2005). The Wisdom of Crowds. Random House.
[2] Thomas D. LaToza, Micky Chen, Luxi Jiang, Mengyao Zhao, and André van der Hoek. (2015). Borrowing from the crowd: a study of recombination in software design competitions. International Conference on Software Engineering, 551-562.
[3] Klaas-Jan Stol and Brian Fitzgerald. (2014). Two's company, three's a crowd: a case study of crowdsourcing software development. International Conference on Software Engineering (ICSE), 187-198.
[4] Thomas D. LaToza, W. Ben Towne, Christian M. Adriano, and André van der Hoek. (2014). Microtask programming: building software with a crowd. Symposium on User Interface Software and Technology (UIST), 43-54.
[5] Thomas D. LaToza, Arturo Di Lecce, Fabio Ricci, W. Ben Towne, and André van der Hoek. (2015). Ask the crowd: scaffolding coordination and knowledge sharing in microtask programming. Symposium on Visual Languages and Human-Centric Computing (VL/HCC), 23-27.
[6] Thomas D. LaToza and André van der Hoek. (2016). Crowdsourcing in Software Engineering: Models, Motivations, and Challenges. IEEE Software, 33, 1 (January 2016), 74-80.
[7] Ke Mao, Licia Capra, Mark Harman, and Yue Jia. (2015). A survey of the use of crowdsourcing in software engineering. Technical Report RN/15/01, Department of Computer Science, University College London.
[8] Thomas D. LaToza and André van der Hoek. (2015). A vision of crowd development. International Conference on Software Engineering (ICSE), 563-566.

Using Biometrics to Assess a Developer's Cognitive and Emotional States

by Sebastian Müller (@smueller9, smueller@ifi.uzh.ch) and Thomas Fritz (fritz@ifi.uzh.ch)
Associate Editor: Christoph Treude (@ctreude)

What difficulties and emotions do software developers experience during their work? When is a developer "in the flow" and making a lot of progress? So far, little is known about these kinds of questions or the cognitive and emotional states of developers during work. Researchers have predominantly focused on the tangible output of developers' work, such as the source code, to identify bug-prone parts. Yet, a better understanding of what developers experience in their work would allow us to provide better and instantaneous support to developers, for instance by avoiding costly interruptions when a developer is in the flow or by intervening when a developer perceives particularly high difficulty levels and might be close to implementing a bug into the code.

Recent advances in biometric (aka psychophysiological) sensor technology in combination with research in psychology show promise that biometric data can fill this gap and provide answers to some of these kinds of questions. For instance, measures of a person's skin conductivity (aka EDA)—how much a person is sweating—can tell us something about the person's stress level and cognitive effort, but also perceived emotions.

In our research, we investigate the use of biometric measurements in the context of software engineering. In particular, we conduct studies ranging from short controlled lab studies to several week long field studies in which we examine the link between a developer's biometric measures and cognitive and emotional aspects of a developer while working on a change task, such as the perceived difficulty of source code elements or tasks, and the emotions and progress the developer experiences. Our vision is to use biometric measurements to gain a better understanding of what developers experience in their work and to use this improved understanding to better support developers.

To study the potential of biometrics to predict developers' perceived progress and emotions (valence), for example, we conducted a lab study [1] with 17 participants from two companies and one university. For this study, we collected biometric data, such as EDA, or the heart rate of participating developers while they were working on two different change tasks. At the same time, we asked developers in periodic intervals to self-report their perceived progress and emotions, specifically valence and arousal (on two scales from -200 to +200). The following figure highlights some of the collected data, with the biometric EDA and heart rate data on top and the self-reported valence and progress on bottom.

The figure also illustrates a visible difference in the biometric data, in particularly in the EDA signal, between the first episode that was rated as medium progress and higher valence compared to the last episode with the developer reporting being stuck and a lower valence. To analyze the data, we performed several steps of data cleaning, noise filtering and feature extraction, before we could feed these features into a machine learning classifier. The results of our analysis show that biometric data can be used well to distinguish between positive and negative emotions, as well as between phases of low and high progress, in most cases.

Overall, the results of our three studies ([1], [2], [3]) show that we are able to predict aspects of developers' cognitive and emotional states in real-time using biometrics. The results also show that we are able to make accurate predictions in the field in the common work environment of professional software developers and over multiple days. Not surprisingly, biometric data is very sensitive to the individual and certain biometric measures work better for some than for others. By using an approach that takes into account these individual differences more explicitly, we should be able to further improve our results in the future.

So what can we do with these insights? Similar to other metrics, such as code metrics or interaction metrics, biometrics can be leveraged to provide better support to developers, even in real-time. The results of our latest study [3] show that we can use biometrics to predict where quality concerns are or will occur while developers are still working on the change task, and that the biometrics even outperform more traditional metrics. These insights can now be used to enhance existing tool support, such as the automatic detection of quality concerns, to highlight and rank places in the code that should be reviewed and/or refactored, or even to indicate to the developer when he/she should reconsider a change to prevent him/her from ever submitting code with errors to the repository. Similarly, we can use biometrics to provide better interruption management support for developers or to provide recommendations when a developer perceives difficulty, ranging from talking to a coworker to taking a break.

In summary, the results of our studies are promising and provide initial evidence that biometrics can be used to assess cognitive and emotional states of a developer and thus help answer some of the open questions. As one example, our results show that we might be able to support developers when they are experiencing difficulties and help them to fix quality concerns early on. With the rapid and continuous advances in sensor technologies and the increasing pervasiveness of smart wearable devices that already collect biometric data, we might soon be able to leverage this kind of data for each professional software developer and fine-tune our classifiers to further increase their accuracy. At the same time, this opens up many ethical and privacy concerns that will have to be addressed and investigated in further studies.

References

1. Sebastian Müller and Thomas Fritz. Stuck and Frustrated or In Flow and Happy: Sensing Developers' Emotions and Progress. In Proceedings of the 37th International Conference on Software Engineering (ICSE 2015), Florence, Italy
2. Thomas Fritz, Andrew Begel, Sebastian Müller, Serap Yigitt-Elliott, and Manuela Züger. Using psycho-physiological measures to assess task difficulty in software development. In Proceedings of the 36th International Conference on Software Engineering (ICSE 2014), Hyderabad, India
3. Sebastian Müller and Thomas Fritz. Using Bio(Metrics) To Predict Code Quality Online. In Proceedings of the 38th International Conference of Software Engineering (ICSE 2016), Austin, USA

If you like this article, you might also enjoy reading:

• Graziotin, D., Wang, X., and Abrahamsson, P. Software Developers, Moods, Emotions, and Performance. In IEEE Software, vol.31, no.4. 2014
• Ikutani, Y. and Uwano, H. Brain activity measurement during program comprehension with NIRS. In Proceedings of the International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing. 2014.
• Nakagawa, T., Kamei, Y., Uwano, H., Monden, A., Matsumoto, K., and German, D. M. Quantifying programmers' mental workload during program comprehension based on cerebral blood flow measurement: A controlled experiment. In Companion Proceedings of International Conference on Software Engineering. 2014.
• Parnin, C. Subvocalization - toward hearing the inner thoughts of developers. In Proceedings of the International Conference on Program Comprehension. 2011.
• Siegmund, J., Kästner, C., Apel, S., Parnin, C., Bethmann, A., Leich, T., Saake, G., and Brechmann, A. Understanding understanding source code with functional magnetic resonance imaging. In Proceedings of the 36th International Conference on Software Engineering. 2014.

Understanding Runtime Value

- A Cost/Benefit Approach to Performance Analysis 

by David Maplesden, The University of Auckland, Auckland, New Zealand (@dmap_nz)
Associate Editor: Zhen Ming (Jack) Jiang, York University, Toronto, Canada

Many large-scale modern applications suffer from performance problems [1] and engineers go to great lengths searching for optimisation opportunities. Most performance engineering approaches focus on understanding an application's cost (i.e., its use of runtime resources). However, understanding cost alone does not necessarily help find optimisation opportunities. One piece of code may take longer than another simply because it is performing more necessary work. For example, it would be no surprise that a routine that sorted a list of elements took longer than another routine that returned the number of elements in the list. The fact that the costs of the two routines are different does not help us understand which may represent an optimisation opportunity. However, if we had two different routines which output the same results (e.g., two different sorting algorithms), then determining which is the more efficient solution becomes a simple cost comparison.

The key is to understand the value provided by the code. It is then possible to find the superfluous activity that characterises poor performance.

Traditionally it has been left to the engineer to determine the value provided by a piece of code through experience, intuition or guesswork. However, the challenge of intuitively divining runtime value is difficult in large-scale applications. These applications have thousands of methods interacting to produce millions of code paths. Establishing the value provided by each method via manual inspection is not practical with such scale and complexity.

To tackle this challenge we are developing an approach to empirically measure runtime value. We can combine this measure with traditional runtime cost information to quantify the efficiency of each method in an application. This allows us to find the most inefficient methods in an application and analyse them for optimisation opportunities.

Our approach to quantifying value is to measure the amount of data created by a method that becomes visible to the rest of the application, i.e., the data that escapes the context of the method. Our rationale is that the value a method is providing can only be imparted by the visible results it creates. Intermediate calculations used to create the data but then discarded do not contribute to this final value. Intuitively two method calls that produce identical results (given the same arguments) are providing the same amount of value, regardless of their internal implementations.

Specifically we track the number of object field updates that escape their enclosing method. An object field update is any assignment to an object field or array element (e.g., foo.value = 1 or bar[0] = 2). A field update escapes a method if the object it is applied to escapes the method i.e. is a global (static), a method parameter or returned.

For example consider the time formatting Java code below:

  public static String formatElapsedTime(long timeInMillis) {
    long seconds = timeInMillis / 1000;
    long minutes = seconds / 60;
    long hours = minutes / 60;

    final StringBuilder sb = new StringBuilder();
    formatTimePart(sb, hours, "hours");
    formatTimePart(sb, minutes, "minutes");
    formatTimePart(sb, seconds, "seconds");
    return sb.toString();
  }  

  public static void formatTimePart(StringBuilder sb, long l, String description) {
    if (l > 0) {
      if (sb.length() > 0) {
        sb.append(' ');
      }
      sb.append(l);
      sb.append(' ');
      sb.append(description);
    }
  }

The formatTimePart method updates the StringBuilder parameter (via calls to sb.append()) and so it has parameter escaping field updates. The formatElapsedTime method has no parameter escaping updates but it does return a newString value (constructed via sb.toString()) and so has returned field updates. Note that the StringBuilder object sb does not escape formatElapsedTime and so the updates applied to it are actually captured by the method, it is only the subsequently constructed String which escapes. We have found captured writes such as these to be a strong indicator of inefficient method implementations.

We have evaluated our approach [2] using the DaCapo benchmark suite, demonstrating our analysis allows us to quantify the efficiency of the code in each benchmark and find real optimisation opportunities, providing improvements of up to 36% in our case studies. For example, we found over 10% of the runtime activity in the h2 benchmark was incurred by code paths such as JdbcConnection.checkClosed() that were checking assertions and not contributing directly to the benchmark result. Many of these checks were repeated unnecessarily and we were able to refactor and remove them.

Our proposed approach allows the discovery of new optimisation opportunities that are not readily apparent from the original profile data. The results of our experiments and the performance improvements we made in our case studies demonstrate that efficiency analysis is an effective technique that can be used to complement existing performance engineering approaches.

References

1. G. Xu, N. Mitchell, M. Arnold, A. Rountev, and G. Sevitsky. Software Bloat Analysis: Finding, Removing, and Preventing Performance Problems in Modern Large-Scale Object-Oriented Applications. Proceedings of the FSE/SDP Workshop on the Future of Software Engineering Research, pages 421-425, 2010.
2. D. Maplesden, E. Tempero, J. Hosking, and J. Grundy. A Cost/Benefit Approach to Performance Analysis. Proceedings of the 7th ACM/SPEC International Conference on Performance Engineering (ICPE), to appear. 2016.