Can We Trust the Stack Overflow Netiquette? Evidence-based Guidelines for Asking Good Technical Questions

by Fabio Calefato, University of Bari, ITALY (@fcalefato), Filippo Lanubile, University of Bari, ITALY (@lanubile), and Nicole Novielli, University of Bari, ITALY (@NicoleNovielli)
Associate Editor: Christoph Treude (@ctreude)

Several thousands of developers daily head to Stack Overflow (SO) for asking technical questions, hoping to receive swift help and fix the issues that they have been facing. To increase the chances of getting help from others, the SO community provides members with detailed guidelines on how to write more effective questions (e.g., see [9]). These official recommendations also include those provided by Jon Skeet, the highest reputation member, whose guidelines have become over time a de facto standard for the community [10].

For example, SO states that the site is “all about getting answers. It's not a discussion forum. There's no chit-chat.” Thus, askers are recommended to avoid “greetings and sign-offs […], as they’re basically a distraction,” which are also supposed to be edited out by other users [10]. Still, many askers finish their questions showing gratitude in advance towards potential helpers. Why do they go against this explicit recommendation? Are they just unaware of it or do they feel that having a positive attitude may attract more potential solutions?

In our work [5], we provide an evidence-based netiquette for writing effective questions by empirically validating several SO guidelines, retrieved from both the community and previous empirical studies on Q&A sites. Specifically, we analyzed a dataset of 87K questions by combining a logistic regression analysis with a user survey, first, to estimate the effect of these guidelines on the probability of receiving a successful answer and, then, to compare their actual effectiveness to that perceived by SO users.

Actionable Factors for Good Questions

Figure 1. Our conceptual framework of success factors for writing good questions in Stack Overflow. The success of a question is defined as the probability of receiving an answer that is accepted as solution. Affect, Presentation Quality, and Time are the three actionable success factors of interest, while Reputation (non-actionable) serves as a control factor. Each success factor is related to several metrics used in turn to inform guidelines.

We developed a conceptual framework (Figure 1) for the analysis of the aspects that may influence the success of a question. We focused on 9 actionable metrics that can be acted upon by developers when writing a question and, therefore, are useful to inform guidelines. These metrics are grouped into 3 success factors, concerning affect (i.e., the positive/negative tone conveyed by a question), presentation quality (i.e., the readability and comprehensibility of its text), and time (i.e., when to post it). As per the asker’s reputation, it is included in our model, but only as a non-actionable control factor. While reputation has been already found to help people receive more and fasters answers in other communities such as Reddit [1], one can’t really do anything to increase their score just when posting a question.

Findings: Evidence vs. User Perception

Table 1. The evidence-based netiquette for effective question writing on Stack Overflow. Guidelines are shown in bold when supported by evidence.

#	Guideline	Success factor	Empirical support	User perception	Source
1	Write questions using a neutral emotional style	Affect	Yes	Effective	Skeet [10], SO Help Center [9], Kucuktunc et al. [7], Bazelli et al. [3]
2	Provide sample code and data	Presentation quality	Yes	Effective	Skeet [10], Asaduzzaman et al. [2], Duijn et al. [6], Treude et al. [11]
3	Use capital letters where appropriate	Presentation quality	Yes	Effective	Skeet [10]
4	Be concise	Presentation quality	Yes	Ineffective	Skeet [10]
5	Use short, descriptive question titles	Presentation quality	No	Ineffective	Skeet [10]
6	Provide context through tags	Presentation quality	No	Effective	Skeet [10]
7	Provide context through URLs	Presentation quality	No	Effective	Ponzanelli et al. (partially) [8]
8	Be aware of low-efficiency hours	Time	Yes	Ineffective	Bosu et al. [4]

Table 1 reports the main findings of our study, from which some interesting observations emerge. For 4 out of the 8 guidelines studied, we found that the perceived and actual effectiveness match. In particular, SO users are correct to think that minding their tone (#1), providing snippets with examples (#2), and avoiding the inappropriate use of capital letters (#3) all increase the probability to receive a successful answer to their questions. Instead, the use of short titles is neither perceived nor found to be an effective guideline (#5).

Perhaps even more interesting considerations arise from the remaining cases of mismatch, though. First, SO users seem to be unaware that writing questions concisely (#4) and posting them during GMT evening hours (#8) increase the chances of a question to be resolved. Regarding efficiency times, Bosu et al. [4] have speculated that these time slices are the most successful because they correspond to the working time in the USA. Second, contrary to users’ perception, we found that providing context for questions by adding extra tags (#6) and including in the text URLs to external resources (#7) have no positive effects on the chance of getting a successful answer.

Final Remarks

One of the greatest issues with Stack Overflow is the sheer number of unresolved questions. Currently, the site hosts almost 15 million questions of which about 7 million are still unresolved. Helping users ask “better” questions can increase the number of those resolved. One way to do so is increasing the awareness among community members about the existence of effective question-writing guidelines while also trimming down the list to only those supported by evidence. 

For more information about our study, please refer to our paper “How to Ask for Technical Help? Evidence-based Guidelines for Writing Questions on Stack Overflow” [5].

References

[1] T. Althoff, T., Danescu-Niculescu-Mizil, C., and Jurafsky, D. How to Ask for a Favor: A Case Study on the Success of Altruistic Requests. In Proc. of the 8th Int’l Conf. on Weblogs and Social Media (ICWSM’14), 2014.
[2] M. Asaduzzaman, A.S Mashiyat, C.K. Roy, K.A. Schneider. 2013. Answering questions about unanswered questions of Stack Overflow. In Proc. of the 10th IEEE Working Conf. on Mining Software Repositories (MSR 2013), pp. 97-100.
[3] B. Bazelli, A. Hindle, and E. Stroulia. On the Personality Traits of StackOverflow Users. In Proc. of Int’l Conference on Software Maintenance (ICSM '13), 2013, pp. 460-463.
[4] A. Bosu, C.S. Corley, D. Heaton, D. Chatterji, D.J. Carver, and N.A. Kraft. Building Reputation in StackOverflow: An Empirical Investigation. In Proc. of 10th IEEE Working Conf. on Mining Software Repositories (MSR’13), 2013, pp. 89-92.
[5] F. Calefato, F. Lanubile, N. Novielli. How to Ask for Technical Help? Evidence-based Guidelines for Writing Questions on Stack Overflow. Information and Software Technology, 2017.
[6] M. Duijn, A. Kučera, and A. Bacchelli. Quality questions need quality code: classifying code fragments on stack overflow. In Proc. of 12th Working Conf. on Mining Software Repositories (MSR '15), 2015, pp. 410-413.
[7] O. Kucuktunc, B.B. Cambazoglu, I. Weber, and H. Ferhatosmanoglu. A large-scale sentiment analysis for Yahoo! answers. In Proc. of 5th Int’l Conf. on Web Search and Data Mining (WSDM '12), 2012, pp. 633-642.
[8] L. Ponzanelli, A. Mocci, A. Bacchelli, and M. Lanza. Understanding and Classifying the Quality of Technical Forum Questions. In Proc. of 14th Int’l Conf. on Quality Software (QSIC’14), 2014, pp. 343-352.
[9] Stack Overflow Help Center, Be nice (last accessed: Nov. 2017).
[10] J. Skeet, 2010. Writing the Perfect Question (last accessed: Nov. 2017).
[11] C. Treude, O. Barzilay, and M-A. Storey. How do programmers ask and answer questions on the web? (NIER track). In Proc. of the 33rd Int’l Conf. on Software Engineering (ICSE '11), 2011, pp. 804-80.

IEEE September/October Issue, Blog, and SE Radio Summary

Associate Editor: Brittany Johnson (@drbrittjay)

The September/October Issue of IEEE Software, top magazine for all things software, again delivers a range of interesting topics for thought and discussion in the SE community. The topics discussed in this issue included software requirements and testing, DevOps, gamification, and software architecture.

The feature topic in this issue of IEEE Software was software testing. This issue featured the following articles related to software testing:

• "State-of-the-Art Software Testing" by Diomidis Spinellis (From The Editor)
• "Adaptive Virtual Gestures for GUI Testing on Smartphones" by Chia-Wei Hsu, Shan-Hsin Lee, and Shiuhpyng Winston Shieh
• "KLOVER: Automatic Test Generation for C and C++ Programs, Using Symbolic Execution" by Hiroaki Yoshida, Guodong Li, Takuki Kamiya, Indradeep Ghosh, Sreeranga Rajan, Susumu Tokumoto, Kazuki Munakata, and Tadahiro Uehara
• "Worlds Apart: Industrial and Academic Focus Areas in Software Testing" by Vahid Garousi and Michael Felderer
• "Software Testing: The State of the Practice" by Mohamad Kassab, Joanna F. DeFranco, and Phillip A. Laplante
• "Replicating Rare Software Failures with Exploratory Visual GUI Testing" by Emil Elégroth, Johan Gustafsson, Henrik Ivarsson, and Robert Feldt

For those interested in getting a quick overview of how testing is used and viewed in the software community, "Software Testing: The State of the Practice" and "Worlds Apart: Industrial and Academic Focus Areas in Software Testing" are great articles to read.

In "Software Testing: The State of the Practice", Kassab and colleagues conducted a comprehensive survey of software practitioners to gain a much needed understanding of the state-of-the-art in software testing and quality assurance practices.

Their ultimate goal was to gather and be able to disseminate best practices to community based on their findings, such as practicing test-driven development.

In "Worlds Apart: Industrial and Academic Focus Areas in Software Testing", Garousi and Felderer discusses the problem of the often disparate efforts between industry and academy when it comes to academia doing research that matters and industry applying this research to their work.

Their focus was on software testing and how we can improve industry-academic collaborations.

They observed titles of presentations given at industrial and academic conferences and, using Wordle to create word clouds, found that these communities often focus on different aspects of testing. 

For example, industry conferences most often talk about automation while academic conferences most often talk about models. The authors make suggestions, such as inviting practitioners to research-intensive SE conferences to gain their perspective on the research we're doing.

Some of the articles in this issue focus on a specific subset of testing, such as GUI testing. 

In "Adaptive Virtual Gestures for GUI Testing on Smartphones", Hsu and colleagues propose an approach to testing mobile software called adaptive GUI testing (AGT). AGT allows for faster cross-device testing and is based on touch events known as visually oriented gestures (VOG). 

Along the same lines, in "Replicating Rare Software Failures with Exploratory Visual GUI Testing", Elégroth and colleagues discuss the benefits of visual GUI testing (VGT) based on their own experiences. They speak about how VGT can be used to replicate failures and push forward analysis of infrequent or nondeterministic failures. 

IEEE Software Blog

Behind on your IEEE Software issues? No worries! Summaries from the May/June and July/August issues of IEEE Software are up in the IEEE Software Blog. On the technical side, this issue features blog posts on merge conflicts and using code metrics to predict fault and change proneness. On the not-so-technical (but obviously still software-related) side, this issue features articles that explore behaviors in OS projects, such as understanding the mindset of one-time contributors and female participation in OS software. Among the not-so-technical blog posts, a developer from the Debian project observes and analyzes the social structure within the project based on the cryptographic keyring of the project.

SE Radio

This issue of SE Radio didn't see any new hosts but lots of interesting discussions on interesting topics, ranging from choosing freelancing as a career path to machine learning for predictive applications.  For the software engineers out there, Haroon Meer, founder of Thinkst, spoke with SE Radio on what's going wrong with network security in our applications and what we can do better.

For aspiring software engineers out there, Evgeny Shadchnev discussed code schools, or programs available that prepare students to become software developers in a few months.

For those in management positions, or aspiring to be a manager one day, Ron Lichty joined SE Radio to discuss difficulties and suggestions for managing programmers. He also provided a link to his blog on the topic. Also useful is the information provided by Harsh Sinha, VP of TransferWise, on another type of management known as product management.

NaPiRE: Global Study on Industrial Practices and Problems

By Daniel Méndez and Birgit Penzenstadler
Edited by Mei Nagappan

40 years after Requirements Engineering (RE) was acknowledged for the first time as an independent discipline in an issue of the Transactions of Software Engineering, it has received much attention in research and practice due to its importance to software project success. The importance of RE cannot be refuted as many decisions in software projects are rooted therein; same holds for the problems. As Nancy Leveson (MIT) was cited in an article in The Atlantic:

The serious problems that have happened with software have to do with requirements, not coding errors.

In fact, it has become conventional wisdom that many problems emerge from "insufficient RE" and that the later the problems are discovered, the harder (and, thus, more expensive) they become to fix. Yet it remains difficult to obtain reliable empirical figures that would describe what "insufficient RE" exactly means, how it manifests in the processes and artefacts created, and what root causes and effects this has. Such figures are, however, critical determinants for a problem-driven research, i.e., to support contributions that are in tune with the problems they intend to solve. As a matter of fact, the state of empirical evidence in RE is still weak and much of everyday industrial practices and research as well are both dominated by wisdom and beliefs rather than being governed by empirical evidence. This results in research contributions with a potentially low practical impact and, in the end, a continuously increasing disconnect between research and practice.

The Naming the Pain in Requirements Engineering Initiative

Motivated by this situation where we need a stronger body of knowledge about the state of the industrial practice in RE , we initiated the Naming the Pain in Requirements Engineering (short: NaPiRE) initiative in 2012. The initiative constitutes a globally distributed family of practitioner surveys on Requirements Engineering (RE) with the overall objective to build a holistic theory on industrial practices, trends, and problems. It is run by the RE research community with the purpose of serving researchers and practitioners alike and is, in fact, the first of its kind. In a nutshell, each survey replication aims at distilling

• the status quo in company practices and industrial experiences,
• problems and how those problems manifest themselves in the process, and
• what potential success factors for RE are.

To achieve the long-term objective of increasing the practical impact of research, the community behind NaPiRE has committed itself to open science principles. All publications, but also results obtained from the studies, are open to the public, including the anonymised raw data, codebooks, and analysis scripts. This shall support other researchers in running independent data analyses, interpretations, and replications, and it helps practitioners in assessing their own current situation in context of a broad picture illustrating overall industrial practices.

Current State of NaPiRE

While our first survey round focused on surveying German companies and had a first replication in the Netherlands, the second replication was run in 2014/15 and already took place in 10 countries with the support of 23 researchers. That run yielded fruitful insights into contemporary problems practitioners experience as well as into the criticality of those problems as shown in the following chart.

 

Top Problems in Requirements Engineering as reported in "Naming the Pain in Requirements Engineering: Contemporary Problems, Causes, and Effects in Practice" (Empirical Software Engineering Journal)[/caption] The colour coding visualises the criticality of the problems in the sense of illustrating the extent to which problems are seen as the main reason for project failure. We can see, for example, that incomplete requirements constitute the most frequently stated problem. At the same time, we can also see that moving targets, although ranked as the fourth most frequent problem, becomes the top priority problem when considering the project failure ratios alone. An overview of the further results including more fine-grained analyses of the root-causes and the effects going beyond a simple notion of "project failure", can be taken from the publications on the project website www.re-survey.org. Motivated by our past success in revealing insights into the status quo, but also in being able to transfer those analytical results into first constructive methods, e.g. in the context of risk management, we have initiated the third replication. By now, the NaPiRE community has grown into an international alliance of nearly 60 researchers who share the vision of contributing our part in increasing the practical impact of research contributions to RE.

Call for Participation

We are reaching out to you, the IEEE Software readers, as a highly relevant community of software practitioners. Please volunteer 20 minutes of your valuable time to contribute your insights and experience by participating in the current run of the survey. To foster problem-driven research with high practical impact, we depend on your input. The link to the survey is: participate.re-survey.org (open until the end of December).