Monday, April 25, 2016

Common Architecture Weakness Enumeration (CAWE)

By Mehdi Mirakhorli (@MehdiMirakhorli), Associate Editor.

Software architecture design is the first and the fundamental step to address quality goals surrounding attributes such as security, privacy, safety, reliability, dependability, and performance. Design Flaws in the architecture of a software system mean that successful attacks could result in enormous consequences. To satisfy a security concern, an architect must consider alternate design solutions, evaluate their trade-offs, identify the risks and select the best solution. Such design decisions are often based on well-known architectural patterns, defined as reusable techniques for achieving specific quality concerns.

Security patterns come in many different shapes and sizes and provide solutions for enforcing the data integrityprivacyaccountabilityavailabilitysafety and non-repudiation requirements, even when the system is under attack.

Previous estimations indicate that roughly 50% of security problems are the result of software design flaws such as miss-understanding architecturally significant requirementspoor architectural implementationviolation of design principles in the source code and degradations of the security architecture. Flaws in the architecture of a software system can have a greater impact on various security concerns in the system and, as a result, giving more space and flexibility for malicious users.

Fundamentally, Design flaws (or only "flaws'') are different from Bugs, as the latter are more code-level while the former are at a deeper level and are much more subtle than bugs such as buffer overflows. Although a software system will always have bugs, recent studies show that the security of many software applications is breached due to flaws in the architecture.

Architectural flaws are results of inappropriate design choices in early stages of software development, incorrect implementation of security patterns, or degradation of security architecture over time.
An example of such architectural flaw is the Use of Client-Side Authentication, in which a client/server product performs authentication within client code, but not in server code, allowing server-side authentication to be bypassed via a modified client that omits the authentication check. 
This design decision to implement authentication creates a flaw in the security architecture. It can  be successfully exploited by an intruder with reverse-engineering skills.

Even though there are many techniques and practices such as threat modeling, static and dynamic code analysis, penetration testing that help developing a secure software system, there have not been many previous research papers in the literature that approach security from the architecture perspective. A recent effort is the IEEE Center for Secure Design launched by IEEE Computer Society. However as of today, there are not many examples or catalog of design flaws obtained or published yet that can help architects and developers learn and avoid such flaws.

Therefore, in our research team, we are working on establishing a catalog of Common Architecture Weakness Enumeration, containing architectural weaknesses that may create security breaches within the software.

This catalog is built on top of the previous library of Common Software Weaknesses Enumeration, which documents about 1000 software weaknesses. These weaknesses, however, are not categorized based on their architectural impacts and do not clearly distinguish between architectural weaknesses (security issues rooted in software architecture) and programming issues. We categorize these weaknesses into architectural and non-architectural and release the resulting catalog for the public. In addition, in a series of real case studies, we demonstrate instances of architectural weaknesses in four systems. These case studies indicate that the catalog of architectural weaknesses will be helpful for architects and designers to adopt a proactive approach to architecture-bases security.

Designing for Security

To ensure an application is secure, the security principles need to be implemented from the ground up. During requirements analysis, malicious practices are taken for granted, and requirements engineers identify all the use cases which are interests of an attacker. During architecture design, architects carefully analyze these requirements and adopts appropriate security patterns to resist, detect and recover from attacks.

Weaknesses in a Security Architecture

A software architecture can be flawed for many reasons resulting fundamental breaches in the system. Such flaws occur because of bad design decisions (flaws of commission), lack of design decisions (flaws of omission) or incorrect implementation of architectural patterns used to make the system secure (flaws of realization). These types of flaws are discussed in the following:
  • Flaws of Omission. Such design flaws result from ignoring a security requirement or potential threats. Such flaws identify decisions that were never made. A common design flaw is to store a password in a file without encryption. In this flaw the architect assumes that attackers would never have access to the file, thereby considering that the password stored in plaintext would not correspond to a compromise of the system. However, such design decision can open the system to attacks, because anyone, who has granted read access to the file, will be able to read all the stored passwords.
  • Flaws of Commission.Such design flaws refer to the design decisions which were made and could lead to undesirable consequences. Examples of such flaws are “Client side authentication” or ”using a weak encryption algorithm” to achieve better performance while maintaining data confidentiality.
  • Flaws of Realization. The design decision is correct but the implementation of that suffers from a coding mistake. For instance, the system was designed to have the Chroot Jail pattern. In this pattern, a controlled environment (“jail”) is created to limit access to system files so, attackers are avoided to exploit files/directories outside a specific directory. A common way to implement this pattern on Unix environments is to invoke the chroot() system function, which creates the jail but does not change the current working directory. Consequently, a developer may incorrect implement it through creating the chroot jail without changing the working directory, which allows that relative path still point to files- side the jail. Thus, attackers would still be able to access files/directories outside the jail even after invoking the chroot() function.


Examples of Weaknesses in Security Architecture

The Secure Session Management pattern is concerned about keeping track of sessions, which are a set of activities performed over a limited time period by a certain user. The main goal of this pattern is to keep track of who is using the system at a given time through managing a session object that contains all relevant data associated to the user and the session. In this pattern, every user is assigned an exclusive identifier (Session ID), which is utilized for both identifying users and retrieving the user-related data. Since session IDs are a sensitive information, this pattern may be affected by two main types of attacks: session hijacking (an attacker impersonate a legitimate user through stealing or predicting a valid session ID) and session fixation (an attacker has a valid session ID and forces the victim to use this ID).

The session hijacking can be facilitating by the architectural flaw of not securing the storage of session identifiers. Such flaw can be observed in the “session” module of the PHP language:


Per this description we note that PHP was designed to store each data session in plain textual files in a temporary directory without using a security mechanism for storing these session files (such as encryption). When closely inspecting the source code of PHP in version 4.0, we observe that the mod file.cc names every session file as "sess_xyz” (where "xyz" is the session ID), as shown in the code snippet presented above (where buf is a variable later used when creating the session files).
Figure 1(a) shows a scenario in which the flaw could be exploited. First, a legitimate user successfully identifies him/herself to the application. This causes the Web application written in PHP to start a session for the user through invoking the session start() from the PHP’s session module. Then, the session module in the PHP assigns a session ID for the user and it creates a new file named as “sess_qEr1bqv1q4V2FGX9C7mvb0” to store the data about the user’s session. At this point, the security of the application is compromised when an attacker observes the session file name and realizes that the user’s session ID is equal to “qEr1bqv1q4V2FGX9C7mvb0”. Subsequently, the attacker is able to impersonate the user through sending a cookie (PHPSESSIONID) in a HTTP request with this stolen Session ID. The Web application, after calling functions from the PHP’s session, verifies that the session ID provided matches with the user’s data so, the application considers that the requests are being made by a legitimate user.



From this scenario we can observe that such architectural weakness can lead to many consequences. First, if the user has an administrative role in the application, the attacker will be able to perform all the administrative tasks. Second, the attacker may be able to read the contents of the session file, thereby accessing the data, which may be sensitive, about the user that the attacker is not supposed to have access. It is important to highlight that such flaw affects not only the Secure Session Management, but also other security patterns (e.g. Authentication and Authorization) which uses the Secure Session Management for performing authentication and access control of users.

An example from PHP of an architectural weakness that facilitates the session fixation is shown below:


When verifying the session implementation in the source code of PHP version 5 we note that there is an incorrect implementation (i.e. a realization flaw) in the PHP’s session module that accept uninitialized session IDs before using it for authentication/authorization purposes. In fact, in the line 158 shown above, the function ps files valid key() does not properly validates the session ID, it only checks whether it contains a valid charset and has a valid length but does not verify whether the ID actually exists associated to the client performing the HTTP request.
Figure 1(a) shows how this architectural vulnerability is exploited. The attack starts with the attacker establishing a valid session ID (steps 1 to 4). Next, the attacker induces the user to authenticate him/herself in the system using the attacker’s session ID (steps 5 and 6).

Application of CAWE Catalog


Given that the CAWE catalog provides detailed information about architectural weaknesses, it can be used to guide architects and developers make appropriate design and implementation decisions to preserve security concerns at the architectural level throughout the software development lifecycle.

For example, code reviews are usually focused on finding bugs through technical discussions and analysis of the source code and other related artifacts (such as a portion of the requirements document, the architecture, etc). However using the CAWE catalog, the reviewers, who are responsible for inspecting the code, can check common security issues in their software. Past experiences in industry lead to the creation of security-driven software development processes, which emphasizes security concerns early in the software development lifecycle, such as CLASP (Comprehensive, Lightweight Application Security Process) and Microsoft’s SDL (Security Development Lifecycle). A common aspect of these processes and practices is the recommendation of providing proper training of the employees to promote a common background about software security. With this respect, our catalog could be used to aid such training and promote the awareness of the potential architectural issues that a system may be exposed to. Moreover, those security-driven processes include two activities for modeling potential threats in the software: threat modeling and design of misuse cases. These two activities are usually done through brainstorming sections. Such brainstorming could be aided with the CAWE for obtaining insights. In fact, practitioners in the security domain, support the usage of threat libraries, built from the MITRE’s catalog, for aiding this threat modeling process. Architectural risk analysis, which is a systematic approach of evaluating de- sign decisions against quality requirements, could also benefit from our catalog as a guidance for the evaluation.

You may also like:

Iván Arce, Kathleen Clark-Fisher, Neil Daswani, Jim DelGrosso, Danny Dhillon, Christoph Kern, Tadayoshi Kohno, Carl Landwehr, Gary McGraw, Brook Schoenfield, Margo Seltzer, Diomidis Spinellis, Izar Tarandach, and Jacob West, Avoiding the Top 10 Software Security Design Flaws, IEEE Cybersecurity, 2015.

Hall, Anthony & Chapman, Roderick. “Correctness by Construction: Developing a Commercial Secure System.” IEEE Software 19, 1 (Jan./Feb. 2002): 18-25.

Linger, R. C. “Cleanroom Process Model.” IEEE Software 11, 2 (March 1994): 50-58.

Acknowledgement:
"This post includes joint work with Joanna Santos and Jairo Pavel Veloz Vidal, graduate students at RIT.

Sunday, April 10, 2016

Dissecting The Myth That Open Source Software Is Not Commercial

By: Karl Fogel (@kfogel)
Associate editor: Stefano Zacchiroli (@zacchiro)

Writing a myth-debunking piece for such an informed audience poses a certain risk. The readers of the IEEE Software Blog already know what open source software is, and many have probably written some. How can I be sure that anyone reading this even holds the belief about to be debunked?

Well, I can't be completely sure, but can at least say that this myth is one I still encounter frequently among software specialists, including people who themselves use free software on a daily basis. (By the way, I will use the terms "free" — as in "free software" — and "open source" interchangeably here, because they are synonyms in the sense that they refer to the same set of software and the same set of pro-sharing software licenses.) The continued prevalence of this myth in many organizations is an obstacle to the adoption and production of open source software.
First, to state it clearly:

Myth: Open source is not commercial, or is even anti-commercial, and is driven mostly by volunteerism.

That's really two myths, but they're closely related and it's best to address them together.

In mainstream journalism, open source is still almost always portrayed as a spare-time activity pursued by irrepressible programmers who band together for the love of coding and for the satisfaction they get from releasing free tools to the world. (See the second letter here for one example, but there are many, many more examples like that.) Surprisingly, this portrayal is widespread within the software industry too, and in tech journalism. There is, to be fair, a grain of truth to the legend of the volunteers: especially in the early days of open source — from the mid 1980s until the late 1990s (a period when it wasn't even called "open source" yet, just "free software") — a higher proportion of open source development could legitimately have been called volunteer than is the case today.

But still not as high a proportion as one might think. Much free software activity was institutionally funded even then, although the institutions in question weren't always aware of it. Programmers and sysadmins frequently launched shared collaborative projects simply to make their day jobs easier. Why should each person have to write the same network log analyzer by themselves, when a few people could just write it once, together, and then maintain it as a common codebase? That's cheaper for everyone, and a lot more enjoyable.

In any case, intentional investment in open source by for-profit outfits started quite early on, and such investment has only been growing since (indeed, to the point now where meta-investment is happening: for example, my company, Open Tech Strategies, flourishes commercially by doing exclusively open source development and by advising other organizations on how to run open source projects). For a long time now, a great deal of widely-used open source software has been written by salaried developers who are paid specifically for their work on that software, and usually paid by for-profit companies. There is not space here to discuss all their business models in depth, nor how erstwhile competitors manage to collaborate successfully on matters of shared concern (though note that no one ever seems to wonder how they manage this when it comes to political lobbying). Suffice it to say that there are many commercial organizations in whose interests it is to have this growing body of code be actively maintained, and who have no need to "own" or otherwise exercise monopolistic control over the results.

A key ingredient in this growth has been the fact that all open source licenses are commercial licenses. That is, they allow anyone to use the software for any purpose, including commercial purposes. This has always been part of the very definition of a "free software" or "open source" license, and that's why there is no such thing as software that is "open source for non-commercial use only", or "open source for academic use only", etc.

An important corollary of this is that open source software automatically meets the standard government and industry definition of "Commercial Off-The-Shelf" (COTS) software: software that is commercially available to the general public. COTS doesn't mean you must pay money — though you might choose to purchase a support contract, which is a fee for service and is very different from a licensing fee. COTS essentially just means something that is equally available to all in the marketplace, and open source software certainly fits that bill.

So: open source is inherently commercial, and the people who write it are often paid for their work via normal market dynamics.

Why, then, is there a persistent belief that open source is somehow non-commercial or anti-commercial, and that it's developed mainly by volunteers?

I think this myth is maintained by several mutually reinforcing factors:
  • Open source's roots are as an ideologically-driven movement (under the name "free software"), opposed to monopoly control over the distribution and modification of code. Although that movement has turned out to be successful in technical and business terms as well, it has not shed its philosophical roots. Indeed, I would argue, though will not here due to space limitations, that its philosophical basis is an inextricable part of its technical and business success. (It is worth considering deeply the fact that merely being anti-monopoly is enough to get a movement a reputation for being anti-commercial; perhaps it is the roots of modern capitalism as actually practiced that need closer examination, not the roots of open source.)
  • For a time, various large tech companies whose revenues depend mainly on selling proprietary software on a fee-for-copying basis made it a standard part of their marketing rhetoric to portray open source as being either anti-commercial or at any rate unconcerned with commercial viability. In other words: don't trust this stuff, because there's no one whose earnings depend on making sure your deployment is successful. This tactic has become less popular in recent years, as many of those companies start to have open-source offerings themselves. I hope to see it gradually fade away entirely, but its legacy lives on in the many corporate and government procurement managers who were led to believe that open source is the opposite of commercial.
  • Many companies now offer software-as-a-service based on open source packages with certain proprietary additions — those additions being their "value-add" (or, less formally, their "secret sauce"), the thing that distinguishes their SaaS offering from you just deploying the open source package on which it is based, and the thing that not coincidentally has the potential to lock you in to that provider.
    Unfortunately, companies with such offerings almost always refer to the open source base package as the "community edition", and their proprietized version as the "commercial edition" or sometimes "enterprise edition". A more accurate way to label the two editions would be "open source" and "proprietary", of course. But, from a marketing point of view, that has the disadvantage of making it plain what is going on.
  • Software developers have multiple motivations, and it's true that in open source, some of their motivation is intrinsic and not just driven by salary. It's actually quite common for open source developers to move from company to company, being paid to work on the same project the whole time; their résumé and work product are fully portable, and they take advantage of that. Open source means that one cannot be alienated from the fruits of one's labor, even when one changes employers. There is nothing anti-commercial about this — indeed, it could be viewed as the un-distortion of a market — but one can certainly see how observers with particular concerns about the mobility of labor might be inclined to fudge that distinction.
Finally, I think people also want to believe in a semi-secret worldwide army of happy paladins acting for the good of humanity. It would be so comforting to know they're out there. But what's actually going on with open source is much more complex and more interesting, and is firmly connected to commerce.

References

Sunday, April 3, 2016

The Descartes Modeling Language for Self-Aware Performance and Resource Management

Samuel Kounev, University of Würzburg, Würzburg, Germany
Associate Editor: Zhen Ming (Jack) Jiang, York University, Toronto, Canada 

Modern software systems have increasingly distributed architectures composed of loosely-coupled services that are typically deployed on virtualized infrastructures. Such system architectures provide increased flexibility by abstracting from the physical infrastructure, which can be leveraged to improve system efficiency. However, these benefits come at the cost of higher system complexity and dynamics. The inherent semantic gap between application-level metrics, on the one hand, and resource allocations at the physical and virtual layers, on the other hand, significantly increase the complexity of managing end-to-end application performance.

To address this challenge, techniques for online performance prediction are needed. Such techniques should make it possible to continuously predict at runtime: a) changes in the application workloads [3], b) the effect of such changes on the system performance, and c) the expected impact of system adaptation actions [1]. Online performance prediction can be leveraged to design systems that proactively adapt to changing operating conditions, thus enabling what we refer to as self-aware performance and resource management [4, 7]. Existing approaches to performance and resource management in the research community are mostly based on coarse-grained performance models that typically abstract systems and applications at a high-level (e.g., [2, 5, 8]). Such models do not explicitly model the software architecture and execution environment, distinguishing performance-relevant behavior at the virtualization level vs. at the level of applications hosted inside the running VMs. Thus, their online prediction capabilities are limited and do not support complex scenarios such as predicting how changes in application workloads propagate through the layers and tiers of the system architecture down to the physical resource layer, or predicting the effect on the response times of different services, if a VM in a given application tier is to be replicated or migrated to another host, possibly of a different type.

To enable online performance prediction in scenarios such as the above, architecture-level modeling techniques are needed, specifically designed for use in online settings. The Descartes Modeling Language (DML) provides such a language for performance and resource management of modern dynamic IT systems and infrastructures. DML is designed to serve as a basis for self-aware systems management during operation, ensuring that system performance requirements are continuously satisfied while infrastructure resources are utilized as efficiently as possible. DML provides appropriate modeling abstractions to describe the resource landscape, the application architecture, the adaptation space, and the adaptation processes of a software system and its IT infrastructure [1, 4, 6]. An overview of the different constituent parts of DML and how they can be leveraged to enable online performance prediction and proactive model-based system adaptation can be found in [6]. A set of related tools and libraries are available from the DML website at: http://descartes.tools/dml.

References

[1]  F. Brosig, N. Huber, and S. Kounev. Architecture-Level Software Performance Abstractions for Online Performance Prediction. Elsevier Science of Computer Programming Journal (SciCo), Vol. 90, Part B:71–92, 2014.

[2] I. Cunha, J Almeida, V. Almeida, and M. Santos. Self-Adaptive Capacity Management for Multi-Tier Virtualized Environments. In IFIP/IEEE Int. Symposium on Integrated Network Management, pages 129–138, 2007.

[3] N. Herbst, N. Huber, S. Kounev, and E. Amrehn. Self-Adaptive Workload Classification and Forecasting for Proactive Resource Provisioning. Concurrency and Computation - Practice and Experience, John Wiley and Sons, 26(12):2053–2078, 2014.

[4] N. Huber, A. van Hoorn, A. Koziolek, F. Brosig, and S. Kounev. Modeling Run-Time Adaptation at the System Architecture Level in Dynamic Service-Oriented Environments. Service Oriented Computing and Applications Journal, 8(1):73–89, 2014.

[5] G. Jung, M.A. Hiltunen, K.R. Joshi, R.D. Schlichting, and C. Pu. Mistral: Dynamically Managing Power, Performance, and Adaptation Cost in Cloud Infrastructures. In IEEE Int. Conf. on Distributed Computing Systems, pages 62 –73, 2010.

[6] S. Kounev, N. Huber, F. Brosig, and X. Zhu. Model-Based Approach to Designing Self-Aware IT Systems and Infrastructures. IEEE Computer Magazine, 2016, IEEE. To appear. http://se2.informatik.uni-wuerzburg.de/pa/uploads/papers/paper-926.pdf

[7] S. Kounev, X. Zhu, J. O. Kephart, and M. Kwiatkowska, editors. Model-driven Algorithms and Architectures for Self-Aware Computing Systems. Dagstuhl Reports. Dagstuhl, Germany, January 2015. http://drops.dagstuhl.de/opus/volltexte/2015/5038/

[8] Qi Zhang, Ludmila Cherkasova, and Evgenia Smirni. A Regression-Based Analytic Model for Dynamic Resource Provisioning of Multi-Tier Applications. In Proceedings of the 4th International Conference on Autonomic Computing, 2007.




If you like this article, you might also enjoy reading:
  • A. Avritzer, J. P. Ros and E. J. Weyuker, "Reliability testing of rule-based systems," IEEE Software, vol. 13, no. 5, pp. 76-82, Sep 1996.
  • E. Dimitrov, A. Schmietendorf, R. Dumke, "UML-Based Performance Engineering Possibilities and Techniques”, IEEE Software, vol. 19, no. 1, pp. 74-83, Jan-Feb, 2002.
     
  • J. Happe, H. Koziolek and R. Reussner, "Facilitating Performance Predictions Using Software Components," in IEEE Software, vol. 28, no. 3, pp. 27-33, May-June 2011.

Sunday, March 27, 2016

Should software developers be replaced by the crowd?

by Thomas D. LaToza (@ThomasLaToza) and André van der Hoek (@awvanderhoek)
Associate Editor: Christoph Treude (@ctreude)

We value our professional software developers, hire them in teams (local or remote), and set off on our merry way producing projects. This has been the status quo for a long time. But does it need to be? Could the crowd supplant the need for explicitly hired teams? More strongly yet: should the crowd perhaps replace hired employees?

These are tantalizing questions, both in terms of excitement about possibilities and in terms of 'fear' regarding the significant disruption that may play out in the workplace. Here, we consider several aspects of the issue of crowds versus hired employees.

1. Isn’t this just open source?

According to Howe [1], crowdsourcing is “the act of a company or institution taking a function once performed by employees and outsourcing it to an undefined (and generally large) network of people in the form of an open call”. In software, open source is one of the oldest and most successful crowdsourcing models. But all crowdsourcing is not open source. Platforms such as TopCoder employ a competition model, in which developers respond to a client request and clients select and reward a winner. Others such as trymyUI offer to the crowd short, well-defined tasks that, when taken together, compose into a larger whole – for instance comprehensively testing the usability of an application. Labor markets such as UpWork offer the promise of on-demand, fluid labor forces with the skills necessary for the job at hand, with workers ready to take on specialized work for an hour, a day, or maybe a week.

2. What about the speed of development?

As the old adage goes, many hands make light work. Fundamental to many crowdsourcing models is the decomposition of a large task into many small tasks. In principle, decomposed tasks enable work to be completed dramatically faster, as they enable parallel distribution to and completion of tasks by the crowd. But decomposition itself brings challenges: communicating tasks to developers, coordinating contributions made by the crowd, and ensuring smooth handoffs between tasks. In practice, these competing forces have given rise to a wide diversity of competing systems and platforms, using different granularities of tasks to crowdsource a variety of software development activities.

3. What about quality?

One common concern about opening contributions to anyone is the quality of the work that will result. As in traditional development, there are many approaches to managing quality. Work is almost always reviewed, either by the client, an agent of the client, or the crowd itself. Many platforms explicitly track reputation, allowing clients to identify those who have demonstrated quality work on similar tasks in the past and motivating developers to do high quality work to maintain their reputation. Competition platforms use the crowd to generate competing alternatives, allowing the client to select the highest quality solution. Through these mechanisms, it may, in fact, be possible to achieve higher quality through crowdsourcing, as some of our work has demonstrated [2]. On the other hand, it has been observed that, without proper management, quality may well suffer; care, thus, must be taken to focus on quality from the start [3].

4. What about crowdsourcing environments?

A significant innovation many crowdsourcing approaches bring is their dedicated support for performing crowdsourced work. Crowdsourcing platforms often provide, directly in the environment, support for contributors to browse and identify tasks matching their interests or that have the greatest chance of a reward. Clients may browse reputation information about potential workers, browse submitted contributions, and use the platform to make and manage payments. Systems that offer fine-grained tasks may go further still, offering workers self-contained tasks that can be quickly completed within the environment itself. For example, CrowdCode [4] offers an online environment for completing programming microtasks.



5. What about more complex development tasks?

Another common question is how a complex task such as architecture or design—requiring knowledge of the project as a whole or with dependencies making decomposition difficult—could ever be crowdsourced? Of course, one approach is simply to not decompose such tasks. For example, some TopCoder tasks are architectural in nature, asking competing workers to create design documents. An experienced crowd worker called a “co-pilot” is then responsible for creating, coordinating, and managing the tasks to implement the architecture. However, this can bring challenges in imposing a waterfall development process, where an architecture is built up-front, independent of future programming work [3]. Alternatively, it may be possible for the crowd itself to be more involved, decomposing architectures into networks of connected decisions [5].

6. Can I help?

Crowdsourcing is already penetrating software development practice today [6][7]. TopCoder has hosted more than 427,000 software design, development, and data science competitions. More than 100,000 testers freelance on uTest. But we believe that this may be just the beginning [8]. Just as open source changed how many organizations do software development and the nature of organizations themselves, crowdsourcing opens the door to new types of contributions and new ways stakeholders may interact that may lead to new models of software development. To help understand where these models may fail and where they may succeed, we have begun to study crowdsourcing approaches through a variety of experiments, from debugging to programming to designing. If you’d like to see how such approaches may work and help discover new ways of developing software, sign up here.

References

[1] James Surowiecki. (2005). The Wisdom of Crowds. Random House.
[2] Thomas D. LaToza, Micky Chen, Luxi Jiang, Mengyao Zhao, and André van der Hoek. (2015). Borrowing from the crowd: a study of recombination in software design competitions. International Conference on Software Engineering, 551-562.
[3] Klaas-Jan Stol and Brian Fitzgerald. (2014). Two's company, three's a crowd: a case study of crowdsourcing software development. International Conference on Software Engineering (ICSE), 187-198.
[4] Thomas D. LaToza, W. Ben Towne, Christian M. Adriano, and André van der Hoek. (2014). Microtask programming: building software with a crowd. Symposium on User Interface Software and Technology (UIST), 43-54.
[5] Thomas D. LaToza, Arturo Di Lecce, Fabio Ricci, W. Ben Towne, and André van der Hoek. (2015). Ask the crowd: scaffolding coordination and knowledge sharing in microtask programming. Symposium on Visual Languages and Human-Centric Computing (VL/HCC), 23-27.
[6] Thomas D. LaToza and André van der Hoek. (2016). Crowdsourcing in Software Engineering: Models, Motivations, and Challenges. IEEE Software, 33, 1 (January 2016), 74-80.
[7] Ke Mao, Licia Capra, Mark Harman, and Yue Jia. (2015). A survey of the use of crowdsourcing in software engineering. Technical Report RN/15/01, Department of Computer Science, University College London.
[8] Thomas D. LaToza and André van der Hoek. (2015). A vision of crowd development. International Conference on Software Engineering (ICSE), 563-566.

Sunday, March 13, 2016

Improving Bug Reporting for Mobile Applications

by: Kevin Moran@P10LeGacY 
Associate Editor: Federica Sarro@f_sarro

We currently live in the era of mobile computing. It is conceivable, as smartphones and tablets gain worldwide popularity, that many individuals’ primary computing device will soon be a mobile “smart” device. A large contributor to this phenomenon is the continued rise of an “App Store” economy where users turn to increasingly complex applications to accomplish mainstream computing tasks that were previously accomplished primarily in a desktop computing environment.
This has created a fiercely competitive software marketplace where, if an application is not performing as expected due to bugs, crashes, or a lack of customer desired features, nearly half of users will simply abandon it for a similar application [1]. Therefore, for an application to be successful in a modern mobile ecosystem, development teams must be extremely efficient and effective during the software maintenance cycle, pushing regular updates to consumers. Unfortunately, several mobile-specific confounding factors such as hardware and platform fragmentation, API-instability and fault-proneness [2], and difficult-to-test features (e.g., sensors) make for a particularly challenging maintenance process.

Perhaps the most important software maintenance activity is bug report resolution. However, the manner in which bugs are currently reported to developers is far from ideal. For apps released in mobile marketplaces, most developers and teams must sort through potentially thousands of user-reviews where users may or may not describe bugs in a manner sufficient to allow bug reproduction and resolution. Additionally, a recent study has shown that only a reduced set of user reviews can be considered useful or informative to developers [3]. Open source applications or commercial applications undergoing internal or beta testing will often make use of existing issue tracking systems (e.g., Bugzilla, Mantis, JIRA, GitHub Issue Tracker) to report information related to mobile bugs and crashes. However, these bug reporting systems rely mostly on unstructured natural language bug descriptions and lack key features that would better support and facilitate accurate bug reporting for mobile applications.

A previous study has shown that there are typically three key types of information that developers consider most helpful when comprehending and fixing issue reports [4]: 1) detailed reproduction steps, 2) stack traces, and 3) re-playable test cases or scenarios. Unfortunately, this information is also typically the most difficult for reporters to provide, and in the context of mobile applications, is further complicated by a highly GUI-event driven nature. This highlights the typical lexical gap that exists between reporters of bugs and the developers that are attempting to read and fix the reports. Reporters, particularly end users and beta testers, typically have only a working functional knowledge of an application, whereas developers have intimate code-level knowledge; thus the primary task that a bug reporting system must accomplish is the bridging of this gap by providing the types of information listed above. The whole of these current issues highlight the need for an improved reporting mechanism for mobile applications.

The SEMERU research team has taken the first step towards addressing these problems by developing a novel bug reporting mechanism called Fusion [5] that operates under the following key insight: automated program analysis techniques can be used to bridge the lexical knowledge gap between reporters and developers. Currently, issue tracking systems operate with essentially no prior knowledge of the application that they support, and given the information that can be gleaned from automated program analysis techniques, we saw a natural opportunity to create a “smart” bug reporting mechanism for mobile apps. Fusion uses fully-automated static and dynamic analysis techniques to gather screenshots, GUI-component information, and app event-flow information, linking dynamically extracted information back to program source code to help facilitate users in reporting issues and aid developers in resolving bugs. During the reporting process, users leverage a smart web-form to fill out the reproduction steps for a bug using an <action, component> formula where the action is the touch event (e.g. tap, long-tap) and the component is the GUI component on the screen (e.g. button, checkbox). The form tracks the location of the user in the high level event-flow of the application, making auto completion suggestions to guide the reporting (e.g. if the user reports that they clicked on a button that leads to a new app screen, the system will only suggest components from the new screen). In the end, the developer is presented with a report that includes detailed reproduction steps, screenshots, and traceability links back to the source code.

Figure 1: An Example Fusion Bug Report

In a user study with over 20 participants conducted by our lab, we found that users of Fusion system generated more reproducible bug reports for 14 real-world open source Android applications than a comparable system, the Google Code Issue Tracker. Particularly encouraging is that a subset of users in our study had no background in coding or computer science, but were still able to construct detailed, actionable reports. We believe that this is the first step towards creating more effective bug reporting systems for mobile and GUI-based applications. However, these reports could enable much more. The rich information contained within the reports could enable more effective methods of supporting other typically expensive or difficult maintenance tasks such as developer triaging or duplicate bug report detection.

You can access the Fusion tool, documentation, and demonstration videos at www.fusion-android.com.

References
[1] Mobile apps: What consumers really need and want https://info.dynatrace.com/rs/compuware/images/Mobile_App_Survey_Report.pdf
[2] Linares-Vásquez, M., Bavota, G., Bernal-Cárdenas, C., Di Penta, M., Oliveto, R., and Poshyvanyk, D., "API Change and Fault Proneness: A Threat to the Success of Android Apps", in Proceedings of 9th Joint Meeting of the European Software Engineering Conference and the 21st ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE’13)
[3] N. Chen, J. Lin, S. Hoi, X. Xiao, and B. Zhang. AR-Miner: Mining informative reviews for developers from mobile app marketplace. In 36th International Conference on Software Engineering (ICSE’14) 

[4] N. Bettenburg, S. Just, A. Schro ̈ter, C. Weiss,
R. Premraj, and T. Zimmermann. What makes a good bug report? In Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of Software Engineering, (SIGSOFT ’08/FSE-16)
[5] 
K. Moran, M. Linares-Vásquez, C. Bernal-Cárdenas, and D. Poshyvanyk. Auto-completing bug reports for android applications. 10th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE’15)







Sunday, March 6, 2016

Release Management of Mobile Apps

Associate Editor: Federica Sarro, @f_sarro

Among all the stories reflected by news agencies and social networks on Syrian refugees, the impact of mobile apps on their journey, in particular WhatsApp and Telegram, was echoed all over. During the tough journey they had, refugees connected to each other, shared their experience, reported the border’s traffic and updated their family about their health using VOIP apps …: “Without a cell phone [and internet] you are lost on the road” [2].

Ranging from Sukey, built to save British student protesters in 2011, to Angry Birds, which brings entertainment, mobile apps have a major impact on business, economy, health, politics, education … in short, on our whole life. Individual software engineers and their products have never been that close to the day to day life of regular people as they are today. This impact has been expected to also change the decision process for software development and evolution [3].

In a recently published study [1], we wanted to better understand the release decision process in mobile apps and their impact on users. Unfortunately, despite a recent surge of research interest in release engineering [6], the release engineering decisions and activities for mobile apps are still not well understood. To answer questions such as "how are release decisions for mobile apps made?", "how do users think about new releases?", we reached out to 36 developers and 674 users and found that release management of mobile apps is different from that of proprietary software.

First of all, the majority of mobile app developers, especially those with a large experience, has a clear rationale for deciding when to release a new version of their mobile app. We identified six release strategies, i.e., a time-based strategy (release schedule) to release at a specific point in time, marketing strategies to get more visibility, quality (test) driven strategies to test with parts of the customer base or achieve a specific test coverage, a feature-based strategy to release hot features ASAP, a size-based strategy to keep the size of the update fixed, and/or an occasional strategy to release an app concurrently with the launch of well-known mobile devices.



Second, as shown in the diagram above, we made a range of observations about how app developers perceive new releases. Developers believe that apps with frequent updates do not necessarily require more development effort, or introduce substantial changes to an app. Indeed, the majority (61.1%) of developers believe that apps with frequent updates (more than once per 3 weeks) deliver less functionality and changes in each version. However, there was no consensus if frequent releases affect customer feedback, i.e., whether they increase or decrease an app’s rating or download volume.



Third, our study on 674 users showed that mobile app users are not only aware of app updates but they also observed the presence of regularity in the release cycle of their apps. Users stated mixed feelings toward frequent app releases. They like apps with frequent updates but at the same time, frequent updates may be discouraging and could provoke them to uninstall apps. This might be why only half of the users turn on automatic updates. The summary of our findings from studying app users is presented in the diagram below.


Also, as part of the study, we identified a range of problems that users faced after updating their apps, as shown in the following picture. Finding solutions for these real problems is definitely of value.

The findings of this study pose several challenges that could be addressed by the research community:
Challenge 1. The impact of release frequency on mobile app users’ needs further study and analysis.
Challenge 2. The relation between release frequency and app quality is unclear.
Challenge 3. While the developers believe that release strategy and frequency do not affect development practices and total effort, empirical analysis needs to go beyond app stores to compare with traditional apps (desktop, server, web, etc.).

To sum up, app release practices are different from non-app software that researchers used to study. App developers have different strategies and considerations compared to non-app software and in some cases those strategies were not known before (such as size-based strategies). Moreover, only half of the users allow automatic updates of apps, hence each app release needs to earn people’s attention. Given that the release strategy is a success factor for mobile apps, the effect of release practices such as frequency of releases needs further elaboration.

You can find more details of this study in our paper [3]. You may also consider attending the paper presentation at the SANER 2016 conference.


References

[1] M. Nayebi, B. Adams, and G. Ruhe. “Release Practices for Mobile Apps – What do Users and Developers Think?”, Proceedings of the 23rd IEEE International Conference on Software Analysis, Evolution, and Re-engineering, SANER 2016.

[2] WhatsApp offers lifeline for Syrian refugees on journey across Europe, http://mashable.com/2015/07/03/syrians-europe-whatsapp-refugees/#jkIaUCHhtsqS, last access February 2016.

[3] W. Maalej, M. Nayebi, T. Johann, and G. Ruhe. "Towards Data-Driven Requirements Engineering", IEEE Software, Jan 2016.

[4] S. McIlroy, N. Ali, and A. E. Hassan. "Fresh apps: An empirical study of frequently-updated mobile apps in the Google play store", Empirical Software Engineering, pp. 1-25, 2015.

[5] M. William, G. Sarro, and M. Harman. "Causal Impact Analysis Applied to App M. in Google Play and Windows Phone Store", RN 15/07, (2015).

[6] B. Adams, and S. McIntosh. "Modern Release Engineering in a Nutshell - Why Researchers Should Care" in Leaders of Tomorrow: Future of Software Engineering, Proceedings of the 23rd IEEE International Conference on Software Analysis, Evolution, and Re-engineering, SANER 2016.

Sunday, February 28, 2016

Do commercial software teams use GitHub? How?

By: Eirini Kalliamvakou, University of Victoria, Canada (@irina_kAl)
Associate Editor: Bogdan Vasilescu, University of California, Davis. USA (@b_vasilescu) 


GitHub is really popular. Right now it has more than 31 million repositories and over 12 million users. It is also growing at an impressive rate, becoming the tool of choice for a lot of development teams. Much empirical research in software engineering is currently focusing on GitHub; the transparent development environment it affords, together with its pull-based workflow, provide a lightweight mechanism for managing code changes. GitHub’s features impact how it is used and the benefits it provides to teams’ development and collaboration. While most of the evidence we have is from GitHub’s use in open source software (OSS) projects, GitHub is also used in an increasing number of commercial projects.


It is difficult to pin down what “good collaboration” is, what tools and practices make it up. So when something as popular as GitHub comes along in the service of “good collaboration” we want to know how it works in practice. In a qualitative study, we investigated GitHub and collaboration by looking at the practices of commercial software teams. That means teams that develop software that is proprietary, built in commercial organizations, and hosted on private repositories. Our study looked both at how these teams use GitHub and how they think about collaboration.


We surveyed and interviewed professional developers that use GitHub in their workplace. The practices that we heard about from the commercial software teams fall under 3 categories: the teams’ workflow, their communication and coordination, and their self-organization.


Workflow
We asked participants to describe every step of the process that takes them from a task list all the way through to a merge. Our finding was that commercial teams follow a “branch & pull” workflow (Figure 1), that is not either the fork & pull, or the shared repository workflows, the two main workflows recognized by GitHub.


Figure 1: Branch & pull workflow


In the fork & pull model, there is a main repository for the project or the team, and developers isolate their work by creating a copy of the repository and making their changes there. When they are done they submit a pull request and that triggers a code review, changes and a merge. The fork & pull model has a distinct team phase — the code review step — that is triggered by the pull request. This is in line with the tradition of open source projects that were the inspiration for this workflow – pull requests act as screening mechanism for code that is coming from unknown contributors. Branch & pull works like fork & pull, with the difference being that work is isolated through the use of branches rather than forks. Instead of users making a copy of the main repository under their own GitHub account, they make a branch inside the main repository. That is an appropriation of an open source-style workflow to a commercial team environment. 


We found the branch & pull workflow to be very popular, reported by 23 out of our 24 interviewees and the reason was because it made the code reviews part of the workflow, instead of an afterthought.


Communication & Coordination
We asked participants to give us examples of circumstances when they found it essential to communicate and coordinate with their team and the mechanisms they used. The overall observation was that although GitHub is not a communication tool, communication is happening on GitHub.


Developers had a preference for awareness rather than direct communication, by looking at the issue list, the commit list, getting notification emails from GitHub, or using a chat client that still integrates with GitHub. Between these mechanisms, the preference was pretty much equal. GitHub was preferred for code-centric discussions. Most developers said that GitHub lacked the space and synchronicity that is essential when discussing ideas, and in those cases they found the need to move their conversations to a communication tool that was external to GitHub. They found however that communication through comments was great for code-centric discussions. Why? Because all the information that is related to an artifact is attached to it and remains so, becoming essentially a record of decisions. 


Self-organization
The primary way self-organization showed up was as self task-assignment. A developer would choose what to work on based on their expertise and availability, and would pick tasks off of GitHub’s issue list (or other issue tracker if that is what the team was using). This is not a practice that is typically associated with commercial projects. However, the manager is still part of the process. Their role is to define bite-size tasks (can be worked on by a single developer), and they are still part of prioritization and estimation.


Does it sound familiar?
All the above work practices that we heard about from commercial software teams using GitHub, are also known open source practices. We know that open source projects that use GitHub use pull request and they screen contributions with them. What’s unexpected here is that the commercial teams do not have the same need for screening - there is trust built into the team - and yet they still prefer to use the pull requests as an opportunity to review the code. Open source projects on GitHub also use comments for providing direct feedback and as part of the code reviews. This is true of open source project in general; lightweight, text-based communication that is automatically archived is the preferred way of communicating. Finally, self-organization is a long known practice in OSS projects.


What do these results mean?
One thing to take away is that we saw GitHub acting as a vehicle for commercial software teams to adopt best practices, styled after open source ones. Our results indicate that GitHub is giving commercial teams the chance to adopt best practices that are tried and true in open source projects when they choose to use it.


What is more, GitHub seems to act not only as a toolkit but also as a process kit. This is based on how consistent we saw GitHub being used — 23 out of the 24 commercial interviewees described the same workflow to us. And that was not the expected one for commercial projects, but the one that open source projects use. GitHub seems to come together with a “way to use it” and that travels together with the tool — very visibly in the open source world but quite organically in the commercial world too.


How? This takes us to the third take away: GitHub users advocate it in the workplace. They are the ones that bring the bundle that is the tool and process and best practice into their organizations. It is a bottom-up approach rather than a top-down one. 

Where to go from here?
Given the overlap between Github and Git, which of them is responsible for the trend we saw? Would Git by itself have the same effect?  The same question applies to other GitHub-like tools. How much of GitHub can we strip away before the effect disappears?

References:


If you liked this post, you might also enjoy reading: